Create a Forum Topic by Logging In or Creating an Account

Consistent mapping from special double values to integers

Fri, 05/11/2018 - 10:51 am

The guy sitting in front of my computer introduced a bug due to passing a NaN where an integer should have been.
And that NaN mapped to a -1 which was kind of hard to track down.

It would be nice to have a fixed way of converting from special double values to integers.
Currently that seems a bit unspecified.

Function DoStuff()

    int a

    int64 b

    uint64 c

    print "Mapping NaN to int"

    a = NaN

    b = NaN

    c = NaN

    printf "%x, ,%x, %x\r", a, b, c

    Make/FREE/B a1 = {NaN}

    Make/FREE/B/U a2 = {NaN}

    Make/FREE/W b1 = {NaN}

    Make/FREE/W/U b2 = {NaN}

    Make/FREE/I c1 = {NaN}

    Make/FREE/I/U c2 = {NaN}

    Make/FREE/L d1 = {NaN}

    Make/FREE/L/U d2 = {NaN}

    print/D a1[0], a2[0], b1[0], b2[0], c1[0], c2[0], d1[0], d2[0]

    print "Mapping inf to int"

    a = Inf

    b = Inf

    c = Inf

    printf "%x, ,%x, %x\r", a, b, c

    Make/FREE/B a1 = {Inf}

    Make/FREE/B/U a2 = {Inf}

    Make/FREE/W b1 = {Inf}

    Make/FREE/W/U b2 = {Inf}

    Make/FREE/I c1 = {Inf}

    Make/FREE/I/U c2 = {Inf}

    Make/FREE/L d1 = {Inf}

    Make/FREE/L/U d2 = {Inf}

    print/D a1[0], a2[0], b1[0], b2[0], c1[0], c2[0], d1[0], d2[0]

    print "Mapping -inf to int"

    a = -inf

    b = -inf

    c = -inf

    printf "%x, ,%x, %x\r", a, b, c

    Make/FREE/B a1 = {-inf}

    Make/FREE/B/U a2 = {-inf}

    Make/FREE/W b1 = {-inf}

    Make/FREE/W/U b2 = {-inf}

    Make/FREE/I c1 = {-inf}

    Make/FREE/I/U c2 = {-inf}

    Make/FREE/L d1 = {-inf}

    Make/FREE/L/U d2 = {-inf}

    print/D a1[0], a2[0], b1[0], b2[0], c1[0], c2[0], d1[0], d2[0]

End


•dostuff()
  Mapping NaN to int
  8000000000000000, ,8000000000000000, 8000000000000000
  -1  255  -1  65535  -1  4294967295  -9223372036854775808  -9223372036854775808
  Mapping inf to int
  8000000000000000, ,8000000000000000, 8000000000000000
  -1  255  -1  65535  -1  4294967295  -9223372036854775808  -9223372036854775808
  Mapping -inf to int
  8000000000000000, ,8000000000000000, 8000000000000000
  0  0  0  0  0  0  -9223372036854775808  -9223372036854775808

IMHO the best solution would be to throw a RTE with rtFunctionErrors=1 but I would expect that this is too costly performance-wise.