Be aware of differences between x86 and x64 floating point arithmetic when writing tests

[abelbraaksma]

Not a bug, but merely a message to anyone writing or updating tests and sharing my own "gotcha" moment. Floating point arithmetic is not IEEE-754 stable and is not always deterministic, that is, it doesn't give the same results when run in x86 or x64 mode.

This is a problem in the following scenario:

1. Run any test in VS Test window that depends on floating point equality, let's say it fails
2. Run the same test from the commandline, let's say it succeeds
3. Run the same test in FSI, it succeeds
4. Run the same in 32 bit in debug mode vs release mode, different outcomes
5. Debug the same in 32 bit in with JIT optimizations on or off, different outcomes

Repro steps

An example of such test could be:

let result = Operators.tanh 0.8
let expected = 0.66403677026784891
let stringResult = result.ToString("G17")
let stringExp = expected.ToString("G17")
printfn "expect: %s, result: %s" stringExp stringResult
printfn "%%f expect: %.18f, result: %.18f" expected result
printfn "Are equal: %b" (result = expected)
Assert.AreEqual(expected, result)

When you run this test from FSI (which I believe defaults to x64), it succeeds and prints:

expect: 0,66403677026784891, result: 0,66403677026784891
%f expect: 0.664036770267849000, result: 0.664036770267849000
Are equal: true
val result : float = 0.6640367703
val expected : float = 0.6640367703
val stringResult : string = "0,66403677026784891"
val stringExp : string = "0,66403677026784891"
val it : unit = ()

When setting FSI to x86, you'll get this:

expect: 0,66403677026784891, result: 0,66403677026784902
%f expect: 0.664036770267849000, result: 0.664036770267849000
Are equal: false
NUnit.Framework.AssertionException: Expected: 0.66403677026784891d
    But was: 0.66403677026784902d

        at NUnit.Framework.Assert.ReportFailure(String message) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.cs:line 394
        at NUnit.Framework.Assert.ReportFailure(ConstraintResult result, String message, Object[] args) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.cs:line 382
        at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.That.cs:line 247
        at <StartupCode$FSI_0005>.$FSI_0005.main@()
Stopped due to error

If you run the above inside a test, like noticed here c3631e2#diff-e8b1e9e6d95b154612c4f36f01b52122R537, then you can get either success or fail depending on what arch your host is running in. In VS that's usually x86 (I finally found the difference by using NCrunch, which allows me to switch between x86 and x64 runners, host and compilers).

Known workarounds

Since this is a poorly defined, but existing "feature" of some parts of some JITs not being IEEE-754 compliant, and while it is still important to find regressions in floating point calculations using exact comparisons, the workaround for such scenarios is as follows:

1. Make sure to check floating point expected results on x86 and x64 (@vzarytovskii, do we currently run all tests in both x86 and x64? If not, we probably should).
2. If there's a difference, either find an outcome that is deterministically the same on both machines, or wrap it in a pointer-check (size of 4 is x86, size of 8 is x64)

Related information

Issue that discusses this to some depth: dotnet/roslyn#7333. Of note are:

• No way to get reliable precision of floating-point operations roslyn#7333 (comment) (difference of 32 bit dep. on JIT optimiz. on or off)
• No way to get reliable precision of floating-point operations roslyn#7333 (comment) (difference between RyuJIT, .NET versions, and Legacy JIT). This also explains that legacy means: 64 bit precision, while RyuJIT means 80 bit precision through SSE instructions.
• Caused by: Re enable tests for operators: OperatorsModule1.fs and OperatorsModule2.fs #9516

[abelbraaksma]

This has now been solved in #9516, where I added a simple, but for this purpose valuable Assert.AreNearEqual, which uses a delta, so that differences due to architecture can be worked around in tests, while still being able to catch regressions.

[abelbraaksma]

I'll close this, as there are no actionable items anymore, and it has been discussed offline with the proper people.