Within mantid, `NaN` compares equal to any floating point value using `CompareWorkspaces`, system tests impacted

[rboston628]

Describe the bug

Within CompareWorkpaces as it was written, the comparison of any value with NaN will evaluate as true. This means a workspace with finite values will be considered "equal" to a workspace with only NaNs in it.

While it might sometimes be okay for NaN == NaN to be true, there is no situation where NaN == 5.0 should be true. That represents an error in a calculation.

The following systems tests rely on errantly comparing finite values to NaN. The CNCS and ARCS tests may be related to old ways of masking bins:

• CNCSReductionTest.CNCSReductionTest
• SANSZOOMTomlFileConversion.ZOOMV1UserFileTest_m8 - Fix and reinstate SANSZOOM system test #38130
• ARCSReductionTest.ARCSReductionTest

The following systems tests rely on comparing NaN values to NaN. Conversation with @AndreiSavici suggests this might be an issue of masks.

• ISISIndirectInelastic.OSIRISMultiFileSummedReduction
• ISIS_PowderPolarisTest.FocusTestAbsorptionPaalmanPings
• DirectILLAutoProcessTest.DirectILLAuto_PANTHER_Powder_Test
• DirectILLAutoProcessTest.DirectILLAuto_SHARP_Powder_Test
• MDNormHYSPECTest.MDNormHYSPECTest
• SANSBeamCentreFinderCoreTest.SANSBeamCentreCoreRunnerTest
• SANSReductionCoreTest.SANSReductionCoreRunnerTest
• SANSILLReductionTest.ILL_D33_LTOF_Test
• D7AbsoluteCrossSectionsTest.ILL_D7_TimeOfFlight_Test
• ReflectometryQuickMultiDetector.ReflectometryQuickMultiDetector
• ILLDirectGeometryReductionTest.IN4
• ILLDirectGeometryReductionTest.IN5_Mask_Non_Overlapping_Bins
• FlipperEfficiencyTest.FlipperEfficiencyPolarisedTest
• FlipperEfficiencyTest.FlipperEfficiencyUnpolarisedTest
• AbinsTest.AbinsCRYSTAL2D
• SANSILLAutoProcessTest.D11_AutoProcess_IQxQy_Test
• SANSILLAutoProcessTest.D11_AutoProcess_Multiple_Transmissions_Test
• SANSILLAutoProcessTest.D33_AutoProcess_IPhiQ_Test
• ISISIndirectEnergyTransferTest.ISISIndirectEnergyTransferTest
• MuonProcessTest.MuonProcessTest

To Reproduce

The following script illustrates the issue

from mantid.simpleapi import CreateEmptyTableWorkspace, CreatwWorkspace, CompareWorkspaces

## TABLES
tab1 = CreateEmptyTableWorkspace()
tab1.addColumn("double", "col1")
tab1.addRow({"col1": 1.0})

tab2 = CreateEmptyTableWorkspace()
tab2.addColumn("double", "col1")
tab2.addRow({"col1": float("nan")})

# the two tables are "equal" accoridng to mantid
res, _ = CompareWorkspaces(tab1, tab2)
assert res

## WORKSPACES
ws1 = CreateWorkspace(DataX = [1.0, 2.0, 3.0], DataY = [2.0, 3.0])
ws2 = CreateWorkspace(DataX = [1.0, 2.0, 3.0], DataY = [float("nan"), float("nan")])

# the two workspaces are "equal" according to mantid
res, _ = CompareWorkspaces(ws1, ws2)
assert res

Expected behavior

There are valid reasons, especially within mantid's use cases (normalizing with empty bins), where NaNs should be considered equal.

There is no possible justification for NaN == 5.0 evaluating as true.

In the above script, ALL of those assertions should fail.

This will be partly fixed in PR #38075 , by adding a flag to CompareWorkspaces to control NaN == NaN. By default, NaN is not equal to anything, but the user can set them to be equal; this is necessary for about twenty system tests to work.

However, what to do about the system tests relying on NaN == 5.0?

Screenshots

Platform/Version (please complete the following information):

• OS: [e.g. Windows, RHEL 7, Ubuntu, macOS]
• OS Version:
• Mantid Version [e.g. 6.0.0]

Additional context

This has been slowly uncovered in the following PRs:

• changes to mantid's assert_almost_equal test fixture #37878
• Fix relative comparison in CompareWorkspaces #37889
• Add single source of truth for absolute and relative differences to Mantid::Kernel::FloatingPointComparison #37931
• Add CheckUncertainty property to CompareWorkspaces to turn off error-comparison #37933
• Replace relative and absolute differences with new operations #38067
• Add flag to CompareWorkspaces so users can specify NaN == NaN behavior #38075