Investigate ISISIndirectInelastic.OSIRISIqtAndIqtFitMulti unreliable system test

[MohamedAlmaki]

A recent update to CompareWorkspace in #37889 has caused ISISIndirectInelastic.OSIRISIqtAndIqtFitMulti to fail randomly. The test fails approximately 70% of the time using the current tolerance. Increasing the tolerance to a larger value does not eliminate the random failures. Upon initial investigation, it was found that the reference workspace is within the error margin boundaries. However, even after replacing the reference workspace with a new one, the random errors persist. This issue requires further investigation to determine how to fix the test.

Here is a screenshot of the comparison between the reference and multiple test runs mismatch output

[rboston628]

I think switching the comparison from relative to absolute (i.e. ToleranceRelErr flag from true to false), at least for the blue/green curves, could likely fix the issue.

This was found to work in the three systems tests that were updated in PR #37889.

Here's why.

The blue/green curves are showing small numbers near zero.

Previously, if CompareWorkspaces was comparing two workspaces using relative tolerance, but the magnitudes of the values being compared were already really small, then it would just perform a comparison against an absolute difference.
The fix in PR #37889 changed that, so that it actually always performs a relative comparison, regardless of the size of the values.

As an example, consider the numbers $x_1=0.001, x_2=0.0012$.
The absolute difference is $\Delta x_{12} = 0.0002$, and the relative difference is 0.18.
If you put this into the previous algorithm with a relative tolerance of $\delta=0.01$, the properly expected behavior of this test should be to fail.
However, the algorithm instead looked if $|x_1+x_2|/2 = 0.0011 \leq \delta$, and then instead performed the (unnecessary) check $|x_1-x_2| \leq \delta$, which succeeds.
With the new behavior, this just checks if the relative difference (0.18) is smaller than the relative tolerance $\delta$, and so fails.

It is possible the failing relative differences are occurring in the near-zero portions of the graph, but that those portions were previously so small in magnitude that CompareWorkspaces did not use the relative difference, and just checked the magnitude.

Therefore switching ToleranceRelErr to false would do the absolute comparison it used to perform, and might pass.

But the left region of the blue curve suggests where the values are larger, the absolute difference is also fairly large, so it might still fail for a different reason.

If it is really desired for CompareWorkspaces to not perform relative difference comparisons for near-zero values, then I suggest a V2 that will have a better interface for controlling that sort of behavior, like a `"IgnoreBelowThreshold" property that tells the algo to ignore comparing values that are small enough.

[rboston628]

I confirmed that switching

self.tolerance_is_rel_err = False

causes this test to pass. However, that is then using an absolute difference of 5.0. Adjusting this value, it still fails if using an absolute difference below about 0.165.

Were the values in the graph above normalized? From that graph, the absolute difference in the y-values is certainly never exceeding 0.165.

Could it be something other than the y-values?

[rboston628]

Editing the test to disable all the other checks (Instrument, Axes, SpectraMap, Masking, Sample), it still fails for a tolerance of 0.1, suggesting it is an issue with the y-values.

[rboston628]

I ran the system test and loaded the mismatch and reference into workbench.

reference_file = "~/mantid/build/ExternalData/Testing/SystemTests/tests/framework/reference/II.OSIRISIqt.nxs"
mismatch_file = "~/mantid/build/Testing/SystemTests/scripts/OSIRISIqtAndIqtFitMulti-mismatch.nxs"

reference = "reference"
mismatch = "mismatch-1"
difference = "difference"
scale = "denominator"

LoadNexus(
    Filename = reference_file,
    OutputWorkspace=reference,
)
LoadNexus(
    Filename = mismatch_file,
    Outputworkspace=mismatch,
)

res, _ = CompareWorkspaces(
    Workspace1=reference,
    Workspace2=mismatch,
)
assert not res

The assert passes, meaning they are not comparing as equal.

Looking at the absolute difference, this is confusing:

Minus(
    LHSWorkspace=reference,
    RHSWorkspace=mismatch,
    OutputWorkspace=difference,
)

Here is the difference across each spectrum, and the scale for the graph is 1.e-15.

Looking at the relative difference, also confusing:

Minus(
    LHSWorkspace=reference,
    RHSWorkspace=mismatch,
    OutputWorkspace=difference,
)
Plus(
    LHSWorkspace=reference,
    RHSWorkspace=mismatch,
    OutputWorkspace=scale,
)
Scale(
    InputWorkspace=scale,
    OutputWorkspace=scale,
    Factor=0.5,
)
Divide(
    LHSWorkspace=difference,
    RHSWorkspace=scale,
    OutputWorkspace=difference,
)

However, running

res, _ = CompareWorkspaces(
    Workspace1=reference,
    Workspace2=mismatch,
    CheckAllData=True,
    Tolerance = 0.1,
)
assert not res

produces debugging output:

Write-locking _
Read-locking reference
Read-locking mismatch-1
Data mismatch at cell (hist#,bin#): (0,136)
 Dataset #1 (X,Y,E) = (0.70306,0.404193,0.626619)
 Dataset #2 (X,Y,E) = (0.70306,0.404193,0.748799)
 Difference (X,Y,E) = (0,0,0.122179)
Data mismatch at cell (hist#,bin#): (0,137)
 Dataset #1 (X,Y,E) = (0.70823,0.512894,0.650339)
 Dataset #2 (X,Y,E) = (0.70823,0.512894,0.766147)
 Difference (X,Y,E) = (0,1.11022e-16,0.115809)
Data mismatch at cell (hist#,bin#): (9,150)
 Dataset #1 (X,Y,E) = (0.775434,0.911837,0.738537)
 Dataset #2 (X,Y,E) = (0.775434,0.911837,0.876133)
 Difference (X,Y,E) = (0,0,0.137596)
Data mismatch at cell (hist#,bin#): (11,134)
 Dataset #1 (X,Y,E) = (0.692721,0.368211,0.581675)
 Dataset #2 (X,Y,E) = (0.692721,0.368211,0.478791)
 Difference (X,Y,E) = (0,0,0.102884)
Data mismatch at cell (hist#,bin#): (11,143)
 Dataset #1 (X,Y,E) = (0.739247,0.822654,0.58926)
 Dataset #2 (X,Y,E) = (0.739247,0.822654,0.696968)
 Difference (X,Y,E) = (0,0,0.107709)
Data mismatch at cell (hist#,bin#): (12,125)
 Dataset #1 (X,Y,E) = (0.646195,0.607105,1.26333)
 Dataset #2 (X,Y,E) = (0.646195,0.607105,1.4144)
 Difference (X,Y,E) = (0,0,0.151073)
Data mismatch at cell (hist#,bin#): (18,124)
 Dataset #1 (X,Y,E) = (0.641026,0.373372,0.69277)
 Dataset #2 (X,Y,E) = (0.641026,0.373372,0.568712)
 Difference (X,Y,E) = (0,0,0.124058)
Data mismatch at cell (hist#,bin#): (23,133)
 Dataset #1 (X,Y,E) = (0.687552,0.969584,0.901905)
 Dataset #2 (X,Y,E) = (0.687552,0.969584,0.762877)
 Difference (X,Y,E) = (0,2.22045e-16,0.139028)
Data mismatch at cell (hist#,bin#): (23,135)
 Dataset #1 (X,Y,E) = (0.697891,0.877629,0.576347)
 Dataset #2 (X,Y,E) = (0.697891,0.877629,0.698025)
 Difference (X,Y,E) = (0,0,0.121678)
Data mismatch at cell (hist#,bin#): (26,144)
 Dataset #1 (X,Y,E) = (0.744417,0.759943,0.875618)
 Dataset #2 (X,Y,E) = (0.744417,0.759943,0.766352)
 Difference (X,Y,E) = (0,0,0.109266)
Data mismatch at cell (hist#,bin#): (30,124)
 Dataset #1 (X,Y,E) = (0.641026,0.967622,0.824505)
 Dataset #2 (X,Y,E) = (0.641026,0.967622,0.717552)
 Difference (X,Y,E) = (0,0,0.106953)
Data mismatch at cell (hist#,bin#): (30,146)
 Dataset #1 (X,Y,E) = (0.754756,0.705452,0.772482)
 Dataset #2 (X,Y,E) = (0.754756,0.705452,0.664443)
 Difference (X,Y,E) = (0,0,0.10804)
The workspaces did not match: Data mismatch

indicating the failure has to do with the error measurements.

Looking at a plot of the spectra errors, sure enough, there is a significant relative difference in some locations.

Same looking at the absolute difference.

Is it necessary for this test to ensure that the error values match? If not, can they be removed prior to the test? There is no flag to CompareWorkspaces, unfortunately, to ignore the error values.

[rboston628]

All of my results in the previous comment were from a separate failing comparison than the one Mohamed is showing. Mohamed is showing the II.OSIRISIqtFitMulti.nxs reference workspace, while I was looking at errors in the II.OSIRISIqt.nxs reference workspace.

To be sure about the error comparison being the issue, I edited L656 to remove the comparison of the errors and ran the system test using a relative error of 1.e-10. Now I find a different error in the workspaces Mohamed was graphing.

The absolute differences look like so:

The relative differences look like so:

If I restore the comparison of the e values and rebuild mantid, then I get the error on the II.OSIRISIqt.nxs reference workspace again.

[rboston628]

I changed CompareWorkspaces to not check uncertainties. Then changed the failing test to perform an absolute difference comparison with a tolerance of 0.05. This seems to consistently pass.

As the comment inside the failing test notes,

# Relative tolerance is used because the calculation of Monte Carlo errors means the Iqt errors are randomized
 # within a set amount

so that comparing errors might not be meaningful for this test.

The solutions could be

• clear out errors from the workspaces prior to comparison
• add a new flag to CompareWorkspaces to disable checking errors

That mixed with using an absolute tolerance of 0.05 should fix this.

[MohamedAlmaki]

Thanks for the thorough investigation. I was able to reproduce the same results. I think adding a new flag is a better solution as this behaviour might happen again. What do you think? @robertapplin

[rboston628]

I am working on a PR now to add the flag, CheckUncertainity, which will turn off checking the E values of MatrixWorkspaces
PR #37933

[robertapplin]

I am working on a PR now to add the flag, CheckUncertainity, which will turn off checking the E values of MatrixWorkspaces PR #37933

Thank you for investigating this, the solution you are going for sounds good to me!