[CI] CDataFrameAnalyzerOutlierTest.testErrors is flakey #1337
Description
CDataFrameAnalyzerOutlierTest.testErrors has failed twice with the same error:
[Exception] - critical check memoryLimitExceed has failed
== [File] - CDataFrameAnalyzerOutlierTest.cc
== [Line] -537
The first of these was on Linux in a nightly debug build of the master branch with assertions enabled and optimisation disabled: http://192.168.62.148:4343/job/MasterBuild-Debug/389/label=vm-centos-77-64-2/testReport/junit/lib.api.CDataFrameAnalyzerOutlierTest/CDataFrameAnalyzerOutlierTest/testErrors/
The second was on Windows in a normal build of the 7.x branch: http://jenkins.prelert.com:4343/job/V7.xBuild/label=vm-win2012r2-64-4/357/testReport/junit/lib.api.CDataFrameAnalyzerOutlierTest/CDataFrameAnalyzerOutlierTest/testErrors/
(London office VPN access is required for those failure links to work.)
It seems to be a race condition. On trying to reproduce, the test passed as the errors were:
2020-06-18 12:32:10,406064 UTC [4260] DEBUG CDataFrameAnalyzerOutlierTest.cc@528 [Input error: memory limit 0MB is too low to perform analysis. You need to give the process at least 1MB, but preferably more., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 66 kb and restart.]
Then trying again under valgrind the errors were:
2020-06-18 12:33:46,293898 UTC [4332] DEBUG CDataFrameAnalyzerOutlierTest.cc@528 [Input error: memory limit 0MB is too low to perform analysis. You need to give the process at least 1MB, but preferably more., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 32 kb exceeds the memory limit 10 kb. Please increase the limit to at least 64 kb and restart., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 33 kb exceeds the memory limit 10 kb. Please increase the limit to at least 65 kb and restart., Input error: required memory 32 kb exceeds the memory limit 10 kb. Please increase the limit to at least 64 kb and restart.]
The longer the test runs the more errors of the form the assertion checks for are seen.
In the failure cases the equivalent line is:
2020-06-18 10:40:00,973797 UTC [11130] DEBUG CDataFrameAnalyzerOutlierTest.cc@528 [Input error: memory limit 0MB is too low to perform analysis. You need to give the process at least 1MB, but preferably more.]
and:
2020-06-18 20:18:13,455373 UTC [3444] DEBUG CDataFrameAnalyzerOutlierTest.cc@528 [Input error: memory limit 0MB is too low to perform analysis. You need to give the process at least 1MB, but preferably more.]
There must be some situation where the test can miss the required error message.