Skip to content

Builds Are Marked as Unstable When Flaky Tests Fail Once Then Pass #38148

@cailafinn

Description

@cailafinn

Describe the bug
Jenkins is marking builds as unstable if certain (I think) flaky tests fail.

As far as I can tell, it's anything that produces an Error: blah blah blah message when the assertion fails.

This is due to the Jenkins Warnings plugin, which reads through the build log to try and find anything that would represent a build error. Unfortunately it's catching flaky tests, rather than anything in the build.

To Reproduce
See this run of pull_requests_conda_linux: https://builds.mantidproject.org/job/pull_requests-conda-linux/9042/

Expected behavior
The plugin can be pointed at a specific file to check for build warnings. So outputting the result of the build and pointing the Warnings plugin at that is probably a good solution.

Metadata

Metadata

Assignees

Labels

BugIssues and pull requests that are regressions or would be considered a bug by users (e.g. crashing)DevOpsIssues and pull requests related to DevOps

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions