-
Notifications
You must be signed in to change notification settings - Fork 25
Description
Description
Current Behavior
Currently, when multiple Cypress tests are associated with a single TestRail case_id, each test result updates the TestRail case status individually. This means the final status in TestRail reflects only the last test that executed, regardless of whether previous tests passed or failed.
For example, if three tests are associated with case_id C12345:
- Test 1: Fails
- Test 2: Fails
- Test 3: Passes
The TestRail case C12345 will show as "Passed" because the last update wins, hiding the fact that some tests actually failed.
Proposed Solution
Implement an aggregation rule for test results when multiple Cypress tests map to the same TestRail case_id:
All associated tests must pass for the TestRail case to be marked as "Passed"
If any associated test fails, the TestRail case should be marked as "Failed"
Update the TestRail case only once with the final aggregated status after all related tests complete
Example Behavior
With the proposed feature:
- Test 1: Fails
- Test 2: Fails
- Test 3: Passes
The TestRail case C12345 would correctly show as "Failed" because not all tests passed.
Implementation Suggestion
This could be implemented as a new configuration option like aggregateResults: true
however, in this case, the aggregation approach takes care of an edge case that was not originally handled so we do not believe there is any reason that one would want to run with aggregateResults: false. Therefore, it should be implemented without a configuration option, and always apply the aggregation rule.