Add failure summary to ConsoleOutputRecorder

[tienquocbui]

Overview

This PR implements a failure summary section for ConsoleOutputRecorder to help users quickly locate and review test failures in large test suites.

Fixes #1355
Fixes rdar://168698184

---

Architectural Changes Implemented

1. State Management

• Uses existing HumanReadableOutputRecorder state with single Locked<Context>
• Fits naturally with existing Graph traversal logic
• No additional state in ConsoleOutputRecorder (stateless design preserved)

2. Data Structure

• Added lightweight TestData.IssueInfo struct with: sourceLocation, description, isKnown, severity
• Issues stored as array in TestData.issues (each issue has its own source location)
• Test metadata (displayName, testCaseArguments) stored once per test in TestData
• Existing issueCount dictionary preserved for efficient parallel tracking

3. New TestRunSummary Type

• Dedicated type for failure summary generation
• Breaks down logic into focused methods: header(), formatFailedTest(), fullyQualifiedName(), formatIssue()
• Collects failures from Graph and formats them independently
• fileprivate initializer uses HumanReadableOutputRecorder.Context types

4. API Improvements

• generateFailureSummary() returns Optional<String> (nil when no failures)
• Uses expandedDescription() for normal output, expandedDebugDescription() when verbose
• ConsoleOutputRecorder calls this at runEnded event with blank line spacing

5. Output Features

• Fully qualified suite/test paths (addresses Add failure summary section to console output for easier debugging #1355)
• Custom display names shown in quotes (e.g., "Custom Display Name")
• Parameterized test arguments displayed once per test (e.g., (value → 7))
• Each issue listed with indentation, description, and source location
• Visual separation with blank lines before and after summary

Example Output

Regular Test Failure:

✗  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26

Parameterized Test Failure (shows which argument failed):

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72

Test with Custom Display Name:

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83

Complete Test Output

Click to expand full test run output

(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % swift test --filter "FailureSummaryDemoTests" --disable-xctest
[1/1] Planning build
Building for debugging...
[257/257] Linking swift-testingPackageTests
Build complete! (39.83s)
􀟈  Test run started.
􀄵  Testing Library Version: 6.3-dev (d8b140d780dc2da - modified)
􀟈  Suite "Failure Summary Demo" started.
􀟈  Suite "Math Operations" started.
􀟈  Suite "String Operations" started.
􀟈  Test "Addition works correctly" started.
􀟈  Test "String comparison passes" started.
􀟈  Test "Multiple string failures" started.
􀟈  Test "String concatenation fails" started.
􀟈  Suite "Array Operations" started.
􀟈  Test "Division fails" started.
􀟈  Test "Array equality" started.
􀟈  Suite "Parameterized Tests" started.
􁁛  Test "Addition works correctly" passed after 0.001 seconds.
􁁛  Test "String comparison passes" passed after 0.001 seconds.
􁁛  Test "Array equality" passed after 0.001 seconds.
􀟈  Test "Array contains element" started.
􀟈  Test "This is a custom display name" started.
􀟈  Test "Check even numbers" started.
􀟈  Test "String length validation" started.
􀟈  Test case passing 1 argument value → 4 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 6 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 7 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 10 to "Check even numbers" started.
􀟈  Test case passing 1 argument text → "hello" to "String length validation" started.
􀟈  Test case passing 1 argument text → "world" to "String length validation" started.
􀟈  Test case passing 1 argument text → "a" to "String length validation" started.
􀟈  Test case passing 1 argument value → 2 to "Check even numbers" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:46:7: Expectation failed: "a" == "b"
􀢄  Test "String concatenation fails" recorded an issue at FailureSummaryDemoTests.swift:36:7: Expectation failed: (greeting + name → "HelloWorld") == "Hello World"
􀢄  Test "This is a custom display name" recorded an issue at FailureSummaryDemoTests.swift:83:7: Expectation failed: (value → 42) == 100
􀢄  Test "String length validation" recorded an issue with 1 argument text → "a" at FailureSummaryDemoTests.swift:77:7: Expectation failed: (text.count → 1) >= 5
􀢄  Test "Array contains element" recorded an issue at FailureSummaryDemoTests.swift:57:7: Expectation failed: (numbers → [1, 2, 3, 4, 5]).contains(10)
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:47:7: Expectation failed: ("hello".count → 5) == 10
􀢄  Test "Division fails" recorded an issue at FailureSummaryDemoTests.swift:26:7: Expectation failed: (result → 3) == 4
􀢄  Test "Check even numbers" recorded an issue with 1 argument value → 7 at FailureSummaryDemoTests.swift:72:7: Expectation failed: (value % 2 → 1) == 0
􀢄  Test "This is a custom display name" failed after 0.003 seconds with 1 issue.
􀢄  Test "Array contains element" failed after 0.004 seconds with 1 issue.
􀟈  Test case passing 1 argument text → "swift" to "String length validation" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:48:7: Expectation failed: ("swift".uppercased() → "SWIFT") == "swift"
􀢄  Test "String concatenation fails" failed after 0.005 seconds with 1 issue.
􀢄  Suite "Array Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Division fails" failed after 0.005 seconds with 1 issue.
􀢄  Test "String length validation" with 4 test cases failed after 0.004 seconds with 1 issue.
􀢄  Test "Check even numbers" with 5 test cases failed after 0.004 seconds with 1 issue.
􀢄  Suite "Math Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Multiple string failures" failed after 0.005 seconds with 3 issues.
􀢄  Suite "Parameterized Tests" failed after 0.005 seconds with 3 issues.
􀢄  Suite "String Operations" failed after 0.005 seconds with 4 issues.
􀢄  Suite "Failure Summary Demo" failed after 0.007 seconds with 9 issues.
􀄵  /// Demo tests showcasing the refactored failure summary feature.

Test run had 7 failed tests with 9 issues:
􀢄  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26
􀢄  TestingTests/FailureSummaryDemoTests/ArrayTests/"Array contains element"
  - (numbers: Swift.Array<Swift.Int> → [1, 2, 3, 4, 5]).contains(10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:57
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testStringLength(text:)/"String length validation"
  (text → "a")
  - (text.count: Swift.Int → 1) >= (5: Swift.Int → 5)
    at TestingTests/FailureSummaryDemoTests.swift:77
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"Multiple string failures"
  - ("a": Swift.String → "a") == ("b": Swift.String → "b")
    at TestingTests/FailureSummaryDemoTests.swift:46
  - ("hello".count: Swift.Int → 5) == (10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:47
  - ("swift".uppercased(): Swift.String → "SWIFT") == ("swift": Swift.String → "swift")
    at TestingTests/FailureSummaryDemoTests.swift:48
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"String concatenation fails"
  - (greeting + name: Swift.String → "HelloWorld") == ("Hello World": Swift.String → "Hello World")
    at TestingTests/FailureSummaryDemoTests.swift:36

􀢄  Test run with 10 tests in 5 suites failed after 0.007 seconds with 9 issues.
(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % 

[grynspan]

Would it make sense to factor the new code into a dedicated TestRunSummary type? (I'm not saying you must, just asking.)

[tienquocbui]

@grynspan thanks for the suggestion! I discussed that with Stuart and I've refactored the failure summary logic into a dedicated TestRunSummary type. Would you mind taking another look when you get a chance? Thanks!

[stmontgomery]

We met about this offline, and I made a handful of commits of my own which I've just pushed to the PR branch after discussion. Beyond that, there is one problem in how the fully-qualified name is formed for parameterized test functions, and cases within them, in the failure summary section, and @tienquocbui it sounds like you will address that (thanks!)

[tienquocbui]

@stmontgomery I've addressed the parameterized test display issue. Here's what changed:

Before:

✗ FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

After:

✗ FailureSummaryDemoTests/ParameterizedTests/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

[grynspan]

If we only store for error or greater, do we need to store the severity in the data structure?

[tienquocbui]

I've kept it in the struct for now since Stuart had asked to track it, it could be useful if we later want to include warnings in the summary or format issues differently by severity. Happy to remove it if you both agree it's not needed.