From 59e71ff9254131876d7dda8867822b7144967ff1 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 11:52:07 -0700 Subject: [PATCH 01/12] New pitch for testing: Polling Expectations --- .../testing/NNNN-polling-expectations.md | 278 ++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 proposals/testing/NNNN-polling-expectations.md diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md new file mode 100644 index 0000000000..4d7816bacf --- /dev/null +++ b/proposals/testing/NNNN-polling-expectations.md @@ -0,0 +1,278 @@ +# Polling Expectations + +* Proposal: [ST-NNNN](NNNN-polling-expectations.md) +* Authors: [Rachel Brindle](https://github.com/younata) +* Review Manager: TBD +* Status: **Awaiting implementation** or **Awaiting review** +* Implementation: (Working on it) +* Review: (Working on it) + +## Introduction + +Test authors frequently need to wait for some background activity to complete +or reach an expected state before continuing. This proposal introduces a new API +to enable polling for an expected state. + +## Motivation + +Test authors can currently utilize the existing [`confirmation`](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)-5mqz2) +APIs or awaiting on an `async` callable in order to block test execution until +a callback is called, or an async callable returns. However, this requires the +code being tested to support callbacks or return a status as an async callable. + +This proposal adds another avenue for waiting for code to update to a specified +value, by proactively polling the test closure until it passes or a timeout is +reached. + +More concretely, we can imagine a type that updates its status over an +indefinite timeframe: + +```swift +actor Aquarium { + var dolphins: [Dolphin] + + func raiseDolphins() async { + // over a very long timeframe + dolphins.append(Dolphin()) + } +} +``` + +## Proposed solution + +This proposal introduces new overloads of the `#expect()` and `#require()` +macros that take, as arguments, a closure and a timeout value. When called, +these macros will continuously evaluate the closure until either the specific +condition passes, or the timeout has passed. The timeout period will default +to 1 second. + +There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. +Passes Once will continuously evaluate the expression until the expression +returns true. If the timeout passes without the expression ever returning true, +then a failure will be reported. Passes Always will continuously execute the +expression until the first time expression returns false or the timeout passes. +If the expression ever returns false, then a failure will be reported. + +Tests will now be able to poll code updating in the background using +either of the new overloads: + +```swift +let subject = Aquarium() +Task { + await subject.raiseDolphins() +} +await #expect(until: .passesOnce) { + subject.dolphins.count() == 1 +} +``` + +## Detailed design + +### New expectations + +We will introduce the following new overloads of `#expect()` and `#require()` to +the testing library: + +```swift +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#expect()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread. +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") + +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#expect()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread, and you +/// expect the expression to throw an error as part of succeeding +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + throws error: E, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool +) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") +where E: Error & Equatable + +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @Sendable () async throws(E) -> Bool, + throws errorMatcher: (E) async throws -> Bool +) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") +where E: Error + +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#require()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread. +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> R? +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where R: Sendable + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + throws error: E, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where E: Error & Equatable + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool, + throwing errorMatcher: (E) async throws -> Bool, +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where E: Error +``` + +### Polling Behavior + +A new type, `PollingBehavior`, to represent the behavior of a polling +expectation: + +```swift +public enum PollingBehavior { + /// Continuously evaluate the expression until the first time it returns + /// true. + /// If it does not pass once by the time the timeout is reached, then a + /// failure will be reported. + case passesOnce + + /// Continuously evaluate the expression until the first time it returns + /// false. + /// If the expression returns false, then a failure will be reported. + /// If the expression only returns true before the timeout is reached, then + /// no failure will be reported. + /// If the expression does not finish evaluating before the timeout is + /// reached, then a failure will be reported. + case passesAlways +} +``` + +### Usage + +These macros can be used with an async test function: + +```swift +@Test func `The aquarium's dolphin nursery works`() async { + let subject = Aquarium() + Task { + await subject.raiseDolphins() + } + await #expect(until: .passesOnce) { + subject.dolphins.count() == 1 + } +} +``` + +With the definition of `Aquarium` above, the closure will only need to be +evaluated a few times before it starts returning true. At which point the macro +will end, and no failure will be reported. + +If the expression never returns a value within the timeout period, then a +failure will be reported, noting that the expression was unable to be evaluated +within the timeout period: + +```swift +await #expect(until: .passesOnce, timeout: .seconds(1)) { + // Failure: The expression timed out before evaluation could finish. + try await Task.sleep(for: .seconds(10)) +} +``` + +In the case of `#require` where the expression returns an optional value, under +`PollingBehavior.passesOnce`, the expectation is considered to have passed the +first time the expression returns a non-nil value, and that value will be +returned by the expectation. Under `PollingBehavior.passesAlways`, the +expectation is considered to have passed if the expression always returns a +non-nil value. If it passes, the value returned by the last time the +expression is evaluated will be returned by the expectation. + +When no error is expected, then the first time the expression throws any error +will cause the polling expectation to stop & report the error as a failure. + +When an error is expected, then the expression is not considered to pass +unless it throws an error that equals the expected error or returns true when +evaluated by the `errorMatcher`. After which the polling continues under the +specified PollingBehavior. + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Integration with supporting tools + +We will expose the polling mechanism under ForToolsIntegrationOnly spi so that +tools may integrate with them. + +## Future directions + +The timeout default could be configured as a Suite or Test trait. Additionally, +it could be configured in some future global configuration tool. + +## Alternatives considered + +Instead of creating the `PollingBehavior` type, we could have introduced more +macros to cover that situation: `#expect(until:)` and `#expect(always:)`. +However, this would have resulted in confusion for the compiler and test authors +when trailing closure syntax is used. + +## Acknowledgments + +This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/). +In particular, thanks to [Jeff Hui](https://github.com/jeffh) for writing the +original implementation of Nimble's Polling Expectations. From 65a0e35869d57ba09e8f7cda7b4ed9d4fd25a987 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:24:29 -0700 Subject: [PATCH 02/12] Add an alternatives considered section on having passesOnce continue to evaluate the expression after it passes --- .../testing/NNNN-polling-expectations.md | 37 ++++++++++++++++++- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 4d7816bacf..7c1860f352 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -62,7 +62,7 @@ Task { await subject.raiseDolphins() } await #expect(until: .passesOnce) { - subject.dolphins.count() == 1 + await subject.dolphins.count == 1 } ``` @@ -212,7 +212,7 @@ These macros can be used with an async test function: await subject.raiseDolphins() } await #expect(until: .passesOnce) { - subject.dolphins.count() == 1 + await subject.dolphins.count == 1 } } ``` @@ -266,11 +266,44 @@ it could be configured in some future global configuration tool. ## Alternatives considered +### Remove `PollingBehavior` in favor of more macros + Instead of creating the `PollingBehavior` type, we could have introduced more macros to cover that situation: `#expect(until:)` and `#expect(always:)`. However, this would have resulted in confusion for the compiler and test authors when trailing closure syntax is used. +### `PollingBehavior.passesOnce` continues to evaluate expression after passing + +Under `PollingBehavior.passesOnce`, we thought about requiring the expression +to continue to pass after it starts passing. The idea is to prevent test +flakiness caused by an expectation that initially passes, but stops passing as +a result of (intended) background activity. For example: + +```swift +@Test func `The aquarium's dolphin nursery works`() async { + let subject = Aquarium() + await subject.raiseDolphins() + Task { + await subject.raiseDolphins() + } + await #expect(until: .passesOnce) { + await subject.dolphins.count == 1 + } +} +``` + +This test is flaky, but will pass more often than not. However, it is still +incorrect. If we were to change `PollingBehavior.passesOnce` to instead check +that the expression continues to pass after the first time it succeeds until the +timeout is reached, then this test would correctly be flagged as failing each +time it's ran. + +We chose to address this by using the name `passesOnce` instead of changing the +behavior. `passesOnce` makes it clear the exact behavior that will happen: the +expression will be evaluated until the first time it passes, and no more. We +hope that this will help test authors to better recognize these situations. + ## Acknowledgments This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/). From 0f21afbc07715bdbc761262bd15cc0a1082ba4b8 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:26:25 -0700 Subject: [PATCH 03/12] Add a link to the pitch thread --- proposals/testing/NNNN-polling-expectations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 7c1860f352..f8c98fa384 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -5,7 +5,7 @@ * Review Manager: TBD * Status: **Awaiting implementation** or **Awaiting review** * Implementation: (Working on it) -* Review: (Working on it) +* Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) ## Introduction From 6d30af3a79755bf7b8877e36d3bfe7b6c8a21f38 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:27:22 -0700 Subject: [PATCH 04/12] Set polling expectation's status to awaiting implementation I'm working on it, but macros are hard. --- proposals/testing/NNNN-polling-expectations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index f8c98fa384..16e54ad972 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -3,7 +3,7 @@ * Proposal: [ST-NNNN](NNNN-polling-expectations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD -* Status: **Awaiting implementation** or **Awaiting review** +* Status: **Awaiting implementation** * Implementation: (Working on it) * Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) From 98c3ede9f96951039141dd76d8aba96460a86263 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Wed, 14 May 2025 23:10:45 -0700 Subject: [PATCH 05/12] Swift Testing Polling Expectations: - Change the default timeout. - Change how unexpected thrown errors are treated. - Add future direction to add change monitoring via Observation - Add alternative considered of Just Use A While Loop - Add alternative considered for shorter default timeouts --- .../testing/NNNN-polling-expectations.md | 105 ++++++++++-------- 1 file changed, 57 insertions(+), 48 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 16e54ad972..7dda74e45f 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -44,7 +44,7 @@ This proposal introduces new overloads of the `#expect()` and `#require()` macros that take, as arguments, a closure and a timeout value. When called, these macros will continuously evaluate the closure until either the specific condition passes, or the timeout has passed. The timeout period will default -to 1 second. +to 1 minute. There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. Passes Once will continuously evaluate the expression until the expression @@ -88,60 +88,34 @@ the testing library: /// changes as the result of activity in another task/queue/thread. @freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> Bool ) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -/// Continuously check an expression until it matches the given PollingBehavior -/// -/// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. -/// -/// Use this overload of `#expect()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread, and you -/// expect the expression to throw an error as part of succeeding @freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, throws error: E, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool -) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") where E: Error & Equatable -@freestanding(expression) public macro expect( +@freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @Sendable () async throws(E) -> Bool, - throws errorMatcher: (E) async throws -> Bool -) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -where E: Error + performing expression: @Sendable () async throws -> Bool, + throws errorMatcher: (any Error) async throws -> Bool +) -> Error = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -/// Continuously check an expression until it matches the given PollingBehavior -/// -/// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. -/// -/// Use this overload of `#require()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread. @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> Bool @@ -149,7 +123,7 @@ where E: Error @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> R? @@ -159,22 +133,21 @@ where R: Sendable @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, throws error: E, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool + expression: @Sendable () async throws -> Bool ) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") where E: Error & Equatable -@freestanding(expression) public macro require( +@freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool, - throwing errorMatcher: (E) async throws -> Bool, + expression: @Sendable () async throws -> Bool, + throwing errorMatcher: (any Error) async throws -> Bool, ) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where E: Error ``` ### Polling Behavior @@ -201,6 +174,11 @@ public enum PollingBehavior { } ``` +### Platform Availability + +Polling expectations will not be available on platforms that do not support +Swift Concurrency, nor on platforms that do not support multiple threads. + ### Usage These macros can be used with an async test function: @@ -240,14 +218,14 @@ expectation is considered to have passed if the expression always returns a non-nil value. If it passes, the value returned by the last time the expression is evaluated will be returned by the expectation. -When no error is expected, then the first time the expression throws any error -will cause the polling expectation to stop & report the error as a failure. - When an error is expected, then the expression is not considered to pass unless it throws an error that equals the expected error or returns true when evaluated by the `errorMatcher`. After which the polling continues under the specified PollingBehavior. +When no error is expected, then this is treated as if the expression returned +false. This is specifically to invert the case when an error is expected. + ## Source compatibility This is a new interface that is unlikely to collide with any existing @@ -264,8 +242,39 @@ tools may integrate with them. The timeout default could be configured as a Suite or Test trait. Additionally, it could be configured in some future global configuration tool. +On the topic of monitoring for changes, we could add a tool integrating with the +Observation module which monitors changes to `@Observable` objects during some +lifetime. + ## Alternatives considered +### Just use a while loop + +Polling could be written as a simple while loop that continuously executes the +expression until it returns, something like: + +```swift +func poll(timeout: Duration, expression: () -> Bool) -> Bool { + let clock: Clock = // ... + let endTimestamp = clock.now + timeout + while clock.now < endTimestamp { + if expression() { return true } + } + return false +} +``` + +Which works in most naive cases, but is not robust. Notably, This approach does +not handle the case when the expression never returns, or does not return within +the timeout period. + +### Shorter default timeout + +Due to the nature of Swift Concurrency scheduling, using short default +timeouts will result in high rates of test flakiness. This is why the default +timeout is 1 minute. We do not recommend that test authors use timeouts any +shorter than this. + ### Remove `PollingBehavior` in favor of more macros Instead of creating the `PollingBehavior` type, we could have introduced more From 29c525c93424543ba0b04db4027242f021c61628 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Wed, 14 May 2025 23:21:09 -0700 Subject: [PATCH 06/12] Add a link to the PR containing polling expectations --- proposals/testing/NNNN-polling-expectations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 7dda74e45f..85d87f51eb 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -3,8 +3,8 @@ * Proposal: [ST-NNNN](NNNN-polling-expectations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD -* Status: **Awaiting implementation** -* Implementation: (Working on it) +* Status: **Awaiting review** +* Implementation: [swiftlang/swift-testing#1115](https://github.com/swiftlang/swift-testing/pull/1115) * Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) ## Introduction From c13df827795dd2f2348d14f7a23182e63bde310f Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Mon, 26 May 2025 16:05:39 -0700 Subject: [PATCH 07/12] Update polling proposal to refer to new confirmation functions --- .../testing/NNNN-polling-expectations.md | 325 +++++++++--------- 1 file changed, 155 insertions(+), 170 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 85d87f51eb..ae86be4f08 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -40,18 +40,14 @@ actor Aquarium { ## Proposed solution -This proposal introduces new overloads of the `#expect()` and `#require()` -macros that take, as arguments, a closure and a timeout value. When called, -these macros will continuously evaluate the closure until either the specific -condition passes, or the timeout has passed. The timeout period will default -to 1 minute. - -There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. -Passes Once will continuously evaluate the expression until the expression -returns true. If the timeout passes without the expression ever returning true, -then a failure will be reported. Passes Always will continuously execute the -expression until the first time expression returns false or the timeout passes. -If the expression ever returns false, then a failure will be reported. +This proposal introduces new members of the `confirmation` family of functions: +`confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in +a closure to be continuously evaluated until the specific condition passes. + +`confirmPassesEventually` will evaluate the closure until the first time it +returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the +closure until it returns false or nil. If neither case happens, evaluation will +continue until the closure has been called some amount of times. Tests will now be able to poll code updating in the background using either of the new overloads: @@ -61,127 +57,141 @@ let subject = Aquarium() Task { await subject.raiseDolphins() } -await #expect(until: .passesOnce) { - await subject.dolphins.count == 1 +await confirmPassesEventually { + subject.dolphins.count == 1 } ``` ## Detailed design -### New expectations +### New confirmation functions -We will introduce the following new overloads of `#expect()` and `#require()` to -the testing library: +We will introduce 4 new members of the confirmation family of functions to the +testing library: ```swift -/// Continuously check an expression until it matches the given PollingBehavior +/// Confirm that some expression eventually returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmPassesEventually( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async + +/// Confirm that some expression eventually returns a non-nil value /// /// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. /// -/// Use this overload of `#expect()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread. -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") - -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - throws error: E, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -where E: Error & Equatable - -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @Sendable () async throws -> Bool, - throws errorMatcher: (any Error) async throws -> Bool -) -> Error = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> R? -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where R: Sendable - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - throws error: E, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where E: Error & Equatable - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool, - throwing errorMatcher: (any Error) async throws -> Bool, -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +/// - Returns: The first non-nil value returned by `body`. +/// +/// - Throws: A `PollingFailedError` will be thrown if `body` never returns a +/// non-optional value +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmPassesEventually( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> R? +) async throws -> R where R: Sendable + +/// Confirm that some expression always returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmAlwaysPasses( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async + +/// Confirm that some expression always returns a non-optional value +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Returns: The value from the last time `body` was invoked. +/// +/// - Throws: A `PollingFailedError` will be thrown if `body` ever returns a +/// non-optional value +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmAlwaysPasses( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> R? +) async throws -> R where R: Sendable ``` -### Polling Behavior +### New Error Type -A new type, `PollingBehavior`, to represent the behavior of a polling -expectation: +A new error type, `PollingFailedError` to be thrown when polling doesn't return +a non-nil value: ```swift -public enum PollingBehavior { - /// Continuously evaluate the expression until the first time it returns - /// true. - /// If it does not pass once by the time the timeout is reached, then a - /// failure will be reported. - case passesOnce - - /// Continuously evaluate the expression until the first time it returns - /// false. - /// If the expression returns false, then a failure will be reported. - /// If the expression only returns true before the timeout is reached, then - /// no failure will be reported. - /// If the expression does not finish evaluating before the timeout is - /// reached, then a failure will be reported. - case passesAlways -} +/// A type describing an error thrown when polling fails to return a non-nil +/// value +public struct PollingFailedError: Error {} ``` +### New Issue Kind + +A new Issue.Kind, `confirmationPollingFailed` will be added to represent the +case here confirmation polling fails. This issue kind will be recorded when +polling fails. + ### Platform Availability Polling expectations will not be available on platforms that do not support -Swift Concurrency, nor on platforms that do not support multiple threads. +Swift Concurrency. ### Usage -These macros can be used with an async test function: +These functions can be used with an async test function: ```swift @Test func `The aquarium's dolphin nursery works`() async { @@ -189,42 +199,24 @@ These macros can be used with an async test function: Task { await subject.raiseDolphins() } - await #expect(until: .passesOnce) { + await confirmPassesEventually { await subject.dolphins.count == 1 } } ``` With the definition of `Aquarium` above, the closure will only need to be -evaluated a few times before it starts returning true. At which point the macro +evaluated a few times before it starts returning true. At which point polling will end, and no failure will be reported. -If the expression never returns a value within the timeout period, then a -failure will be reported, noting that the expression was unable to be evaluated -within the timeout period: - -```swift -await #expect(until: .passesOnce, timeout: .seconds(1)) { - // Failure: The expression timed out before evaluation could finish. - try await Task.sleep(for: .seconds(10)) -} -``` - -In the case of `#require` where the expression returns an optional value, under -`PollingBehavior.passesOnce`, the expectation is considered to have passed the -first time the expression returns a non-nil value, and that value will be -returned by the expectation. Under `PollingBehavior.passesAlways`, the -expectation is considered to have passed if the expression always returns a -non-nil value. If it passes, the value returned by the last time the -expression is evaluated will be returned by the expectation. - -When an error is expected, then the expression is not considered to pass -unless it throws an error that equals the expected error or returns true when -evaluated by the `errorMatcher`. After which the polling continues under the -specified PollingBehavior. +Polling will be stopped in the following cases: -When no error is expected, then this is treated as if the expression returned -false. This is specifically to invert the case when an error is expected. +- After the expression has been evaluated 1 million times. +- If the task that started the polling is cancelled. +- For `confirmPassesEventually`: The first time the closure returns true or a + non-nil value +- For `confirmAlwaysPasses`: The first time the closure returns false or nil. +- The first time the closure throws an error. ## Source compatibility @@ -232,26 +224,18 @@ This is a new interface that is unlikely to collide with any existing client-provided interfaces. The typical Swift disambiguation tools can be used if needed. -## Integration with supporting tools - -We will expose the polling mechanism under ForToolsIntegrationOnly spi so that -tools may integrate with them. - ## Future directions -The timeout default could be configured as a Suite or Test trait. Additionally, -it could be configured in some future global configuration tool. - -On the topic of monitoring for changes, we could add a tool integrating with the -Observation module which monitors changes to `@Observable` objects during some +We plan to add support for more push-based monitoring, such as integrating with +the Observation module to monitor changes to `@Observable` objects during some lifetime. ## Alternatives considered -### Just use a while loop +### Use timeouts -Polling could be written as a simple while loop that continuously executes the -expression until it returns, something like: +Polling could be written in such a way that it stops after some amount of time +has passed. Naively, this could be written as: ```swift func poll(timeout: Duration, expression: () -> Bool) -> Bool { @@ -264,27 +248,27 @@ func poll(timeout: Duration, expression: () -> Bool) -> Bool { } ``` -Which works in most naive cases, but is not robust. Notably, This approach does -not handle the case when the expression never returns, or does not return within -the timeout period. +Unfortunately, while this could work reasonably well in an environment where +tests are executed serially, the concurrent test runner the testing library uses +means that timeouts are inherently unreliable. Importantly, timeouts become more +unreliable the more tests in the test suite. -### Shorter default timeout +### Use macros instead of functions -Due to the nature of Swift Concurrency scheduling, using short default -timeouts will result in high rates of test flakiness. This is why the default -timeout is 1 minute. We do not recommend that test authors use timeouts any -shorter than this. +Instead of adding new bare functions, polling could be written as additional +macros, something like: -### Remove `PollingBehavior` in favor of more macros +```swift +#expectUntil { ... } +#expectAlways { ... } +``` -Instead of creating the `PollingBehavior` type, we could have introduced more -macros to cover that situation: `#expect(until:)` and `#expect(always:)`. -However, this would have resulted in confusion for the compiler and test authors -when trailing closure syntax is used. +However, there's no additional benefit to doing this, and it may even lead test +authors to use polling when other mechanisms would be more appropriate. -### `PollingBehavior.passesOnce` continues to evaluate expression after passing +### `confirmPassesEventually` continues to evaluate expression after passing -Under `PollingBehavior.passesOnce`, we thought about requiring the expression +For `confirmPassesEventually`, we thought about requiring the expression to continue to pass after it starts passing. The idea is to prevent test flakiness caused by an expectation that initially passes, but stops passing as a result of (intended) background activity. For example: @@ -296,22 +280,23 @@ a result of (intended) background activity. For example: Task { await subject.raiseDolphins() } - await #expect(until: .passesOnce) { + await confirmPassesEventually { await subject.dolphins.count == 1 } } ``` This test is flaky, but will pass more often than not. However, it is still -incorrect. If we were to change `PollingBehavior.passesOnce` to instead check +incorrect. If we were to change `confirmPassesEventually` to instead check that the expression continues to pass after the first time it succeeds until the timeout is reached, then this test would correctly be flagged as failing each time it's ran. -We chose to address this by using the name `passesOnce` instead of changing the -behavior. `passesOnce` makes it clear the exact behavior that will happen: the -expression will be evaluated until the first time it passes, and no more. We -hope that this will help test authors to better recognize these situations. +We chose to address this by using the name `confirmPassesEventually` instead of +changing the behavior. `confirmPassesEventually` makes it clear the exact +behavior that will happen: the expression will be evaluated until the first time +it passes, and no more. We hope that this will help test authors to better +recognize these situations. ## Acknowledgments From 357dd9089ddcd1889f011c79c93667176ba3ca07 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sat, 7 Jun 2025 21:15:05 -0700 Subject: [PATCH 08/12] Polling Confirmations: Renamed from Polling Expectations - Include commenting on why not pure counts (i.e. why counts + interval) - Specify the configuration traits - Specify callsite configuration of counts + interval --- ...tions.md => NNNN-polling-confirmations.md} | 202 +++++++++++++++--- 1 file changed, 171 insertions(+), 31 deletions(-) rename proposals/testing/{NNNN-polling-expectations.md => NNNN-polling-confirmations.md} (54%) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-confirmations.md similarity index 54% rename from proposals/testing/NNNN-polling-expectations.md rename to proposals/testing/NNNN-polling-confirmations.md index ae86be4f08..8c587545a4 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -1,6 +1,6 @@ -# Polling Expectations +# Polling Confirmations -* Proposal: [ST-NNNN](NNNN-polling-expectations.md) +* Proposal: [ST-NNNN](NNNN-polling-confirmations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD * Status: **Awaiting review** @@ -42,7 +42,9 @@ actor Aquarium { This proposal introduces new members of the `confirmation` family of functions: `confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in -a closure to be continuously evaluated until the specific condition passes. +a closure to be repeatedly evaluated until the specific condition passes, +waiting at least some amount of time - specified by `pollingInterval` and +defaulting to 1 millisecond - before evaluating the closure again. `confirmPassesEventually` will evaluate the closure until the first time it returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the @@ -75,6 +77,21 @@ testing library: /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -87,6 +104,8 @@ testing library: @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmPassesEventually( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool @@ -97,6 +116,20 @@ public func confirmPassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -114,6 +147,8 @@ public func confirmPassesEventually( @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmPassesEventually( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> R? @@ -124,6 +159,19 @@ public func confirmPassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -135,36 +183,12 @@ public func confirmPassesEventually( @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmAlwaysPasses( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool ) async - -/// Confirm that some expression always returns a non-optional value -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// - Returns: The value from the last time `body` was invoked. -/// -/// - Throws: A `PollingFailedError` will be thrown if `body` ever returns a -/// non-optional value -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmAlwaysPasses( - _ comment: Comment? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> R? -) async throws -> R where R: Sendable ``` ### New Error Type @@ -184,9 +208,99 @@ A new Issue.Kind, `confirmationPollingFailed` will be added to represent the case here confirmation polling fails. This issue kind will be recorded when polling fails. +### New Traits + +Two new traits will be added to change the default values for the +`maxPollingIterations` and `pollingInterval` arguments. Test authors often +want to poll for the `passesEventually` behavior more than they poll for the +`alwaysPasses` behavior, which is why there are separate traits for configuring +defaults for these functions. + +```swift +/// A trait to provide a default polling configuration to all usages of +/// ``confirmPassesEventually`` within a test or suite. +/// +/// To add this trait to a test, use the +/// ``Trait/confirmPassesEventuallyDefaults`` function. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public struct ConfirmPassesEventuallyConfigurationTrait: TestTrait, SuiteTrait { + public var maxPollingIterations: Int? + public var pollingInterval: Duration? + + public var isRecursive: Bool { true } + + public init(maxPollingIterations: Int?, pollingInterval: Duration?) +} + +/// A trait to provide a default polling configuration to all usages of +/// ``confirmPassesAlways`` within a test or suite. +/// +/// To add this trait to a test, use the ``Trait/confirmAlwaysPassesDefaults`` +/// function. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public struct ConfirmAlwaysPassesConfigurationTrait: TestTrait, SuiteTrait { + public var maxPollingIterations: Int? + public var pollingInterval: Duration? + + public var isRecursive: Bool { true } + + public init(maxPollingIterations: Int?, pollingInterval: Duration?) +} + +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +extension Trait where Self == ConfirmPassesEventuallyConfigurationTrait { + /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. + /// + /// - Parameters: + /// - maxPollingIterations: The maximum amount of times to attempt polling. + /// If nil, polling will be attempted up to 1000 times. + /// `maxPollingIterations` must be greater than 0. + /// - pollingInterval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `pollingInterval` must be greater than 0. + public static func confirmPassesEventuallyDefaults( + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil + ) -> Self +} + +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +extension Trait where Self == ConfirmPassesAlwaysConfigurationTrait { + /// Specifies defaults for ``confirmAlwaysPasses`` in the test or suite. + /// + /// - Parameters: + /// - maxPollingIterations: The maximum amount of times to attempt polling. + /// If nil, polling will be attempted up to 1000 times. + /// `maxPollingIterations` must be greater than 0. + /// - pollingInterval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `pollingInterval` must be greater than 0. + public static func confirmAlwaysPassesDefaults( + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil + ) -> Self +} +``` + +Specifying `maxPollingIterations` or `pollingInterval` directly on either +`confirmPassesEventually` or `confirmAlwaysPasses` will override any value +provided by the trait. + +### Default Polling Configuration + +For both `confirmPassesEventually` and `confirmsAlwaysPasses`, the Testing +library will default `maxPollingIterations` to 1000, and `pollingInterval` to +1 millisecond. This allows for tests on lightly-loaded systems such as developer +workstations to run in a little over 1 second wall-clock time, while still +being able to gracefully handle running on large loads. + ### Platform Availability -Polling expectations will not be available on platforms that do not support +Polling confirmations will not be available on platforms that do not support Swift Concurrency. ### Usage @@ -211,7 +325,8 @@ will end, and no failure will be reported. Polling will be stopped in the following cases: -- After the expression has been evaluated 1 million times. +- After the expression has been evaluated up to the count specified in + `maxPollingInterations`. - If the task that started the polling is cancelled. - For `confirmPassesEventually`: The first time the closure returns true or a non-nil value @@ -253,6 +368,31 @@ tests are executed serially, the concurrent test runner the testing library uses means that timeouts are inherently unreliable. Importantly, timeouts become more unreliable the more tests in the test suite. +### Use only polling iterations + +Another option considered was only using polling iterations. Naively, this +would write the main polling loop as: + +```swift +func poll(iterations: Int, expression: () -> Bool) async -> Bool { + for _ in 0.. Date: Sat, 7 Jun 2025 21:42:31 -0700 Subject: [PATCH 09/12] Polling Confirmations: Add a couple sentences on why pollingInterval must be positive --- proposals/testing/NNNN-polling-confirmations.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 8c587545a4..0e83f78737 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -393,6 +393,14 @@ well under a millisecond. Because of this, we decided to add on the polling interval argument: a minimum duration to wait between polling, to make it much easier for test authors to predict a good-enough guess for when to stop polling. +### Allow `pollingInterval` to be `.zero` + +We could allow test authors to set `pollingInterval` as `Duration.zero`, making +polling behave as if only polling iterations is counted. +We chose not to allow this for the same reason we chose to add a wait between +polling: this makes it much easier for test authors to predict when to stop +polling. + ### Use macros instead of functions Instead of adding new bare functions, polling could be written as additional From 4512c0c4b719ef3d9f6a2007890b9254a4bebbb8 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sat, 7 Jun 2025 22:10:02 -0700 Subject: [PATCH 10/12] Polling Confirmations: Add requirePassesEventually and requireAlwaysPasses --- .../testing/NNNN-polling-confirmations.md | 94 +++++++++++++++++-- 1 file changed, 87 insertions(+), 7 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0e83f78737..0a95c475bb 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -68,7 +68,7 @@ await confirmPassesEventually { ### New confirmation functions -We will introduce 4 new members of the confirmation family of functions to the +We will introduce 5 new members of the confirmation family of functions to the testing library: ```swift @@ -111,6 +111,47 @@ public func confirmPassesEventually( _ body: @escaping () async throws -> Bool ) async +/// Require that some expression eventually returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` will be thrown if the expression never +/// returns true. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func requirePassesEventually( + _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async throws + /// Confirm that some expression eventually returns a non-nil value /// /// - Parameters: @@ -138,7 +179,7 @@ public func confirmPassesEventually( /// - Returns: The first non-nil value returned by `body`. /// /// - Throws: A `PollingFailedError` will be thrown if `body` never returns a -/// non-optional value +/// non-optional value. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For @@ -189,6 +230,45 @@ public func confirmAlwaysPasses( sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool ) async + +/// Require that some expression always returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` will be thrown if the expression ever +/// returns false. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func requireAlwaysPasses( + _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async throws ``` ### New Error Type @@ -197,16 +277,16 @@ A new error type, `PollingFailedError` to be thrown when polling doesn't return a non-nil value: ```swift -/// A type describing an error thrown when polling fails to return a non-nil -/// value -public struct PollingFailedError: Error {} +/// A type describing an error thrown when polling fails. +public struct PollingFailedError: Error, Equatable {} ``` ### New Issue Kind A new Issue.Kind, `confirmationPollingFailed` will be added to represent the -case here confirmation polling fails. This issue kind will be recorded when -polling fails. +case where a polling confirmation failed. This issue kind will be recorded when +`confirmPassesEventually` and `confirmAlwaysPasses` fail, but not when +`requirePassesEventually` or `requireAlwaysPasses` fail. ### New Traits From 0c851959724bbe94d1eb5e9e3b8094fc55844c34 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Mon, 21 Jul 2025 00:09:54 -0700 Subject: [PATCH 11/12] Polling Confirmations: rewrite, again, for new API design --- .../testing/NNNN-polling-confirmations.md | 530 ++++++++---------- 1 file changed, 240 insertions(+), 290 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0a95c475bb..0a223d3084 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -41,25 +41,36 @@ actor Aquarium { ## Proposed solution This proposal introduces new members of the `confirmation` family of functions: -`confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in -a closure to be repeatedly evaluated until the specific condition passes, -waiting at least some amount of time - specified by `pollingInterval` and -defaulting to 1 millisecond - before evaluating the closure again. - -`confirmPassesEventually` will evaluate the closure until the first time it -returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the -closure until it returns false or nil. If neither case happens, evaluation will -continue until the closure has been called some amount of times. - -Tests will now be able to poll code updating in the background using -either of the new overloads: +`confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)`. These +functions take in a closure to be repeatedly evaluated until the specific +condition passes, waiting at least some amount of time - specified by +`pollingEvery`/`interval` and defaulting to 1 millisecond - before evaluating +the closure again. + +Both of these use the new `PollingStopCondition` enum to determine when to end +polling: `PollingStopCondition.firstPass` configures polling to stop as soon +as the `body` closure returns `true` or a non-`nil` value. At this point, +the confirmation will be marked as passing. +`PollingStopCondition.stopsPassing` configures polling to stop once the `body` +closure returns `false` or a `nil` value. At this point, the confirmation will +be marked as failing: an error will be thrown, and an issue will be recorded. + +Under both `PollingStopCondition` cases, when the early stop condition isn't +reached, polling will continue up until approximately the `within`/`duration` +value has elapsed. When `PollingStopCondition.firstPass` is specified, reaching +the duration stop point will mark the confirmation as failing. +When `PollingStopCondition.stopsPassing` is specified, reaching the duration +stop point will mark the confirmation as passing. + +Tests will now be able to poll code updating in the background using either of +the stop conditions: ```swift let subject = Aquarium() Task { await subject.raiseDolphins() } -await confirmPassesEventually { +await confirmation(until: .firstPass) { subject.dolphins.count == 1 } ``` @@ -68,85 +79,52 @@ await confirmPassesEventually { ### New confirmation functions -We will introduce 5 new members of the confirmation family of functions to the +We will introduce 2 new members of the confirmation family of functions to the testing library: ```swift -/// Confirm that some expression eventually returns true +/// Poll expression within the duration based on the given stop condition /// /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or -/// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, waiting on some state to change that cannot be easily confirmed -/// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmPassesEventually( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async - -/// Require that some expression eventually returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. /// - body: The function to invoke. /// -/// - Throws: A `PollingFailedError` will be thrown if the expression never -/// returns true. +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func requirePassesEventually( +@available(_clockAPI, *) +public func confirmation( _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool @@ -157,232 +135,219 @@ public func requirePassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. /// - body: The function to invoke. /// -/// - Returns: The first non-nil value returned by `body`. +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. /// -/// - Throws: A `PollingFailedError` will be thrown if `body` never returns a -/// non-optional value. +/// - Returns: The last non-nil value returned by `body`. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmPassesEventually( +@available(_clockAPI, *) +@discardableResult +public func confirmation( _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> R? -) async throws -> R where R: Sendable + _ body: @escaping () async throws -> sending R? +) async throws -> R +``` -/// Confirm that some expression always returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmAlwaysPasses( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async +### New `PollingStopCondition` enum -/// Require that some expression always returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// - Throws: A `PollingFailedError` will be thrown if the expression ever -/// returns false. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func requireAlwaysPasses( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async throws +A new enum type, `PollingStopCondition` will be defined, specifying when to stop +polling before the duration has elapsed. Additionally, if the early stop +condition isn't fulfilled before the duration elapses, then this also defines +how the confirmation should be handled. + +```swift +/// A type defining when to stop polling early. +/// This also determines what happens if the duration elapses during polling. +public enum PollingStopCondition: Sendable { + /// Evaluates the expression until the first time it returns true. + /// If it does not pass once by the time the timeout is reached, then a + /// failure will be reported. + case firstPass + + /// Evaluates the expression until the first time it returns false. + /// If the expression returns false, then a failure will be reported. + /// If the expression only returns true before the timeout is reached, then + /// no failure will be reported. + /// If the expression does not finish evaluating before the timeout is + /// reached, then a failure will be reported. + case stopsPassing +} ``` ### New Error Type -A new error type, `PollingFailedError` to be thrown when polling doesn't return -a non-nil value: +A new error type, `PollingFailedError` to be thrown when the polling +confirmation doesn't pass: ```swift /// A type describing an error thrown when polling fails. -public struct PollingFailedError: Error, Equatable {} +public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} ``` -### New Issue Kind - -A new Issue.Kind, `confirmationPollingFailed` will be added to represent the -case where a polling confirmation failed. This issue kind will be recorded when -`confirmPassesEventually` and `confirmAlwaysPasses` fail, but not when -`requirePassesEventually` or `requireAlwaysPasses` fail. - ### New Traits Two new traits will be added to change the default values for the -`maxPollingIterations` and `pollingInterval` arguments. Test authors often -want to poll for the `passesEventually` behavior more than they poll for the -`alwaysPasses` behavior, which is why there are separate traits for configuring -defaults for these functions. +`duration` and `interval` arguments. Test authors will often want to poll for +the `firstPass` stop condition for longer than they poll for the +`stopsPassing` stop condition, which is why there are separate traits for +configuring defaults for these functions. ```swift /// A trait to provide a default polling configuration to all usages of -/// ``confirmPassesEventually`` within a test or suite. +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.firstPass`` +/// stop condition. /// /// To add this trait to a test, use the -/// ``Trait/confirmPassesEventuallyDefaults`` function. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public struct ConfirmPassesEventuallyConfigurationTrait: TestTrait, SuiteTrait { - public var maxPollingIterations: Int? - public var pollingInterval: Duration? +/// ``Trait/pollingUntilFirstPassDefaults`` function. +@available(_clockAPI, *) +public struct PollingUntilFirstPassConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? public var isRecursive: Bool { true } - - public init(maxPollingIterations: Int?, pollingInterval: Duration?) } /// A trait to provide a default polling configuration to all usages of -/// ``confirmPassesAlways`` within a test or suite. +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.stopsPassing`` +/// stop condition. /// -/// To add this trait to a test, use the ``Trait/confirmAlwaysPassesDefaults`` +/// To add this trait to a test, use the ``Trait/pollingUntilStopsPassingDefaults`` /// function. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public struct ConfirmAlwaysPassesConfigurationTrait: TestTrait, SuiteTrait { - public var maxPollingIterations: Int? - public var pollingInterval: Duration? +@available(_clockAPI, *) +public struct PollingUntilStopsPassingConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? public var isRecursive: Bool { true } - - public init(maxPollingIterations: Int?, pollingInterval: Duration?) } -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -extension Trait where Self == ConfirmPassesEventuallyConfigurationTrait { +@available(_clockAPI, *) +extension Trait where Self == PollingUntilFirstPassConfigurationTrait { /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. /// /// - Parameters: - /// - maxPollingIterations: The maximum amount of times to attempt polling. - /// If nil, polling will be attempted up to 1000 times. - /// `maxPollingIterations` must be greater than 0. - /// - pollingInterval: The minimum amount of time to wait between polling + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling /// attempts. - /// `pollingInterval` must be greater than 0. - public static func confirmPassesEventuallyDefaults( - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil + /// `interval` must be greater than 0. + public static func pollingUntilFirstPassDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil ) -> Self } -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -extension Trait where Self == ConfirmPassesAlwaysConfigurationTrait { - /// Specifies defaults for ``confirmAlwaysPasses`` in the test or suite. +@available(_clockAPI, *) +extension Trait where Self == PollingUntilStopsPassingConfigurationTrait { + /// Specifies defaults for ``confirmPassesAlways`` in the test or suite. /// /// - Parameters: - /// - maxPollingIterations: The maximum amount of times to attempt polling. - /// If nil, polling will be attempted up to 1000 times. - /// `maxPollingIterations` must be greater than 0. - /// - pollingInterval: The minimum amount of time to wait between polling + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling /// attempts. - /// `pollingInterval` must be greater than 0. - public static func confirmAlwaysPassesDefaults( - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil + /// `interval` must be greater than 0. + public static func pollingUntilStopsPassingDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil ) -> Self } ``` -Specifying `maxPollingIterations` or `pollingInterval` directly on either -`confirmPassesEventually` or `confirmAlwaysPasses` will override any value -provided by the trait. +Specifying `duration` or `interval` directly on either new `confirmation` +function will override any value provided by the relevant trait. Additionally, +when multiple of these configuration traits are specified, the innermost or +last trait will be applied. ### Default Polling Configuration -For both `confirmPassesEventually` and `confirmsAlwaysPasses`, the Testing -library will default `maxPollingIterations` to 1000, and `pollingInterval` to -1 millisecond. This allows for tests on lightly-loaded systems such as developer -workstations to run in a little over 1 second wall-clock time, while still -being able to gracefully handle running on large loads. +For all polling confirmations, the Testing library will default `duration` to +1 second, and `interval` to 1 millisecond. ### Platform Availability Polling confirmations will not be available on platforms that do not support Swift Concurrency. +### Duration and Concurrent Execution + +It is an unfortunate side effect that directly using the `duration` to determine +when to stop polling (i.e. `while duration has not elapsed { poll() }`) is +unreliable in a parallel execution environment. Especially on systems that are +under-resourced, under very high load, or both - such as CI systems. This is +especially the case for the Testing library, which, at time of writing, submits +every test at once to the concurrency system for scheduling. Under this +environment, with heavily-burdened machines running test suites with a very +large amount of tests, there is a very real case that a polling confirmation's +`duration` might elapse before the `body` has had a chance to return even once. + +To prevent this, the Testing library will calculate how many times to poll the +`body`. This is done by dividing the `duration` by the `interval`. For example, +with the default 1 second duration and 1 millisecond interval, the Testing +library will poll 1000 times, waiting 1 millisecond between polling attempts. +This works and is immune to the issues posed by concurrent execution on +heavily-burdened systems. +This is also very easy for test authors to understand and predict, even if it is +not fully accurate - each poll attempt takes some amount of time, even for very +fast `body` closures. Which means that the real-time duration of a polling +confirmation will always be longer than the value specified in the `duration` +argument. + ### Usage These functions can be used with an async test function: @@ -393,7 +358,7 @@ These functions can be used with an async test function: Task { await subject.raiseDolphins() } - await confirmPassesEventually { + await confirmation(until: .firstPass) { await subject.dolphins.count == 1 } } @@ -405,12 +370,12 @@ will end, and no failure will be reported. Polling will be stopped in the following cases: -- After the expression has been evaluated up to the count specified in - `maxPollingInterations`. +- The specified `duration` has elapsed. - If the task that started the polling is cancelled. -- For `confirmPassesEventually`: The first time the closure returns true or a - non-nil value -- For `confirmAlwaysPasses`: The first time the closure returns false or nil. +- For `PollingStopCondition.firstPass`: The first time the closure returns true + or a non-nil value +- For `PollingStopCondition.stopsPassing`: The first time the closure returns + false or nil. - The first time the closure throws an error. ## Source compatibility @@ -421,13 +386,49 @@ if needed. ## Future directions +### More `confirmation` types + We plan to add support for more push-based monitoring, such as integrating with the Observation module to monitor changes to `@Observable` objects during some lifetime. +These are out of scope for this proposal, and may be part of future proposals. + +### Adding timeouts to existing `confirmation` APIs + +One common request for the existing `confirmation` APIs is a timeout: wait +either until the condition is met, or some amount of time has passed. Adding +that would require additional consideration outside of the context of this +proposal. As such, adding timeouts to the existing (or future) `confirmation` +APIs may be part of a future proposal. + +### More Stop Conditions + +One possible future direction is adding additional stop conditions. For example, +a stop condition where we expect the body closure to initially be false, but to +continue passing once it starts passing. Or a `custom` stop condition, allowing +test authors to define their own stop conditions. + +In order to keep this proposal focused, I chose not to add them yet. They may +be added as part of future proposals. + ## Alternatives considered -### Use timeouts +### Use separate functions instead of the `PollingStopCondition` enum + +Instead of the `PollingStopCondition` enum, we could have created different +functions for each stop condition. This would double the number new confirmation +functions being added, and require additional `confirmation` functions to be +added as we define new stop conditions. In addition to ballooning the number +of `confirmation` functions, this would also harm usability: to differentiate +polling confirmations from the other `confirmation` functions, there needs to be +at least one named argument without a default which isn't the `body` closure. +I was unwilling to compromise on the `duration` and `interval` arguments, +because being able to fall back to defaults is important to usability. +Instead, I created the `stopCondition` argument as the one named argument +without a default. + +### Directly use timeouts Polling could be written in such a way that it stops after some amount of time has passed. Naively, this could be written as: @@ -448,38 +449,16 @@ tests are executed serially, the concurrent test runner the testing library uses means that timeouts are inherently unreliable. Importantly, timeouts become more unreliable the more tests in the test suite. -### Use only polling iterations +### Use polling iterations -Another option considered was only using polling iterations. Naively, this -would write the main polling loop as: - -```swift -func poll(iterations: Int, expression: () -> Bool) async -> Bool { - for _ in 0.. Date: Mon, 28 Jul 2025 14:25:27 -0700 Subject: [PATCH 12/12] Mention the new Issue.Kind case, as well as why polling confirmations do not have a configurable clock --- .../testing/NNNN-polling-confirmations.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0a223d3084..31fcc2be45 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -217,6 +217,25 @@ confirmation doesn't pass: public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} ``` +### New `Issue.Kind` case + +A new issue kind will be added to report specifically when a test fails due to +a failed polling confirmation. + +```swift +public struct Issue { + public enum Kind { + /// An issue due to a polling confirmation having failed. + /// + /// This issue can occur when calling ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` + /// or + /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` + /// whenever the polling fails, as described in ``PollingStopCondition``. + case pollingConfirmationFailed + } +} +``` + ### New Traits Two new traits will be added to change the default values for the @@ -460,6 +479,16 @@ to predict a good-enough polling iterations value. Most test authors will think in terms of a duration, and we would expect nearly all test authors to add helpers to compute a polling iteration for them. +### Take in a `Clock` instance + +Polling confirmations could take in and use an custom Clock by test authors. +This is not supported because Polling is often used to wait out other delays, +which may or may not use the specified Clock. By staying with the default +continuous clock, Polling confirmations will continue to work even if a test +author otherwise uses a non-standard clock, such as one that skips all sleep +calls, or a clock that allows test authors to specifically control how it +advances. + ### Use macros instead of functions Instead of adding new bare functions, polling could be written as additional