diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md new file mode 100644 index 0000000000..31fcc2be45 --- /dev/null +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -0,0 +1,512 @@ +# Polling Confirmations + +* Proposal: [ST-NNNN](NNNN-polling-confirmations.md) +* Authors: [Rachel Brindle](https://github.com/younata) +* Review Manager: TBD +* Status: **Awaiting review** +* Implementation: [swiftlang/swift-testing#1115](https://github.com/swiftlang/swift-testing/pull/1115) +* Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) + +## Introduction + +Test authors frequently need to wait for some background activity to complete +or reach an expected state before continuing. This proposal introduces a new API +to enable polling for an expected state. + +## Motivation + +Test authors can currently utilize the existing [`confirmation`](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)-5mqz2) +APIs or awaiting on an `async` callable in order to block test execution until +a callback is called, or an async callable returns. However, this requires the +code being tested to support callbacks or return a status as an async callable. + +This proposal adds another avenue for waiting for code to update to a specified +value, by proactively polling the test closure until it passes or a timeout is +reached. + +More concretely, we can imagine a type that updates its status over an +indefinite timeframe: + +```swift +actor Aquarium { + var dolphins: [Dolphin] + + func raiseDolphins() async { + // over a very long timeframe + dolphins.append(Dolphin()) + } +} +``` + +## Proposed solution + +This proposal introduces new members of the `confirmation` family of functions: +`confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)`. These +functions take in a closure to be repeatedly evaluated until the specific +condition passes, waiting at least some amount of time - specified by +`pollingEvery`/`interval` and defaulting to 1 millisecond - before evaluating +the closure again. + +Both of these use the new `PollingStopCondition` enum to determine when to end +polling: `PollingStopCondition.firstPass` configures polling to stop as soon +as the `body` closure returns `true` or a non-`nil` value. At this point, +the confirmation will be marked as passing. +`PollingStopCondition.stopsPassing` configures polling to stop once the `body` +closure returns `false` or a `nil` value. At this point, the confirmation will +be marked as failing: an error will be thrown, and an issue will be recorded. + +Under both `PollingStopCondition` cases, when the early stop condition isn't +reached, polling will continue up until approximately the `within`/`duration` +value has elapsed. When `PollingStopCondition.firstPass` is specified, reaching +the duration stop point will mark the confirmation as failing. +When `PollingStopCondition.stopsPassing` is specified, reaching the duration +stop point will mark the confirmation as passing. + +Tests will now be able to poll code updating in the background using either of +the stop conditions: + +```swift +let subject = Aquarium() +Task { + await subject.raiseDolphins() +} +await confirmation(until: .firstPass) { + subject.dolphins.count == 1 +} +``` + +## Detailed design + +### New confirmation functions + +We will introduce 2 new members of the confirmation family of functions to the +testing library: + +```swift +/// Poll expression within the duration based on the given stop condition +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. +/// If nil, this uses whatever value is specified under the last +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. +/// If nil, this uses whatever value is specified under the last +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(_clockAPI, *) +public func confirmation( + _ comment: Comment? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async throws + +/// Confirm that some expression eventually returns a non-nil value +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. +/// If nil, this uses whatever value is specified under the last +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. +/// If nil, this uses whatever value is specified under the last +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. +/// +/// - Returns: The last non-nil value returned by `body`. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(_clockAPI, *) +@discardableResult +public func confirmation( + _ comment: Comment? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> sending R? +) async throws -> R +``` + +### New `PollingStopCondition` enum + +A new enum type, `PollingStopCondition` will be defined, specifying when to stop +polling before the duration has elapsed. Additionally, if the early stop +condition isn't fulfilled before the duration elapses, then this also defines +how the confirmation should be handled. + +```swift +/// A type defining when to stop polling early. +/// This also determines what happens if the duration elapses during polling. +public enum PollingStopCondition: Sendable { + /// Evaluates the expression until the first time it returns true. + /// If it does not pass once by the time the timeout is reached, then a + /// failure will be reported. + case firstPass + + /// Evaluates the expression until the first time it returns false. + /// If the expression returns false, then a failure will be reported. + /// If the expression only returns true before the timeout is reached, then + /// no failure will be reported. + /// If the expression does not finish evaluating before the timeout is + /// reached, then a failure will be reported. + case stopsPassing +} +``` + +### New Error Type + +A new error type, `PollingFailedError` to be thrown when the polling +confirmation doesn't pass: + +```swift +/// A type describing an error thrown when polling fails. +public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} +``` + +### New `Issue.Kind` case + +A new issue kind will be added to report specifically when a test fails due to +a failed polling confirmation. + +```swift +public struct Issue { + public enum Kind { + /// An issue due to a polling confirmation having failed. + /// + /// This issue can occur when calling ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` + /// or + /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` + /// whenever the polling fails, as described in ``PollingStopCondition``. + case pollingConfirmationFailed + } +} +``` + +### New Traits + +Two new traits will be added to change the default values for the +`duration` and `interval` arguments. Test authors will often want to poll for +the `firstPass` stop condition for longer than they poll for the +`stopsPassing` stop condition, which is why there are separate traits for +configuring defaults for these functions. + +```swift +/// A trait to provide a default polling configuration to all usages of +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.firstPass`` +/// stop condition. +/// +/// To add this trait to a test, use the +/// ``Trait/pollingUntilFirstPassDefaults`` function. +@available(_clockAPI, *) +public struct PollingUntilFirstPassConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? + + public var isRecursive: Bool { true } +} + +/// A trait to provide a default polling configuration to all usages of +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.stopsPassing`` +/// stop condition. +/// +/// To add this trait to a test, use the ``Trait/pollingUntilStopsPassingDefaults`` +/// function. +@available(_clockAPI, *) +public struct PollingUntilStopsPassingConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? + + public var isRecursive: Bool { true } +} + +@available(_clockAPI, *) +extension Trait where Self == PollingUntilFirstPassConfigurationTrait { + /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. + /// + /// - Parameters: + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `interval` must be greater than 0. + public static func pollingUntilFirstPassDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil + ) -> Self +} + +@available(_clockAPI, *) +extension Trait where Self == PollingUntilStopsPassingConfigurationTrait { + /// Specifies defaults for ``confirmPassesAlways`` in the test or suite. + /// + /// - Parameters: + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `interval` must be greater than 0. + public static func pollingUntilStopsPassingDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil + ) -> Self +} +``` + +Specifying `duration` or `interval` directly on either new `confirmation` +function will override any value provided by the relevant trait. Additionally, +when multiple of these configuration traits are specified, the innermost or +last trait will be applied. + +### Default Polling Configuration + +For all polling confirmations, the Testing library will default `duration` to +1 second, and `interval` to 1 millisecond. + +### Platform Availability + +Polling confirmations will not be available on platforms that do not support +Swift Concurrency. + +### Duration and Concurrent Execution + +It is an unfortunate side effect that directly using the `duration` to determine +when to stop polling (i.e. `while duration has not elapsed { poll() }`) is +unreliable in a parallel execution environment. Especially on systems that are +under-resourced, under very high load, or both - such as CI systems. This is +especially the case for the Testing library, which, at time of writing, submits +every test at once to the concurrency system for scheduling. Under this +environment, with heavily-burdened machines running test suites with a very +large amount of tests, there is a very real case that a polling confirmation's +`duration` might elapse before the `body` has had a chance to return even once. + +To prevent this, the Testing library will calculate how many times to poll the +`body`. This is done by dividing the `duration` by the `interval`. For example, +with the default 1 second duration and 1 millisecond interval, the Testing +library will poll 1000 times, waiting 1 millisecond between polling attempts. +This works and is immune to the issues posed by concurrent execution on +heavily-burdened systems. +This is also very easy for test authors to understand and predict, even if it is +not fully accurate - each poll attempt takes some amount of time, even for very +fast `body` closures. Which means that the real-time duration of a polling +confirmation will always be longer than the value specified in the `duration` +argument. + +### Usage + +These functions can be used with an async test function: + +```swift +@Test func `The aquarium's dolphin nursery works`() async { + let subject = Aquarium() + Task { + await subject.raiseDolphins() + } + await confirmation(until: .firstPass) { + await subject.dolphins.count == 1 + } +} +``` + +With the definition of `Aquarium` above, the closure will only need to be +evaluated a few times before it starts returning true. At which point polling +will end, and no failure will be reported. + +Polling will be stopped in the following cases: + +- The specified `duration` has elapsed. +- If the task that started the polling is cancelled. +- For `PollingStopCondition.firstPass`: The first time the closure returns true + or a non-nil value +- For `PollingStopCondition.stopsPassing`: The first time the closure returns + false or nil. +- The first time the closure throws an error. + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Future directions + +### More `confirmation` types + +We plan to add support for more push-based monitoring, such as integrating with +the Observation module to monitor changes to `@Observable` objects during some +lifetime. + +These are out of scope for this proposal, and may be part of future proposals. + +### Adding timeouts to existing `confirmation` APIs + +One common request for the existing `confirmation` APIs is a timeout: wait +either until the condition is met, or some amount of time has passed. Adding +that would require additional consideration outside of the context of this +proposal. As such, adding timeouts to the existing (or future) `confirmation` +APIs may be part of a future proposal. + +### More Stop Conditions + +One possible future direction is adding additional stop conditions. For example, +a stop condition where we expect the body closure to initially be false, but to +continue passing once it starts passing. Or a `custom` stop condition, allowing +test authors to define their own stop conditions. + +In order to keep this proposal focused, I chose not to add them yet. They may +be added as part of future proposals. + +## Alternatives considered + +### Use separate functions instead of the `PollingStopCondition` enum + +Instead of the `PollingStopCondition` enum, we could have created different +functions for each stop condition. This would double the number new confirmation +functions being added, and require additional `confirmation` functions to be +added as we define new stop conditions. In addition to ballooning the number +of `confirmation` functions, this would also harm usability: to differentiate +polling confirmations from the other `confirmation` functions, there needs to be +at least one named argument without a default which isn't the `body` closure. +I was unwilling to compromise on the `duration` and `interval` arguments, +because being able to fall back to defaults is important to usability. +Instead, I created the `stopCondition` argument as the one named argument +without a default. + +### Directly use timeouts + +Polling could be written in such a way that it stops after some amount of time +has passed. Naively, this could be written as: + +```swift +func poll(timeout: Duration, expression: () -> Bool) -> Bool { + let clock: Clock = // ... + let endTimestamp = clock.now + timeout + while clock.now < endTimestamp { + if expression() { return true } + } + return false +} +``` + +Unfortunately, while this could work reasonably well in an environment where +tests are executed serially, the concurrent test runner the testing library uses +means that timeouts are inherently unreliable. Importantly, timeouts become more +unreliable the more tests in the test suite. + +### Use polling iterations + +Another option considered was using polling iterations, either solely or +combined with the interval value. + +However, while this works and is resistant to many of the issues timeouts face +in concurrent testing environments, it is extremely difficult for test authors +to predict a good-enough polling iterations value. Most test authors will think +in terms of a duration, and we would expect nearly all test authors to +add helpers to compute a polling iteration for them. + +### Take in a `Clock` instance + +Polling confirmations could take in and use an custom Clock by test authors. +This is not supported because Polling is often used to wait out other delays, +which may or may not use the specified Clock. By staying with the default +continuous clock, Polling confirmations will continue to work even if a test +author otherwise uses a non-standard clock, such as one that skips all sleep +calls, or a clock that allows test authors to specifically control how it +advances. + +### Use macros instead of functions + +Instead of adding new bare functions, polling could be written as additional +macros, something like: + +```swift +#expectUntil { ... } +#expectAlways { ... } +``` + +However, there's no additional benefit to doing this, and it may even lead test +authors to use polling when other mechanisms would be more appropriate. + +## Acknowledgements + +This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/). +In particular, thanks to [Jeff Hui](https://github.com/jeffh) for writing the +original implementation of Nimble's Polling Expectations. + +Additionally, I'd like to thank [Jonathan Grynspan](https://github.com/grynspan) +for his help with API design during the pitch phase of this proposal.