-
Notifications
You must be signed in to change notification settings - Fork 624
Description
Testing frameworks like pytest can skip tests dynamically using e.g. pytest.skip("skipped dynamically")
. Hypothesis re-raises these exceptions to the testing framework, if they match skip_exceptions_to_reraise
.
We want to continue doing this. But we should also start to save database entries for these failures, so that Hypothesis retries them immediately next time, instead of having to rediscover the particular inputs which trigger the dynamic exception.
We'll want to make sure this follows standard eviction logic: the entry is not evicted if it reproduces the skip exception, and is evicted if it doesn't reproduce. I'm noting this because we will re-raise the skip exception if it reproduces, which means we skip .freeze()
-ing the ConjectureData
and associated checks. I expect the direct implementation to work, but we should include tests to future-proof this new code path.
(prompted by Zac-HD/hypofuzz#202 (comment)).