snapshots to be tied to a hash of the search path?

[MichaelChirico]

We recently updated to rlang 1.0.0, and it has broken a ton of snapshot tests, e.g.:

- "Error: Can't recycle `bar` (size 2) to size 1."
+ "Error in stop_vctrs(x_size = x_size, y_size = size, x_arg = x_arg, class = c(\"vctrs_error_incompatible_size\", : Can't recycle `bar` (size 2) to size 1."

I have spent a lot of time chasing these down to disable them (updating the snapshots would be equal if not more work).

This has made me really pessimistic about the viability of snapshot testing -- they become extremely brittle as the superficial aspects of dependency packages (e.g. error message decoration/presentation) migrate slowly over time.

It would also be a huge pain for test authors to manually inject skip() whenever this comes up -- besides the issue that it might be hard to anticipate when this can happen in the first place without diligently vetting the snapshots in different environments regularly.

Requesting variants has a similar problem (which aspects of variation are important to capture?), as well as a dimensionality explosion (R version x rlang version x ...).

One thought is to do a pip freeze-like encapsulation of the environment where the snapshot is run, and only attempt to match the snapshot when run in an identical environment. sort(search()) --> hash64() as the default variant. This needs to be fleshed out more, and far from perfect, but I think it's a step in a better direction.

[hadley]

I think we are getting a better sense of how to use snapshot tests so hopefully this is less of a problem. But I think this is a fundamental challenge of testing output that is aimed at humans.

[MichaelChirico]

This also came from a time when we were trying to run way more tests by setting !on_cran, which we don't do anymore.

The original main reason was this issue with snapshots, but even if this were fixed I don't think we'd re-enable !on_cran tests -- fundamentally they are more brittle and cause a lot of churn headache.

Thanks!