Skip to content

Golden images in /build/paparazzi/failures are not stored in some error cases #1716

@SimonMarquis

Description

@SimonMarquis

I'm not sure if this should be classified as a bug of a feature request:

Description
Golden images are stored in /build/paparazzi/failures when there is a regression (next to the delta-*.png files) but, they are not when:

  • The reference image (in /test/snapshots/images) is not present:

        java.lang.AssertionError: File module/ScreenshotTesting/src/test/snapshots/images/...ScreenshotTest_test.png does not exist
            at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:60)
            at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
            at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
            at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
            at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
    
  • The reference image has a different size:

        java.lang.IllegalStateException: Images are different sizes
            at app.cash.paparazzi.internal.OffByTwo.compare(OffByTwo.kt:11)
            at app.cash.paparazzi.internal.ImageUtils.compareImages(ImageUtils.kt:144)
            at app.cash.paparazzi.internal.ImageUtils.assertImageSimilar(ImageUtils.kt:55)
            at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:73)
            at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
            at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
            at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
            at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
    

Steps to Reproduce
I can create a simple project for that if it is really necessary, but I think the description is fairly simple, and easy to reproduce, without special setup. Please let me know.

Expected behavior
Having golden screenshots generated even when the test fail because it does not match the reference (or if the reference is missing) would be very valuable in CI scenarios, where the newly created golden images could be immediately extracted without having to run an entire pipeline with a new execution of recordPaparazzi Gradle task.
Also, running the recordPaparazzi might lead to unrelated golden images being updated (with invisible variations) that would flood the review diff.

Additional information:

  • Paparazzi Version: 1.3.5

Note:
In previous versions (pre-OffByTwo), when there was a regression with different image sizes, the delta and golden images were stored to disk on /out/failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions