Some similar snapshots falsely pass even with maxPercentDifference set to zero

[matejdro]

Description

It seems that in some cases, Paparazzi will ignore the maxPercentDifference = 0.0 and will falsely pass the tests, even when the recorded image differs from the new image.

This seems to be happening when the result of the comparion is the DiffResult.Similar. It appears percentage is not forwarded in this case, but always hardcoded as zero:

paparazzi/paparazzi/src/main/java/app/cash/paparazzi/internal/ImageUtils.kt

Line 144 in ca94122

is Similar -> result.delta to 0f

Steps to Reproduce

1. Record the snapshot with the following test:

class BugDemo {

    @get:Rule
    val paparazzi: Paparazzi = Paparazzi(deviceConfig = DeviceConfig.NEXUS_5, maxPercentDifference = 0.0)

    @Test
    fun test() {
        paparazzi.snapshot {
            Box(Modifier.size(1024.dp).background(Color.Black).padding(100.dp)) {
                Box(Modifier.size(400.dp, 1.dp).background(Color.White))
            }
        }
    }
}

2. Change the width of the inner box from 400.dp to the 390.dp
3. Run the verify task

Expected behavior

When maxPercentDifference is set to 0.0, then any change in the image should fail the verification, no matter how small

Additional information:

• Paparazzi Version: 1.3.5
• OS: Linux
• Compile SDK: 35
• Gradle Version: 8.11.1
• Android Gradle Plugin Version: 8.10.0

Screenshots

Paparazzi does not detect line slightly shrinking in the above image

[geoff-powell]

So when testing this in paparazzi and preview, it is working as Preview works.

For whatever reason this composable you are testing doesn't actually change the output image when changing from
400 -> 390.

This is due because NEXUS_5 device pixel size is 1080x1920.

If the outer Box is 1024.dp then that would be way too large.

The following test fails as expected

class BugDemo {

  @get:Rule
  val paparazzi: Paparazzi = Paparazzi(deviceConfig = DeviceConfig.NEXUS_5, maxPercentDifference = 0.0)

  @Test
  fun test() {
    paparazzi.snapshot {
      Box(Modifier.fillMaxSize().background(Color.Black).padding(50.dp)) {
        Box(Modifier.size(100.dp, 1.dp).background(Color.White))
      }
    }
  }
}

Change 100.dp to 99.dp like your example and test fails with a 4px diff

[matejdro]

@geoff-powell you are right, sorry, my example was bad. It appears that the actual issue happens when there is a transparent background and the black (not white) line shrinks. Here is the better reproducer:

class BugDemo {

    @get:Rule
    val paparazzi: Paparazzi = Paparazzi(
        deviceConfig = DeviceConfig.NEXUS_5,
        maxPercentDifference = 0.0,
        theme = "TranslucentBackground",
    )

    @Test
    fun test() {
        paparazzi.snapshot {
            Box(
                Modifier
                    .fillMaxSize()
                    .padding(50.dp),
            ) {
                Box(
                    Modifier
                        .size(100.dp, 1.dp)
                        .background(Color.Black),
                )
            }
        }
    }
}

in themes.xml:

<style name="TranslucentBackground">
    <item name="android:windowBackground">@null</item>
</style>

Same steps: record, change the line to 90.dp, verify

[geoff-powell]

Thanks, yes that indeed has a diff a 4 pixels but verify passes when it should fail.

[matejdro]

Do we reopen the ticket then?