Gradle transform cache exploding with 1.3.5

[rschattauer]

I am using paparazzi 1.3.3 and wanted to upgrade, 1.3.4 had issues that are resolved with 1.3.5, so I was implementing the latest snapshot of it. Since using that snapshot my ~/.gradle/caches/transform-4 (or -3 or transforms within wrapper, depending on your gradle version, I am using 8.6 and tried 8.10.2 too) has been exploding on all of our CI instances. For context: I am using paparazzi in a multi module project and have plenty of modules where the paparazzi plugin is applied.

I checked the transforms within transforms using du -shc -t 30M * | sort -h and the biggest outliers are as follow:
...
166M d48bc9770ee795bbdf7528986243b391 -> unzipped-distribution
169M 21d464295408edaac324e1d752a8cb06 -> layoutlib-14.0.11
233M 2cc4b7b4cb854d20097a6a9acbbc7ee7 -> nativeruntime-dist-compat-1.0.12
233M ee68dff6cd2ae85b45f9654696932b97 -> nativeruntime-dist-compat-1.0.12
234M ea4827d0659cd67a24b31cca3f330702 -> nativeruntime-dist-compat-1.0.12

nativeruntime-dist-compat would come from robolectric and layoutlib from paparazzi (or both? no idea)

So I downgraded to 1.3.3 and couldn't replicate this issue, my transforms folder didn't make my disk explode.
After the downgrade I only had ea4827d0659cd67a24b31cca3f330702, so a single nativeruntime-dist-compat-1.0.12

I can't wrap my head around why nativeruntime-dist-compat is downloaded multiple times using paparazzi 1.3.5-SNAPSHOT but this is blocking us from 1.3.5

[rschattauer]

Hey @jrodbx , maybe you can confirm this before hitting the release button for 1.3.5 :-)

[jrodbx]

nativeruntime-dist-compat would come from robolectric

Correct!

However, this doesn't read like a Paparazzi issue, but rather one with Gradle transforms. Mind filing an issue with the Gradle team and linking here? Fwiw, I have also seen multiple layoutlib instances in the transforms folder, but haven't dug into the Gradle internals to figure out why.

but this is blocking us from 1.3.5

Why is it blocking? Could your CI scripts be updated to do a find and cleanup of transform directories that contains nativeruntime-dist-compat or layoutlib?

[rschattauer]

Will do. Could you give me a little more information about what changed from 1.3.3 to 1.3.5 that has to do with gradle, so I can write a better issue.

[jrodbx]

Here's the comparison for now: 1.3.3...master.
Once 1.3.5 is tagged, this becomes the better link: 1.3.3...1.3.5.

[rschattauer]

Raised an issue with Gradle. I sadly don't know if this is an issue at all or solvable at all.
Sorry for the confusion it has something to do with snapshots.
Using the 1.3.5 release that exists since today I wasn't able to reproduce the disk space flooding anymore -> Closing this issue.

[rschattauer]

I have to reopen this issue. It still is an issue with 1.3.5 flooding our transforms folder within ~/.gradle/... We have a lot of modules with screenshot tests, maybe that's the issue here.

[rschattauer]

I did some testing comparing 1.3.3 and 1.3.5 and want to share my results here.

First, we have have around 180 modules in our project, where around 1/3 have a single Paparazzi-ScreenshotTest that crawls through our code using reflection to get all the Previews within that module and runs Paparazzi on those. We run on gradle 8.12, but as checked with the 1.3.5-SNAPSHOT some months ago, I had the same issue with lower gradle versions and I no longer think it's gradle related.

First, before testing every version I ran ./gradlew clean to wipe all my projects build folder, then I fully removed the ~/.gradle/caches folder and logged my total disk space before starting multiple runs.
All runs ran exclusively on terminal using ./gradlew recordPapaprazziDebug without any unnecessary program running in the background.
Once a test was completed, I changed a single color using TextEdit to keep it lightweight and ran another run. The color I changed is being used within a single component that affected around 200 screenshots of our ~3.000 screenshots we have within our project.

Here is my result, with 7 runs doing this for 1.3.3 and 1.3.5

Stage	Remaining Disk Space - 1.3.3	Remaining Disk Space - 1.3.5
Before	220.65 GB	222.11 GB
Run 1	209.90 GB	201.45 GB
Run 2	209.73 GB	201.02 GB
Run 3	209.67 GB	200.80 GB
Run 4	209.62 GB	200.45 GB
Run 5	209.56 GB	200.49 GB
Run 6	209.51 GB	200.20 GB
Run 7	209.79 GB	199.99 GB

Some thoughts about the results:

• Yes, it's "just" ~250MB per run compared to ~50MB
• Yes, on Run 7 (1.3.3) and Run 5 (1.3.5) my disk space increased. No idea why, maybe gradle wiped something 🤷
• Yes, it's total disk space, but I know that 1.3.5 drastically increased disk usage within ~/.gradle/caches/8.12/transforms
• No, this is not what we see while we develop on our project. There we can investigate even bigger numbers, maybe it's verifyPaparazziDebug related, I can't tell, all I can say that from the 220GB remaining disk space my disk is full after just around 5 days
• Using 1.3.3 we have the rule mentioned in Getting OutOfMemoryErrors after ~500 screenshots #915 implemented as we would otherwise run into memory issues. This rule is no longer required using 1.3.5, so that's one difference on our codebase
• Another thing is that we have the workaround from the following issue using 1.3.3 where I am currently unsure if that is fixed with 1.3.3, 1.3.4 or 1.3.5 -> 1.3.1 -> 1.3.2 java.lang.IllegalAccessError: class com.android.resources.ResourceType tried to access method 'java.util.stream.Collector  #1231

We are now downgrading to 1.3.3 as 1.3.5 is killing our CI's disk space flooding gradle cache.

If I can help you somehow on resolving this, please feel free to hit me up.

[rschattauer]

Hi, due to #1877 we can't update our compile sdk to 36 and this issue here prevents us from going to 1.3.5 or higher. Any chance you may take a new look into this?