Improvements to GitHub Actions package caching

[trexfeathers]

The problem

Our existing use of caching Conda content in GitHub Actions workflows clashes with Conda's own caching mechanism. Packages are stored in the Conda package-cache, and the packages are then included in individual environments via hard-links. GHA caching does not parse these hard-links, but we cache both the Conda package-cache AND the environment directories, causing two problems:

• Each package is cached TWICE (once per directory). With recent changes to GitHub, our older caches are being wiped almost daily - every time we blow our space limit - so workflows running less frequently (e.g. benchmarks) store caches that are NEVER used before disappearing.
• The restored package-cache is never used - each workflow run is associated with ~500MB of pointless IO. This is because there is no connection between the restored package-cache and the restored environments. The only time the package-cache would be used is when downloading new packages, but we invalidate the GHA caches when the lock-files change.

My proposed solution

• Only cache the Conda package-cache. Conda will populate the environments from this cache automatically.
• Share the package cache between workflows of the same Python version. I.e. for the latest Python version this would be tests, wheels and benchmarks.

Pros and cons

Pros

• Reduced cache size - sharing a single copy of a package between multiple workflows.
• Reduced downloads - sharing a single copy of a package between multiple workflows.
• Reduced IO - no redundant content is getting cached/restored.

Cons

• Slight runtime increase - Conda must recreate the environments each time.
  This is largely mitigated:
  • Cached package-cache avoids needing to re-download
  • Lock-files avoid needing to re-resolve dependencies
  • GHA's file system is fast

[scitools-ci]

• Please see: SciTools/.github#227

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.39%. Comparing base (c794e8e) to head (45dc273).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6803   +/-   ##
=======================================
  Coverage   90.39%   90.39%           
=======================================
  Files          91       91           
  Lines       24761    24761           
  Branches     4639     4639           
=======================================
  Hits        22382    22382           
  Misses       1608     1608           
  Partials      771      771           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[trexfeathers]

This could stand some improvement. The wheels job always finishes first and populates the cache. When the tests job - this one - finishes, it does not overwrite the cache. For the top level Nox environment, the Python version is not controlled, so the dependency stack will often be for a different Python version than what is in the cache, leading to lots of new downloads.

Solutions

• Make the Nox environment use matrix.python-version, so that the Cartopy dependency stack comes from the cache.
• Give the tests workflow 'control' of the cache, having wheels etc only use the cache but not populate it.

[trexfeathers]

Should benchmarking also be using the Cartopy cache?