Self hosted runners running out of disk space

[mdboom]

The pystats self-hosted runner ran out of disk space and therefore stopped fulfilling requests. While this issue is most acute on that runner, since it's a VM with only 30GB of disk space, these issues probably apply equally to all self-hosted runners, so we should have a solution that applies to both.

Here are some sources of disk space usage that should be addressed:

1. The .git directory for the benchmarking repo. We keep the repo size down by periodically manually deleting old history. However, since the checkout is retained on the runners, this doesn't actually remove the old content. This was initially 12GB, and after running git gc it went to 4GB. This can be automated.

2. There are multiple versions of the dependencies for Github Actions installed, left behind when GHA self-updates. These have the names externalsX.YY and are roughly .5GB each. There is an issue for this that claims a fix has been merged, but nonetheless these dirs seem to remain.

3. The pip cache is about 1GB. Unfortunately, I don't think it's possible to control the amount of space used for the pip cache. The best we can probably do is periodically purge it.

[mdboom]

Step 1. has been implemented.

Step 2. requires an upstream fix from Github, IMHO.

Step 3. is still TBD.

[chris-eibl]

Regarding step3: until limiting the cache pypa/pip#3138 is possible, pip cache purge or pip cache remove <pattern> seem to be the only options.

Options --editable and --no-cache-dir are sometimes mentioned to workaround too big caches, but I think not applicable for pyperformance.