Support for PyPI packages

[mvgijssel]

Hi folks! First of all, thanks this amazing library, I've always been looking for something exactly like this to make setting up developer environments easier ❤. Currently I use Trunk CLI with Trunk Tools to install project specific packages, which has the downside having to remember to run trunk install from time to time (similar to asdf/mise/etc). Hermit solves this as it gives the user binstubs with just-in-time-installation so you can just git clone and start working immediately! 🎉

I'm trying to migrate some Python packages (dbt / mkdocs / ansible) from Trunk to Hermit, but these packages don't offer a .pyz or .pex single binary executable. So I was trying to hack something together which:

1. Downloads a Python wheel using Hermit
2. Installs Python wheel using pipx using Hermit
3. Exposes created binary

This works only if pipx is already installed and otherwise I get an error

fatal:hermit: mkdocs-publisher-1.4.4: failed to execute "/Users/maarten/Library/Caches/hermit/pkg/mkdocs-publisher-1.4.4/pipx_install.sh /Users/maarten/Library/Caches/hermit/pkg/mkdocs-publisher-1.4.4 mkdocs_publisher-1.4.4-py3-none-any.whl": fatal:hermit: failed to acquire lock: timed out acquiring lock /Users/maarten/Library/Caches/hermit/.lock after 30.00233075s, locked by pid {20910 downloading and extracting mkdocs-publisher-1.4.4}: downloading and extracting mkdocs-publisher-1.4.4: lock timed out
: exit status 1

I believe the error comes from doing a Hermit invocation from within a Hermit invocation.

So now my question to you: thoughts on how to make this work? I realise doing a just-in-time build is less deterministic than just installing pre-packaged binaries, but using pipx/pex to install a downloaded wheel to me is an OK tradeoff.

Hackery

pipx.hcl

description = "Install and Run Python Applications in Isolated Environments"
binaries = ["pipx"]
test = "pipx --version"
source = "https://github.com/pypa/pipx/releases/download/${version}/pipx.pyz"
runtime-dependencies = ["python3@3.10"]
dont-extract = true

on "unpack" {
  rename {
    from = "${root}/pipx.pyz"
    to = "${root}/pipx"
  }

  chmod {
    file = "${root}/pipx"
    mode = 448
  }
}

version "1.7.1" {
  auto-version {
    github-release = "pypa/pipx"
  }
}

sha256sums = {
"https://github.com/pypa/pipx/releases/download/1.7.1/pipx.pyz": "1d4f46f86830640f1d7c4e29b280a7a42265d6e8af2c063f40baed4513f03ae8"
}

mkdocs.hcl

description = "Publisher for MkDocs - a set of plugins for content creators"
binaries = [
    ".pipx/venvs/mkdocs-publisher/bin/mkdocs",
]
source = "https://files.pythonhosted.org/packages/c8/48/6334c850f7e4953a2dc330904a136b91e53b5c053dda094df5779dfe4da5/mkdocs_publisher-1.4.4-py3-none-any.whl"

requires = ["pipx"]
dont-extract = true

on "unpack" {
    copy {
        from = "pipx_install.sh"
        to = "${root}/pipx_install.sh"
    }

    chmod {
        file = "${root}/pipx_install.sh"
        mode = 493
    }

    run {
        cmd = "${root}/pipx_install.sh ${root} mkdocs_publisher-1.4.4-py3-none-any.whl"
    }
}

version "1.4.4" {
  auto-version {
    github-release = "mkdocs-publisher/mkdocs-publisher"
  }
}

sha256sums = {
"https://files.pythonhosted.org/packages/c8/48/6334c850f7e4953a2dc330904a136b91e53b5c053dda094df5779dfe4da5/mkdocs_publisher-1.4.4-py3-none-any.whl": "714f683f7875cf07ca0a14277daf1c0def2d6e629aed99862a123fc8c9b7b104"
}

pipx_install.sh

#!/bin/bash
set -euo pipefail

root="$1"
wheel="$2"
wheel="$root/$wheel"

cd "${root}"
export PIPX_HOME=$root/.pipx

### <<< PROBLEM IS HERE >>>
#
# pipx comes from the same Hermit environment as mkdocs. If it's missing it will try 
# to install pipx which means a Hermit installation within a Hermit installation happens
# resulting in a lock error
#
pipx install $wheel

[mvgijssel]

If I set the HERMIT_STATE_DIR to an alternative location before invoking pipx I can do a nested hermit invocation successfully

export HERMIT_STATE_DIR=$root/.hermit

[alecthomas]

Hi @mvgijssel,

While Hermit deliberately doesn't support language-specific toolchains, the best approach I've come up with, is plonking a wrapper script in a directory that looks something like this:

bin/mkdocs:

#!/bin/bash
set -euo pipefail
exe="$PYTHONUSERBASE/bin/mkdocs"
test -x "$exe" || pip install mkdocs 1>&2
exec "$bin" "$@"

This will install mkdocs on-demand.

[nickajacks1]

It seems like the real problem in this issue is that the cache lock is non-reentrant, meaning you cannot use hermit-installed binaries in a package trigger if they are not yet unpacked.

1. hermit cache is clean, project has both openjdk and foo provide by hermit.
2. user execs foo binary
3. hermit locks the cache dir and starts to download and unpack foo
4. foo has an on "unpack" script that executes java
5. hermit tries to lock the cache and deadlocks.

I can think of two ways to fix this.

1. Make it so that when a package needs to be downloaded/unpacked, all of its requires packages get downloaded/unpacked first.
2. Implement a re-entrant flock by somehow tracking parent hermit PIDs, for example with env vars.

[mvgijssel]

After tinkering with this a bit, I decided the "Hermit-way" is probably the best way. So I created a separate repo https://github.com/vgijssel/hermit-python-packages which packages Python tools using Pex and serves them from GitHub releases using Hermit manifests. Pex's are built for macOS and Linux amd64 + arm64.

This has the same user experience as other Hermit served binaries and works great so far! Currently there tools are supported:

• sqlfluff
• aider-chat
• ansible
• datafold-sdk
• black

It's also relatively easy to extend to support more tools and to automatically update with new tool versions