feature request: add CONCIERGE_EXTRA_PIPS to concierge

[taurus-forever]

Hi,

Data team uses CONCIERGE_EXTRA_ variables for CI tests:

    environment:
      CONCIERGE_EXTRA_SNAPS: charmcraft
      CONCIERGE_EXTRA_DEBS: pipx

The pipx is being used to install tox, poetry, charmcraftcache (manually: #1, #2).
It would be great to introduce CONCIERGE_EXTRA_PIPS to simply do:

    environment:
      CONCIERGE_EXTRA_SNAPS: charmcraft
      CONCIERGE_EXTRA_DEBS: pipx
      CONCIERGE_EXTRA_PIPS: tox poetry charmcraftcache

Tnx!

P.S. can of worms: how to guaranty pipx installation before tox?

[tonyandrewmeyer]

I don't think we want to bake pipx into concierge. The Python packages could be installed with python -m pip instead, but then either we're installing into the system Python environment, which is generally a bad idea, or we have to have a virtualenv to install into.

There's also the ordering issue you bring up: concierge tries to install everything at once, so introducing a requirement that a specific deb package has to go first wouldn't be great. I think that's a strong argument against a pipx list (or a uv one, etc).

One solution (perhaps the "Ubuntu way"?) would be to make snaps of poetry and charmcraftcache and then install the snap instead (and either also tox, or use the existing out-of-date tox deb).

Something that would work now is to use pipx run rather than pipx install. I think pipx caches so that running twice isn't any slower than running once. Is there any reason not to just do that?

[jnsgruk]

One solution (perhaps the "Ubuntu way"?) would be to make snaps of poetry and charmcraftcache and then install the snap instead (and either also tox, or use the existing out-of-date tox deb).

I think this is the way - creating a snap for charmcraftcache.

In the specific case of charmcraftcache, much of that will be solved in coming moths anyway with our new build/test machinery. It may also be possible to eschew that particular package in place of something like this in some cases: https://github.com/jnsgruk/zinc-k8s-operator/blob/a6c61d880c513e22e85825d4ea4b67fe71037105/.github/workflows/build-and-test.yaml#L59-L69

[taurus-forever]

Hi,

I agree about race-condition in ordering for deb/pipx calls. Hard to solve this in a beautiful way.
In my PoC, I have separated prepare and run concierge configs as different YAML files,
so pipx deb package has been installed once initially and then pipx ensure tox/poetry is available on later stages.

I hope into charmcraftcache migration to the upcoming machinery,
it looks inefficient to invest into snap here nowadays.

I am happy to use tox/poetry as snap as well! Who can snap them (as classic?)?
It could be not-trivial task due to long-term support cost.

For the record, SQL charm devels still prefer to run tests via tox locally (avoiding spread), time saving on bootstrap.
Tox used to be the recommended way for charmers, is it deprecated now?

Tnx!