Export Poetry's lockfile as pip requirements.txt

[fabcor-maxiv]

This will let the pre-commit hooks ensure that Poetry's lockfile is kept up-to-date and that it is exported to a pip requirements.txt file.

GitHub: fix #1268

[fabcor-maxiv]

I feel like maybe this should be discussed, so I keep it as draft for a bit. In particular it feels like attempts at using some of those Poetry pre-commit hooks have been made in the past, and I can assume it was not entirely successful, so maybe there is some background info that I am missing.

[marcus-oscarsson]

Hm, yes. I suspect so as well

[fabcor-maxiv]

I am wondering how we could move this forward...

What could be an issue here?

Updates to requirements.txt would happen only when poetry.lock changes. Typically changes to poetry.lock happen only when Dependabot does it, or when we do changes to the dependency list in pyproject.toml. So all in all, it feels like we can keep the Dependabot part under control, and on the other hand changing the dependency list in pyproject.toml is relatively rare that one of the more experienced contributors can easily intervene if things get messy. If I checked everything correctly, the pre-commit hooks should do most of the work and will inform correctly when files are not in sync.

Maybe I could document this a bit...

[fabcor-maxiv]

@oldfielj-ansto If I am not mistaken, you were the one to introduce pre-commit hooks and Poetry. Maybe you have some opinion on this...

[marcus-oscarsson]

What does this hook do, does it check if the lock file is up to date or does it also update and commit it ?

[fabcor-maxiv]

As far as I can tell it runs poetry check --lock. The documentation is not super clear on what this does, I assume there is an issue in the documentation, because the behavior does not match the documentation.

Anyway what actually happens in this hook is that it checks that:

1. the content of pyproject.toml is valid (from the point of view of Poetry)
2. there is a poetry.lock file
3. the content of poetry.lock is in sync with pyproject.toml

The next hook runs poetry lock --no-update, which does syncs the lock file. The hook will fail if the hook changes the file, this is important to have on the CI side, for the case when someone commits without running the hooks locally.

Then the third hook does the export to requirements.txt. The hook will fail if the hook changes the file, this is important to have on the CI side, for the case when someone commits without running the hooks locally.

There is some redundancy between the first two steps, and I assume we can simplify over time, depending on how things go.

[marcus-oscarsson]

It would probably nice with some documentation :)

[fabcor-maxiv]

It would probably nice with some documentation :)

I added a bit of documentation. Mostly in comments in .pre-commit-config.yaml, let me know if you think I should add more documentation and/or move it somewhere else.

[marcus-oscarsson]

How come that the status checks are pending for such a long time ?

[fabcor-maxiv]

How come that the status checks are pending for such a long time ?

Merging #1289 broke the GitHub Actions workflows completely. See mxcube/mxcubecore#960

[marcus-oscarsson]

@fabcor-maxiv: Yes, I think its fixed now. Seems like the poetry-export hook reacts to something I guess its the update of certifi

[fabcor-maxiv]

Seems like the poetry-export hook reacts to something I guess its the update of certifi

Oh, nice! That is very good that the issue is caught, that is what we want. : )

I forced the pre-commit hook to regenerate the exported requirements.txt (pre-commit run -a).

[fabcor-maxiv]

This is now rebased, it takes into account the newly added dependency (argon2-cffi #1327).