Applying the cookiecutter on an existing repo

[sfmig]

Is Your Feature Request Related to a Problem? Please Describe

When @samcunliffe and I had to apply a cookiecutter to an existing repo, we ended up creating the template on a separate directory and copying files across. Is there a better way to do this? And should we document this usecase in this repo?

@matt-graham mentioned that in the ARC festival workshop they tried a different approach to keep the git history.

Describe the Solution You'd Like

Mention this case in the readme and add some steps with the best way to do this.

Describe Alternatives You've Considered

No response

Additional Context

No response

[samcunliffe]

Should it be on the website too? Perhaps a bit out of scope: the website is "just" recommending tools, the root-level README is instructions for how to use the cookie-cutter template.

[paddyroddy]

Do we know if this is a common use-case? Obviously, it would be very handy, but perhaps out of scope? Would be good to document either way, perhaps the website could have a blog section where we write this kind of thing up?

[matt-graham]

In the workshop we worked on applying the cookiecutter package structure to the existing package in zmek/patientflow. We initially tried following a git merge approach following this guide @paddyroddy pointed me to, with the cookiecutter package created in a new repository to which we added the existing (local) repository as a remote. However we subsequently realised in this case this seemed overkill as there was no history from the new package repository we wanted to preserve so just doing what @sfmig suggested above of copying across the new package structure in to the existing repository in a new commit made sense (we did this in this commit on patientflow).

Some thoughts / comments on doing this for this specific case:

• The patientflow package previously had a flat rather than src layout which simplified copying in the new package structure without a risk overwriting any existing top-level __init__.py, with we then just git mving the top-level patientflow directory in to src.
• There was an existing .gitignore file in the repository that we decided to retain in favour of the cookiecutter version as it had some customisations as well as the standard Python ignores. I forgot when doing this initially though that I would need to add a rule for the automatically generated _version.py file that setuptools_scm creates.
• I also forgot initially to recreate some logic in the previous setup.py file for pulling in package dependencies from a requirements.txt file by setting 'dynamically' in the pyproject.toml.
• The ruff configuration we have set up for the cookiecutter template is I would say on the strict side for an existing project, and resulted in a very large wall of errors for patientflow. I recommended to Zella to switch initially to the rules selected in the default configuration and then maybe add in other rules incrementally later to avoid having to fix so many linting errors at once.

[dstansby]

FWIW https://cruft.github.io/cruft/ might be of interest, which I've seen used for exactly this elsewhere (https://github.com/OpenAstronomy/packaging-guide)

[paddyroddy]

Just FYI @matt-graham, one big advantage of having all ruff rules on is that, for example, NPY201 is automatically turned on, which will help with numpy>=2.0

[sfmig]

From an RSECon24 workshop: copier does seem to deal with this nicely, as it allows you to apply the template to an existing project.

Some other interesting features:

• quite modular, you can define different profiles or variants of a template
  • e.g. minimum, recommended, let me choose
• you can gradually (after creation or during the life of the package) add new features to your template
  • e.g. start off with a minimal template, then add skeleton for unit tests, github actions, linting etc separately.
  • You can select individual github actions, type of citation files etc too via submenus.
• you can pull features/fixes implemented to the copier template anytime

One downside may be that it abstracts away too much for beginners. It seems better to gradually increase complexity, rather than abstract everything away from the start. But they do have the option to add selected features, which is maybe more aligned with that second approach.

Relatedly, the neuroinformatics team pointed me to a guide Laura did a while ago on how to use cruft to apply a cookiecutter template to an existing repo.

[samcunliffe]

Should we just point to Laura's guide?
(@dstansby ... cruft)

[dstansby]

I have no opinions or useful advice other than I've seen cruft being used before 😄