[WIP] Documentation for using spack developer workflow #515

mauneyc-LANL · 2025-06-19T17:36:10Z

PR Summary

This is a tutorial/walkthrough/instructions on using Spack to do iterative development.

Based on this spack tutorial, but focused on singularity-eos and friends. This can also be used as a template/base for getting a reasonably complete build environment without much hassle.

Currently a draft and incomplete. Looking for input. I've gotten the output directly from going through the steps on darwin. If you'd like to replicate it, I've included the "external" YAMLs that give you the configuration for x86 darwin (this was all done on the frontend).

external_packages.yaml.txt
external_compilers.yaml.txt

(be sure to remove the .txt suffix if you use).

NOTE: This is done with a fresh spack clone, which I would recommend. This also means it's done with the new 1.0 schema and may not be compatible with most current spack installations.

PR Checklist

Adds a test for any bugs fixed. Adds tests for new features.
Format your changes by using the make format command after configuring with cmake.
Document any new features, update documentation for changes made.
Make sure the copyright notice on any files you modified is up to date.
After creating a pull request, note it in the CHANGELOG.md file.
LANL employees: make sure tests pass both on the github CI and on the Darwin CI

If preparing for a new release, in addition please check the following:

Update the version in cmake.
Move the changes in the CHANGELOG.md file under a new header for the new release, and reset the categories.
Ensure that any when='@main' dependencies are updated to the release version in the package.py

… folder

jhp-lanl

I went through this process and got it to work! Thank you!

Just a few comments

jhp-lanl · 2025-06-19T18:13:30Z

README.spack_devflow.md

+
+This can be done with the `spack develop` command 
+```bash
+$ seos-dev > spack develop singularity-eos


Without specifying the version, I get an error here

==> Error: Packages to develop must have a concrete version

Adding @main fixes this

Interesting. I started this with using spack develop singuarity-eos@main, but playing around a bit it seemed like if I included the version in the spack.yaml specs, I didn't need to include it in the develop command.

May be something polluted on my side, but does your spack.yaml have a @main (or any version) in the specs section?

spack: # add package specs to the `specs` list specs: - singularity-eos@main - spiner@main - ports-of-call@main

and

develop: singularity-eos: spec: singularity-eos@main%gcc@12.3+tests+spiner+closure+hdf5+eospac+mpi+kokkos+kokkos-kernels^eospac@6.5.9 spiner: spec: spiner@=main ports-of-call: spec: ports-of-call@=main

I think we should suggest (for developers anyway) to always point to @main

@Yurlungur the spec has @main already, but for whatever reason my spack required specifically saying @main for the develop command as well. It didn't just find it from the spec.

jhp-lanl · 2025-06-19T18:27:29Z

README.spack_devflow.md

+-rw-rw-r--@ 1 jack  dudes  24K Mar 28 11:10 spack.lock
+-rw-r--r--@ 1 jack  dudes  230 Mar 28 10:29 spack.yaml
+```
+


Since the upstream package.py is a bit older, I have to do a spack repo add here pointing at the checkout in order to get some of our newer variants, e.g. if I want +tests in singularity-eos.

Is this the correct way to do this? Or is there a better way in this workflow?

There would be a couple ways to do this, I'm not sure which would be best;

The way you've hit on isn't incorrect, tho the obvious downside here is you need to be careful to avoid breaking the environment (e.g. never do spack undevelop)

I haven't gotten to this yet in the text, but it's possible to point to a manually cloned source path spack develop -p <path>. This elides the spack clone step and will use whatever directory you point it to. This way the source clone always exists and you don't break the environment, but it's a layer of complexity I deliberately didn't want to include in the first few sections.

Could just be blunt and throw the spack-repo stuff into the spack instance directly, or the .spack. Not very elegant or illuminating, but it would do the job.

Naively, option 2 seems like an appropriate solution

jhp-lanl · 2025-06-19T19:38:43Z

README.spack_devflow.md

+
+Great! We can now commit and request a merge.
+
+And there we go! We've gotten a build with our expected environment, made changes and built all with spack!


How would you run the unit tests in this workflow?

I get a symlink to a spack build directory, but when I try ccmake . there, it looks like the build tests option isn't on despite me specifying +tests in the spec. I also can do a ctest -V and it says there are no tests.

One way I've been able to run the tests is to do something like this (assuming I'm in the seos-dev folder):

spack install spack load singularity-eos@main cd singularity-eos mkdir build ; cd build cmake .. -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment ccmake . # configure my options cmake --build . -j ./sesame2spiner/sesame2spiner -p ../sesame2spiner/examples/unit_tests/*.dat ctest -V

Is this the best way?

Also in Coming Soon™, the way you would do this - or rather, in general the way you would start doing the "classic" build steps is

spack build-env singularity-eos@main -- bash

This launches a new bash session with the build environment loaded, so all you would need to do is cd singularity-eos; mkdir build && cd build; cmake #... as you have after the spack commands. (This is what our CI does)

Spack actually has some neat functionality here too:

$> spack build-env singularity-eos@main -- bash $> source $SPACK_ROOT/share/spack/setup-env.sh # the spack docs explain why but we need to get wrappers into this new shell $> spack cd -c singularity-eos # will take you to the source dir; if you didn't `spack develop`, this could be the `/tmp` dir spack extracted to $> spack cd -b singularity-eos # to the build dir; unless otherwise specified, a dir in `/tmp`

So you can "manually" run the Spack stages as Spack would run them. This isn't necessary (e.g. as above, once in the new shell you don't need to even stay in the directory, you could cd to /scratch or wherever and just do a new git clone and proceed with a standard CMake build routine), but as part of this documentation I do want to have a Spack focus to de-mystify a bit of what Spack is doing.

Also, another neat thing here is you can get a build environment for a dependency too. So for example, if you want to do some work on spiner, you don't have to leave or change this environment at all, just

$> spack build-env spiner@main -- bash

And, as an added benefit, when you spack install (you'd need to exit the build env first), it builds all the way through, so you know your changes didn't muck up singularity-eos

Thank you! That's exactly what I was missing.

jhp-lanl · 2025-06-19T19:57:57Z

README.spack_devflow.md

+    mpi:
+      require: openmpi
+  specs:
+  - singularity-eos@main


If I want to add the +tests variant, should I do so here or use

spack config add "packages:all:variants: +tests"

?

You can do either, but the latter will add +tests to all built dependencies, if those dependencies accept the +tests variant.

I'd guess this isn't what you would want most of the time, but also it wouldn't really effect much beyond making the environment build-out take longer to build and run the tests. So likely what you want is to just add +tests to the singularity-eos spec.

(there's some Spack machinary here: spack change singularity-eos+tests, but I actually haven't ever used this so ATM I can't say "do this")

I've just been manually editing the spack.yaml file to input the spec, but I'm happy to use a spack command if that's the better way.

These commands just update the spack.yaml file, and probably that's the best, simplest approach.

*if you know the schema, and yaml

jhp-lanl · 2025-06-19T21:04:35Z

README.spack_devflow.md

+    unify: true
+```
+
+(Here, I've added some `include:` sections for external packages and external compilers. These will be platform specific. See - - below for an example to generate these files)


Have you written this up? Is it just spack external find --all --not-buildable?

So that's the basic approach, but spack external find will throw everything into the spack.yaml and in running this out I ended up just doing some cut-paste into new files that became external_*.yaml. There are some portability and engineering considerations here, but really I just wanted to have a simple spack.yaml in the docs.

The actual commands where spack external find (this gets basic build utils and some linux stuff), spack external find openmpi to generate openmpi configs (after a module load openmpi/<verison><compiler>), and spack compiler find (after a module load gcc/<version>). This should be sufficent on any machine with a normal linux and competent module environment.

There may be more work involved on Cray/HPE machines.

jhp-lanl · 2025-06-19T22:05:34Z

One other comment: how should this information mix with the information in the other spack readme? https://github.com/lanl/singularity-eos/blob/main/README.spack.md#installing-the-dependency-package

Yurlungur · 2025-06-20T13:28:14Z

README.spack_devflow.md

+Spack has become a standard package management solution for software development HPC environments. While a powerful tool, even experienced users have found it's use to be occasionally frustrating. Furthermore, much of the downstream use of Spack tends to emphasize it's use as means to dev-ops. This is, of course, Spack's main context, but this can mislead developers looking for a flexible, extensible programming environment.
+
+
+This walkthrough is intended to provide users with experience in using Spack for a developer workflow, with the goal of showing 


sentence fragment

Yurlungur

I haven't gone through this carefully the way @jhp-lanl has but I am very happy to have this here. Thanks for putting it together @mauneyc-LANL ! I would like to include this in the docs as a tutorial once we're happy with it.

Yurlungur · 2025-06-20T13:28:32Z

README.spack_devflow.md

+
+This walkthrough is intended to provide users with experience in using Spack for a developer workflow, with the goal of showing 
+
+Assume familiarty with basic Spack usage (for example, defining a package's dependencies and installing those dependencies)


sentence fragment

mauneyc-LANL · 2025-06-20T21:48:37Z

@Yurlungur I've added a section on using Spack as an alternative to Git submodules, would welcome your suggestions

jhp-lanl · 2025-07-21T20:23:11Z

README.spack_devflow.md

+
+```bash
+$ ~/dev_top > mkdir seos-dev && cd seos-dev
+$ ~/dev_top > cd seos-dev


Suggested change

$ ~/dev_top > cd seos-dev

Already in the line above

draft of walkthrough

ca370fa

mauneyc-LANL requested review from rbberger, jonahm-LANL and jhp-lanl June 19, 2025 17:36

mauneyc-LANL self-assigned this Jun 19, 2025

mauneyc-LANL added 2 commits June 19, 2025 11:48

fixed a command

372a3a6

added step 0 to clone new spack instance and point to seperate .spack…

36eb3be

… folder

jhp-lanl reviewed Jun 19, 2025

View reviewed changes

Yurlungur reviewed Jun 20, 2025

View reviewed changes

Rewrite, fixes, and new sections

a5f5c1c

jhp-lanl reviewed Jul 21, 2025

View reviewed changes


		Great! We can now commit and request a merge.

		And there we go! We've gotten a build with our expected environment, made changes and built all with spack!

		Spack has become a standard package management solution for software development HPC environments. While a powerful tool, even experienced users have found it's use to be occasionally frustrating. Furthermore, much of the downstream use of Spack tends to emphasize it's use as means to dev-ops. This is, of course, Spack's main context, but this can mislead developers looking for a flexible, extensible programming environment.


		This walkthrough is intended to provide users with experience in using Spack for a developer workflow, with the goal of showing


		This walkthrough is intended to provide users with experience in using Spack for a developer workflow, with the goal of showing

		Assume familiarty with basic Spack usage (for example, defining a package's dependencies and installing those dependencies)

[WIP] Documentation for using spack developer workflow #515

Are you sure you want to change the base?

[WIP] Documentation for using spack developer workflow #515

Uh oh!

Conversation

mauneyc-LANL commented Jun 19, 2025

PR Summary

PR Checklist

Uh oh!

jhp-lanl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhp-lanl Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhp-lanl Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mauneyc-LANL Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mauneyc-LANL Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhp-lanl commented Jun 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yurlungur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mauneyc-LANL commented Jun 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jhp-lanl Jun 20, 2025 •

edited

Loading

jhp-lanl Jun 19, 2025 •

edited

Loading

mauneyc-LANL Jun 19, 2025 •

edited

Loading

mauneyc-LANL Jun 20, 2025 •

edited

Loading