Skip to content

Commit ce146c0

Browse files
brittonsmithbrittonsmith
authored
Merge pull request #320 from mabruzzo/binary-wheel
Packaging Pygrackle
2 parents bfe0476 + 0f2c183 commit ce146c0

11 files changed

+2162
-10
lines changed

‎.github/dependabot.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
version: 2
2+
updates:
3+
# Maintain dependencies for GitHub Actions
4+
- package-ecosystem: "github-actions"
5+
directory: "/"
6+
schedule:
7+
interval: "weekly"
8+
groups:
9+
actions:
10+
patterns:
11+
- "*"

‎.github/workflows/wheels.yml‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# inspired by
2+
# - https://learn.scientific-python.org/development/guides/gha-wheels/
3+
# - https://cibuildwheel.pypa.io/en/stable/setup/#github-actions
4+
# - https://github.com/yt-project/yt/blob/main/.github/workflows/wheels.yaml
5+
#
6+
# the numpy action that does the exact same thing
7+
# https://github.com/numpy/numpy/blob/main/.github/workflows/wheels.yml
8+
# illustrates a nifty trick for configuring the action to run whenever a commit
9+
# is pushed where the commit-message is prefixed with [wheel build]
10+
11+
name: Wheel
12+
13+
on:
14+
schedule:
15+
# ┌───────────── minute (0 - 59)
16+
# │ ┌───────────── hour (0 - 23)
17+
# │ │ ┌───────────── day of the month (1 - 31)
18+
# │ │ │ ┌───────────── month (1 - 12 or JAN-DEC)
19+
# │ │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
20+
# │ │ │ │ │
21+
- cron: "17 0 * * MON"
22+
push:
23+
branches:
24+
- main
25+
- binary-wheel # just for initial testing purposes
26+
tags:
27+
- 'grackle-*'
28+
pull_request:
29+
paths:
30+
- '.github/workflows/wheels.yaml'
31+
workflow_dispatch:
32+
33+
jobs:
34+
make_sdist:
35+
name: Make SDist
36+
runs-on: ubuntu-latest
37+
steps:
38+
- uses: actions/checkout@v4
39+
with:
40+
# fetch the full git history to let us run all our tests
41+
fetch-depth: 0
42+
# fetch the submodules to let us run the tests
43+
submodules: true
44+
45+
- name: Set up Python
46+
uses: actions/setup-python@v5
47+
with:
48+
python-version: '3.10'
49+
50+
- name: Build SDist
51+
run: pipx run build --sdist
52+
53+
- name: Check README rendering for PyPI
54+
run: |
55+
python -mpip install twine
56+
twine check dist/*
57+
58+
- name: Test sdist
59+
run: |
60+
sudo apt-get install libhdf5-dev
61+
# it may not be necessary to setup a venv
62+
python -m venv my-venv
63+
source my-venv/bin/activate
64+
pip install --upgrade pip
65+
python -m pip install --group test "$(echo dist/*.tar.gz)"
66+
python -m pip list
67+
pytest
68+
69+
- uses: actions/upload-artifact@v4
70+
with:
71+
name: cibw-sdist
72+
path: dist/*.tar.gz
73+
74+
build_wheels:
75+
name: Build wheels on ${{ matrix.os }}
76+
runs-on: ${{ matrix.os }}
77+
strategy:
78+
fail-fast: false # don't exit abruptly if another wheel can't be built
79+
matrix:
80+
os: [
81+
ubuntu-latest,
82+
# we should revisit arm-based builds in the future
83+
# -> the tests run REALLY slowly because wheels aren't shipped by yt/matplotlib
84+
#ubuntu-24.04-arm,
85+
macos-13, # (runs on intel-CPUs)
86+
macos-14 #(runs on arm cpus)
87+
]
88+
89+
steps:
90+
- uses: actions/checkout@v4
91+
with:
92+
# fetch the full git history to let us run all our tests
93+
fetch-depth: 0
94+
# fetch the submodules to let us run the tests
95+
submodules: true
96+
97+
- name: Set MACOSX_DEPLOYMENT_TARGET
98+
if: startsWith( matrix.os, 'macos-' )
99+
run: |
100+
# we may be able to reduce the following version numbers (or pick
101+
# something consistent b/t Intel & Arm) once all fortran code is
102+
# removed
103+
RUNNER=${{ matrix.OS }}
104+
if [ "$RUNNER" = "macos-14" ]; then # ARM-builds
105+
echo "CIBW_ENVIRONMENT=MACOSX_DEPLOYMENT_TARGET=12.3" >> "$GITHUB_ENV"
106+
elif [ "$RUNNER" = "macos-13" ]; then # Intel-builds
107+
echo "CIBW_ENVIRONMENT=MACOSX_DEPLOYMENT_TARGET=10.14" >> "$GITHUB_ENV"
108+
fi
109+
110+
111+
- name: Build wheels
112+
uses: pypa/cibuildwheel@v2.23.3
113+
with:
114+
output-dir: dist
115+
116+
- uses: actions/upload-artifact@v4
117+
with:
118+
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
119+
path: ./dist/*.whl
120+
121+
upload_all:
122+
name: Publish to PyPI
123+
needs: [build_wheels, make_sdist]
124+
environment: testpypi # CHANGEME to pypi (to start uploading to PyPI)
125+
permissions:
126+
id-token: write
127+
attestations: write
128+
contents: read
129+
130+
runs-on: ubuntu-latest
131+
# upload to PyPI on every tag starting with 'grackle-'
132+
# uncomment the following line when we start uploading to PyPI
133+
#if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/grackle-')
134+
steps:
135+
- uses: actions/download-artifact@v4
136+
with:
137+
pattern: cibw-*
138+
path: dist
139+
merge-multiple: true
140+
141+
- name: Generate artifact attestations
142+
uses: actions/attest-build-provenance@v2
143+
with:
144+
subject-path: "dist/*"
145+
146+
- name: Publish to pypi
147+
uses: pypa/gh-action-pypi-publish@release/v1
148+
with: # DELETEME (and the next line) to start uploading to PyPI
149+
repository-url: https://test.pypi.org/legacy/

‎pyproject.toml‎

Lines changed: 81 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,22 +38,26 @@ classifiers=[
3838
"Operating System :: POSIX :: Linux",
3939
"Operating System :: Unix",
4040
"Natural Language :: English",
41-
"Programming Language :: Python :: 3.8",
42-
"Programming Language :: Python :: 3.9",
4341
"Programming Language :: Python :: 3.10",
4442
"Programming Language :: Python :: 3.11",
43+
"Programming Language :: Python :: 3.12",
44+
"Programming Language :: Python :: 3.13",
4545
]
4646
keywords=[
4747
"simulation", "chemistry", "cooling", "astronomy", "astrophysics"
4848
]
49-
requires-python = ">=3.7"
49+
requires-python = ">=3.10"
5050
dependencies = [
5151
'h5py',
5252
'numpy',
5353
'matplotlib',
5454
'yt>=4.0.2'
5555
]
5656

57+
[project.readme]
58+
file = "README.md"
59+
content-type = "text/markdown"
60+
5761
[project.license]
5862
text = "BSD 3-Clause"
5963

@@ -111,7 +115,29 @@ minimum-version = "build-system.requires"
111115

112116
# Files to exclude from the SDist (even if they're included by default).
113117
# Supports gitignore syntax.
114-
sdist.exclude = [".circleci",".readthedocs.yml"]
118+
sdist.exclude = [
119+
# exclude continuous integration:
120+
".circleci", ".readthedocs.yml", ".github",
121+
# exclude files related to creating precompiled wheels
122+
"scripts/wheels",
123+
# exclude data files since we want the sdist to follow the same rules as wheels
124+
# when it comes to data file distribution (for simplicity).
125+
# -> the main value of including the files would be if somebody wanted to download
126+
# the sdist, untar it, and run tests. But, I would argue that they should be
127+
# cloning the git repository for that purpose
128+
# -> this would also make the sdist ~65 times bigger
129+
"input", "grackle_data_files",
130+
# if we aren't including data-files, there's no reason to include pygrackle-tests
131+
# (all meaningful tests will currently fail)
132+
"src/python/tests",
133+
# if we aren't including data-files, there's no reason to include the pygrackle
134+
# code examples (they currently require editable installations)
135+
"src/python/examples",
136+
# for consistency, we exclude the core-library tests and examples
137+
"tests", "src/example",
138+
# exclude miscellaneous classic-build-system machinery
139+
"configure", "src/Makefile", "src/examples/Make*", "src/clib/Make*"
140+
]
115141

116142
# A list of packages to auto-copy into the wheel.
117143
wheel.packages = ["./src/python/pygrackle"]
@@ -125,3 +151,54 @@ wheel.exclude = [
125151
# No need to package template files
126152
"**.py.in"
127153
]
154+
155+
[[tool.scikit-build.overrides]]
156+
# we are using scikit-build-core's override-functionality to provide a detailed
157+
# error message when a build from a sdist fails
158+
if.from-sdist = true
159+
# maybe we should move most of this message to the website and provide a link
160+
# to the website
161+
messages.after-failure = """
162+
{bold.red}Your build of pygrackle from a sdist failed.{normal} (You should
163+
ignore the rest of this message if you were explicitly want to install from an
164+
sdist).
165+
166+
For more context:
167+
168+
- The sdist and wheel formats are the 2 modern standardized formats for python
169+
package distribution. At a high-level, both of them handle a package's python
170+
code in a similar manner. The largest differences occurs when packages (like
171+
grackle) have extension modules, written in compiled languages (like C).
172+
1. A sdist (source-distribution) includes the extension modules' source code.
173+
When installing an sdist, your package manager (e.g. pip) directly compiles
174+
the extension module & links it against dependencies, as part of process.
175+
2. A wheel ships a precompiled copy of extension module (& copies of external
176+
dependencies). During installation, your package manager copies the
177+
precompiled extension module & any dependencies to the installation
178+
179+
- Since your package manager tried to install pygrackle from a sdist, that
180+
probably means that a wheel isn't available for your current python version
181+
on the {platform.platform} (with the {platform.machine}). If this is a common
182+
Unix platform, please let the us (the Grackle developers know) and we can
183+
consider adding wheels for this platform in the future.
184+
185+
- Your build probably failed because you are missing a compiler, your compiler
186+
is too old, you are missing a dependency (e.g. hdf5), or the build-system
187+
can't automatically infer some of this info.
188+
189+
- We recommend installing pygrackle directly from the git repository. We provide
190+
detailed instructions on our website (https://grackle.readthedocs.io). If you
191+
encounter further problems, please reach out.
192+
"""
193+
194+
195+
[tool.cibuildwheel]
196+
build = "cp310-* cp311-* cp312-* cp313-*"
197+
skip = ["*_i686", "*_ppc64le", "*_s390x", "*_universal2"]
198+
# we compile high-level hdf5 api purely so we can run the test-command on musllinux
199+
# -> h5py doesn't ship binary-wheels for that platform
200+
# -> depending on how long h5py takes to compile, it may be better to simply use
201+
# test-skip = "*-musllinux*"
202+
before-build = "python {project}/scripts/wheels/cibw_before_build.py --compile-hl-h5 3rdparty {project}"
203+
test-groups = "test" # <- this installs the test dependencies
204+
test-command = "bash {project}/scripts/wheels/cibw_test_command.sh {project}"

‎scripts/configure_file.py‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ def is_valid_varname(s, start = None, stop = None):
2323
return re.fullmatch(_VALID_VARNAME_STR, s[slice(start, stop)]) is not None
2424

2525

26-
def configure_file(lines, variable_map, out_fname):
26+
def configure_file(lines, variable_map, out_fname, literal_linenos):
2727
"""
2828
Writes a new file to out_fname, line-by-line, while performing variable
2929
substituions
@@ -53,14 +53,17 @@ def replace(matchobj):
5353
# make sure to drop any trailing '\n'
5454
assert line[-1] == '\n', "sanity check!"
5555
line = line[:-1]
56-
match_count = 0
57-
58-
out_f.write(_PATTERN.sub(replace,line))
56+
if line_num in literal_linenos:
57+
subbed = line
58+
else:
59+
match_count = 0
60+
subbed = _PATTERN.sub(replace,line)
61+
out_f.write(subbed)
5962
out_f.write('\n')
6063
if err_msg is not None:
6164
out_f.close()
6265
os.remove(out_fname)
63-
raise RuntimeError(rslt)
66+
raise RuntimeError(err_msg)
6467

6568
unused_variables = used_variable_set.symmetric_difference(variable_map)
6669

@@ -134,12 +137,16 @@ def main(args):
134137
_parse_variables(variable_map, args.variable_use_file_contents,
135138
val_is_file_path = True)
136139

140+
literal_linenos = set()
141+
if args.literal_linenos is not None:
142+
literal_linenos = set(args.literal_linenos)
137143
# use variable_map to actually create the output file
138144
with open(args.input, 'r') as f_input:
139145
line_iterator = iter(f_input)
140146
configure_file(lines = line_iterator,
141147
variable_map = variable_map,
142-
out_fname = out_fname)
148+
out_fname = out_fname,
149+
literal_linenos=literal_linenos)
143150

144151
return 0
145152

@@ -165,6 +172,10 @@ def main(args):
165172
"--clobber", action = "store_true",
166173
help = "overwrite the output file if it already exists"
167174
)
175+
parser.add_argument(
176+
"--literal-linenos", nargs="*", type=int,
177+
help = "line numbers corresponding to lines that are treated as literals"
178+
)
168179

169180
if __name__ == '__main__':
170181
sys.exit(main(parser.parse_args()))

‎scripts/wheels/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
This directory holds scripts/resources used for precompiling wheels. They are all launched by cibuildwheel.
2+
3+
## Background
4+
5+
To precompile wheels, we make use of [cibuildwheel](https://cibuildwheel.pypa.io/en/stable/), which is a tool maintained by the [Python Packaging Authority](https://www.pypa.io/en/latest/). ``cibuildwheel`` is an extremely popular tool used for creating binary wheels (e.g. numpy, scipy, h5py, yt, pandas, astropy).
6+
7+
cibuildwheel is run by GitHub Actions, and its configuration values are stored within **pyproject.toml**.
8+
9+
## About the scripts
10+
11+
Pygrackle is similar to packages like numpy/scipy/h5py in the sense that its extension modules rely upon external shared libraries that aren't are always provided by external platforms (or external platforms may provide incompatible versions of the dependencies). Consequently, we need to retrieve/compile external dependencies and distribute precompiled-copies of these dependencies as part of the binary wheel. Currently, we redistribute libhdf5, HDF5's runtime dependencies, and gfortran's runtime libraries (the precise details vary with platform). Of course, we also need to distribute the associated licenses in the binary wheel.
12+
13+
To properly configure wheels to do all of this, we instruct cibuildwheel to invoke the scripts in this directory. Specifically, the scripts with the prefix "cibw_" are the ones that are directly invoked from cibuildwheel.

0 commit comments

Comments
 (0)