Skip to content

Pin and automate doing isolated bumps of hub image dependencies' major versions #3565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 1, 2024
Merged

Pin and automate doing isolated bumps of hub image dependencies' major versions #3565

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6e6c4bc
Pin and automate doing isolated bumps of hub image dependencies' majo…
consideRatio Oct 31, 2024
0d46c77
Run ci/refreeze
consideRatio Oct 31, 2024
70e33c1
remove major pins for transitive hub dependencies
minrk Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/dependabot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,15 @@ updates:
interval: monthly
time: "05:00"
timezone: Etc/UTC
- package-ecosystem: pip
directory: /images/hub/unfrozen
labels: [breaking]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice isolation!

groups:
major-versions:
update-types: [major]
exclude-patterns:
- jupyterhub # bumped by other automation
schedule:
interval: daily
time: "05:00"
timezone: Etc/UTC
16 changes: 8 additions & 8 deletions .github/workflows/watch-dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# - Watch multiple images tags referenced in values.yaml to match the latest
# image tag.
#
# - Watch the jupyterhub pinning in images/*/requirements.in to match the
# - Watch the jupyterhub pinning in images/*/unfrozen/requirements.txt to match the
# latest jupyterhub version available on PyPI, and if doing this, also
# refreeze images/*/requirements.txt.
#
Expand All @@ -20,7 +20,7 @@ name: Watch dependencies
on:
push:
paths:
- "images/*/requirements.in"
- "images/*/unfrozen/requirements.txt"
- ".github/workflows/watch-dependencies.yaml"
branches: ["main"]
schedule:
Expand Down Expand Up @@ -142,10 +142,10 @@ jobs:
- name: Install Python dependencies
run: pip install packaging requests

- name: Get images/hub/requirements.in pinned version of jupyterhub
- name: Get images/hub/unfrozen/requirements.txt pinned version of jupyterhub
id: local
run: |
local_version=$(cat images/hub/requirements.in | grep 'jupyterhub==' | sed 's/jupyterhub==//')
local_version=$(cat images/hub/unfrozen/requirements.txt | grep 'jupyterhub==' | sed 's/jupyterhub==//')
echo "version=$local_version" >> $GITHUB_OUTPUT

- name: Get latest version of jupyterhub
Expand All @@ -168,11 +168,11 @@ jobs:
if: steps.local.outputs.version != steps.latest.outputs.version
run: |
for img in hub singleuser-sample; do
sed --in-place 's/jupyterhub==${{ steps.local.outputs.version }}/jupyterhub==${{ steps.latest.outputs.version }}/g' images/$img/requirements.in
sed --in-place 's/jupyterhub==${{ steps.local.outputs.version }}/jupyterhub==${{ steps.latest.outputs.version }}/g' images/$img/unfrozen/requirements.txt
done
sed --in-place 's/appVersion: "${{ steps.local.outputs.version }}"/appVersion: "${{ steps.latest.outputs.version }}"/g' jupyterhub/Chart.yaml

- name: Refreeze images/*/requirements.txt based on images/*/requirements.in
- name: Refreeze images/*/requirements.txt based on images/*/unfrozen/requirements.txt
if: steps.local.outputs.version != steps.latest.outputs.version
run: ci/refreeze

Expand Down Expand Up @@ -208,7 +208,7 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Refreeze images/*/requirements.txt based on images/*/requirements.in
- name: Refreeze images/*/requirements.txt based on images/*/unfrozen/requirements.txt
run: ci/refreeze

- name: git diff
Expand All @@ -227,4 +227,4 @@ jobs:
title: "hub image: refreeze requirements.txt"
body: >-
The hub image's requirements.txt has been refrozen based on
requirements.in.
unfrozen/requirements.txt.
2 changes: 1 addition & 1 deletion ci/refreeze
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@ for img in ${IMAGES}; do
--workdir=/io \
--user=root \
python:3.12-bookworm \
sh -c 'pip install pip-tools==7.* && pip-compile --allow-unsafe --strip-extras --upgrade'
sh -c 'pip install pip-tools==7.* && pip-compile --allow-unsafe --strip-extras --upgrade --output-file=requirements.txt unfrozen/requirements.txt'
popd
done
2 changes: 1 addition & 1 deletion images/hub/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
The Dockerfile in this folder is built by
[chartpress](https://github.com/jupyterhub/chartpress#readme), using the
requirements.txt file. The requirements.txt file is updated based on the
requirements.in file using [`pip-compile`](https://pip-tools.readthedocs.io).
unfrozen/requirements.txt file using [`pip-compile`](https://pip-tools.readthedocs.io).

## How to update requirements.txt

Expand Down
32 changes: 0 additions & 32 deletions images/hub/requirements.in
View file
Open in desktop

This file was deleted.

30 changes: 15 additions & 15 deletions images/hub/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,29 +81,29 @@ jupyter-events==0.10.0
# via jupyterhub
jupyterhub==5.2.1
# via
# -r requirements.in
# -r unfrozen/requirements.txt
# jupyterhub-firstuseauthenticator
# jupyterhub-kubespawner
# jupyterhub-ldapauthenticator
# jupyterhub-ltiauthenticator
# jupyterhub-nativeauthenticator
# oauthenticator
jupyterhub-firstuseauthenticator==1.1.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-hmacauthenticator==1.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-idle-culler==1.4.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-kubespawner==7.0.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-ldapauthenticator==2.0.1
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-ltiauthenticator==1.6.2
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-nativeauthenticator==1.3.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterhub-tmpauthenticator==1.0.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
kubernetes-asyncio==31.1.0
# via jupyterhub-kubespawner
ldap3==2.9.1
Expand All @@ -121,7 +121,7 @@ multidict==6.1.0
mwoauth==0.4.0
# via oauthenticator
oauthenticator==17.1.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
oauthlib==3.2.2
# via
# jupyterhub
Expand All @@ -141,7 +141,7 @@ prometheus-client==0.21.0
propcache==0.2.0
# via yarl
psycopg2==2.9.10
# via -r requirements.in
# via -r unfrozen/requirements.txt
pyasn1==0.6.1
# via
# ldap3
Expand All @@ -152,7 +152,7 @@ pyasn1-modules==0.4.1
pycparser==2.22
# via cffi
pycurl==7.45.3
# via -r requirements.in
# via -r unfrozen/requirements.txt
pydantic==2.9.2
# via jupyterhub
pydantic-core==2.23.4
Expand All @@ -163,7 +163,7 @@ pyjwt==2.9.0
# mwoauth
# oauthenticator
pymysql==1.1.1
# via -r requirements.in
# via -r unfrozen/requirements.txt
python-dateutil==2.9.0.post0
# via
# arrow
Expand Down Expand Up @@ -224,9 +224,9 @@ sqlalchemy==2.0.36
# jupyterhub
# sqlalchemy-cockroachdb
sqlalchemy-cockroachdb==2.0.2
# via -r requirements.in
# via -r unfrozen/requirements.txt
statsd==4.0.1
# via -r requirements.in
# via -r unfrozen/requirements.txt
text-unidecode==1.3
# via python-slugify
tornado==6.4.1
Expand Down
33 changes: 33 additions & 0 deletions images/hub/unfrozen/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# This file is the input to requirements.txt, which is a frozen version of this.
#
# To update:
# - the jupyterhub version or the frozen requirements.txt file, use the
# "Run workflow" button at https://github.com/jupyterhub/zero-to-jupyterhub-k8s/actions/workflows/watch-dependencies.yaml.
# - the major version pins in this file, trigger dependabot to run via
# https://github.com/jupyterhub/zero-to-jupyterhub-k8s/network/updates
#

# JupyterHub itself
jupyterhub==5.2.1

# JupyterHub Spawner, kubernetes specific
jupyterhub-kubespawner==7.*

# JupyterHub Authenticator choices
jupyterhub-firstuseauthenticator==1.*
jupyterhub-hmacauthenticator==1.*
jupyterhub-ldapauthenticator==2.*
jupyterhub-ltiauthenticator==1.*
jupyterhub-nativeauthenticator==1.*
jupyterhub-tmpauthenticator==1.*
oauthenticator[googlegroups,mediawiki]==17.*

# JupyterHub service shutting servers after a period of inactivity
jupyterhub-idle-culler==1.*

# Other optional dependencies for additional features
pymysql==1.* # mysql
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, I wouldn't consider a bump of any of these to be a breaking change, but that's not a big deal. Perhaps they shouldn't be pinned?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah hmmm I agree, they aren't meant to be a breaking change for the user.

It could be relevant to be given a chance to read the changelog before merging though. I'm fine with removing pins or retaining them, but I agree they are reasonably not to be listed as breaking in the end.

I think for all major bumps here, some may not be breaking in the end, so generally a manual consideration about breaking label or not this is probably needed.

psycopg2==2.* # postgres
pycurl==7.* # internal http requests handle more load with pycurl
sqlalchemy-cockroachdb==2.* # cocroachdb
statsd==4.* # statsd metrics collection (TODO: remove soon, since folks use prometheus)
17 changes: 0 additions & 17 deletions images/singleuser-sample/requirements.in
View file
Open in desktop

This file was deleted.

8 changes: 4 additions & 4 deletions images/singleuser-sample/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ jupyter-server==2.14.2
jupyter-server-terminals==0.5.3
# via jupyter-server
jupyterhub==5.2.1
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterlab==4.3.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
jupyterlab-pygments==0.3.0
# via nbconvert
jupyterlab-server==2.27.3
Expand All @@ -157,7 +157,7 @@ matplotlib-inline==0.1.7
mistune==3.0.2
# via nbconvert
nbclassic==1.1.0
# via -r requirements.in
# via -r unfrozen/requirements.txt
nbclient==0.10.0
# via nbconvert
nbconvert==7.16.4
Expand All @@ -168,7 +168,7 @@ nbformat==5.10.4
# nbclient
# nbconvert
nbgitpuller==1.2.1
# via -r requirements.in
# via -r unfrozen/requirements.txt
nest-asyncio==1.6.0
# via
# ipykernel
Expand Down
17 changes: 17 additions & 0 deletions images/singleuser-sample/unfrozen/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# This file is the input to requirements.txt, which is a frozen version of this.
#
# To update:
# - the jupyterhub version or the frozen requirements.txt file, use the
# "Run workflow" button at https://github.com/jupyterhub/zero-to-jupyterhub-k8s/actions/workflows/watch-dependencies.yaml.
#

# JupyterHub itself, update this version pinning by running the workflow
# mentioned above.
jupyterhub==5.2.1

# UI
jupyterlab
nbclassic

# plugins
nbgitpuller