Running SwarmSpawner with c.SwarmSpawner.extra_create_kwargs = {"user": "root"} fails #497
Description
Bug description
I have SwarmSpawner working on a cluster (details below), but would like to run JupyterStacks containers with the logged in username and UID. Running a container directly as per the docs works correctly, e.g. something like:
docker run -it --rm \
-p 803:8888 \
-e NB_USER="testUser" \
-e CHOWN_HOME=yes \
-e NB_UID=1045 \
-e NB_GID=100 \
jupyter/datascience-notebook
With SwarmSpawner I can pass the env settings without issues (see note below on the method), but my spawn fails when I add c.SwarmSpawner.extra_create_kwargs = {"user": "root"} (as specified for changing the user in the SwarmSpawner docs)
Note: for user management I'm using LDAP, and pulling the user info with a pre-spawn-hook routine (see config section below). The working environmental settings go via lines like spawner.environment['NB_UID'] = str(spawner.userdata['uidNumber'][0]), where userdata is populated for the given user from the LDAP server.
In my case I'm stuck running a fairly old version of Docker (20.10.7, API 1.41), so I'm guessing this is the cause of the issue, but it would be good to confirm this, and not just that I'm doing something stupid!
Work-around: building images from Docker Stacks locally and setting the user to root avoids the need to pass --user root, and works as desired with the SwarmSpawner and passed env settings in my case. A minimal Dockerfile example for this is simply:
ARG JUPYTERSTACKS_TAG="2023-09-11"
ARG JUPYTERSTACKS_NOTEBOOK="datascience-notebook"
FROM jupyter/${JUPYTERSTACKS_NOTEBOOK}:${JUPYTERSTACKS_TAG}
USER root
In this case, containers spawn as expected, and accept the additional user environmental settings passed at spawn.
How to reproduce
Add the line c.SwarmSpawner.extra_create_kwargs = {"user": "root"} to a working SwarmSpawner jupyterhub_config.py and test!
Expected behaviour
Equivalent to docker run --user root as per the documentation.
Actual behaviour
Error on Spawn, specifically TypeError: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'. (I also tested for "--user", "u" and "-u" instead of "user", but got the same result.)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | [E 2023-09-14 22:32:49.530 JupyterHub user:884] Unhandled error starting testUser's server: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | Traceback (most recent call last):
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 798, in spawn
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1270, in start
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | obj = await self.create_object()
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/local/lib/python3.10/dist-packages/dockerspawner/swarmspawner.py", line 239, in create_object
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | service = await self.docker("create_service", **create_kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | result = self.fn(*self.args, **self.kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 948, in _docker
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | return m(*args, **kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | File "/usr/local/lib/python3.10/dist-packages/docker/utils/decorators.py", line 34, in wrapper
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | return f(self, *args, **kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER | TypeError: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'
Your personal set up
- OS: cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
- Docker version:
Client: Docker Engine - Community
Version: 20.10.7
API version: 1.41
Go version: go1.13.15
Git commit: f0df350
Built: Wed Jun 2 11:56:47 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:54:58 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
- Jupyter Hub container build from v4.0.2 Docker Hub image (Python 3.10), with current versions of LDAP authenticator (v1.3.2) and Docker Spawner (v12.1.0) pulled from Pypi:
ARG JUPYTERHUB_VERSION=4.0.2
FROM jupyterhub/jupyterhub:${JUPYTERHUB_VERSION}
RUN pip install \
dockerspawner \
jupyterhub-ldapauthenticator
# For LDAP testing
RUN apt-get update && apt install -y slapd ldap-utils \
&& apt install -y iproute2 iputils-ping
Configuration
# [...Messy test config - SwarmSpawner settings only below...]
# **** SWARM Spawner settings
# Based on example from https://zozoo.io/how-to-run-jupyterhub-in-docker-swarm-environment-using-swarmspawner/
# NOT sure all settings are required, but this config worked in testing.
c.ConfigurableHTTPProxy.should_start = True
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.network_name = network_name
c.SwarmSpawner.extra_host_config = { 'network_mode': network_name }
#c.SwarmSpawner.extra_create_kwargs.update({ 'volume_driver': 'local' })
c.SwarmSpawner.remove_containers = True
c.SwarmSpawner.debug = True
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = jhub_port
c.SwarmSpawner.host_ip = "0.0.0.0"
# TLS config
c.JupyterHub.port = 8000
#c.JupyterHub.ssl_key = os.environ['SSL_KEY']
#c.JupyterHub.ssl_cert = os.environ['SSL_CERT']
c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 3000 # Set longer timeout here to allow for disemination of containers to nodes
c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] }
# GET additional user info via LDAP
# See https://github.com/jupyterhub/kubespawner/issues/298#issuecomment-466440455
# Note auth_state from ldapauth should have this, but not working in testing, so use custom routine.
def pre_spawn_start_lookup(spawner):
user = spawner.user.name
# Try custom lookup, LDAPauth doesn't seem to be doing as advertised? (No additional data pulled, and nothing in logs)
from ldap3 import Server, Connection, ALL
server = Server(serverName, get_info=ALL)
conn = Connection(server, auto_bind=True)
conn.search(f'dc={serverName},dc=local', f'(&(objectclass=person)(uid={user}))',attributes=['cn', 'givenName',
'uidNumber','homeDirectory','gidNumber'])
spawner.userdata = conn.entries[0]
spawner.log.info(f"Got {user} additional props.")
spawner.log.info(spawner.userdata)
# setup environment
spawner.environment = {
"CHOWN_HOME": "yes",
"CHOWN_EXTRA": "/home/jovyan",
"CHOWN_HOME_OPTS": "-R",
}
# Works if JupyterLab runs as root user
spawner.environment['NB_UID'] = str(spawner.userdata['uidNumber'][0])
spawner.environment['NB_USER'] = user
c.SwarmSpawner.debug = True
c.Spawner.pre_spawn_hook = pre_spawn_start_lookup
# Specify root user? THIS FAILS
# c.SwarmSpawner.extra_create_kwargs = {"user": "root"}