SwarmSpawner volumes not working as expected

[kreuzert]

Bug description

When using SwarmSpawner in a DockerSwarm environment, volume mounting does not work as expected.

How to reproduce

Create a local DockerSwarm Setup

docker swarm init
docker network create --driver=overlay --attachable jupyterhub-net
docker pull jupyter/base-notebook:latest

Docker Compose File

Minimal docker compose file:

# jupyterhub-compose.yaml
version: "3.8"

services:
  jupyterhub:
    entrypoint:
      - /bin/bash
      - -c
      - "pip install dockerspawner==13 ; jupyterhub"
    image: jupyterhub/jupyterhub:latest
    ports:
      - "8000:8000"
      - "8081:8081"
    volumes:
      - jupyterhub-data:/srv/jupyterhub
      - /var/run/docker.sock:/var/run/docker.sock

    environment:
      DOCKER_NETWORK_NAME: jupyterhub-net
      SPAWNER_CLASS: dockerspawner.SwarmSpawner
      HUB_IP: jupyterhub
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]
    networks:
      - jupyterhub-net

volumes:
  jupyterhub-data:

networks:
  jupyterhub-net:
    external: true

JupyterHub Configuration

Example jupyterhub_config.py file:

# jupyterhub_config.py
c = get_config()

from dockerspawner import SwarmSpawner

c.JupyterHub.authenticator_class = "dummy"

c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.hub_connect_url = "http://jupyterhub:8081"
c.JupyterHub.hub_bind_url = "http://0.0.0.0:8081"
c.JupyterHub.log_level = 10

c.JupyterHub.spawner_class = SwarmSpawner
c.SwarmSpawner.image = 'jupyter/base-notebook:latest'
c.SwarmSpawner.network_name = 'jupyterhub-net'
c.SwarmSpawner.debug = True

### Use persistent storage example from https://jupyterhub-dockerspawner.readthedocs.io/en/latest/data-persistence.html
notebook_dir = '/home/jovyan/work'
c.SwarmSpawner.notebook_dir = notebook_dir
# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.SwarmSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }
# Mount a directory on the host to the notebook user's notebook directory in the container
c.SwarmSpawner.mounts = [ 
  {'source': '/jupyterhub-user-{username}', 'target': notebook_dir, 'type': 'volume'}
]

Start JupyterHub

docker stack deploy -c jupyterhub-compose.yaml jupyterhub
# Wait a few seconds until it's running
docker stack rm jupyterhub
# I'm no expert in Docker / DockerSwarm enviroments. I had to copy the config manually into the volume on my local machine. There are smarter ways for that, for sure
cp jupyterhub_config.py /var/lib/docker/volumes/jupyterhub_jupyterhub-data/_data/jupyterhub_config.py
docker stack deploy -c jupyterhub-compose.yaml jupyterhub

When browsing to http://127.0.0.1:8000/hub/home , logging in and starting a Server it does not start.

docker service ps jupyter-a --no-trunc
ID                          NAME              IMAGE                          NODE      DESIRED STATE   CURRENT STATE             ERROR                                                      
                PORTS                                                                         
9kpe7i4k7jrj9w5m20fwsku94   jupyter-a.1       jupyter/base-notebook:latest   <node>    Ready           Rejected 2 seconds ago    "invalid bind mount source, must be an absolute path: jupyt
erhub-user-a"

Expected behaviour

SwarmSpawner should use the correct mount type.

Actual behaviour

SwarmSpawner uses type bind, but I think it should use type "volume" ?

Your personal set up

Minimal DockerSwarm setup on my local machine.

• OS: Ubuntu 24.04.2.

• Version(s):
  Docker version 28.0.4, build b8034c0.
• Packages:
  JupyterHub 5.3.0
  dockerspawner==13

Logs

Logs:

[D 2025-05-13 07:58:56.475 JupyterHub user:913] Calling Spawner.start for a                                                                
[D 2025-05-13 07:58:56.475 JupyterHub dockerspawner:1247] Skipping pull of jupyter/base-notebook:latest                                    
[D 2025-05-13 07:58:56.475 JupyterHub dockerspawner:1027] Getting service 'jupyter-a'                                                      
[I 2025-05-13 07:58:56.482 JupyterHub dockerspawner:1033] Service 'jupyter-a' is gone                                                      
[I 2025-05-13 07:58:56.488 JupyterHub dockerspawner:1311] Created service jupyter-a (id: oyhhp87) from image jupyter/base-notebook:latest  
[I 2025-05-13 07:58:56.488 JupyterHub dockerspawner:1335] Starting service jupyter-a (id: oyhhp87)
[D 2025-05-13 07:58:56.488 JupyterHub swarmspawner:155] Getting task of service 'jupyter-a'
[D 2025-05-13 07:58:56.488 JupyterHub dockerspawner:1027] Getting service 'jupyter-a'   
[D 2025-05-13 07:58:56.490 JupyterHub swarmspawner:280] Service oyhhp87 state: pending     
[I 2025-05-13 07:58:57.446 JupyterHub log:192] 302 GET /hub/spawn/a -> /hub/spawn-pending/a (a@10.0.0.2) 1005.98ms                   
[D 2025-05-13 07:58:57.462 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/a via scope servers!server=a/
[I 2025-05-13 07:58:57.462 JupyterHub pages:400] a is pending spawn 
[I 2025-05-13 07:58:57.466 JupyterHub log:192] 200 GET /hub/spawn-pending/a (a@10.0.0.2) 11.64ms
[D 2025-05-13 07:58:57.491 JupyterHub swarmspawner:155] Getting task of service 'jupyter-a'     
[D 2025-05-13 07:58:57.491 JupyterHub dockerspawner:1027] Getting service 'jupyter-a'               
[D 2025-05-13 07:58:57.498 JupyterHub swarmspawner:280] Service oyhhp87 state: rejected
...
# and state rejected repeats until spawner gives up on server

[kreuzert]

I'm not sure if we've configured the SwarmSpawner incorrectly. Originally we've noted this behaviour when using internal_ssl=True. Debugging this led me to the minimal example shown above.

I would suggest to add update the SwarmSpawner.mounts property to this:

    volume_binds_mount_type = Unicode(
        default_value="bind",
        allow_None=False,
        config=True,
        help="""
        Use this mount type for creating mounts of volume_binds.
        Check https://docs.docker.com/engine/storage/volumes/ for allowed mount types.

        Default: "bind"
        """
    )

    @property
    def mounts(self):
        if len(self.volume_binds):
            driver = self.mount_driver_config
            return [
                Mount(
                    target=vol["bind"],
                    source=host_loc,
                    type=self.volume_binds_mount_type,
                    read_only=vol["mode"] == "ro",
                    driver_config=driver,
                )
                for host_loc, vol in self.volume_binds.items()
            ]

        else:
            return []

When setting SwarmSpawner.volume_binds_mount_type = "volume" the volume created in the DockerSpawner.move_certs function and the normal user volumes are mounted as expected, and docker does no longer reject the service due to invalid bind mount source