Issuse with running more than one dind job in parallel

[freedomfury]

gitlab-ci-local Docker-in-Docker TLS Troubleshooting

This document summarizes the environment configuration, troubleshooting commands, and outputs from attempts to run Docker-in-Docker (dind) in parallel on a local host using gitlab-ci-local.

---

Example Pipeline YAML

# .gitlab-ci.yml
stages:
  - test

dind_job_1:
  stage: test
  image: docker:latest
  services:
    - name: docker:dind
      alias: docker
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker info

dind_job_2:
  stage: test
  image: docker:latest
  services:
    - name: docker:dind
      alias: docker
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker info

---

Example Variable Files

.gitlab-ci-local-env

# shellcheck shell=bash

PRIVILEGED=true
ULIMIT=8000:16000
VOLUME=certs:/certs/client
VARIABLE="DOCKER_TLS_CERTDIR=/certs"

.gitlab-ci-local-variables.yml

CI_REGISTRY: gitlab-ci-local-registry:5000

---

Troubleshooting Commands and Outputs

1. Compare CA Cert SHA1 Sums

First Pair (Works)

docker exec vibrant_williams sha1sum /certs/server/ca.pem
# Output:
601b24954f25a212bcb732993dfee3cfaf845dfc  /certs/server/ca.pem

docker exec adoring_allen sha1sum /certs/client/ca.pem
# Output:
601b24954f25a212bcb732993dfee3cfaf845dfc  /certs/client/ca.pem

Second Pair (Fails)

docker exec gracious_kalam sha1sum /certs/server/ca.pem
# Output:
21475f6e70402ebef8c5235dad35a344f18768b3  /certs/server/ca.pem

docker exec agitated_shirley sha1sum /certs/client/ca.pem
# Output:
601b24954f25a212bcb732993dfee3cfaf845dfc  /certs/client/ca.pem

---

2. Show CA Certificate Details

vibrant_williams (dind, works)

docker exec vibrant_williams openssl x509 -in /certs/server/ca.pem -noout -serial
# Output:
serial=7E5C55666CF686AE4FBCB178CECFACF03BB542B7

adoring_allen (client, works)

docker exec adoring_allen openssl x509 -in /certs/client/ca.pem -noout -serial
# Output:
serial=7E5C55666CF686AE4FBCB178CECFACF03BB542B7

gracious_kalam (dind, fails)

docker exec gracious_kalam openssl x509 -in /certs/server/ca.pem -noout -serial
# Output:
serial=2B76F691BEF8870EF48477FFA8A6830BB776DED5

agitated_shirley (client, fails)

docker exec agitated_shirley openssl x509 -in /certs/client/ca.pem -noout -serial
# Output:
serial=7E5C55666CF686AE4FBCB178CECFACF03BB542B7

---

3. Verify Client Certificate

adoring_allen (works)

docker exec adoring_allen openssl verify -CAfile /certs/client/ca.pem /certs/client/cert.pem
# Output:
/certs/client/cert.pem: OK

agitated_shirley (fails)

docker exec agitated_shirley openssl verify -CAfile /certs/client/ca.pem /certs/client/cert.pem
# Output:
/certs/client/cert.pem: OK

---

4. Inspect Container Mounts

adoring_allen

docker inspect --format '{{range .Mounts}}{{println .Destination}}{{end}}' adoring_allen
# Output:
/builds/user/devops
/tmp/gitlab-ci-local-file-variables-user-devops-776734
/certs/client
/var/lib/docker

vibrant_williams

docker inspect --format '{{range .Mounts}}{{println .Destination}}{{end}}' vibrant_williams
# Output:
/builds/user/devops
/tmp/gitlab-ci-local-file-variables-user-devops-776734
/var/lib/docker
/certs/client

agitated_shirley

docker inspect --format '{{range .Mounts}}{{println .Destination}}{{end}}' agitated_shirley
# Output:
/builds/user/devops
/tmp/gitlab-ci-local-file-variables-user-devops-956815
/certs/client
/var/lib/docker

gracious_kalam

docker inspect --format '{{range .Mounts}}{{println .Destination}}{{end}}' gracious_kalam
# Output:
/tmp/gitlab-ci-local-file-variables-user-devops-956815
/var/lib/docker
/certs/client
/builds/user/devops

---

5. Docker Info

adoring_allen (works)

docker exec adoring_allen docker info
# Output: (Docker info, no errors)

agitated_shirley (fails)

docker exec agitated_shirley docker info
# Output:
error during connect: Get "https://docker:2376/v1.51/info": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "docker:dind CA")
Command exited with code 1

---

Summary of Findings

• Only the first job/service pair (adoring_allen + vibrant_williams) shares matching CA certs and works.
• Subsequent pairs (agitated_shirley + gracious_kalam) have mismatched CA certs between client and dind, causing TLS errors.
• All containers have /certs/client mounted, but the contents of the volume differ between pairs.
• This appears to be a bug in gitlab-ci-local's handling of named volumes and certificate generation for parallel jobs/services.

---

Expected Behavior

• All job/service pairs should share the same CA certs when using a named volume for /certs/client.
• Docker TLS should work for all pairs, not just the first.

Actual Behavior

• Only the first pair works; others fail with TLS verification errors due to CA mismatch.

[firecow]

This issue seems related #918

[freedomfury]

I spent a good of time trying to get this to work. Ultimately, the only solution that worked consistently was it inelegant set of sleep statements, staggering of the execution entry point in the docker container. I combined these timings with custom certificates that I created and exposed as a docker volume.

dind entry point
cert generation

In theory just the custom certificates being presented as a volume to all the container should've been enough. However, it seems like a sleep is necessary in order to allow docker to mount the volume. If not the entry point runs and creates new certificates that are not aligned with your custom certificates.

Note: Aliases were also needed to get the solution to work they follow the same sleep cadence.

---
stages:
  - build
alpine-image-0:
  services:
    - name: docker:dind
      alias: dind-0
      entrypoint: ["/bin/sh", "-c", "sleep 0; exec dockerd-entrypoint.sh"] <=== sleep 0
  image: docker:cli
  stage: build
  script:
    - ls -lah /dind-certs
    - docker info
alpine-image-1:
  services:
    - name: docker:dind
      alias: dind-1
      entrypoint: ["/bin/sh", "-c", "sleep 1; exec dockerd-entrypoint.sh"] <=== sleep 1
  image: docker:cli
  stage: build
  script:
    - ls -lah /dind-certs
    - docker info

alpine-image-2:
  services:
    - name: docker:dind
      alias: dind-2
      entrypoint: ["/bin/sh", "-c", "sleep 2; exec dockerd-entrypoint.sh"] <=== sleep 2
  image: docker:cli
  stage: build
  script:
    - ls -lah /dind-certs
    - docker info