Need support for k8s / microk8s local image registry / mirror

[fnordahl]

In some CI regions, access to external image registries such as docker is restricted.

One concrete example is the PS7 region for Canonical self hosted GitHub runners.

When running in that region, initialization of the microk8s provider fails and hangs indefinitely (fix provided in #74).

In that region, the environment variable DOCKERHUB_MIRROR provides a URL to a local registry.

If we set this up manually, following the instructions in the microk8s documentation, the setup succeeds:

sudo snap install microk8s --channel 1.32-strict/stable
sudo mkdir -p /var/snap/microk8s/current/args/certs.d/docker.io
cat <<EOF | sudo tee /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml
server = "$DOCKERHUB_MIRROR"
[host."$DOCKERHUB_MIRROR"]
capabilities = ["pull", "resolve"]
EOF
sudo microk8s stop
sudo microk8s start

Currently concierge will install the snap and immediately wait for setup to succeed, which does not allow for the end user to interject image registry configuration.

To me this suggests that concierge needs to handle this type of setup.

[SimoneDutto]

thanks from the JAAS/Juju Terraform team for this workaround :)

[benhoyt]

Hmm, this is a pretty specific thing (and specific environment variable). We could allow for a script inserted at the right place in Concierge config, but it's no better (and in some ways worse) than just the workaround you have. I'm leaning towards we shouldn't add anything to Concierge here.

[jnsgruk]

I agree that adding support for random scripts probably isn't what we want.

Is this a generic enough thing that we could have a more "first class" option for defining registries for k8s/microk8s providers?

[benhoyt]

Yeah, I think so. If we had an image-registry setting in concierge.yaml under the microk8s provider settings (and we could do the same for k8s), that would allow users to specify an alternative image registry or mirror.

However, most users probably don't want to hardcode this value, so we'd also need support to getting concierge.yaml values from environment variables as Frode has shown, for example image-registry: $DOCKERHUB_MIRROR. We could do it for this one setting, though it'd almost certainly be better to make that $INTERPOLATION available for all string settings.

@fnordahl Do you think we need support for multiple registries/mirrors? For example, if you have containers from dockerhub and ghcr? Or is a single setting enough?

[dimaqq]

I'm curious if I perhaps understood the underlying problem correctly.

Is it about PS7 egress limitations?
If that's the case, would proxy environment variables be a valid alternative solution: https://microk8s.io/docs/install-proxy ?

Or is it about Docker Hub pull limit per hour (unauthenticated, by ip; or authenticated per user)?
The feature requests seems to suggest so, in line with https://microk8s.io/docs/dockerhub-limits#for-microk8s-version-123-or-newer
In which case, could there perhaps be a global ghcr.io or quay.io mirror provided by Canonical for the microk8s project, so that the setting doesn't have to be parametrised?

[carlcsaposs-canonical]

Or is it about Docker Hub pull limit per hour (unauthenticated, by ip; or authenticated per user)?

Yes, the mirror was created because of Docker Hub rate limits. AFAIK, the rate limit hit was unauthenticated by IP address—and I believe all of the IS-hosted runners share one or two public IP addresses

could there perhaps be a global ghcr.io or quay.io mirror provided by Canonical for the microk8s project, so that the setting doesn't have to be parametrised?

I think Maksim was looking into that awhile ago, but I don't know if anything happened there:

mk8s 100% pulls from docker. I am working with them to update the default registry.

https://chat.canonical.com/canonical/pl/76bu8mcmbj8gdnec8bxhiaeymy

Do you think we need support for multiple registries/mirrors? For example, if you have containers from dockerhub and ghcr? Or is a single setting enough?

Specifically for PS, the current configuration most people are using is a mirror just for docker.io
GHCR doesn't have the same aggressive rate limit issues, so there's no need for a mirror on PS

(I think /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml only applies to docker.io images, not all images)