"minikube image load" silently fails when repo/image:tag combination is already in use

[holograph]

What Happened?

I've verified that the following happens on:

• Windows 11, minikube v1.29.0, Docker Desktop v20.10.23
• MacOS 13.2.1 (Apple M1), minikube v1.29.0, Docker Desktop v20.10.23

When attempting to overwrite an image that already exists and is in use by a running container, the following happens:

1. The image listed in the minikube Docker context doesn't change (verified via minikube docker-env as well as directly via minikube ssh; the running container has the same image ID listed even after restart).
2. The command actually fails silently (no output or exit code). Adding --alsologtostderr -v=2 showcases the real problem, see listing below (edited for brevity and identifying information).

Expected results:

1. Image is updated in the minikube Docker context (fresh image ID in docker inspect + updated images when restarting containers)
2. Loading a new repo/image:tag combination makes it available in the local Docker context without affecting existing versions (or at least not pruning images in use), i.e. the operation should be idempotent and not affect running containers in the cluster

Attach the log file

C:\work>call minikube image load -v=2 --alsologtostderr :latest
I0312 12:23:03.452060 13268 config.go:180] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.26.1
I0312 12:23:03.452060 13268 localpath.go:146] windows sanitize: C:\Users\Tomer Gabel.minikube\cache\images\amd64<elided>:latest -> C:\Users\Tomer Gabel.minikube\cache\images\amd64<elided>_latest
I0312 12:23:03.518637 13268 cache.go:107] acquiring lock: {Name:mke5e77430a5115bfec9d416d57a1ce605f867e8 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I0312 12:23:03.518637 13268 image.go:134] retrieving image: :latest
I0312 12:23:03.523927 13268 image.go:173] found :latest locally: &{ref:{Repository:{Registry:{insecure:false registry:} repository:} tag:latest original::latest} opener:0xc000ef4b60 tarballImage: id:0xc0005a1180 once:{done:0 m:{state:0 sema:0}} err:}
I0312 12:23:03.523951 13268 cache.go:162] opening: \?\Volume{2a631cad-8831-474c-990c-9d491179baea}\Users\Tomer Gabel.minikube\cache\images\amd64<elided>_latest
I0312 12:24:03.083014 13268 cache.go:157] \?\Volume{2a631cad-8831-474c-990c-9d491179baea}\Users\Tomer Gabel.minikube\cache\images\amd64<elided>_latest exists
I0312 12:24:03.083506 13268 cache.go:96] cache image ":latest" -> "C:\Users\Tomer Gabel\.minikube\cache\images\amd64\_latest" took 59.6309196s
I0312 12:24:03.084947 13268 cache.go:80] save to tar file :latest -> C:\Users\Tomer Gabel.minikube\cache\images\amd64<elided>_latest succeeded
I0312 12:24:03.085976 13268 cache.go:87] Successfully saved all images to host disk.
I0312 12:24:03.086493 13268 config.go:180] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.26.1
I0312 12:24:03.092144 13268 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0312 12:24:03.180442 13268 ssh_runner.go:195] Run: systemctl --version
I0312 12:24:03.183008 13268 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0312 12:24:03.269653 13268 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:52378 SSHKeyPath:C:\Users\Tomer Gabel.minikube\machines\minikube\id_rsa Username:docker}
I0312 12:24:03.349854 13268 cache_images.go:88] LoadImages start: [:latest]
I0312 12:24:03.357620 13268 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} :latest
I0312 12:24:03.370608 13268 cache_images.go:116] ":latest" needs transfer: ":latest" does not exist at hash "sha256:8fc1f5d25500253504025a2a65f852ae5c80c224c9f2b7553608d3f43bc4149e" in container runtime
I0312 12:24:03.371124 13268 docker.go:306] Removing :latest
I0312 12:24:03.373706 13268 ssh_runner.go:195] Run: docker rmi :latest
I0312 12:24:03.385569 13268 cache_images.go:92] LoadImages completed in 35.7153ms
W0312 12:24:03.385569 13268 cache_images.go:254] Failed to load cached images for profile minikube. make sure the profile is running. loading cached images: removing image: remove image docker: docker rmi :latest: Process exited with status 1
stdout:

stderr:
Error response from daemon: conflict: unable to remove repository reference ":latest" (must force) - container 42cd2bcf6d2b is using its referenced image 5c02f5b8a047
I0312 12:24:03.385569 13268 cache_images.go:262] succeeded pushing to:
I0312 12:24:03.386079 13268 cache_images.go:263] failed pushing to: minikube

Operating System

Windows

Driver

Docker

[afbjorklund]

I think there was supposed to be some kind of Id comparison, for the "latest" use case. Not sure what happened to it.

• Getting the Id of an image and not just the name:tag #11347

[afbjorklund]

I think the mechanism in kind is reasonable, need to use crictl to get around some quirks in the containerd runtime.

i.e. when talking docker to it, it will reply with a different number compared to when talking crictl or kubectl with it...

REPOSITORY    TAG       IMAGE ID        CREATED         PLATFORM       SIZE       BLOB SIZE
busybox       latest    c118f5383653    9 months ago    linux/amd64    4.8 MiB    2.5 MiB

IMAGE                                           TAG                 IMAGE ID            SIZE
docker.io/library/busybox                       latest              66ba00ad3de86       2.6MB

[holograph]

I think I'm missing a lot of context to understand that @afbjorklund. Especially with the Docker runtime, is there any reason to remove preexisting images? Is there any reason not to load a new layer (or load the whole image with an anonymous/generated name) and update the target tag instead?

[afbjorklund]

It is an optimization, to not transfer images that have already been uploaded. But then it needs to compare the Id, and not just go with "foo:latest". At the very least it should have a special case, like Kubernetes, to always load :latest. I am not sure why it removes existing images, I thought that was a separate command. Normally, only the tag is moved ? The old image stays, as "<none>"

32a91d6

• image load: overwrite same image unless --overwrite=false #11366

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[eolivelli]

I have the same problem

[aclevername]

Also been hitting this problem regularly

[baelter]

/remove-lifecycle stale
/remove-lifecycle rotten

minikube image load -v=2 --alsologtostderr  amqp-gateway
I0822 13:22:33.940407 2386894 out.go:296] Setting OutFile to fd 1 ...
I0822 13:22:33.940538 2386894 out.go:348] isatty.IsTerminal(1) = true
I0822 13:22:33.940544 2386894 out.go:309] Setting ErrFile to fd 2...
I0822 13:22:33.940548 2386894 out.go:348] isatty.IsTerminal(2) = true
I0822 13:22:33.940687 2386894 root.go:338] Updating PATH: /home/anders/.minikube/bin
I0822 13:22:33.941070 2386894 config.go:182] Loaded profile config "minikube": Driver=podman, ContainerRuntime=docker, KubernetesVersion=v1.27.4
I0822 13:22:33.941125 2386894 cache.go:107] acquiring lock: {Name:mkb04478fd5b28310b2bf4026f694e6c272867f3 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0822 13:22:33.941188 2386894 image.go:134] retrieving image: amqp-gateway:latest
I0822 13:22:33.941198 2386894 image.go:140] checking repository: index.docker.io/library/amqp-gateway
W0822 13:22:35.078026 2386894 image.go:147] remote: HEAD https://index.docker.io/v2/library/amqp-gateway/manifests/latest: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
I0822 13:22:35.078067 2386894 image.go:148] short name: amqp-gateway:latest
I0822 13:22:35.079098 2386894 image.go:173] found index.docker.io/library/amqp-gateway:latest locally: &{ref:{Repository:{Registry:{insecure:false registry:index.docker.io} repository:amqp-gateway} tag:latest original:amqp-gateway:latest} opener:0xc0000f0000 tarballImage:<nil> computed:false id:0xc00066e000 configFile:<nil> once:{done:0 m:{state:0 sema:0}} err:<nil>}
I0822 13:22:35.079119 2386894 cache.go:162] opening:  /home/anders/.minikube/cache/images/amd64/amqp-gateway
I0822 13:22:35.934475 2386894 cache.go:157] /home/anders/.minikube/cache/images/amd64/amqp-gateway exists
I0822 13:22:35.934505 2386894 cache.go:96] cache image "amqp-gateway:latest" -> "/home/anders/.minikube/cache/images/amd64/amqp-gateway" took 1.993392139s
I0822 13:22:35.934517 2386894 cache.go:80] save to tar file amqp-gateway -> /home/anders/.minikube/cache/images/amd64/amqp-gateway succeeded
I0822 13:22:35.934532 2386894 cache.go:87] Successfully saved all images to host disk.
I0822 13:22:35.934704 2386894 config.go:182] Loaded profile config "minikube": Driver=podman, ContainerRuntime=docker, KubernetesVersion=v1.27.4
I0822 13:22:35.935115 2386894 cli_runner.go:164] Run: sudo -n podman container inspect minikube --format={{.State.Status}}
I0822 13:22:35.987629 2386894 ssh_runner.go:195] Run: systemctl --version
I0822 13:22:35.987716 2386894 cli_runner.go:164] Run: sudo -n podman version --format {{.Version}}
I0822 13:22:36.034206 2386894 cli_runner.go:164] Run: sudo -n podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0822 13:22:36.085347 2386894 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:42879 SSHKeyPath:/home/anders/.minikube/machines/minikube/id_rsa Username:docker}
I0822 13:22:36.173081 2386894 cache_images.go:88] LoadImages start: [amqp-gateway]
I0822 13:22:36.174869 2386894 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} amqp-gateway
I0822 13:22:36.197513 2386894 cache_images.go:116] "amqp-gateway" needs transfer: "amqp-gateway" does not exist at hash "sha256:d15faa95b0f468a82d6c24f564b45b6ae5497801541e68b1f0d08d771955b3ad" in container runtime
I0822 13:22:36.197560 2386894 docker.go:316] Removing image: amqp-gateway
I0822 13:22:36.197602 2386894 ssh_runner.go:195] Run: docker rmi amqp-gateway
I0822 13:22:36.208516 2386894 cache_images.go:92] LoadImages completed in 35.399189ms
W0822 13:22:36.208533 2386894 cache_images.go:254] Failed to load cached images for profile minikube. make sure the profile is running. loading cached images: removing image: remove image docker: docker rmi amqp-gateway: Process exited with status 1
stdout:

stderr:
Error response from daemon: conflict: unable to remove repository reference "amqp-gateway" (must force) - container b954ebefc57b is using its referenced image b9a67f3b0f3c
I0822 13:22:36.208544 2386894 cache_images.go:262] succeeded pushing to: 
I0822 13:22:36.208548 2386894 cache_images.go:263] failed pushing to: minikube