Deployment to AKS uses old image

[Crawenlil]

I'm trying to update an endpoint, that is already deployed at AKS. Here is my workflow:

1. I'm updating a custom image - e.g. update installed libraries. It is registered in private ACR.
2. I call environment.build() - forcing an azureml image to be rebuilt, so that changes in source image will be updated in final image. Please notice, that the built image has the same name: d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb. The only thing that changes is the source docker, which is reflected with a new manifest.
3. Deployment of the new model:
   webservice = Model.deploy( workspace=ws, name=service_name, models=[model], inference_config=inference_config, deployment_config=deployment_config, show_output=True, overwrite=True )

Deployment goes well, however the app runs on the previous image.
In the ACR the latest version is:
d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb:latest
and sha256:19b9c6dfaa3cd0ad4364719fcdf2a170606b9e43106225748f6785291c43d7f4
while if I describe a pod in K8 it uses the previous version from cache:
d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb@sha256:bc7911cd9109ea3748feba3a56b1f8f8bc8c65b154d75a54bf96b87d80b5e472
I think that's due to imagePullPolicy: IfNotPresent.
In the kubernetes describe pod I can see, that an image was not pulled but used from cache instead.
I couldn't find any way to change this setting to Always.
I have patched the deployment map and correct image has been pulled, so I know that this option ("Always") would fix this issue for me. Thanks for any help!

[M1F1]

I've got the same problem, any progress on this bug?

[KeliGui]

Did you specify user-managed dependencies? If yes, then conda dependencies are omitted. So the built image will be same. https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments#building-environments-as-docker-images

[Crawenlil]

Yes, I have specified user-managed dependencies. However when the base image is changed, eg. some libraries were updated, a new build is registered in the acr. And, due to pull policy the old version is used for the pod on k8 instead of the newest one. For now I made a workaround - I've added a versioning system to image tags, so that azure is forced to use the latest image. But I think that it should work for the "latest" tag as well.

[itstalmeez]

Hi,

spec:
containers:

• name: your-container-name
  image: d2bbe17668204c4f957f17cca62a7db4.azurecr.io/azureml/azureml_02fba29521dca17addff8c3b5f3d93cb:latest
  imagePullPolicy: Always

Let me know if this resolves your issue or if you have any further questions!

Best regards,
Talmeez Fuaad