Image updater unable to update image tags through ApplicationSet

[eric09240]

Describe the bug
Using write-back-target to deploy an application through an ApplicationSet fails to update the image parameters. This works when deploying as an Application.

Also referenced this example from a contributor.

 level=info msg="git checkout --force main" dir=/tmp/git-fake-webserver execID=6f9b7
 level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=412.68863500000003
level=info msg="git clean -ffdx" dir=/tmp/git-fake-webserver execID=2ba29
level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=1.492445
level=error msg="Could not update application spec: failed to set image parameter version value: unexpected type  for root" application=fake-webserver

To Reproduce
Steps to reproduce the behavior:

values.yaml

containers:
  fakewebserver:
    name: fake-webserver
    image: "xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver"
    tag: "main-cbc1056"

AppSet

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: fake-webserver
  namespace: argo-cd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: "https://foo.com/foo/argo-tools"
        revision: main
        directories:
        - path: "apps/fake-webserver/*/*"
        pathParamPrefix: app
        values:
          region: "{{index .path.segments 2}}"
          environment: "{{index .path.segments 3}}" 
  template:
    metadata:
      name: "fake-webserver-{{.values.region}}-{{.values.environment}}"
      namespace: argo-cd   
      labels:
        environment: "{{.values.environment}}"
        region: "{{.values.region}}"
      annotations:
        argocd-image-updater.argoproj.io/image-list: fake-webserver=xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver
        argocd-image-updater.argoproj.io/fake-webserver.update-strategy: newest-build
        argocd-image-updater.argoproj.io/fake-webserver.allow-tags: regexp:^main-
        argocd-image-updater.argoproj.io/fake-webserver.force-update: "true"
        argocd-image-updater.argoproj.io/fake-webserver.ignore-tags: regexp:^sha256
        argocd-image-updater.argoproj.io/write-back-method: git
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-name: containers.fakewebserver.image 
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-tag: containers.fakewebserver.tag 
        argocd-image-updater.argoproj.io/git-repository: https://foo.com/foo/argo-tools
        argocd-image-updater.argoproj.io/write-back-target: "helmvalues:/{{ .app.path.path }}/values.yaml"
        notifications.argoproj.io/subscribe.app-deployed.github: ""
        argocd-image-updater.argoproj.io/git-branch: main         
    spec: 
      project: argocd
      sources:
        - repoURL: https://foo.com/foo/argo-tools
          targetRevision: main
          ref: values
        - repoURL: https://foo.com/artifactory/argocd-virtual  
          chart: foo
          targetRevision: 3.*.*
          helm:
            releaseName: "fake-webserver-{{.values.environment}}"
            valueFiles:
              - "$values/{{ .app.path.path }}/values.yaml"
      destination:
        name: "{{.values.region}}-{{.values.environment}}"
        namespace: "fake-webserver-{{.values.environment}}"          
      syncPolicy:
        automated:
          prune: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideDiff=false
          - ServerSideApply=false

 level=info msg="git checkout --force main" dir=/tmp/git-fake-webserver execID=6f9b7
 level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=412.68863500000003
level=info msg="git clean -ffdx" dir=/tmp/git-fake-webserver execID=2ba29
level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=1.492445
level=error msg="Could not update application spec: failed to set image parameter version value: unexpected type  for root" application=fake-webserver

Error

Expected behavior
Expect Image Updater to write back to the values.yaml with the latest tag.

Additional context
Can confirm that Image Updater recognizes the latest tags.

Version
v16.0
v15.x

Logs

 level=info msg="git checkout --force main" dir=/tmp/git-fake-webserver execID=6f9b7
 level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=412.68863500000003
level=info msg="git clean -ffdx" dir=/tmp/git-fake-webserver execID=2ba29
level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/git-fake-webserver operation_name="exec git" time_ms=1.492445
level=error msg="Could not update application spec: failed to set image parameter version value: unexpected type  for root" application=fake-webserver

[chengfang]

the error message "unexpected type for root" is from:

argocd-image-updater/pkg/argocd/update.go

Lines 615 to 626 in 7e4c52a

	func setHelmValue(currentValues *yaml.Node, key string, value interface{}) error {
	current := currentValues
	// an unmarshalled document has a DocumentNode at the root, but
	// we navigate from a MappingNode.
	if current.Kind == yaml.DocumentNode {
	current = current.Content[0]
	}
	if current.Kind != yaml.MappingNode {
	return fmt.Errorf("unexpected type %s for root", nodeKindString(current.Kind))
	}

, which was recently introduced in PR #1055

It seems the helm values file root element is not what the image updater was expecting, i.e., the root element node type is not a MappingNode (the actual node kind is "", a unrecognized kind). Do you see anything special in your helm values file?

[eric09240]

the root element node type is not a MappingNode (the actual node kind is "", a unrecognized kind)

I don't fully understand. Can you explain more?

I double checked and both helm values and they're nearly the same copy in both AppSet and Application deployment. I made a new image and the Application deployment picked up the new image and made the commit back to the repo whereas the AppSet deployment just hangs.

My Application deployment.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/fake-webserver.force-update: "true"
    argocd-image-updater.argoproj.io/fake-webserver.helm.image-name: containers.fake-webserver.image
    argocd-image-updater.argoproj.io/fake-webserver.helm.image-tag: containers.fake-webserver.tag
    argocd-image-updater.argoproj.io/fake-webserver.ignore-tags: regexp:^sha256
    argocd-image-updater.argoproj.io/fake-webserver.update-strategy: newest-build
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/git-repository: https://foo.com/foo/argo-tools.git
    argocd-image-updater.argoproj.io/image-list: fake-webserver=xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver
    argocd-image-updater.argoproj.io/write-back-method: git
    notifications.argoproj.io/subscribe.app-deployed.github: ""
    argocd-image-updater.argoproj.io/write-back-target: "helmvalues:./apps/fake-webserver-two/us-east-1/dev/values.yaml"
  labels:
    environment: dev
    owner: developer1
    region: useast1
  name: fake-webserver-two-dev
  namespace: argo-cd
spec:
  destination:
    name: useast1-dev
    namespace: fake-webserver-two-dev
  project: myproject
  sources:
  - ref: values
    repoURL: https://foo.com/foo/argo-tools
    targetRevision: HEAD
  - chart: webserver
    helm:
      releaseName: fake-webserver-two-dev
      valueFiles:
      - $values/apps/fake-webserver-two/us-east-1/dev/values.yaml
    repoURL: https://foo.com/artifactory/myproject-virtual
    targetRevision: 3.*.*
  syncPolicy:
    automated:
      prune: false
    syncOptions:
    - CreateNamespace=true
    - ServerSideDiff=false
    - ServerSideApply=false

My ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: fake-webserver
  namespace: argo-cd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: "https://foo.com/foo/argo-tools"
        revision: main
        directories:
        - path: "apps/fake-webserver/*/*"
        pathParamPrefix: app
        values:
          region: "{{index .path.segments 2}}"
          environment: "{{index .path.segments 3}}" 
  template:
    metadata:
      name: "fake-webserver-{{.values.region}}-{{.values.environment}}"
      namespace: argo-cd   
      labels:
        environment: "{{.values.environment}}"
        region: "{{.values.region}}"
        owner: developer1
      annotations:
        argocd-image-updater.argoproj.io/image-list: fake-webserver=xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver
        argocd-image-updater.argoproj.io/fake-webserver.update-strategy: newest-build
        argocd-image-updater.argoproj.io/fake-webserver.allow-tags: regexp:^main-
        argocd-image-updater.argoproj.io/fake-webserver.force-update: "true"
        argocd-image-updater.argoproj.io/fake-webserver.ignore-tags: regexp:^sha256
        argocd-image-updater.argoproj.io/write-back-method: git
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-name: containers.fake-webserver.image 
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-tag: containers.fake-webserver.tag 
        argocd-image-updater.argoproj.io/git-repository: https://xxxxxxxx/foo/argo-tools
        argocd-image-updater.argoproj.io/write-back-target: "helmvalues:./{{ .app.path.path }}/values.yaml"
        notifications.argoproj.io/subscribe.app-deployed.github: ""
        argocd-image-updater.argoproj.io/git-branch: main         
    spec: 
      project: argo-cd
      sources:
        - repoURL: https://xxxxxxxx/foo/argo-tools
          targetRevision: main
          ref: values
        - repoURL: https://foo.com/artifactory/myproject-virtual
          chart: webserver
          targetRevision: 3.*.*
          # path: "apps/fake-webserver/useast1/{{.values.environment}}"
          helm:
            releaseName: "fake-webserver-{{.values.environment}}"
            valueFiles:
              - "$values/{{ .app.path.path }}/values.yaml"
              # - $values/apps/fake-webserver/{{.values.region}}/{{.values.environment}}/values.yaml
      destination:
        name: "{{.values.region}}-{{.values.environment}}"
        namespace: "fake-webserver-{{.values.environment}}"          
      syncPolicy:
        automated:
          prune: false
        syncOptions:
          - CreateNamespace=true
          - ServerSideDiff=false
          - ServerSideApply=false

[chengfang]

@dkarpele is looking into this.

[dkarpele]

Hi @eric09240, I was trying to reproduce your issue but without success. I guess that there are some unexpected (for image-updater) values in your values.yaml file.

I prepared an example as similar as possible to yours but image-updater doesn't fail. It successfully updates tag to 0.2@sha256... under containers.fakewebserver.tag. Here I use my local git repo but you can use any repo you want.

Please have a look (first of all to values.yaml files) and try to find the difference between my file and yours. Probably try some tests with my example.

If you can paste the full values.yaml files it will help us a lot.

Looking forward to your reply.

[eric09240]

I've parred down my values.yaml(below) to the minimal needed to deploy the app and still getting an error. It seems like the image tag is being seen and it attempts to commit it to the repo but stuck in pending state:

level=info msg="Committing 1 parameter update(s) for application fake-webserver-useast1-dev" application=fake-webserver-useast1-dev
level=debug msg="Using commit message: build: automatic update of fake-webserver-useast1-dev\n\nupdates image argo-tools/fake-webserver tag 'main-d27bc40' to 'main-7cb5d77'\n"
level=info msg="Successfully updated image ‘xxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver:main-d27bc40' to ‘xxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver:main-7cb5d77', but pending spec update (dry run=false)" alias=fake-webserver application=fake-webserver-useast1-dev image_name=argo-tools/fake-webserver image_tag=main-d27bc40 registry=xxxxxx.dkr.ecr.us-east-1.amazonaws.com

Any chance there's weirdness with the appset template.spec? My applicationSet pretty much tracks the example you gave except that I'm referencing the application values.yaml from one repo to a helm chart located in another repo.

spec: 
      project: "argocd"
      sources:
        - repoURL: https://foo.com/foo/argo-tools
          targetRevision: HEAD
          ref: values
        - repoURL: xxxxxxx.dkr.ecr.us-east-1.amazonaws.com
          chart: webserver
          targetRevision: 3.*.*
          helm:
            releaseName: "fake-webserver-{{.values.environment}}"
            valueFiles:
            - $values/{{ .app.path.path }}/values.yaml

full Values.yaml

global:
  multiRegion:
    enabled: false
    regions:
    - us-east-1
    - eu-west-1
  application:
    data_classification: other_data
    environment: dev
    external_access: true
    name: fake-webserver
    region: us-east-1
    zone: internal

application:
  data_classification: other_data
  environment: dev
  external_access: true
  name: fake-webserver
  region: us-east-1
  zone: internal

containers:
  fake-webserver:
    name: fake-webserver
    image: xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver
    tag: main-d27bc40
    imagePullPolicy: Always
    # volumeMounts:
    # - name: efsvolume
    #   mountPath: /opt/efs
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 10
      tcpSocket:
        port: 8080
      timeoutSeconds: 3
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 3
    ports:
    - name: https
      port: 8080
      protocol: TCP
    resources:
      limits:
        cpu: 200m
        memory: 98M
      requests:
        cpu: 100m
        memory: 75M
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: false

deployment:
  enabled: true
  replicas: 2
  nodeAffinity:
    enabled: false
  strategy:
    maxSurge: 3
    maxUnavailable: 1
    type: RollingUpdate

autoscaling:
  minReplicas: 2
  maxReplicas: 3
  metrics:
    cpuutilization: 70
    memoryutilization: 70

poddisruptionbudget:
  enabled: true
  minAvailable: 1

[seanlucey]

I;ve been experiencing this issue since upgrade Helm version to 0.12.1. The only solution I found was to downgrade back to (in my case) 0.11.0

[dkarpele]

@eric09240 but are you still observing the original problem?

level=error msg="Could not update application spec: failed to set image parameter version value: unexpected type  for root" application=fake-webserver

UPD. Have you tried this config in the past? Maybe with previous updater/argocd version. I mean, you created an issue 2 weeks ago, has this scenario worked before?

[eric09240]

I was using v15.x for months before upgrading to v16 and can confirm that using all those versions deploying via Application worked while ApplicationSet caused me to run into errors.

I'm no longer seeing the initial issue but I have seen this error before but did not reported it in the initial post because I thought it was a downstream issue of the one error I reported.

evel=info msg="Successfully updated image ‘xxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver:main-d27bc40' to ‘xxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver:main-7cb5d77', but pending spec update (dry run=false)" alias=fake-webserver application=fake-webserver-useast1-dev image_name=argo-tools/fake-webserver image_tag=main-d27bc40 registry=xxxxxx.dkr.ecr.us-east-1.amazonaws.com

[chengfang]

the above INFO message is normal and we've seen this in normal image-updater runs. What should follow next is another message indicating pushing commits to git repo: "Committing 1 parameter update(s) for application ...".

If you're not seeing this "Committing 1 param..." message, it could be

• the git repo authentication is not properly configured, which is blocking the git commit operation;
• or there is a contention for the lock:

  argocd-image-updater/pkg/argocd/update.go

  Line 365 in 7e4c52a

  err := commitChangesLocked(&updateConf.UpdateApp.Application, wbc, state, changeList)

  , since there are multiple applications generated.

[eric09240]

This has been resolved. The first issue was that I didn't notice there was a configuration error in the way I use path in the git generator section. Then the next error I had to fix was making sure image updater had the right github credentials to pull/commit from/to the targeted repo. Finally the last error I had to overcome was the fact that argocd-image-updater.argoproj.io/git-repository had to match exactly with spec.sources.repourl. This line is the reason for the error. In the end it looks like this now.

Thanks @chengfang @dkarpele

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: fake-webserver
  namespace: argo-cd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: "https://github.foo.com/foo/argo-tools"
        revision: HEAD
        directories:
        - path: apps/fake-webserver/*/*
        pathParamPrefix: app
        values:
          region: "{{index .app.path.segments 2}}"
          environment: "{{index .app.path.segments 3}}" 
  template:
    metadata:
      name: "fake-webserver-{{.values.region}}-{{.values.environment}}"
      namespace: argo-cd   
      annotations:
        argocd-image-updater.argoproj.io/image-list: fake-webserver=xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/argo-tools/fake-webserver
        argocd-image-updater.argoproj.io/fake-webserver.update-strategy: newest-build
        argocd-image-updater.argoproj.io/fake-webserver.allow-tags: regexp:^main-
        argocd-image-updater.argoproj.io/fake-webserver.force-update: "true"
        argocd-image-updater.argoproj.io/fake-webserver.ignore-tags: regexp:^sha256
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-name: containers.fake-webserver.image 
        argocd-image-updater.argoproj.io/fake-webserver.helm.image-tag: containers.fake-webserver.tag
        argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd-image-updater/citadel-ci-creds 
        argocd-image-updater.argoproj.io/git-repository: https://github.foo.com/foo/argo-tools
        argocd-image-updater.argoproj.io/write-back-target: "helmvalues:/{{ .app.path.path }}/values.yaml"
        argocd-image-updater.argoproj.io/fake-webserver.git-branch: main         
    spec: 
      project: "nucleus"
      sources:
        - repoURL: https://github.foo.com/foo/argo-tools
          targetRevision: main
          ref: values
        - repoURL: https://artifactory.foo.com/artifactory/helm-virtual 
          chart: nucleus
          targetRevision: 3.10.*
          helm:
            releaseName: "fake-webserver-{{.values.environment}}"
            valueFiles:
            - $values/{{ .app.path.path }}/values.yaml

[JeferCatarina]

I'm facing the same issue on v0.16.0. I'm not using appset. After downgrading the image updater to v0.15.2, it worked properly.
Interestingly, the error was only happening with an image on a GCP Artifact Registry which uses an external script to generate the access token. Other images hosted on ghrc didn't have any problems.

This is my app definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: appname
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: appname=us-central1-docker.pkg.dev/project/registry-name/appname
    argocd-image-updater.argoproj.io/appname.update-strategy: newest-build
    argocd-image-updater.argoproj.io/appname.allow-tags: regexp:^sha-[0-9a-f]{7}$
    argocd-image-updater.argoproj.io/appname.ignore-tags: latest, master, main
    argocd-image-updater.argoproj.io/appname.helm.image-name: image.repository
    argocd-image-updater.argoproj.io/appname.helm.image-tag: image.tag
    argocd-image-updater.argoproj.io/git-repository: https://github.com/account/repo-name.git
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/secret-name
    argocd-image-updater.argoproj.io/write-back-target: helmvalues:/appname/staging/image-updater.yaml
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  destination:
    namespace: appname
    name: destination-cluster
  project: appname
  sources:
    - repoURL: https://chart.onechart.dev
      targetRevision: 0.73.0
      chart: onechart
      helm:
        valueFiles:
          - $values/appname/common/values.yaml
          - $values/appname/staging/image-updater.yaml
          - $values/appname/staging/values.yaml
        ignoreMissingValueFiles: true
    - repoURL: https://github.com/account/repo-name.git
      targetRevision: HEAD
      ref: values
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

And these are the values files:

# $values/appname/common/values.yaml
containerPort: 8080
image:
  repository: us-central1-docker.pkg.dev/project/registry-name/appname
  tag: latest
ports:
  - name: http
    containerPort: 8080
svcPort: 8080
resources:
  requests:
    cpu: 250m
    memory: 256M
  limits:
    cpu: "1"
    memory: 1G

# $values/appname/staging/values.yaml
ingresses:
  - host: host.domain.xyz
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: letsencrypt
      external-dns.alpha.kubernetes.io/hostname: "host.domain.xyz"
      external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
      external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
      nginx.ingress.kubernetes.io/whitelist-source-range: whitelisted-address/32
    tlsEnabled: true

[JeferCatarina]

I figured what my problem was: the image-updater.yaml didn't exist.
Prior v0.16.0, the file would be created with the helm.image-name and helm.image-tag values. I'm unsure what the desired behavior is, but even if the image-updater is not supposed to create the file, the logs should return a better message.

I can investigate the issue (if it is an issue and not an intended behavior) or improve the log message. Let me know how I can help with this.