Skip to content

[BUG] Stop tag enumeration on Docker Hub 429 causes multi-year back-off in v0.9.x #376

Open
@gitfeber

Description

@gitfeber

Describe the bug
When running version-checker v0.9.0 or v0.9.1 against Docker Hub without authentication, tag enumeration stops as soon as a 429 Too Many Requests is returned. The process enters an extremely long back-off (e.g. “retrying in 485009h…”), so you only see a small subset of your images in Prometheus. This did not happen in v0.8.6, where all tags were fetched successfully.

To Reproduce

  1. Deploy version-checker v0.9.1 with only these args (no --docker-token for Docker Hub):
containers:
- name: version-checker
  image: quay.io/jetstack/version-checker:v0.9.1
  args:
    - "--test-all-containers"
    - "--metrics-serving-address=0.0.0.0:8181"
  1. Wait for the first scrape cycle to finish.

  2. Inspect the logs and you’ll see entries like:

time="2025-04-30T17:58:52Z" level=info msg="[DEBUG] GET https://registry.hub.docker.com/v2/repositories/bitnami/oauth2-proxy/tags?page=19&page_size=100 (status: 429): retrying in 485009h59m52s (10 left)" client=docker component=client

  1. Check Prometheus: only the first pages were fetched; all remaining repositories are missing.

Expected behavior
Version-checker should either retry after a short delay (e.g. a few seconds) and continue enumeration, or upon the first 429, skip only that single request and proceed with the next repository, instead of entering a multi-year back-off.

Screenshots
time="2025-04-30T17:58:52Z" level=info msg="[DEBUG] GET https://registry.hub.docker.com/v2/repositories/bitnami/oauth2-proxy/tags?page=19&page_size=100 (status: 429): retrying in 485009h59m52s (10 left)" client=docker component=client

Environment (please complete the following information):

  • Version-Checker: v0.9.0 & v0.9.1
  • Kubernetes: v1.31.8
  • Prometheus: v3.3.0
  • Grafana: v9.5.0
    Container Registry: registry.hub.docker.com (anonymous pull and docker token pull)

Additional context

  • In v0.8.6, the complete tag listing worked without any rate-limit interruptions.
  • Docker Hub enforces an “abuse limit” on unauthenticated API requests, which currently halts the scan completely when hit.
  • A more graceful retry strategy or skipping on 429 would restore full coverage.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions