Skip to content

[BUG] Stop tag enumeration on Docker Hub 429 causes multi-year back-off in v0.9.x #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gitfeber opened this issue Apr 30, 2025 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@gitfeber
Copy link

Describe the bug
When running version-checker v0.9.0 or v0.9.1 against Docker Hub without authentication, tag enumeration stops as soon as a 429 Too Many Requests is returned. The process enters an extremely long back-off (e.g. “retrying in 485009h…”), so you only see a small subset of your images in Prometheus. This did not happen in v0.8.6, where all tags were fetched successfully.

To Reproduce

  1. Deploy version-checker v0.9.1 with only these args (no --docker-token for Docker Hub):
containers:
- name: version-checker
  image: quay.io/jetstack/version-checker:v0.9.1
  args:
    - "--test-all-containers"
    - "--metrics-serving-address=0.0.0.0:8181"
  1. Wait for the first scrape cycle to finish.

  2. Inspect the logs and you’ll see entries like:

time="2025-04-30T17:58:52Z" level=info msg="[DEBUG] GET https://registry.hub.docker.com/v2/repositories/bitnami/oauth2-proxy/tags?page=19&page_size=100 (status: 429): retrying in 485009h59m52s (10 left)" client=docker component=client

  1. Check Prometheus: only the first pages were fetched; all remaining repositories are missing.

Expected behavior
Version-checker should either retry after a short delay (e.g. a few seconds) and continue enumeration, or upon the first 429, skip only that single request and proceed with the next repository, instead of entering a multi-year back-off.

Screenshots
time="2025-04-30T17:58:52Z" level=info msg="[DEBUG] GET https://registry.hub.docker.com/v2/repositories/bitnami/oauth2-proxy/tags?page=19&page_size=100 (status: 429): retrying in 485009h59m52s (10 left)" client=docker component=client

Environment (please complete the following information):

  • Version-Checker: v0.9.0 & v0.9.1
  • Kubernetes: v1.31.8
  • Prometheus: v3.3.0
  • Grafana: v9.5.0
    Container Registry: registry.hub.docker.com (anonymous pull and docker token pull)

Additional context

  • In v0.8.6, the complete tag listing worked without any rate-limit interruptions.
  • Docker Hub enforces an “abuse limit” on unauthenticated API requests, which currently halts the scan completely when hit.
  • A more graceful retry strategy or skipping on 429 would restore full coverage.
@gitfeber gitfeber added the bug Something isn't working label Apr 30, 2025
@davidcollom
Copy link
Collaborator

Hi, thanks for raising this issue, I had noticed this during testing and had repaired #354 in response - This has been merged and I'll work to get a 1.9.2 release out today! 👍

@davidcollom
Copy link
Collaborator

@gitfeber - 0.9.2 should now be released, If you could please test and report back if this is resolved so we can keep issues clear it would be great! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants