Skip to content

Commit d9ba50f

Browse files
authored
[AutoBuild] Allow limiting concurrency of ghr uploads (#1334)
When uploading a large number of artifacts, like with Enzyme, we've observed errors in `ghr` like Failed to upload one of assets: one of the goroutines failed: failed to upload asset: /tmp/jl_tLeFr2/Enzyme-logs.v0.0.128.i686-linux-gnu-cxx11-llvm_version+15.tar.gz: failed to upload release asset: /tmp/jl_tLeFr2/Enzyme-logs.v0.0.128.i686-linux-gnu-cxx11-llvm_version+15.tar.gz: POST https://uploads.github.com/repos/JuliaBinaryWrappers/Enzyme_jll.jl/releases/163097283/assets?name=Enzyme-logs.v0.0.128.i686-linux-gnu-cxx11-llvm_version%2B15.tar.gz: 403 You have exceeded a secondary rate limit. Please wait a few minutes before you try again. Passing the `-p` option to reduce parallelism is an attempt to avoid this issue.
1 parent 178ba22 commit d9ba50f

File tree

1 file changed

+8
-5
lines changed

1 file changed

+8
-5
lines changed

src/AutoBuild.jl

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -488,13 +488,16 @@ function upload_to_github_releases(repo, tag, path; gh_auth=Wizard.github_auth(;
488488
attempts::Int = 3, verbose::Bool = false)
489489
for attempt in 1:attempts
490490
try
491-
ghr() do ghr_path
492-
run(`$ghr_path -u $(dirname(repo)) -r $(basename(repo)) -t $(gh_auth.token) $(tag) $(path)`)
493-
end
491+
# Note: in some cases we may want to reduce/avoid concurrency to avoid exceeding
492+
# secondary rate limits:
493+
# https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api?apiVersion=2022-11-28#avoid-concurrent-requests
494+
# https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits
495+
concurrency = get(ENV, "BINARYBUILDER_GHR_CONCURRENCY", string(Sys.CPU_THREADS))
496+
run(`$(ghr()) -u $(dirname(repo)) -r $(basename(repo)) -t $(gh_auth.token) -p $(concurrency) $(tag) $(path)`)
494497
return
495498
catch
496-
if verbose
497-
@info("`ghr` upload step failed, beginning attempt #$(attempt)...")
499+
if verbose && attempt < attempts
500+
@info("`ghr` upload step failed, beginning attempt #$(attempt + 1)...")
498501
end
499502
end
500503
end

0 commit comments

Comments
 (0)