Support authentication when downloading files (e.g. artifacts)

[giordano]

It has been requested several times to make it possible to have artifacts in private GitHub repositories, or other services such as S3 which would require authentication. For GitHub and GitLab (I don't know if this is the same for all other services) the solution is simple: add the entry "Authorization" => "Bearer ${TOKEN}" to the headers. But some of the problems to implement such a solution are:

• how to let the user provide the token?
• how to determine when the tokens are required?
• related to the point above, the authentication method may not be a single one for a Pkg.add: an environment may have artifacts hosted on different private services, we should have a (user-defined?) mechanism to identify which token to use for each service.

To further complicate things, authentication could be optional: GitHub in general allows unauthenticated downloads from public repositories (also, there's no way to programmatically tell whether a repo is public or private based on the URL only without querying the server), but it recently started aggressively throttling such requests, so that even downloading public artifacts could need a token.

I have the feeling that the Artifacts.toml should optionally have at least some hints for "how to authenticate" for each artifact: for example the name of the environment variable to use as token if it exists? I'm not a fan of this solution, but this to give an idea of the type of approach we may want.

I'm opening the ticket to start the discussion about this, since the new rate limiting also affects Julia's own CI and of many downstream packages.

[vchuravy]

I was hoping artifacts were immune to that, but that would require them to be proxied through the package servers?

For reference in Trixi.jl we did trixi-framework/Trixi.jl#2415

[giordano]

I was hoping artifacts were immune to that, but that would require them to be proxied through the package servers?

Yes, but for example

Pkg.jl/test/test_packages/ArtifactInstallation/Artifacts.toml

Lines 145 to 162 in 88629b5

	[socrates]
	git-tree-sha1 = "43563e7631a7eafae1f9f8d9d332e3de44ad7239"
	lazy = true
	[[socrates.download]]
	url = "https://github.com/staticfloat/small_bin/raw/91f3ecf327d1de943fe076657833252791ba9f60/socrates.tar.gz"
	sha256 = "e65d2f13f2085f2c279830e863292312a72930fee5ba3c792b14c33ce5c5cc58"
	[[socrates.download]]
	url = "https://github.com/staticfloat/small_bin/raw/91f3ecf327d1de943fe076657833252791ba9f60/socrates.tar.bz2"
	sha256 = "13fc17b97be41763b02cbb80e9d048302cec3bd3d446c2ed6e8210bddcd3ac76"
	[collapse_the_symlink]
	git-tree-sha1 = "69a468bd51751f4ed7eda31c240e775df06d6ee6"
	[[collapse_the_symlink.download]]
	url = "https://github.com/staticfloat/small_bin/raw/91f3ecf327d1de943fe076657833252791ba9f60/collapse_the_symlink/collapse_the_symlink.tar.gz"
	sha256 = "956c1201405f64d3465cc28cb0dec9d63c11a08cad28c381e13bb22e1fc469d3"

isn't on the PkgServer and that's used in CI here and in Julia. And we still need a way to allow users host private artifacts.