Testing docs are very confusing

[ZedThree]

The docs for target based test specific dependencies don't mention that this only works when running ]test from the REPL, and doesn't work on the command line. It's not clear if this is a bug in the implementation or just the docs.

The alternative way, using test/Project.toml, is implied to still be unstable, which is very off-putting for a Julia novice. #3953 even claims there are two further ways to do this!

Here's a minimal example, doing my best to follow the docs:

Project.toml can be auto-generated, but the [extras] and [targets] sections have to be manually added:

name = "mvce"
uuid = "5c5c69df-07ce-4eec-9db9-de18ccd496a9"
authors = ["Peter Hill <peter.hill@york.ac.uk>"]
version = "0.1.0"

[extras]
TestItemRunner = "f8b46487-2199-4994-9208-9a1283c18c0a"
TestItems = "1c621080-faea-4a02-84b6-bbd5e436b8fe"

[targets]
test = ["TestItems", "TestItemRunner"]

test/runtests.jl:

using TestItemRunner

and running from the command line fails:

$ julia --project=. test/runtests.jl
ERROR: LoadError: ArgumentError: Package TestItemRunner not found in current path.
- Run `import Pkg; Pkg.add("TestItemRunner")` to install the TestItemRunner package.

If I instead use the alternative, possibly unstable test/Project.toml method:

Project.toml:

name = "mvce"
uuid = "5c5c69df-07ce-4eec-9db9-de18ccd496a9"
authors = ["Peter Hill <peter.hill@york.ac.uk>"]
version = "0.1.0"

test/Project.toml:

[dep]
TestItemRunner = "f8b46487-2199-4994-9208-9a1283c18c0a"
TestItems = "1c621080-faea-4a02-84b6-bbd5e436b8fe"

(do I need the top-level items? It's unclear from the docs)

Then after activating the project and test environments and instantiating inside the REPL, I can run:

$ julia --project=test test/runtests.jl

---

There generally seems to be bigger meta-issue around the docs for testing. Coming from Rust and Python, I've found it very confusing getting tests set up in Julia!
See also:

• [weakdep] [extensions] [extras] and [targets] not mentioned in documentation #3688
• docs: add manifest field documentation to Project.toml guide in Pkg.jl manual #3694
• Document two new ways of specifying test-specific dependencies #3953
• Feature request: more flexibility for Project extras and targets #3297
• target option for Pkg.add #3673