[WIP] feat: maximize artifact reuse between different modes

[weihanglo]

What does this PR try to resolve?

As of this writing, it aims to reuse rmeta between build and check modes.
(one emits rmeta + rlib, and the other emits rmeta)

This has a caveat that DefId in rmeta files might not match between rustc
invocations with different --emits values, and that might lead to rustc
rejecting the input rmeta. To avoid failures in rustc, this is currently
guarded by fingerprint to ensure all output files present, if not, rerun.
This make running check after build safe: build's rmeta files might
have more information and rustc is correct on rejection.

How to test and review this PR?

Proof of concept. NOT READY FOR REVIEW

cc #3501

[weihanglo]

proc-macro build has generated rmeta first, so skipping checking.

[weihanglo]

Artifact lib build generated rmeta first, so doc build can skip checking dependencies.

[epage]

This make running check after build safe: build's rmeta files might
have more information and rustc is correct on rejection.

Is this meant to be incremental to eventually allow cargo check && cargo build? Otherwise, cargo build && cargo check seems rare enough to not justify this.

[weihanglo]

Is this meant to be incremental to eventually allow cargo check && cargo build?

I would love to see it happen. Still experimenting with rustc.

Otherwise, cargo build && cargo check seems rare enough to not justify this.

You may still run cargo build && cargo doc which the latter would benefit.

[weihanglo]

…to eventually allow cargo check && cargo build?

rust-lang/compiler-team#738 is partially related

[celinval]

This make running check after build safe: build's rmeta files might
have more information and rustc is correct on rejection.

Is this meant to be incremental to eventually allow cargo check && cargo build? Otherwise, cargo build && cargo check seems rare enough to not justify this.

Allowing cargo check && cargo build to reuse artifacts will require changes to the compiler, and it could potentially slow down cargo check (as @bjorn3 explained here), so it's not clear at this point if it is something feasible or desirable. I can see that fitting in with selective compilation, which is also highly experimental.

That said, I think a lot of people run cargo clippy before commit or in CI, and they would benefit from cargo build && cargo check not requiring 2 completely separate build cycles.

What is the cost / risks of going through this change though?

[bjorn3]

A check and regular build will produce a different crate hash, so if you do cargo check; cargo build -p some_dep; cargo check you are now forced to rebuild everything that depends on some_dep as the crate hash of some_dep changed.

[celinval]

Hummm.. is that because of the -p?

[bjorn3]

The is used -p to only rebuild part of the crates in build mode. The same would apply if you run cargo build from inside the working directory of a single crate in the workspace or if you interrupt the build halfway through.

[celinval]

would the same happen if you run cargo check from inside the working directory of a single crate in the workspace?

[weihanglo]

A check and regular build will produce a different crate hash, so if you do cargo check; cargo build -p some_dep; cargo check you are now forced to rebuild everything that depends on some_dep as the crate hash of some_dep changed.

This is a good point. By sharing it might make rust-analyzer a bit worse to work with cargo.

I think the real world situation is more complicated for partial build, as other factors like feature unification may kick in. I just tried cargo +stable build -p rusqlite first and then do a full cargo +stable build in Cargo, smallvec is used but not libc. So it is not only an issue of crate hash difference. By the way, can you clarify a bit which “crate hash” you are referring to?

[bjorn3]

By the way, can you clarify a bit which “crate hash” you are referring to?

The crate hash field in the crate metadata. This field is also recorded in the crate metadata for all dependencies. If it changes, then next time rustc has to load it as indirect dependency, it will reject the .rlib/.rmeta file as having the wrong crate hash, thus forcing all dependents of a crate to be recompiled whenever the crate hash changes.

[rustbot]

☔ The latest upstream changes (possibly 36bc1d1) made this pull request unmergeable. Please resolve the merge conflicts.

[weihanglo]

This is not going to work in short term. Need a more comprehensive design and maybe work with compiler team. See #15627 (comment) and #15627 (comment). Closing.