Treat action results with missing mandatory outputs as cache misses

[fmeum]

This allows builds to recover from a polluted remote cache that contains an action result with an exit code of 0 that hasn't produced the required output files.

The check added in a151116 is moved closer to the lookup so that the cached result can be ignored instead of resulting in an error, with forced re-execution in the case of remote execution.

[sluongng]

Worth noting the background on this change for folks who are out of the loop (@fmeum, please feel free to correct me where im wrong):

There is no way in the current Remote Execution API spec for the client to specify which among the expected outputs are "mandatory" and which are "optional". Because of this, the remote RBE workers have to solely rely on the exit code being zero to determine whether an action was completed successfully and upload the result to Action Cache(AC). As a result, if the remote action is badly configured to exit with code zero despite the unsuccessful run, an action cache entry would still be produced, but potentially without the outputs Bazel expected.

This, in our experience, is one of the highest causes of accidental "remote cache poisoning" for newer users. Traditionally, the fix involves either changing --remote_instance_name= value to switch to a new remote cache name space, or using --noremote_accept_cached flag in a clean build. Both options essentially discard all existing AC entries by rebuilding from scratch.

So, can we implement the validation on the worker side to fail the remote action, despite an exit code zero, when missing outputs are detected?

Today, all outputs specified by Bazel Starlark rules are considered mandatory by Bazel. So that could be a reasonable way to fix this issue. However, there is a certain set of "native" actions that are defined inside Bazel Java code that may behave differently:

• TestRunner: All outputs are optional
• CppCompile / ObjcCompile: some outputs can be optional when coverage is enabled
• Javac / JavacTurbine: in cases where reduced-classpath is enabled, only .jdeps output is required for the first pass

Because of this, the remote worker/server implementation may fail some of Bazel's special actions for not producing enough outputs, even though they are completely valid. Hence, worker-side validation will be hard to maintain in the long term.

This PR introduces the alternative solution: when Bazel sees an Action Cache result with missing mandatory outputs, it will treat it as a remote cache miss and try to re-run the action to produce the missing outputs instead. This means that the "poisoned" Action Cache entry will be discarded by Bazel automatically instead of all existing Action Cache entries. The tradeoff is that if the build action is badly constructed, users will experience a lack of remote caching and thus, an increase in remote action executed.

[coeuvre]

Importing.

[fmeum]

@bazel-io fork 8.5.0