Allow use of remote execution for optional remote fetch

[Qinusty]

The remote-asset spec specifies an optional feature (quoted below) of the remote-asset Fetch server which bazel requires for a remote downloader to be useful.

  // Servers *MAY* fetch content that they do not already have cached, for any
  // URLs they support.

When integrated into a remote execution deployment it makes sense for these operations to be performed on remote-execution workers. To do this, we can leverage the work in #3 to form an Action representing the URI/Qualifiers and perform these fetches remotely through Execute requests.

This functionality should be a configurable option to enable these remote fetch operations to be distributed through an REv2 Execution service. In the case of bb-scheduler, this provides deduplication and metrics surrounding the relevant actions being executed.

This will require the Action used for storage of Assets to be formed using a Command which can be executed to successfully perform the fetch operation. Due to the flexibility of Qualifiers, it is likely that some pre-processing of qualifiers will be required to determine the appropriate command script to wrap the URI/Qualfiiers with.

[tomcoldrick-ct]

Due to the flexibility of Qualifiers, it is likely that some pre-processing of qualifiers will be required to determine the appropriate command script to wrap the URI/Qualfiiers with.

This is where the majority of the complexity lives, it seems to me. Thanks to the flexibility of Qualifiers it's difficult to design a solution that allows for the correct generation of a Command. My starter for 10 would be an interface

type QualifierTranslator {
    QualifierToCommand(q *remoteasset.Qualifier) (string, error)
}

which will convert a Qualifier to a format string into which a URI may be passed. This should at least cover the case of converting a Qualifier into a command to download a specific source type, for example:

{
    "name": "resource_type",
    "value": "application/octet-stream"
}

could be transformed into something along the lines of wget %s, which later gets applied to each URI.

However, some qualifiers specify that certain post-processing steps should take place. For example, vcs.commit requires that a commit be checked out of a VCS repository or checksum.sri requires the verification of a checksum. It's possible we can still do this using this model, and joining commands into a script. There are almost certainly still edge-cases to this though, where a single command is specified by multiple qualifiers. Perhaps the interface should rather take a slice of qualifiers.