Add a real download approach to the object service

[djmitche]

#32 added a basic "full" download approach to the object service, but in light of developments since that time we probably need something more sophisticated. Good thing we implemented a way to negotiate different download approaches!

Features required:

• support client-side validation of hashes (#4608)
• support transparent compression where possible
• support range downloads where available (for faster retries and parallel downloads)

Implement that more sophisticated thing. Also, implement a binary tool for using that download approach, either as part of the shell client or as a standalone binary.

[djmitche]

I think the next step is to outline the things that this download approach needs to support, based on some issue and bug archaeology.

[djmitche]

The bug archaeology is complete.

Regarding parallel downloads, it's not clear that's a good idea. This AWS blog entry suggests using it but only for objects that were uploaded in multiple parts. Nothing in the AWS docs suggests that those two things need to be related. This paper finds that parallel downloads are not much better than sequential downloads. I suspect we'd want to do some benchmarking to determine that parallel downloads are a good idea, and balance that with the resilience that comes from retrying specific ranges instead of a full object.

Parallel downloads would also require a different API in all of the clients. Likely it would be a modification to the download_to_file method that allows writing to multiple spans of the file at the same time. That's an improvement that can wait.

I think the bit of parallel downloads that will be important for now is support for the range header. Let's just signal that in the response to startDownload, with most backends just hard-coding it (S3 and GCS both support it).

[djmitche]

As for compression, we want to support compression for two reasons: reduced backend storage, and reduced bandwidth. HTTP provides two encoding options: Content-Encoding and Transfer-Encoding.

Transfer-Encoding is a property of the message, and is specific to each hop. So, for example, a CDN could use transfer-encoding to compress data between the client and the CDN, even if the origin served the data uncompressed. Some testing shows that S3 does not apply any transfer-encoding. And most of the hits I find looking for this only talk about Transfer-Encoding: chunked. I suspect Transfer-Encoding is a dead end.

Content-Encoding is a property of the representation, so it is carried through multiple hops (e.g., CDN) without change. S3 only barely supports this: it's possible to assign a specific Content-Encoding header on upload which is then parroted in GET responses. But this is incorrect for all encodings besides identity, as a GET request may not indicate support for the content-encoding chosen on upload. The default value for Accept-Encoding is identity. GCS supports decompressive transcoding, meaning that an object stored with Content-Encoding: gzip will be automatically decompressed for GET requests not indicating support for gzip content-encoding. This transcoding support disables support for Range headers, meaning parallel downloads are not supported, but that's probably OK -- it would only be used in the simple download method, anyway. GCS does not support compressive transcoding (i.e., gzipping during download).

So I think most of the above needs to be handled with the upload process, and I'll make the necessary notes there.

But, on download, we should support automatic decompression. It should send Accept-Encoding: gzip (and whatever else it wants) and then look at the resulting Content-Encoding header.