Add ETag handling and cache for pre-compressed .gz files

[JosePineiro]

This update brings improved standards-compliant handling of pre-compressed .gz files, providing ETag support and conditional responses, helping optimize client-side caching and server resource usage. Fully backward compatible.

ETag Generation for Pre-compressed .gz Files

• When a .gz file is served via AsyncFileResponse, the server now generates an ETag header based on the CRC32 checksum found ((first 4 bytes) in the gzip trailer (last 8 bytes of gz file).
• A Cache-Control: no-cache header is also added to enforce proper cache validation.

Conditional Request Support in send()

• The send() method now checks for the If-None-Match header when a .gz version of the requested file exists.
• If the client's ETag matches the server's ETag (derived from the gzip trailer), the server responds with 304 Not Modified, saving bandwidth and improving efficiency.
• If the ETag does not match or no valid ETag is present, the normal file response is sent.

These changes only apply to pre-compressed .gz files and do not alter the behavior for uncompressed files.

Why this is useful?

• For static resources, improves client-side caching efficiency by allowing conditional requests with If-None-Match
• Reduces unnecessary re-downloads of static resources when they haven't changed
• Aligns with common web best practices for serving compressed assets with proper cache validation.
• ETag values are derived from the CRC32 checksum embedded in the gzip trailer, ensuring the ETag reflects the original uncompressed content. If change the uncompressed content, change the Etag.
• It is fully backward compatible. It has no adverse effects on library users and does not require them to modify their code.

[mathieucarbou]

@JosePineiro : thank you for this improvement! We will review and merge soon

[mathieucarbou]

@vortigont : would you be able to have a look since you worked on this part ?

[Copilot]

Pull Request Overview

This PR introduces ETag handling and caching improvements for serving pre‐compressed .gz files, enabling conditional GET responses to optimize bandwidth and server resource usage.

• Adds ETag header generation based on the CRC32 checksum from the gzip trailer.
• Implements conditional 304 responses in the send() method when the client's ETag matches the server's.
• Enhances both asynchronous file response handling and web server request processing for .gz files.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/WebResponses.cpp	Adds ETag generation and cache-control headers for .gz files
src/ESPAsyncWebServer.h	Integrates conditional response logic based on the ETag and implements similar ETag extraction logic

[me-no-dev]

Please have a look at the copilot suggestions. Also please move the implementations of send and _getEtag to CPP instead of the header, as they are now not the very short snippets they used to be. Otherwise LGTM

[JosePineiro]

I'm torn between these two versions of the _getEtag function.
Version 1 runs on 8 bits.

static void _getEtag_v1(uint8_t trailer[4], char* serverETag) {
    static constexpr char hexChars[] = "0123456789ABCDEF";
    
    serverETag[0] = '"';
    for (int i = 0; i < 4; ++i) {
        uint8_t byte = trailer[i];
        serverETag[1 + i * 2] = hexChars[byte >> 4];
        serverETag[2 + i * 2] = hexChars[byte & 0x0F];
    }
    serverETag[9] = '"';
    serverETag[10] = '\0';
  }

Version 2 is 25% faster, but requires a 32-bit CPU and uses 20 more bits of flash memory.
Optimization techniques used in v2:

• 32-bit aligned memory copy using memcpy for better cache utilization
• Loop unrolling to eliminate branch overhead
• Direct bit shifting operations without intermediate variables
• Instructions are independent and allow the use of pipeline architecture.

 static void _getEtag_v2(uint8_t trailer[4], char* serverETag) {
    static constexpr char hexChars[] = "0123456789ABCDEF";
    
    uint32_t data;
    memcpy(&data, trailer, 4);
    
    serverETag[0] = '"';
    serverETag[1] = hexChars[(data >> 4) & 0x0F];
    serverETag[2] = hexChars[data & 0x0F];
    serverETag[3] = hexChars[(data >> 12) & 0x0F];
    serverETag[4] = hexChars[(data >> 8) & 0x0F];
    serverETag[5] = hexChars[(data >> 20) & 0x0F];
    serverETag[6] = hexChars[(data >> 16) & 0x0F];
    serverETag[7] = hexChars[data >> 28];
    serverETag[8] = hexChars[(data >> 24) & 0x0F];
    serverETag[9] = '"';
    serverETag[10] = '\0';
  }

What do you consider a priority?

[mathieucarbou]

For a web server I would Favour speed over flash size

[mathieucarbou]

why public ? should be private