Fix unicode corruption in agent downloading endpoint #330
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When agent responses containing non-ASCII characters were sent, these characters were getting corrupted on the way through the proxy. We were decoding the body into a JS string and then re-encoding it as
binary, which breaks UTF-8.❓ What Changed?
binary/utf-8toggle because it inferred fromcontent-encodingheader.Buffer[](fromany[]).⚙️ How This Works?
Keeping everything as raw buffers avoids issues and problems with string encodings. No assumptions about
content-encoding. Compressed and uncompressed bodies are preserved byte by byte.🧑🔬 How To Test?
1. Smoke
Build the new CloudFront worker from this branch and deploy it to your AWS environment. Try to use the agent downloading with a basic client. This test will ensure everything works fine (no breaking).
2. Use Mock Server
Use mock server and run
agent response body integrity protected with 200 status codetest with a CloudFront proxy integration (which points to your mock server). This test will ensure the new changes fix potential Unicode problems.Shareable mock server form value (only test selected, empty fields):
✨ Potential Improvements
Buffer.isBufferand directly push chunks to the accumulator body. (response.on('data', (chunk) => data.push(chunk)))content-lengthfrom forwarded headers and let CloudFront handle the calculation.gzip/brcompression.