exim: Handle a wider range of request types in Sites Tree export

[mhmdiaa]

Overview

Purpose

Improve the Sites Tree export to handle a wider range of request types, including JSON and XML API requests.

Goals

• Export request bodies for all HTTP methods, not just POST.
• Include the full raw body in the 'data' field without parsing parameters.
• Apply URL encoding to the entire body instead of just parameter names.

Changes

• Removed the POST-only check so all HTTP requests follow the same export logic regardless of the method (except for multipart form data)
• Stopped parsing request bodies as application/x-www-form-urlencoded; instead, the full raw body is exported after being URL-encoded
• Updated tests to expect fully URL-encoded bodies in exported POST requests, including parameter values
• Added tests to assert that non-POST request bodies are exported, and that JSON/XML bodies are handled correctly.

Related Issues

Fixes zaproxy/zaproxy#8941

Checklist

• Update help
• Update changelog
• Run ./gradlew spotlessApply for code formatting
• Write tests
• Check code coverage
• Sign-off commits
• Squash commits
• Use a descriptive title

For more details, please refer to the developer rules and guidelines.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[mhmdiaa]

I have read the CLA Document and I hereby sign the CLA

[thc202]

What are you trying to achieve? (i.e. your use cases)

[mhmdiaa]

@thc202 I'm trying to use ZAP to test apps that mostly use APIs (so most requests are JSON or XML), and export site trees that show which endpoints were found and what their requests looked like.

[psiinon]

Checkmarx One – Scan Summary & Details – fafa13ce-8acf-4978-aa6a-40eaddd46c8f

Great job, no security vulnerabilities found in this Pull Request

[psiinon]

Thank you for this PR!
As I mentioned on the issue I dont think we should include the whole body unless it is needed to recreate the node name.
I'd also like to see some more unit tests covering all of the cases (like multipart/form-data).

@thc202 I'm trying to use ZAP to test apps that mostly use APIs (so most requests are JSON or XML), and export site trees that show which endpoints were found and what their requests looked like.

Thats not the use case for the Site Tree Export. If you want the full body then you should use another option.
But we should still fix the problems you reported, and this PR is a v good step towards that 😁

[mhmdiaa]

Thats not the use case for the Site Tree Export. If you want the full body then you should use another option.

This might be a bit off-topic for the PR, but I'd really appreciate any suggestions here. I looked into the HAR export, which does include all the data I want, but it has a couple of drawbacks for my use case:

1. It includes response bodies, which make the file much larger without adding much value for me.
2. It only supports the history and all sources. history doesn't work with automation plans (since there are no manual/proxied requests), and all adds a lot of noise (like spider- or active scanner-generated traffic, which include non-existent or duplicate endpoints)

That's why sitestree seemed like the best fit: no response bodies, only real endpoints, and no duplicates. I'm still exploring this, so let me know if I've got something wrong.

I agree that including full request bodies in the Site Tree Export is not ideal either, so I'm happy to take this PR in whatever direction fits best with ZAP's goals. Just let me know what you think makes sense, and I'll give it a shot 😄

[kingthorin]

HAR should be highly compressible given that its text with high amounts of repetition. However, I guess HAR export could be updated to make response inclusion optional.

[psiinon]

Re your specific usecase, why not write a script to output whatever you want in whatever format you want?
We have examples which go through the Sites Tree and History table here https://github.com/zaproxy/community-scripts/tree/main/standalone