Skip to content

Support Non-Form GET/POST Requests in Sites Tree Export #8941

@mhmdiaa

Description

@mhmdiaa

Describe the bug:

The Sites Tree File Format export produced by the exim add-on treats all request with bodies as if they are application/x-www-form-urlencoded POST requests using the format key1=value1&key2=value2. It attempts to parse them, strip values, and produce an output like key1=&key2=.

This produces inconsistencies when the request body uses other formats, such as JSON or XML:

  1. Multi-line bodies are truncated and only the first line is preserved in the data field.
  2. Bodies with = characters are misinterpreted. For example, an XML declaration like <?xml version="1.0"?> becomes <?xml version= after parsing.
  3. The bodies of non-POST requests with bodies (e.g. PUT, PATCH) are not included, since only POST requests go through the body-extraction logic.

Steps to reproduce the behavior:

Run the following automation plan, which uses a requestor job to send different types of HTTP requests, and an export job to export the site tree.

env:
  contexts:
  - name: Default Context
    urls:
    - https://httpbingo.org
jobs:
- type: requestor
  requests:
    - name: GET Request
      url: https://httpbingo.org/get?key1=value1&key2=value2
      method: GET
    - name: POST Request With Form Data
      url: https://httpbingo.org/post
      method: POST
      data: 'key1=value1&key2=value2'
      headers:
        - "Content-Type: application/x-www-form-urlencoded"
    - name: PUT Request
      url: https://httpbingo.org/put
      method: PUT
      data: '{"key1": "value1",  "key2": "value2"}'
      headers:
        - 'Content-Type: application/json'
    - name: POST Request With Multipart Form Data
      url: https://httpbingo.org/post
      method: POST
      data: |
        --boundary
        Content-Disposition: form-data; name="key1"

        value1
        --boundary
        Content-Disposition: form-data; name="key2"

        value2
        --boundary--
      headers:
        - "Content-Type: multipart/form-data; boundary=boundary"
    - name: PATCH Request
      url: https://httpbingo.org/patch
      method: PATCH
      data: '{"key1": "value1",  "key2": "value2"}'
      headers:
        - 'Content-Type: application/json'
    - name: POST Request With JSON Data
      url: https://httpbingo.org/post
      method: POST
      data: '{"key1": "value1",  "key2": "value2"}'
      headers:
        - 'Content-Type: application/json'
    - name: POST Request With XML Data
      url: https://httpbingo.org/post
      method: POST
      data: |
        <?xml version="1.0"?>
        <key1>value1</key1>
        <key2>value2</key2>
      headers:
        - "User-Agent: test"
        - 'Content-Type: application/xml'
- type: export
  parameters:
    type: yaml
    source: sitestree
    fileName: sitestree.yaml

Note: httpbingo.org requires a User-Agent header to respond correctly, but it's omitted here for brevity since the response is not relevant to this report.

The resulting sitestree.yaml file should look like this:

- node: Sites
  children:
  - node: https://httpbingo.org
    url: https://httpbingo.org
    method: GET
    children:
    - node: "GET:get(key1,key2)"
      url: https://httpbingo.org/get?key1=value1&key2=value2
      method: GET
      responseLength: 930
      statusCode: 200
    - node: PATCH:patch
      url: https://httpbingo.org/patch
      method: PATCH
      responseLength: 1080
      statusCode: 200
    - node: POST:post()(<?xml version)
      url: https://httpbingo.org/post
      method: POST
      responseLength: 1100
      statusCode: 200
      data: '%3C%3Fxml+version='
    - node: "POST:post()(key1,key2)"
      url: https://httpbingo.org/post
      method: POST
      responseLength: 1103
      statusCode: 200
      data: key1=&key2=
    - node: POST:post()(multipart/form-data)
      url: https://httpbingo.org/post
      method: POST
      responseLength: 1238
      statusCode: 200
    - node: "POST:post()({\"key1\": \"value1\",  \"key2\": \"value2\"})"
      url: https://httpbingo.org/post
      method: POST
      responseLength: 1078
      statusCode: 200
      data: '%7B%22key1%22%3A+%22value1%22%2C++%22key2%22%3A+%22value2%22%7D='
    - node: PUT:put
      url: https://httpbingo.org/put
      method: PUT
      responseLength: 1076
      statusCode: 200

Note the following behavior:

Expected

  • GET Request: no body, no data field.
  • POST Request with Form Data: keys are kept, values are stripped.
  • POST Request with Multipart Form Data: the body is present but intentionally left out in the export.

Unexpected

  • PUT Request and PATCH Request are missing the data field, even though they have bodies.
  • POST Request with JSON Data is mis-parsed (ends with an = character.)
  • POST Request with XML Data is mis-parsed (only the first line is kept, and everything after the first = is removed.)

Expected behavior:

  • PUT and PATCH requests include a data field as long as they have bodies.
  • Request bodies are exported with minimal parsing to support non-form requests.

Software versions:

ZAP
Version: 2.16.1

Installed Add-ons: [...[id=exim, version=0.14.0]...]

Screenshots:

No response

Errors from the zap.log file:

No response

Additional context:

Omitting the values of each key in x-www-form-urlencoded bodies seems reasonable, but this approach could lead to inconsistencies with JSON, XML, and possibly other formats. These would need to either:

  • Contain the full body
  • Or require parsing and sanitizing based on their formats, which (IMHO) adds unnecessary complexity

It might be worth considering keeping the entire body with minimal parsing, unless there's a strong reason to do otherwise.

Would you like to help fix this issue?

  • Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions