Description
On Apache, ModSecurity's response body scanning can conflict with mod_deflate.
If the response body is compressed by mod_deflate, it's possible that ModSecurity scans the compressed response, instead of the original response. If this happens, outgoing rules don't behave as desired.
As an example, CRS rule 970013 (OWASP_CRS/LEAKAGE/INFO_DIRECTORY_LISTING) detects open dirs ("Index of...").
To test this rule, create an open dir in a web root served by Apache:
mkdir opendir
chmod a+rx opendir
echo 'Options Indexes' > opendir/.htaccess
Under the CRS, this url now should return a 403 error instead of the directory index.
By default, the curl client uses no HTTP compression, and when we request the open dir, we can verify that the response is blocked correctly:
# curl -s -o /dev/null -w '%{http_code}' http://localhost/opendir/
403
This snippet printed just the HTTP response code. Now, when enabling HTTP compression, the same response is - incorrectly - passed through without blocking:
# curl -s -o /dev/null -w '%{http_code}' --compressed http://localhost/opendir/
200
Extra check to see that we really get the response body that should have been blocked:
# curl -s --compressed http://localhost/opendir/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /opendir</title>
</head>
[rest removed]
The other effect is that the compressed response body seems to be processed by the rules, which sometimes can give false positives, as a CRS anti-XSS or anti-SQLi rule fires on part of the binary data by accident. A lot of binary data is then logged in modsec_audit.log.
As most browsers support compression, the problem manifests in normal usage situations.
The problem goes away when disabling the mod_deflate module.
Whether this problem happens or not seems dependent on the Apache configuration. On Ubuntu with Apache package 2.4.16-4+deb.sury.org~trusty+4 the bug happens. But on FreeBSD with package apache24-2.4.16_1 outgoing content is always scanned correctly, regardless of compression.
Possibly the problem is due to module loading/activation order. Maybe it's the specific ordering of Apache configuration directives in the default .conf files for the various distributions, which are very different. If so, hopefully it might be possible to have ModSecurity hook later in a phase, e.g. by tweaking an APR_HOOK_ constant. If the condition can not be remedied but can be detected, I would propose giving at least a startup error if SecResponseBodyAccess is on, with a hint to fix the Apache configuration.
Tested the problem to be present in ModSecurity 2.7.7 and 2.9.0 compiled from source, on Ubuntu 14.04 with Apache 2.4.16 from ondrej PPA.