[Perf] Don't read full response into mem if not needed NGINX

[jeremyjpj0916]

Regarding this bit here:

https://github.com/SpiderLabs/ModSecurity-nginx/blob/master/src/ngx_http_modsecurity_header_filter.c#L450

Got me thinking, for users that just want to waf protect inbound client requests we really can drop all the phases that execute after the proxy_pass directive(besides the log phase of course) if we are talking nginx as a gateway or lb. So I did something like this in code:

Basically removed all references to _body and _header filter logic

config

# vim: filetype=sh

# If $NGX_IGNORE_RPATH is set to "YES", we will ignore explicit
# library path specification on resulting binary, allowing libmodsecurity.so
# to be relocated across configured library pathes (adjust /etc/ld.so.conf
# or set $LD_LIBRARY_PATH environment variable to manage them)
#
# $YAJL_LIB variable may need to be populated in case of non-standard
# path of libyajl.so's installation

ngx_feature_name=
ngx_feature_run=no
ngx_feature_incs="#include <modsecurity/modsecurity.h>"
ngx_feature_libs="-lmodsecurity"
ngx_feature_test='printf("hello");'
ngx_modsecurity_opt_I=
ngx_modsecurity_opt_L=

YAJL_EXTRA=
if test -n "$YAJL_LIB"; then
    YAJL_EXTRA="-L$YAJL_LIB -lyajl"
fi

# If $MODSECURITY_INC is specified, lets use it. Otherwise lets try
# the default paths
#
if [ -n "$MODSECURITY_INC" -o -n "$MODSECURITY_LIB" ]; then
    # explicitly set ModSecurity lib path
    ngx_feature="ModSecurity library in \"$MODSECURITY_LIB\" and \"$MODSECURITY_INC\" (specified by the MODSECURITY_LIB and MODSECURITY_INC env)"
    ngx_feature_path="$MODSECURITY_INC"
    ngx_modsecurity_opt_I="-I$MODSECURITY_INC"
    ngx_modsecurity_opt_L="-L$MODSECURITY_LIB $YAJL_EXTRA"

    if [ $NGX_RPATH = YES ]; then
        ngx_feature_libs="-R$MODSECURITY_LIB -L$MODSECURITY_LIB -lmodsecurity $YAJL_EXTRA"
    elif [ "$NGX_IGNORE_RPATH" != "YES" -a $NGX_SYSTEM = "Linux" ]; then
        ngx_feature_libs="-Wl,-rpath,$MODSECURITY_LIB -L$MODSECURITY_LIB -lmodsecurity $YAJL_EXTRA"
    else
        ngx_feature_libs="-L$MODSECURITY_LIB -lmodsecurity $YAJL_EXTRA"
    fi

    . auto/feature

    if [ $ngx_found = no ]; then
        cat << END
        $0: error: ngx_http_modsecurity_module requires the ModSecurity library and MODSECURITY_LIB is defined as "$MODSECURITY_LIB" and MODSECURITY_INC (path for modsecurity.h) "$MODSECURITY_INC", but we cannot find ModSecurity there.
END
        exit 1
    fi
else
    # auto-discovery
    ngx_feature="ModSecurity library"
    ngx_feature_libs="-lmodsecurity"

    . auto/feature

    if [ $ngx_found = no ]; then
        ngx_feature="ModSecurity library in /usr/local/modsecurity"
        ngx_feature_path="/usr/local/modsecurity/include"
        if [ $NGX_RPATH = YES ]; then
            ngx_feature_libs="-R/usr/local/modsecurity/lib -L/usr/local/modsecurity/lib -lmodsecurity"
        elif [ "$NGX_IGNORE_RPATH" != "YES" -a $NGX_SYSTEM = "Linux" ]; then
            ngx_feature_libs="-Wl,-rpath,/usr/local/modsecurity/lib -L/usr/local/modsecurity/lib -lmodsecurity"
        else
            ngx_feature_libs="-L/usr/local/modsecurity/lib -lmodsecurity"
        fi

        . auto/feature

    fi
fi



if [ $ngx_found = no ]; then
 cat << END
 $0: error: ngx_http_modsecurity_module requires the ModSecurity library.
END
 exit 1
fi


ngx_addon_name=ngx_http_modsecurity_module

# We must place ngx_http_modsecurity_module after ngx_http_gzip_filter_module
# in load order list to be able to read response body before it gets compressed
# (for filter modules later initialization means earlier execution).
#
# Nginx implements load ordering only for dynamic modules and only a BEFORE part
# of "ngx_module_order". So we list all of the modules that come after
# ngx_http_gzip_filter_module as a BEFORE dependency for
# ngx_http_modsecurity_module.
#
# For static compilation HTTP_FILTER_MODULES will be patched later.

modsecurity_dependency="ngx_http_postpone_filter_module \
                        ngx_http_ssi_filter_module \
                        ngx_http_charset_filter_module \
                        ngx_http_xslt_filter_module \
                        ngx_http_image_filter_module \
                        ngx_http_sub_filter_module \
                        ngx_http_addition_filter_module \
                        ngx_http_gunzip_filter_module \
                        ngx_http_userid_filter_module \
                        ngx_http_headers_filter_module \
                        ngx_http_copy_filter_module"


if test -n "$ngx_module_link"; then
	ngx_module_type=HTTP_FILTER
	ngx_module_name="$ngx_addon_name"
	ngx_module_srcs="$ngx_addon_dir/src/ngx_http_modsecurity_module.c \
            $ngx_addon_dir/src/ngx_http_modsecurity_pre_access.c \
            $ngx_addon_dir/src/ngx_http_modsecurity_log.c \
            $ngx_addon_dir/src/ngx_http_modsecurity_rewrite.c \
            "
	ngx_module_deps="$ngx_addon_dir/src/ddebug.h \
            $ngx_addon_dir/src/ngx_http_modsecurity_common.h \
            "
        ngx_module_libs="$ngx_feature_libs"
        ngx_module_incs="$ngx_feature_path"

        ngx_module_order="ngx_http_chunked_filter_module \
                          ngx_http_v2_filter_module \
                          ngx_http_range_header_filter_module \
                          ngx_http_gzip_filter_module \
                          $ngx_module_name \
                          $modsecurity_dependency";

	. auto/module
else
	CFLAGS="$ngx_modsecurity_opt_I $CFLAGS"
	NGX_LD_OPT="$ngx_modsecurity_opt_L $NGX_LD_OPT"

	CORE_INCS="$CORE_INCS $ngx_feature_path"
	CORE_LIBS="$CORE_LIBS $ngx_feature_libs"

	HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES ngx_http_modsecurity_module"
	NGX_ADDON_SRCS="\
	    $NGX_ADDON_SRCS \
	    $ngx_addon_dir/src/ngx_http_modsecurity_module.c \
	    $ngx_addon_dir/src/ngx_http_modsecurity_pre_access.c \
	    $ngx_addon_dir/src/ngx_http_modsecurity_log.c \
	    $ngx_addon_dir/src/ngx_http_modsecurity_rewrite.c \
	    "

	NGX_ADDON_DEPS="\
	    $NGX_ADDON_DEPS \
	    $ngx_addon_dir/src/ddebug.h \
	    $ngx_addon_dir/src/ngx_http_modsecurity_common.h \
	    "
fi

#
# Nginx does not provide reliable way to introduce our module into required
# place in static ($ngx_module_link=ADDON) compilation mode, so we must
# explicitly update module "ordering rules".
#
if [ "$ngx_module_link" != DYNAMIC ] ; then
    # Reposition modsecurity module to satisfy $modsecurity_dependency
    # (this mimics dependency resolution made by ngx_add_module() function
    # though less optimal in terms of computational complexity).
    modules=
    found=
    for module in $HTTP_FILTER_MODULES; do
        # skip our module name from the original list
        if [ "$module" = "$ngx_addon_name" ]; then
            continue
        fi
        if [ -z "${found}" ]; then
            for item in $modsecurity_dependency; do
                if [ "$module" = "$item" ]; then
                    modules="${modules} $ngx_addon_name"
                    found=1
                    break
                fi
            done
        fi
        modules="${modules} $module"
    done
    if [ -z "${found}" ]; then
        # This must never happen since ngx_http_copy_filter_module must be in HTTP_FILTER_MODULES
        # and we stated dependency on it in $modsecurity_dependency
        echo "$0: error: cannot reposition modsecurity module in HTTP_FILTER_MODULES list"
        exit 1
    fi
    HTTP_FILTER_MODULES="${modules}"
fi

ngx_http_modsecurity_common.h

/*
 * ModSecurity connector for nginx, http://www.modsecurity.org/
 * Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/)
 *
 * You may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * If any of the files related to licensing are missing or if you have any
 * other questions related to licensing please contact Trustwave Holdings, Inc.
 * directly using the email address security@modsecurity.org.
 *
 */


#ifndef _NGX_HTTP_MODSECURITY_COMMON_H_INCLUDED_
#define _NGX_HTTP_MODSECURITY_COMMON_H_INCLUDED_

#include <nginx.h>
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>

#include <modsecurity/modsecurity.h>
#include <modsecurity/transaction.h>


/* #define MSC_USE_RULES_SET 1 */

#if defined(MODSECURITY_CHECK_VERSION)
#if MODSECURITY_VERSION_NUM >= 304010
#define MSC_USE_RULES_SET 1
#endif
#endif

#if defined(MSC_USE_RULES_SET)
#include <modsecurity/rules_set.h>
#else
#include <modsecurity/rules.h>
#endif


/**
 * TAG_NUM:
 *
 * Alpha  - 001
 * Beta   - 002
 * Dev    - 010
 * Rc1    - 051
 * Rc2    - 052
 * ...    - ...
 * Release- 100
 *
 */

#define MODSECURITY_NGINX_MAJOR "1"
#define MODSECURITY_NGINX_MINOR "0"
#define MODSECURITY_NGINX_PATCHLEVEL "1"
#define MODSECURITY_NGINX_TAG ""
#define MODSECURITY_NGINX_TAG_NUM "100"

#define MODSECURITY_NGINX_VERSION MODSECURITY_NGINX_MAJOR "." \
    MODSECURITY_NGINX_MINOR "." MODSECURITY_NGINX_PATCHLEVEL \
    MODSECURITY_NGINX_TAG

#define MODSECURITY_NGINX_VERSION_NUM MODSECURITY_NGINX_MAJOR \
    MODSECURITY_NGINX_MINOR MODSECURITY_NGINX_PATCHLEVEL \
    MODSECURITY_NGINX_TAG_NUM

#define MODSECURITY_NGINX_WHOAMI "ModSecurity-nginx v" \
    MODSECURITY_NGINX_VERSION

typedef struct {
    ngx_str_t name;
    ngx_str_t value;
} ngx_http_modsecurity_header_t;


typedef struct {
    ngx_http_request_t *r;
    Transaction *modsec_transaction;
    ModSecurityIntervention *delayed_intervention;

#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    /*
     * Should be filled with the headers that were sent to ModSecurity.
     *
     * The idea is to compare this set of headers with the headers that were
     * sent to the client. This check was placed because we don't have control
     * over other modules, thus, we may partially inspect the headers.
     *
     */
    ngx_array_t *sanity_headers_out;
#endif

    unsigned waiting_more_body:1;
    unsigned body_requested:1;
    unsigned processed:1;
} ngx_http_modsecurity_ctx_t;


typedef struct {
    void                      *pool;
    ModSecurity               *modsec;
    ngx_uint_t                 rules_inline;
    ngx_uint_t                 rules_file;
    ngx_uint_t                 rules_remote;
} ngx_http_modsecurity_main_conf_t;


typedef struct {
    void                      *pool;
    /* RulesSet or Rules */
    void                      *rules_set;

    ngx_flag_t                 enable;
#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    ngx_flag_t                 sanity_checks_enabled;
#endif

    ngx_http_complex_value_t  *transaction_id;
} ngx_http_modsecurity_conf_t;


extern ngx_module_t ngx_http_modsecurity_module;

/* ngx_http_modsecurity_module.c */
int ngx_http_modsecurity_process_intervention (Transaction *transaction, ngx_http_request_t *r);
ngx_http_modsecurity_ctx_t *ngx_http_modsecurity_create_ctx(ngx_http_request_t *r);
char *ngx_str_to_char(ngx_str_t a, ngx_pool_t *p);
ngx_pool_t *ngx_http_modsecurity_pcre_malloc_init(ngx_pool_t *pool);
void ngx_http_modsecurity_pcre_malloc_done(ngx_pool_t *old_pool);

/* ngx_http_modsecurity_log.c */
void ngx_http_modsecurity_log(void *log, const void* data);
ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r);

/* ngx_http_modsecurity_pre_access.c */
ngx_int_t ngx_http_modsecurity_pre_access_handler(ngx_http_request_t *r);

/* ngx_http_modsecurity_rewrite.c */
ngx_int_t ngx_http_modsecurity_rewrite_handler(ngx_http_request_t *r);


#endif /* _NGX_HTTP_MODSECURITY_COMMON_H_INCLUDED_ */

ngx_http_modsecurity_module.c

/*
 * ModSecurity connector for nginx, http://www.modsecurity.org/
 * Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/)
 *
 * You may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * If any of the files related to licensing are missing or if you have any
 * other questions related to licensing please contact Trustwave Holdings, Inc.
 * directly using the email address security@modsecurity.org.
 *
 */

#ifndef MODSECURITY_DDEBUG
#define MODSECURITY_DDEBUG 0
#endif
#include "ddebug.h"

#include "ngx_http_modsecurity_common.h"
#include "stdio.h"
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>

static ngx_int_t ngx_http_modsecurity_init(ngx_conf_t *cf);
static void *ngx_http_modsecurity_create_main_conf(ngx_conf_t *cf);
static char *ngx_http_modsecurity_init_main_conf(ngx_conf_t *cf, void *conf);
static void *ngx_http_modsecurity_create_conf(ngx_conf_t *cf);
static char *ngx_http_modsecurity_merge_conf(ngx_conf_t *cf, void *parent, void *child);
static void ngx_http_modsecurity_cleanup_instance(void *data);
static void ngx_http_modsecurity_cleanup_rules(void *data);


/*
 * PCRE malloc/free workaround, based on
 * https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_pcrefix.c
 */

static void *(*old_pcre_malloc)(size_t);
static void (*old_pcre_free)(void *ptr);
static ngx_pool_t *ngx_http_modsec_pcre_pool = NULL;

static void *
ngx_http_modsec_pcre_malloc(size_t size)
{
    if (ngx_http_modsec_pcre_pool) {
        return ngx_palloc(ngx_http_modsec_pcre_pool, size);
    }

    fprintf(stderr, "error: modsec pcre malloc failed due to empty pcre pool");

    return NULL;
}

static void
ngx_http_modsec_pcre_free(void *ptr)
{
    if (ngx_http_modsec_pcre_pool) {
        ngx_pfree(ngx_http_modsec_pcre_pool, ptr);
        return;
    }

#if 0
    /* this may happen when called from cleanup handlers */
    fprintf(stderr, "error: modsec pcre free failed due to empty pcre pool");
#endif

    return;
}

ngx_pool_t *
ngx_http_modsecurity_pcre_malloc_init(ngx_pool_t *pool)
{
    ngx_pool_t  *old_pool;

    if (pcre_malloc != ngx_http_modsec_pcre_malloc) {
        ngx_http_modsec_pcre_pool = pool;

        old_pcre_malloc = pcre_malloc;
        old_pcre_free = pcre_free;

        pcre_malloc = ngx_http_modsec_pcre_malloc;
        pcre_free = ngx_http_modsec_pcre_free;

        return NULL;
    }

    old_pool = ngx_http_modsec_pcre_pool;
    ngx_http_modsec_pcre_pool = pool;

    return old_pool;
}

void
ngx_http_modsecurity_pcre_malloc_done(ngx_pool_t *old_pool)
{
    ngx_http_modsec_pcre_pool = old_pool;

    if (old_pool == NULL) {
        pcre_malloc = old_pcre_malloc;
        pcre_free = old_pcre_free;
    }
}

/*
 * ngx_string's are not null-terminated in common case, so we need to convert
 * them into null-terminated ones before passing to ModSecurity
 */
ngx_inline char *ngx_str_to_char(ngx_str_t a, ngx_pool_t *p)
{
    char *str = NULL;

    if (a.len == 0) {
        return NULL;
    }

    str = ngx_pnalloc(p, a.len+1);
    if (str == NULL) {
        dd("failed to allocate memory to convert space ngx_string to C string");
        /* We already returned NULL for an empty string, so return -1 here to indicate allocation error */
        return (char *)-1;
    }
    ngx_memcpy(str, a.data, a.len);
    str[a.len] = '\0';

    return str;
}


ngx_inline int
ngx_http_modsecurity_process_intervention (Transaction *transaction, ngx_http_request_t *r)
{
    char *log = NULL;
    ModSecurityIntervention intervention;
    intervention.status = 200;
    intervention.url = NULL;
    intervention.log = NULL;
    intervention.disruptive = 0;

    dd("processing intervention");

    if (msc_intervention(transaction, &intervention) == 0) {
        dd("nothing to do");
        return 0;
    }

    log = intervention.log;
    if (intervention.log == NULL) {
        log = "(no log message was specified)";
    }

    ngx_log_error(NGX_LOG_ERR, (ngx_log_t *)r->connection->log, 0, "%s", log);

    if (intervention.log != NULL) {
        free(intervention.log);
    }

    if (intervention.url != NULL)
    {
        dd("intervention -- redirecting to: %s with status code: %d", intervention.url, intervention.status);

        if (r->header_sent)
        {
            dd("Headers are already sent. Cannot perform the redirection at this point.");
            return -1;
        }

        /**
         * Not sure if it sane to do this indepent of the phase
         * but, here we go...
         *
         * This code cames from: http/ngx_http_special_response.c
         * function: ngx_http_send_error_page
         * src/http/ngx_http_core_module.c
         * From src/http/ngx_http_core_module.c (line 1910) i learnt
         * that location->hash should be set to 1.
         *
         */
        ngx_http_clear_location(r);
        ngx_str_t a = ngx_string("");

        a.data = (unsigned char *)intervention.url;
        a.len = strlen(intervention.url);

        ngx_table_elt_t *location = NULL;
        location = ngx_list_push(&r->headers_out.headers);
        ngx_str_set(&location->key, "Location");
        location->value = a;
        r->headers_out.location = location;
        r->headers_out.location->hash = 1;

#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
        ngx_http_modescurity_store_ctx_header(r, &location->key, &location->value);
#endif

        return intervention.status;
    }

    if (intervention.status != 200)
    {
        if (r->header_sent)
        {
            dd("Headers are already sent. Cannot perform the redirection at this point.");
            return -1;
        }
        dd("intervention -- returning code: %d", intervention.status);
        return intervention.status;
    }
    return 0;
}


void
ngx_http_modsecurity_cleanup(void *data)
{
    ngx_http_modsecurity_ctx_t *ctx;

    ctx = (ngx_http_modsecurity_ctx_t *) data;

    msc_transaction_cleanup(ctx->modsec_transaction);

#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    /*
     * Purge stored context headers.  Memory allocated for individual stored header
     * name/value pair will be freed automatically when r->pool is destroyed.
     */
    ngx_array_destroy(ctx->sanity_headers_out);
#endif
}


ngx_inline ngx_http_modsecurity_ctx_t *
ngx_http_modsecurity_create_ctx(ngx_http_request_t *r)
{
    ngx_str_t                          s;
    ngx_pool_cleanup_t                *cln;
    ngx_http_modsecurity_ctx_t        *ctx;
    ngx_http_modsecurity_conf_t       *mcf;
    ngx_http_modsecurity_main_conf_t  *mmcf;

    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_modsecurity_ctx_t));
    if (ctx == NULL)
    {
        dd("failed to allocate memory for the context.");
        return NULL;
    }

    mmcf = ngx_http_get_module_main_conf(r, ngx_http_modsecurity_module);
    mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module);

    dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec);

    if (mcf->transaction_id) {
        if (ngx_http_complex_value(r, mcf->transaction_id, &s) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
        ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, r->connection->log);

    } else {
        ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, r->connection->log);
    }

    dd("transaction created");

    ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module);

    cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t));
    if (cln == NULL)
    {
        dd("failed to create the ModSecurity context cleanup");
        return NGX_CONF_ERROR;
    }
    cln->handler = ngx_http_modsecurity_cleanup;
    cln->data = ctx;

#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t));
    if (ctx->sanity_headers_out == NULL) {
        return NGX_CONF_ERROR;
    }
#endif

    return ctx;
}


char *
ngx_conf_set_rules(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    int                                res;
    char                              *rules;
    ngx_str_t                         *value;
    const char                        *error;
    ngx_pool_t                        *old_pool;
    ngx_http_modsecurity_conf_t       *mcf = conf;
    ngx_http_modsecurity_main_conf_t  *mmcf;

    value = cf->args->elts;
    rules = ngx_str_to_char(value[1], cf->pool);

    if (rules == (char *)-1) {
        return NGX_CONF_ERROR;
    }

    old_pool = ngx_http_modsecurity_pcre_malloc_init(cf->pool);
    res = msc_rules_add(mcf->rules_set, rules, &error);
    ngx_http_modsecurity_pcre_malloc_done(old_pool);

    if (res < 0) {
        dd("Failed to load the rules: '%s' - reason: '%s'", rules, error);
        return strdup(error);
    }

    mmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_modsecurity_module);
    mmcf->rules_inline += res;

    return NGX_CONF_OK;
}


char *
ngx_conf_set_rules_file(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    int                                res;
    char                              *rules_set;
    ngx_str_t                         *value;
    const char                        *error;
    ngx_pool_t                        *old_pool;
    ngx_http_modsecurity_conf_t       *mcf = conf;
    ngx_http_modsecurity_main_conf_t  *mmcf;

    value = cf->args->elts;
    rules_set = ngx_str_to_char(value[1], cf->pool);

    if (rules_set == (char *)-1) {
        return NGX_CONF_ERROR;
    }

    old_pool = ngx_http_modsecurity_pcre_malloc_init(cf->pool);
    res = msc_rules_add_file(mcf->rules_set, rules_set, &error);
    ngx_http_modsecurity_pcre_malloc_done(old_pool);

    if (res < 0) {
        dd("Failed to load the rules from: '%s' - reason: '%s'", rules_set, error);
        return strdup(error);
    }

    mmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_modsecurity_module);
    mmcf->rules_file += res;

    return NGX_CONF_OK;
}


char *
ngx_conf_set_rules_remote(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    int                                res;
    ngx_str_t                         *value;
    const char                        *error;
    const char                        *rules_remote_key, *rules_remote_server;
    ngx_pool_t                        *old_pool;
    ngx_http_modsecurity_conf_t       *mcf = conf;
    ngx_http_modsecurity_main_conf_t  *mmcf;

    value = cf->args->elts;
    rules_remote_key = ngx_str_to_char(value[1], cf->pool);
    rules_remote_server = ngx_str_to_char(value[2], cf->pool);

    if (rules_remote_server == (char *)-1) {
        return NGX_CONF_ERROR;
    }

    if (rules_remote_key == (char *)-1) {
        return NGX_CONF_ERROR;
    }

    old_pool = ngx_http_modsecurity_pcre_malloc_init(cf->pool);
    res = msc_rules_add_remote(mcf->rules_set, rules_remote_key, rules_remote_server, &error);
    ngx_http_modsecurity_pcre_malloc_done(old_pool);

    if (res < 0) {
        dd("Failed to load the rules from: '%s'  - reason: '%s'", rules_remote_server, error);
        return strdup(error);
    }

    mmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_modsecurity_module);
    mmcf->rules_remote += res;

    return NGX_CONF_OK;
}


char *ngx_conf_set_transaction_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) {
    ngx_str_t                         *value;
    ngx_http_complex_value_t           cv;
    ngx_http_compile_complex_value_t   ccv;
    ngx_http_modsecurity_conf_t *mcf = conf;

    value = cf->args->elts;

    ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));

    ccv.cf = cf;
    ccv.value = &value[1];
    ccv.complex_value = &cv;
    ccv.zero = 1;

    if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    mcf->transaction_id = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t));
    if (mcf->transaction_id == NULL) {
        return NGX_CONF_ERROR;
    }

    *mcf->transaction_id = cv;

    return NGX_CONF_OK;
}


static ngx_command_t ngx_http_modsecurity_commands[] =  {
  {
    ngx_string("modsecurity"),
    NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_FLAG,
    ngx_conf_set_flag_slot,
    NGX_HTTP_LOC_CONF_OFFSET,
    offsetof(ngx_http_modsecurity_conf_t, enable),
    NULL
  },
  {
    ngx_string("modsecurity_rules"),
    NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_rules,
    NGX_HTTP_LOC_CONF_OFFSET,
    offsetof(ngx_http_modsecurity_conf_t, enable),
    NULL
  },
  {
    ngx_string("modsecurity_rules_file"),
    NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_rules_file,
    NGX_HTTP_LOC_CONF_OFFSET,
    offsetof(ngx_http_modsecurity_conf_t, enable),
    NULL
  },
  {
    ngx_string("modsecurity_rules_remote"),
    NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE2,
    ngx_conf_set_rules_remote,
    NGX_HTTP_LOC_CONF_OFFSET,
    offsetof(ngx_http_modsecurity_conf_t, enable),
    NULL
  },
  {
    ngx_string("modsecurity_transaction_id"),
    NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_1MORE,
    ngx_conf_set_transaction_id,
    NGX_HTTP_LOC_CONF_OFFSET,
    0,
    NULL
  },
  ngx_null_command
};


static ngx_http_module_t ngx_http_modsecurity_ctx = {
    NULL,                                  /* preconfiguration */
    ngx_http_modsecurity_init,             /* postconfiguration */

    ngx_http_modsecurity_create_main_conf, /* create main configuration */
    ngx_http_modsecurity_init_main_conf,   /* init main configuration */

    NULL,                                  /* create server configuration */
    NULL,                                  /* merge server configuration */

    ngx_http_modsecurity_create_conf,      /* create location configuration */
    ngx_http_modsecurity_merge_conf        /* merge location configuration */
};


ngx_module_t ngx_http_modsecurity_module = {
    NGX_MODULE_V1,
    &ngx_http_modsecurity_ctx,             /* module context */
    ngx_http_modsecurity_commands,         /* module directives */
    NGX_HTTP_MODULE,                       /* module type */
    NULL,                                  /* init master */
    NULL,                                  /* init module */
    NULL,                                  /* init process */
    NULL,                                  /* init thread */
    NULL,                                  /* exit thread */
    NULL,                                  /* exit process */
    NULL,                                  /* exit master */
    NGX_MODULE_V1_PADDING
};


static ngx_int_t
ngx_http_modsecurity_init(ngx_conf_t *cf)
{
    ngx_http_handler_pt *h_rewrite;
    ngx_http_handler_pt *h_preaccess;
    ngx_http_handler_pt *h_log;
    ngx_http_core_main_conf_t *cmcf;
    int rc = 0;

    cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
    if (cmcf == NULL)
    {
        dd("We are not sure how this returns, NGINX doesn't seem to think it will ever be null");
        return NGX_ERROR;
    }
    /**
     *
     * Seems like we cannot do this very same thing with
     * NGX_HTTP_FIND_CONFIG_PHASE. it does not seems to
     * be an array. Our next option is the REWRITE.
     *
     * TODO: check if we can hook prior to NGX_HTTP_REWRITE_PHASE phase.
     *
     */
    h_rewrite = ngx_array_push(&cmcf->phases[NGX_HTTP_REWRITE_PHASE].handlers);
    if (h_rewrite == NULL)
    {
        dd("Not able to create a new NGX_HTTP_REWRITE_PHASE handle");
        return NGX_ERROR;
    }
    *h_rewrite = ngx_http_modsecurity_rewrite_handler;

    /**
     *
     * Processing the request body on the preaccess phase.
     *
     * TODO: check if hook into separated phases is the best thing to do.
     *
     */
    h_preaccess = ngx_array_push(&cmcf->phases[NGX_HTTP_PREACCESS_PHASE].handlers);
    if (h_preaccess == NULL)
    {
        dd("Not able to create a new NGX_HTTP_PREACCESS_PHASE handle");
        return NGX_ERROR;
    }
    *h_preaccess = ngx_http_modsecurity_pre_access_handler;

    /**
     * Process the log phase.
     *
     * TODO: check if the log phase happens like it happens on Apache.
     *       check if last phase will not hold the request.
     *
     */
    h_log = ngx_array_push(&cmcf->phases[NGX_HTTP_LOG_PHASE].handlers);
    if (h_log == NULL)
    {
        dd("Not able to create a new NGX_HTTP_LOG_PHASE handle");
        return NGX_ERROR;
    }
    *h_log = ngx_http_modsecurity_log_handler;

    return NGX_OK;
}


static void *
ngx_http_modsecurity_create_main_conf(ngx_conf_t *cf)
{
    ngx_pool_cleanup_t                *cln;
    ngx_http_modsecurity_main_conf_t  *conf;

    conf = (ngx_http_modsecurity_main_conf_t *) ngx_pcalloc(cf->pool,
                                    sizeof(ngx_http_modsecurity_main_conf_t));

    if (conf == NULL)
    {
        return NGX_CONF_ERROR;
    }

    /*
     * set by ngx_pcalloc():
     *
     *     conf->modsec = NULL;
     *     conf->pool = NULL;
     *     conf->rules_inline = 0;
     *     conf->rules_file = 0;
     *     conf->rules_remote = 0;
     */

    cln = ngx_pool_cleanup_add(cf->pool, 0);
    if (cln == NULL) {
        return NGX_CONF_ERROR;
    }

    cln->handler = ngx_http_modsecurity_cleanup_instance;
    cln->data = conf;

    conf->pool = cf->pool;

    /* Create our ModSecurity instance */
    conf->modsec = msc_init();
    if (conf->modsec == NULL)
    {
        dd("failed to create the ModSecurity instance");
        return NGX_CONF_ERROR;
    }

    /* Provide our connector information to LibModSecurity */
    msc_set_connector_info(conf->modsec, MODSECURITY_NGINX_WHOAMI);
    msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);

    dd ("main conf created at: '%p', instance is: '%p'", conf, conf->modsec);

    return conf;
}


static char *
ngx_http_modsecurity_init_main_conf(ngx_conf_t *cf, void *conf)
{
    ngx_http_modsecurity_main_conf_t  *mmcf;
    mmcf = (ngx_http_modsecurity_main_conf_t *) conf;

    ngx_log_error(NGX_LOG_NOTICE, cf->log, 0,
                  "%s (rules loaded inline/local/remote: %ui/%ui/%ui)",
                  MODSECURITY_NGINX_WHOAMI, mmcf->rules_inline,
                  mmcf->rules_file, mmcf->rules_remote);

    return NGX_CONF_OK;
}


static void *
ngx_http_modsecurity_create_conf(ngx_conf_t *cf)
{
    ngx_pool_cleanup_t           *cln;
    ngx_http_modsecurity_conf_t  *conf;

    conf = (ngx_http_modsecurity_conf_t *) ngx_pcalloc(cf->pool,
                                         sizeof(ngx_http_modsecurity_conf_t));

    if (conf == NULL)
    {
        dd("Failed to allocate space for ModSecurity configuration");
        return NGX_CONF_ERROR;
    }

    /*
     * set by ngx_pcalloc():
     *
     *     conf->enable = 0;
     *     conf->sanity_checks_enabled = 0;
     *     conf->rules_set = NULL;
     *     conf->pool = NULL;
     *     conf->transaction_id = NULL;
     */

    conf->enable = NGX_CONF_UNSET;
    conf->rules_set = msc_create_rules_set();
    conf->pool = cf->pool;
    conf->transaction_id = NGX_CONF_UNSET_PTR;
#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    conf->sanity_checks_enabled = NGX_CONF_UNSET;
#endif

    cln = ngx_pool_cleanup_add(cf->pool, 0);
    if (cln == NULL) {
        dd("failed to create the ModSecurity configuration cleanup");
        return NGX_CONF_ERROR;
    }

    cln->handler = ngx_http_modsecurity_cleanup_rules;
    cln->data = conf;

    dd ("conf created at: '%p'", conf);

    return conf;
}


static char *
ngx_http_modsecurity_merge_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_modsecurity_conf_t *p = parent;
    ngx_http_modsecurity_conf_t *c = child;
#if defined(MODSECURITY_DDEBUG) && (MODSECURITY_DDEBUG)
    ngx_http_core_loc_conf_t *clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
#endif
    int rules;
    const char *error = NULL;

    dd("merging loc config [%s] - parent: '%p' child: '%p'",
        ngx_str_to_char(clcf->name, cf->pool), parent,
        child);

    dd("                  state - parent: '%d' child: '%d'",
        (int) c->enable, (int) p->enable);

    ngx_conf_merge_value(c->enable, p->enable, 0);
    ngx_conf_merge_ptr_value(c->transaction_id, p->transaction_id, NULL);
#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS)
    ngx_conf_merge_value(c->sanity_checks_enabled, p->sanity_checks_enabled, 0);
#endif

#if defined(MODSECURITY_DDEBUG) && (MODSECURITY_DDEBUG)
    dd("PARENT RULES");
    msc_rules_dump(p->rules_set);
    dd("CHILD RULES");
    msc_rules_dump(c->rules_set);
#endif
    rules = msc_rules_merge(c->rules_set, p->rules_set, &error);

    if (rules < 0) {
        return strdup(error);
    }

#if defined(MODSECURITY_DDEBUG) && (MODSECURITY_DDEBUG)
    dd("NEW CHILD RULES");
    msc_rules_dump(c->rules_set);
#endif
    return NGX_CONF_OK;
}


static void
ngx_http_modsecurity_cleanup_instance(void *data)
{
    ngx_pool_t                        *old_pool;
    ngx_http_modsecurity_main_conf_t  *mmcf;

    mmcf = (ngx_http_modsecurity_main_conf_t *) data;

    dd("deleting a main conf -- instance is: \"%p\"", mmcf->modsec);

    old_pool = ngx_http_modsecurity_pcre_malloc_init(mmcf->pool);
    msc_cleanup(mmcf->modsec);
    ngx_http_modsecurity_pcre_malloc_done(old_pool);
}


static void
ngx_http_modsecurity_cleanup_rules(void *data)
{
    ngx_pool_t                   *old_pool;
    ngx_http_modsecurity_conf_t  *mcf;

    mcf = (ngx_http_modsecurity_conf_t *) data;

    dd("deleting a loc conf -- RuleSet is: \"%p\"", mcf->rules_set);

    old_pool = ngx_http_modsecurity_pcre_malloc_init(mcf->pool);
    msc_rules_cleanup(mcf->rules_set);
    ngx_http_modsecurity_pcre_malloc_done(old_pool);
}


/* vi:set ft=c ts=4 sw=4 et fdm=marker: */

And in my build process I removed the filter reference c files:

### PERF notes(dropped in modified ngx_http_modsecurity_common.h file and ngx_http_modsecurity_module.c file to drop _header + _body filter references.
RUN rm /ModSecurity-nginx/src/ngx_http_modsecurity_body_filter.c
RUN rm /ModSecurity-nginx/src/ngx_http_modsecurity_header_filter.c
RUN rm /ModSecurity-nginx/config
COPY patches/ngx-connector/config /ModSecurity-nginx/config

Then we tested with a 20MB response body json payload(as this mostly should help on perf when dealing with large response/header data). Seemingly we saved about 20ms on average per request doing 11 TPS with the big response with preliminary data.

Rather than forcing folks to gut the code, some kinda flags we can set to achieve what I did above ^^ with the files may be good.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[jeremyjpj0916]

This should bot have been closed. Honestly the bot should not even be installed on any of the ModSec repos for how dead the community is around things currently.

[zimmerle]

@jeremyjpj0916 by adding the 'nostale' label the issues won't be stalled.

[zimmerle]

@jeremyjpj0916 It is very unusual to receive a contribution with pasted code along with the issue. We rather prefer to have it in the format of a pull request, even if that is only there for fomenting discussion. There is a guideline on how to do it here: Creating a pull request.

About the inspection in different phases, ModScerutiy has a mechanism to control the flow of phases execution, including the allow action that can skip phases. Specifically for request body and response body, there are some configuration options that can be used:

SecRequestBodyAccess
SecRequestBodyInMemoryLimit
SecRequestBodyLimit
SecRequestBodyNoFilesLimit
SecRequestBodyLimitAction
SecResponseBodyLimit
SecResponseBodyLimitAction
SecResponseBodyMimeType
SecResponseBodyMimeTypesClear
SecResponseBodyAccess
SecResponseBodyLimit
SecResponseBodyLimitAction
SecResponseBodyMimeType
SecResponseBodyMimeTypesClear
SecResponseBodyAccess

Those may not be fully integrated within the connector yet, some are still a work in progress. Having said that, what is the benefit of what you have suggested atop of the existent features? And how exactly it is addressing the possibility of having a rule in a phase that is not being executed?

[jeremyjpj0916]

what i did was a hack, im suggesting those competent with this lib make proper flags in the code to achieve it(I am not competent for those changes). And yes those features mentioned are not integrated within the connector yet.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[jeremyjpj0916]

Bad bot.