[INSTALLATION] Nginx Proxy Manager

[maxdd]

Did you look through existing ISSUES ?

Yes, Nginx Proxy Manager docker is not mentioned

Describe the problem you are experiencing

I would like to setup an nginx proxy manager docker + nginx bad bot blocker
The current folder layout (not decided by me) is as follow:

# default from the docker container
/etc/nginx/nginx.conf
/etc/nginx/fastcgi.conf
/etc/nginx/conf.d/default.conf
/etc/nginx/conf.d/include/block-exploits.conf
/etc/nginx/conf.d/include/proxy.conf
/etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
/etc/nginx/conf.d/include/assets.conf
/etc/nginx/conf.d/include/ip_ranges.conf
/etc/nginx/conf.d/include/ssl-ciphers.conf
/etc/nginx/conf.d/include/force-ssl.conf
/etc/nginx/conf.d/include/resolvers.conf
/etc/nginx/conf.d/production.conf
/data/nginx/default_host/site.conf

# once you create via the web-ui panel the first proxy host
/data/nginx/proxy_host/1.conf

and the wiki is saying one can also use the following

You can add your custom configuration snippet files at /data/nginx/custom as follow:

/data/nginx/custom/root.conf: Included at the very end of nginx.conf
/data/nginx/custom/http_top.conf: Included at the top of the main http block
/data/nginx/custom/http.conf: Included at the end of the main http block
/data/nginx/custom/events.conf: Included at the end of the events block
/data/nginx/custom/stream.conf: Included at the end of the main stream block
/data/nginx/custom/server_proxy.conf: Included at the end of every proxy server block
/data/nginx/custom/server_redirect.conf: Included at the end of every redirection server block
/data/nginx/custom/server_stream.conf: Included at the end of every stream server block
/data/nginx/custom/server_stream_tcp.conf: Included at the end of every TCP stream server block
/data/nginx/custom/server_stream_udp.conf: Included at the end of every UDP stream server block

The download went fine

./install-ngxblocker -x
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Creating directory: /etc/nginx/bots.d

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /etc/nginx/conf.d/globalblacklist.conf...OK
Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /etc/nginx/conf.d/botblocker-nginx-settings.conf...OK

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /etc/nginx/bots.d/blockbots.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /etc/nginx/bots.d/ddos.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /etc/nginx/bots.d/custom-bad-referrers.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /etc/nginx/bots.d/bad-referrer-words.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /etc/nginx/bots.d/blacklist-ips.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /etc/nginx/bots.d/blacklist-user-agents.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /etc/nginx/bots.d/whitelist-domains.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /etc/nginx/bots.d/whitelist-ips.conf...OK

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/setup-ngxblocker      [TO]=>  /usr/local/sbin/setup-ngxblocker...OK
Downloading [FROM]=>  [REPO]/update-ngxblocker     [TO]=>  /usr/local/sbin/update-ngxblocker...OK
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker

and i was expecting to launch the setup on /data/nginx/proxy_host/1.conf which is my proxy conf

./setup-ngxblocker -v /data/nginx/proxy_host/ -e conf

but this is what i get

./setup-ngxblocker -v /data/nginx/proxy_host/ -e conf
WARN: ./setup-ngxblocker optionally requires: 'dig' => cannot whitelist public ip address.
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

** Dry Run ** | not updating files | run  as 'setup-ngxblocker -x' to setup files.

INFO:      /etc/nginx/conf.d/* detected               => /etc/nginx/nginx.conf
setup will fix conflict from: 'server_names_hash_bucket_size' in /etc/nginx/conf.d/botblocker-nginx-settings.conf

Checking for missing includes:

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /etc/nginx/conf.d
Nothing to update for directory: /etc/nginx/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker

so other than my current value for server_names_hash_bucket_size which is 1024 i would have expected this

include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;

to be mentioned for /data/nginx/proxy_host/1.conf but that's not the case.

This is the nginx.conf

# run nginx in foreground
daemon off;
pid /run/nginx/nginx.pid;
user npm;

# Set number of worker processes automatically based on number of CPU cores.
worker_processes auto;

# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;

error_log /data/logs/fallback_error.log warn;

# Includes files with directives to load dynamic modules.
include /etc/nginx/modules/*.conf;

events {
        include /data/nginx/custom/events[.]conf;
}

http {
        include                       /etc/nginx/mime.types;
        default_type                  application/octet-stream;
        sendfile                      on;
        server_tokens                 off;
        tcp_nopush                    on;
        tcp_nodelay                   on;
        client_body_temp_path         /tmp/nginx/body 1 2;
        keepalive_timeout             90s;
        proxy_connect_timeout         90s;
        proxy_send_timeout            90s;
        proxy_read_timeout            90s;
        ssl_prefer_server_ciphers     on;
        gzip                          on;
        proxy_ignore_client_abort     off;
        client_max_body_size          2000m;
        server_names_hash_bucket_size 1024;
        proxy_http_version            1.1;
        proxy_set_header              X-Forwarded-Scheme $scheme;
        proxy_set_header              X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header              Accept-Encoding "";
        proxy_cache                   off;
        proxy_cache_path              /var/lib/nginx/cache/public  levels=1:2 keys_zone=public-cache:30m max_size=192m;
        proxy_cache_path              /var/lib/nginx/cache/private levels=1:2 keys_zone=private-cache:5m max_size=1024m;

        log_format proxy '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';
        log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

        access_log /data/logs/fallback_access.log proxy;

        # Dynamically generated resolvers file
        include /etc/nginx/conf.d/include/resolvers.conf;

        # Default upstream scheme
        map $host $forward_scheme {
                default http;
        }

        # Real IP Determination

        # Local subnets:
        set_real_ip_from 10.0.0.0/8;
        set_real_ip_from 172.16.0.0/12; # Includes Docker subnet
        set_real_ip_from 192.168.0.0/16;
        # NPM generated CDN ip ranges:
        include conf.d/include/ip_ranges.conf;
        # always put the following 2 lines after ip subnets:
        real_ip_header X-Real-IP;
        real_ip_recursive on;

        # Custom
        include /data/nginx/custom/http_top[.]conf;

        # Files generated by NPM
        include /etc/nginx/conf.d/*.conf;
        include /data/nginx/default_host/*.conf;
        include /data/nginx/proxy_host/*.conf;
        include /data/nginx/redirection_host/*.conf;
        include /data/nginx/dead_host/*.conf;
        include /data/nginx/temp/*.conf;

        # Custom
        include /data/nginx/custom/http[.]conf;
}

stream {
        # Files generated by NPM
        include /data/nginx/stream/*.conf;

        # Custom
        include /data/nginx/custom/stream[.]conf;
}

# Custom
include /data/nginx/custom/root[.]conf;

which apparently already contains include /etc/nginx/conf.d/*.conf;

[maxdd]

For sake of completness after adding manually in 1.conf the following and rebooting the container

server {
...
include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;
...
}

this is what i receive

nginx: [warn] duplicate network "138.199.57.151", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18889
nginx: [warn] duplicate network "143.244.38.129", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18905
nginx: [warn] duplicate network "195.181.163.194", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19000
nginx: [warn] duplicate network "5.188.120.15", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19127
nginx: [warn] duplicate network "89.187.173.66", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19174

as well as

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

altough i'm wondering whether this is a messy approach

[accessiblepixel]

I'm looking to integrate the two as well, and it appears (according to NPM's website) that you could just add the include to /data/nginx/custom/http_top.conf and that would include it before any proxied sites as the easiest way to enable it for all of them.

Since in NPM it loads that file just before, eg

        # Custom
        include /data/nginx/custom/http_top[.]conf;

        # Files generated by NPM
        include /etc/nginx/conf.d/*.conf;
        include /data/nginx/default_host/*.conf;
        include /data/nginx/proxy_host/*.conf;

and then it would only be included once?

Edit: Hmm... No. It needs to be in a server block, and it doesn't look like there's a way to add it early in the server block using the include files of NPM, only at the end
/data/nginx/custom/server_proxy.conf: Included at the end of every proxy server block.
Dang.

Would be curious if you found a better way to integrate it though.

[maxdd]

Have you tried this?

/data/nginx/custom/server_proxy.conf: Included at the end of every proxy server block

I honestly don't know whether it is better to have it at the top or at the bottom.
Are the same value overwritten by the latest?

The way i did was simply to add from the npm web ui in the proxy host advanced tab these two after running the install script

include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;

of course this will be valid only for the specific proxy and not widely available to every created proxy.
The reason i'm using the web-ui is because every now and then npm overwrite the proxy.conf file so manual "edit" is a no go.
I also did not look into the "whitelist" feature since it requires a package which npm docker does not have.

[arhyneRWU]

I have no idea if this is helpful/useful, but here is what I did.

I built a docker container for this repo that runs git pull updates every 12 hours, which links to a persistent directory. I then feed those directories into my proxy docker. I provided the included paths to the bots.d files I'm using and symlinked the conf.d files. This allows me to use the conf.d files, not the test files.

This method will allow me to keep my own white list and blacklists and not have those over written.

  botblocker:
    volumes:
      - ./botblocker/files:/bot-blocker

  proxy:
    volumes:
      - ./certbot/www:/var/www/certbot/:ro
      - ./certbot/conf/:/etc/letsencrypt/:ro
      - ./botblocker/files/bots.d:/etc/nginx/bots.d:ro
      - ./botblocker/files/deny.d:/etc/nginx/deny.d:ro
      - ./botblocker/files/conf.d:/etc/nginx/bots.conf.d:ro. # You have to create this dir separately to make the symlink work. 
      - ./files/nginx/conf:/etc/nginx/conf.d:ro

files/nginx/conf#

botblocker-nginx-settings.conf -> /etc/nginx/bots.conf.d/botblocker-nginx-settings.conf
globalblacklist.conf -> /etc/nginx/bots.conf.d/globalblacklist.conf
domain.conf

website.conf

server {
    listen 80;
    server_name domain.com;

    # Include bot blocker configuration
    include /etc/nginx/bots.d/blockbots.conf;
    include /etc/nginx/bots.d/ddos.conf;
    include /etc/nginx/deny.d/deny.conf;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot/;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name domain.com;

    # Include bot blocker configuration
    include /etc/nginx/bots.d/blockbots.conf;
    include /etc/nginx/bots.d/ddos.conf;
    include /etc/nginx/deny.d/deny.conf;

    ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass http://service-name:port
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

bots banging on it minutes after I restarted the services, deny.conf works nicely.

57.129.23.166 - - [31/Dec/2023:22:27:23 -0500] "GET /.env HTTP/1.1" 444 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"

[maxdd]

I'm getting s6 overlay error when using

- ./files/nginx/conf:/etc/nginx/conf.d:ro

the error is

chown: changing ownership of '/etc/nginx/conf.d': Read-only file system
s6-rc: warning: unable to start service prepare: command exited 1
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

and even by removing "ro" i get

/etc/s6-overlay/s6-rc.d/prepare/40-dynamic.sh: line 16: /etc/nginx/conf.d/include/resolvers.conf: No such file or directory
s6-rc: warning: unable to start service prepare: command exited 1
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

[arhyneRWU]

You're mounting files to your conf.d. Do you have a resolver.config in your files directory? I

./files/nginx/conf/
├── ...
└── include/
    └── resolvers.conf

[maxdd]

No i don't, normally npm adds it automatically as 127.0.0.11 in the file. Shall i create it?

[arhyneRWU]

yes, you'll have to create the directory and the resolver in that dir. permission should be the same.

like this?
resolver 127.0.0.11 valid=30s;

[maxdd]

Yeah but then also for letsencrypt-acme-challenge.conf file would be the same

[accessiblepixel]

Alright, I think I worked out a way to set it up correctly :D

On my docker host that has nginxproxymanager installed with docker-compose, I made a folder to hold everything, say /root/loadbalancer

I made a new directory called botblocker under this, and a conf.d and a bots.d

Then I got the installer from the repo with

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker
sudo chmod +x /usr/local/sbin/install-ngxblocker
/usr/local/sbin/install-ngxblocker -b /root/loadbalancer/botblocker/bots.d/ -c /root/loadbalancer/botblocker/conf.d/

And setup my docker-compose.yml like this

version: '3.8'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - /root/loadbalancer/botblocker/conf.d/botblocker-nginx-settings.conf:/etc/nginx/conf.d/botblocker-nginx-setting
s.conf
      - /root/loadbalancer/botblocker/conf.d/globalblacklist.conf:/etc/nginx/conf.d/globalblacklist.conf
      - ./botblocker/bots.d:/etc/nginx/bots.d

And then in my data/nginx/custom/server_proxy.conf I added

include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/bots.d/blockbots.conf;

And it all seems to work as intended :D

Might have to restart the container when doing a rule update with the ultimate-bad-block updater scripts, but think I finally solved it for me... Hopefully this will help anyone else - Thanks to everyone else in the thread posting their ideas which helped me find mine, much appreciated! :>

Edit: Oh, and yeah you have to comment out #server_names_hash_bucket_size 256; in botblocker-nginx-settings.conf since it's already defined somewhere else in nginxproxymanager.