Flask-socketio multi workers with nginx

[aktoak]

hello

i followed exactly the official documentation on how to set up flask-socketio multi worker with nginx:

1. i monkey_patched eventlet
2. added the message queue to SocketIO as following
   socketio.init_app(app, async_mode='eventlet', manage_session=False, message_queue='redis://')
3. in nginx i added the upstream as following with my own server instances

upstream socketio_nodes {
    ip_hash;

    server 127.0.0.1:5000;
    server 127.0.0.1:5001;
    server 127.0.0.1:5002;
    # to scale the app, just add more nodes here!
}

and updated the /socketio location as specified

location /socket.io {
        include proxy_params;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://socketio_nodes/socket.io;
    }

my website is working fine but when i monitor flask-socket using redis-cli monitor i keep constantly getting this publish disconnect message from flask-socketio

and when i check how many subscribers to the flask-socketio channel using redis-cli PUBSUB NUMSUB flask-socketio
which i expect it to be equal to the number of workers
i keep getting 1

what am i missing here?
why flask-socket keeps constantly publishing messages
and why i can't see all the subscribers.

Thanks

[miguelgrinberg]

Flask-SocketIO does not publish anything to the message queue. Those disconnect messages must be coming from your application, which must be calling the disconnect method.

I don't have an answer for you regarding the number of subscribers. I agree that it should match the number of workers that are actively holding clients. With the information that you provided I can't explain why you see only one.

[aktoak]

Thanks for the quick answer.
regarding the flooding of disconnect event it was a problem with python-socketio found the solution here python-socketio #441
so upgrading the package solved the problem.

regarding the number of subscribers i still get 1 all the time.
does workers subscribe to flask-socketio channel if they are not yet holding any clients?

my nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
<------># worker_connections 768;
<------>worker_connections 5000;
<------># multi_accept on;
}

http {

<------>##
<------># Basic Settings
<------>##

<------>sendfile on;
<------>tcp_nopush on;
<------>tcp_nodelay on;
<------>keepalive_timeout 65;
<------>types_hash_max_size 2048;
<------># server_tokens off;

<------># server_names_hash_bucket_size 64;
<------># server_name_in_redirect off;

<------>include /etc/nginx/mime.types;
<------>default_type application/octet-stream;

<------>##
<------># SSL Settings
<------>##

<------>ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
<------>ssl_prefer_server_ciphers on;

<------>##
<------># Logging Settings
<------>##

<------>access_log /var/log/nginx/access.log;
<------>error_log /var/log/nginx/error.log;

<------>##
<------># Gzip Settings
<------>##

<------>gzip on;

<------># gzip_vary on;
<------># gzip_proxied any;
<------># gzip_comp_level 6;
<------># gzip_buffers 16 8k;
<------># gzip_http_version 1.1;
<------># gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

<------>##
<------># Virtual Host Configs
<------>##

<------>include /etc/nginx/conf.d/*.conf;
<------>include /etc/nginx/sites-enabled/*;
}


#mail {
#<-----># See sample authentication script at:
#<-----># http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#.
#<-----># auth_http localhost/auth.php;
#<-----># pop3_capabilities "TOP" "USER";
#<-----># imap_capabilities "IMAP4rev1" "UIDPLUS";
#.
#<----->server {
#<-----><------>listen     localhost:110;
#<-----><------>protocol   pop3;
#<-----><------>proxy      on;
#<----->}
#.
#<----->server {
#<-----><------>listen     localhost:143;
#<-----><------>protocol   imap;
#<-----><------>proxy      on;
#<----->}
#}

my nginx balance-loader settings:

##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration

# upstream (stick session with ip_hash to allow multipe instance workers)
upstream socketio_nodes {
<------>ip_hash;

<------>server unix:/run/instance1.socket;
<------>server unix:/run/instance2.socket;
<------>server unix:/run/instance3.socket;
<------># to scale the pp, just add more nodes here!
}

# redirect http to https
server {
<------>listen 80;
<------>listen [::]:80;
<------>server_name www.mysite.com
<------>return 301 https://mysite.com$request_uri;
}

# redirect https with www to https without www
server {
<------>listen 443 ssl;
<------>listen [::]:443 ssl;
<------>server_name www.mysite.com;
<------>ssl_certificate fullchain.pem;
<------>ssl_certificate_key privkey.pem;
<------>return 301 https://mysite$request_uri;
}

# listen to actuall https calls and handling location/
server {
<------>listen 443 ssl;
<------>listen [::]:443 ssl;
<------>#listen 80;
<------>#listen [::]:80;
<------>ssl_certificate fullchain.pem;
<------>ssl_certificate_key privkey.pem;
<------>ssl_stapling on;

<------># SSL configuration
<------>#
<------># listen 443 ssl default_server ;
<------># listen [::]:80 ssl default_server;
<------>#
<------># Note: You should disable gzip for SSL traffic.
<------># See: https://bugs.debian.org/773332
<------>#
<------># Read up on ssl_ciphers to ensure a secure configuration.
<------># See: https://bugs.debian.org/765782
<------>#
<------># Self signed certs generated by the ssl-cert package
<------># Don't use them in a production server!
<------>#
<------># include snippets/snakeoil.conf;

<------>root /var/www/mysite.com;

<------># Add index.php to the list if you are using PHP
<------>index index.html index.htm index.nginx-debian.html;

<------>server_name mysite.com;
<------># if ($scheme != https) {
<------>#    return 301 https://mysite.com$request_uri;
<------># }

<------>error_page 500 501 502 503 504 /maintenance.html;
<------>error_page 429 /too-many-requests.html;

<------>location = /maintenance.html {
<------><------>index maintenance.html;
<------><------>internal;
<------>}

<------>location = /too-many-requests.html {
<------><------>index too-many-requests.html;
<------><------>internal;
<------>}

<------>location /login {
<------><------>include /etc/nginx/proxy_params;
<------><------>proxy_pass http://unix:/run/instance1.socket;
<------><------>try_files $uri $uri/ /index.html =503; # return 503 if index.html not found
<------>}

<------>location /socket.io {
<------><------>include /etc/nginx/proxy_params;
<------><------>proxy_pass http://socketio_nodes/socket.io;
<------><------>proxy_http_version 1.1;
<------><------>proxy_set_header Upgrade $http_upgrade;
<------><------>proxy_set_header Connection 'upgrade';
<------><------>proxy_buffering off;
<------>}
}

[miguelgrinberg]

I think you are looking at this in the wrong way. You mentioned above that every works. Is this still correct?

So you have those 3 workers, and some clients, each randomly connected to one of them. What happens if you emit with the broadcast=True option? If all your clients get the event, that means that your workers are receiving messages from the pub/sub queue. If they weren't connected they would not receive the messages, and that would cause some of your connected clients to miss events. If none of your clients are missing events, then it is clear that all your workers are connected to the queue, so your observation that there is only one connected worker has to be wrong.

[aktoak]

so turns out the problem was with my webserver instances, they weren't working probably.
problem solved.
and now i can see 3 subscribers for flask-socketio channel

Thanks for your insights it made me look at the problem from a different aspect.

[hemmokarja]

This is undoubtedly not the right forum to ask this, but I'll do it anyway since I'm facing the same issue with seeing only a single sub with multiple workers and can't seem to debug the issue. What was wrong with your webserver instances and how did you fix it?