bidict.ValueDuplicationError

[gil0109]

I am not sure why this is occurring and there are a large number of these exceptions and I was hoping you can help. I am waiting to upgrade to the latest version of Flask-SocketIO to see if it resolves it but we need to either wait for a fix you have to be implemented or change our room to text instead of int.

Using Flask-SocketIO, Redis, Gunicorn, Eventlet

Logs
I can try to get more detailed logs if needed..

[2021-06-10 01:02:48,926] ERROR (engineio.server) <server.py>._trigger_event: message handler error
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.8/site-packages/engineio/server.py", line 596, in _trigger_event
return self.handlersevent
File "/opt/app-root/lib64/python3.8/site-packages/socketio/server.py", line 740, in _handle_eio_message
self._handle_connect(eio_sid, pkt.namespace, pkt.data)
File "/opt/app-root/lib64/python3.8/site-packages/socketio/server.py", line 625, in _handle_connect
sid = self.manager.connect(eio_sid, namespace)
File "/opt/app-root/lib64/python3.8/site-packages/socketio/base_manager.py", line 47, in connect
self.enter_room(sid, namespace, None, eio_sid=eio_sid)
File "/opt/app-root/lib64/python3.8/site-packages/socketio/base_manager.py", line 112, in enter_room
self.rooms[namespace][room][sid] = eio_sid
File "/opt/app-root/lib64/python3.8/site-packages/bidict/_mut.py", line 78, in setitem
self._put(key, val, self.on_dup)
File "/opt/app-root/lib64/python3.8/site-packages/bidict/_base.py", line 209, in _put
dedup_result = self._dedup_item(key, val, on_dup)
File "/opt/app-root/lib64/python3.8/site-packages/bidict/_base.py", line 258, in _dedup_item
raise ValueDuplicationError(val)
bidict.ValueDuplicationError: WwsK4Dk147yY9y-TAATu

Any help would be appreciated..

[miguelgrinberg]

What client are you using? This error would appear when a connection request is received for a namespace that is already connected, at least that's the only reason I can think of. If you can provide a small client and server that I can use to reproduce the error I can investigate more.

[gil0109]

The client is VueJS frontend. socket.io-client": "4.0.0"
The backend is python using: Flask-SocketIO==5.0.2

We are running in Openshift which uses HA proxy to manage load balancing. With multiple frontend containers with NGINX and python containers using redis to sync the python containers together.

Our repo is: https://github.com/bcgov/queue-management

I will see if I can provide a small client/server - not sure how to do this.

[miguelgrinberg]

You may want to enable detailed logs for Socket.IO, maybe that will give us more clues. See the troubleshooting section of the docs if you don't know how to do this.

[gil0109]

@miguelgrinberg thank.. I have attached a log that has some more detailed logs. I still don't understand it but I believe that this error only happens when we have more than one instance of the python that sync use Redis (or RabbitMQ) to sync with each other. Yes, I tried both.

In the logs, please search: bidict.ValueDuplicationError: koOnrpN0-Cf29cF9AAAC

Any help is appreciated, thank you..
log.txt

[miguelgrinberg]

Do you have sticky sessions on your load balancer? That is required if you use long-polling.

[gil0109]

Yes, Openshift builds their routes by default with sticky sessions via cookies.

[miguelgrinberg]

Okay, so this is what is strange:

koOnrpN0-Cf29cF9AAAC: Received packet MESSAGE data 0
[2021-06-12 16:18:19,234] INFO     (engineio.server) <socket.py>.receive: koOnrpN0-Cf29cF9AAAC: Received packet MESSAGE data 0
koOnrpN0-Cf29cF9AAAC: Sending packet MESSAGE data 0{"sid":"e0W1DhgqAnVmD71CAAAD"}
[2021-06-12 16:18:19,234] INFO     (engineio.server) <socket.py>.send: koOnrpN0-Cf29cF9AAAC: Sending packet MESSAGE data 0{"sid":"e0W1DhgqAnVmD71CAAAD"}
koOnrpN0-Cf29cF9AAAC: Received packet MESSAGE data 0
[2021-06-12 16:18:19,310] INFO     (engineio.server) <socket.py>.receive: koOnrpN0-Cf29cF9AAAC: Received packet MESSAGE data 0
message handler error

The 0 packet is sent by the client when it asks to join a namespace. Here you can see that a request for the main namespace was made, and the server responded with another 0 package that includes a new sid. But a moment later, the same client asks to join the namespace again, by sending the 0 packet again. This is exactly what I described in my first response to you. Any idea why the client will connect twice to the same namespace?

[gil0109]

I am not sure but I will get a couple of dev's to see if they can find the code could be doing that. Thank you.

[gil0109]

My problem is that is not consistent. It only happens sometimes so it's hard to trace back as I can't replicate this in localhost. Maybe network issue??

[miguelgrinberg]

In any case, the server should not crash because of this, so I'll add code to handle this as a bad request. The client will receive an ERROR packet in response.

[gil0109]

Thank you. I will try to trace it down.

On another topic, you mentioned I was using long-polling. I thought I was setup for web-sockets and not long-polling. Where do I set that? Do I need to move to gevent for that?

https://github.com/bcgov/queue-management/blob/a44ebfa617d5fd49556a829b940eacd9e807c3c2/frontend/src/views/Socket.vue#L55

https://github.com/bcgov/queue-management/blob/a44ebfa617d5fd49556a829b940eacd9e807c3c2/frontend/src/components/smartboard/board-socket.vue#L36

Thanks

[miguelgrinberg]

That was a generic comment. Looking at your logs it does look like your clients are connecting directly over WebSocket. My comment comes from the fact that by default Socket.IO clients initiate the connection over long-polling, and then upgrade to WebSocket.

[gil0109]

I am still trying to track down the cause of the issue. I am unsure if it is unstable Wifi for our digital signage but that is what I am suspecting at this point. I will also look at how it calls the websocket connections.

[miguelgrinberg]

A fix that prevents this crash is now committed: miguelgrinberg/python-socketio@cb1b8ec. With this fix, the client will receive an ERROR packet if it attempts to connect to an already connected namespace.

[gil0109]

Thank you. I am still tracking down the issue on our side. My best guess is our intrusion protection system is doing something weird..

[gil0109]

@miguelgrinberg Out of curiousity, when do you think it will make it into pypi as a new version release?

[miguelgrinberg]

I don't have an exact date, I have a couple more things to do before the next release.