Emitting messages consumed from external Kafka topic

[anthnyprschka]

Hi there,

I've been digging for a while and would appreciate some input.

I have a Flask-SocketIO server making websocket connections with clients.

Now I want to consume messages from a Kafka topic and emit them to the clients.

I've read about "Emitting from External Process" (https://flask-socketio.readthedocs.io/en/latest/deployment.html#emitting-from-an-external-process) and also this question (#1595) that suggests to register an HTTP endpoint to which the Kafka consumer can post which in turn emits to the clients.

I experimented with background tasks but for some reason these don't work in my setup (running the server inside gunicorn with worker class geventwebsocket.gunicorn.workers.GeventWebSocketWorker. No idea why. I tried start_background_task, threading, gevent.spawn and multiprocessing.Process but none seem to work.

So the simplest solution might just be adding another HTTP endpoint to the flask app that emits to the clients, create another script that consumes the Kafka topic and posts these to the flask app.

Ideally i could just start a background task in the same process with the same SocketIO instance.

Or i create an External Process connected via Kafka (or any other message queue but Kafka makes sense because I already use it) that consumes my Kafka topic and emits its messages. Are there any examples available for this use case?

Thanks!

[miguelgrinberg]

The Kafka support was written by a contributor. I don't use it myself, so I unfortunately don't have any examples share.

If you want an example that uses Redis, you can look at my microflack application: https://github.com/miguelgrinberg/microflack_admin. This is a fairly complex application that I demonstrated at a PyCon class on microservices.

[anthnyprschka]

Thanks @miguelgrinberg for the link, resorted to a workaround but will get back to this 👍