Emitting messages consumed from external Kafka topic #1661
Unanswered
anthnyprschka
asked this question in
Q&A
Replies: 1 comment 1 reply
-
The Kafka support was written by a contributor. I don't use it myself, so I unfortunately don't have any examples share. If you want an example that uses Redis, you can look at my microflack application: https://github.com/miguelgrinberg/microflack_admin. This is a fairly complex application that I demonstrated at a PyCon class on microservices. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi there,
I've been digging for a while and would appreciate some input.
I have a Flask-SocketIO server making websocket connections with clients.
Now I want to consume messages from a Kafka topic and emit them to the clients.
I've read about "Emitting from External Process" (https://flask-socketio.readthedocs.io/en/latest/deployment.html#emitting-from-an-external-process) and also this question (#1595) that suggests to register an HTTP endpoint to which the Kafka consumer can post which in turn emits to the clients.
I experimented with background tasks but for some reason these don't work in my setup (running the server inside gunicorn with worker class geventwebsocket.gunicorn.workers.GeventWebSocketWorker. No idea why. I tried start_background_task, threading, gevent.spawn and multiprocessing.Process but none seem to work.
So the simplest solution might just be adding another HTTP endpoint to the flask app that emits to the clients, create another script that consumes the Kafka topic and posts these to the flask app.
Ideally i could just start a background task in the same process with the same SocketIO instance.
Or i create an External Process connected via Kafka (or any other message queue but Kafka makes sense because I already use it) that consumes my Kafka topic and emits its messages. Are there any examples available for this use case?
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions