This project demonstrates a simple real-time streaming pipeline using Apache Kafka and Python. It generates random sentences using the wonderwords
library, publishes them to Kafka, and then consumes them for real-time sentiment analysis. The pipeline is containerized using Docker for easier setup and deployment.
.gitignore
: Specifies files and directories to be ignored by Git.Readme.md
: Documentation of the project.analytics.py
: Consumes messages from Kafka and performs sentiment analysis.docker-compose.yml
: Configuration for running Kafka, Zookeeper, and other services using Docker.requirements.txt
: List of Python dependencies required for the project.sentences_producer.py
: Produces random sentences and sends them to Kafka.sentiment_analysis.py
: Contains functions or classes for sentiment analysis.
- Docker and Docker Compose
- Python 3.x
- Kafka
-
Clone the repository:
git clone <repository-url> cd stream-analytics-main
-
Install Python dependencies:
pip install -r requirements.txt
-
Start Docker services:
docker-compose up
-
Run the Kafka producer to generate and send random sentences:
python sentences_producer.py
-
Run the Kafka consumer to consume messages and perform sentiment analysis:
python analytics.py
- sentences_producer.py: Generates random sentences and sends them to a Kafka topic named
sentences
. - analytics.py: Consumes messages from the
sentences
topic in Kafka and performs sentiment analysis using functions defined insentiment_analysis.py
. - sentiment_analysis.py: Contains the logic for analyzing the sentiment of given sentences.
-
Producer Output:
Produced message on topic sentences with value of {"sentence": "The placid prayer changes carry."}
-
Consumer Output:
{'text': 'The placid prayer changes carry.', 'sentiment': 'negative'}
📸 The left panel shows the Kafka producer streaming random sentences. The right panel shows the Kafka consumer performing real-time sentiment analysis on each sentence.
- Wonderwords Library: Quickstart Guide — used in this project to generate varied random sentences for stream testing
- Apache Kafka Documentation: https://kafka.apache.org/