MLP2. [Streaming] Data Engineering End-to-End Project - Spark, Kafka, Airflow, Cassandra, Python

[nnthanh101]

MLP2. [Streaming] Data Engineering End-to-End Project - Spark, Kafka, Airflow, Cassandra, Python

1. Tech Stack

• Apache Cassandra
• Python & API
• Apache Airflow
• Apache Kafka
• Apache Spark

2. 🚀 Deliverables:

• Reproducible Local Development with Dev Containers
• Architecture Diagram
• [Streaming] Data Engineering End-to-End Project - Spark, Kafka, Airflow, Cassandra, Python

References:

• https://medium.com/@dogukannulu/data-engineering-end-to-end-project-1-7a7be2a3671
• https://github.com/dogukannulu/kafka_spark_structured_streaming