My Second DBT (Data Build Tool) Project

Based on the course: End-to-End Data Engineering Project,
by LinkedIn Learning.

Note: because we added invalid FKs, this repo has an altered dump of the original dataset for education purposes only. The original data and repo's licence can be found at: https://github.com/LinkedInLearning/end-to-end-data-engineering-project-4413618

Architecture

The current implementation follows a classic star schema–based data warehouse architecture. We may consider migrating to a Medallion-style approach in future iterations.

Setup

Prerequisites

• Python Poetry
• GNU Make
• Podman or Docker
• (Optional) Any SQL client of your choice

Environment and dependencies

make env-setup

Database and data

make db-run db-reset 

Note: The db-reset command will drop and recreate the database, so use it with caution.

Test dbt-DB connection

make dbt-debug

Note: If you encounter connection issues, ensure that the database container is running and that your connection settings in .env are correct.

Run DBT

Run models

make dbt-build