Skip to content

Commit 196bd3b

Browse files
Update README.md
1 parent e71f856 commit 196bd3b

File tree

1 file changed

+41
-23
lines changed

1 file changed

+41
-23
lines changed

README.md

Lines changed: 41 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
1-
# Intro
1+
# 🚀 Portable Data Stack with dbt, DuckDB and Sheets!
2+
3+
All the context in this blog post: [📦 Portable Data Stack Concept With dbt, DuckDB, Sheets and Looker Studio](https://open.substack.com/pub/thepipeandtheline/p/portable-data-stack-concepts-with-dbt-duckdb-sheets-looker-studio)
24

35
A portable data-stack with:
46
- Custom Python ETL
@@ -7,37 +9,53 @@ A portable data-stack with:
79
- Google Sheets for Warehouse
810
- Data Studio for Visualization
911

10-
# Draft
11-
- requisites
12-
- cuenta demo de piwikpro
13-
- crear conectar service account. Add your service_account.json file in /reporting path
14-
- crear un sheets vacío, darle acceso al mail del service account
15-
- crear un data studio y conectarlo al sheets (hablar de caveats si se pierde la hoja)
16-
- duckdb / Docker Desktop
17-
- rename .env.example to .env and fill in your values
18-
19-
## Steps
20-
- make build
21-
- make run (generate data, save to duckdb, run dbt and export to sheets) -> container will keep running
22-
- connect data studio to sheets
23-
24-
# if you want to query data
12+
## 🙋🏻‍♂️ Pre-requisites
13+
- Docker Desktop
14+
- PiwikPRO demo account.
15+
- GCP Service account to work with Google Sheets. Add your service_account.json file in /reporting path.
16+
- An Empty Spreadsheet that shares access to the email generated by the GCP Service account.
17+
- A Looker Studio connected to the Spreadhsheet.
18+
- DuckDB to run local queries.
19+
- rename .env.example to .env and fill in your values if you want to test it!
20+
21+
## 📝 Steps
22+
- To get started with the Docker Image:
23+
```sh
24+
make build
25+
```
26+
- This will generate data, save to DuckDB, run dbt and export to sheets. Container will keep running:
2527

26-
Uncomment line 45, 46, 47 and replace the command value with:
28+
```sh
29+
make run
30+
```
31+
- Once the data is present on the Google Sheet, connect Looker Studio to it.
32+
- If you want to query data. Uncomment line 45, 46, 47 and replace the command value with:
2733

28-
 ```sh
34+
```sh
2935
sh -c "python model_export.py && tail -f /dev/null
3036
```
31-
This will leave the container open so you can copy the files to query.
3237
33-
- make duckdb-files (if you want to query data)
34-
- make duckdb (query duckdb)
38+
- This will leave the container open so you can copy the files to query and use DuckDB querying engine:
39+
40+
```sh
41+
make duckdb-files
42+
make duckdb
43+
```
3544
3645
```sql
3746
show tables;
3847
select * from attribution limit 5;
3948
````
49+
- If you want to remove all dependencies from the container, run:
50+
51+
```sh
52+
make clean
53+
```
54+
55+
- To run in Github Actions, create your Actions secrets for the .env file.
4056
41-
- make clean (removes all generated files)
57+
### 😎 [Follow me on Linkedin](https://www.linkedin.com/in/alejandro-aboy/)
58+
- Get tips, learnings and tricks for your Data career!
4259
43-
- To run in Github Actions, create your secrets for the .env file
60+
### 📩 [Subscribe to The Pipe & The Line](https://thepipeandtheline.substack.com/?utm_source=github&utm_medium=referral)
61+
- Join the Substack newsletter to get similar content to this one and more to improve your Data career!

0 commit comments

Comments
 (0)