You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# 🚀 Portable Data Stack with dbt, DuckDB and Sheets!
2
+
3
+
All the context in this blog post: [📦 Portable Data Stack Concept With dbt, DuckDB, Sheets and Looker Studio](https://open.substack.com/pub/thepipeandtheline/p/portable-data-stack-concepts-with-dbt-duckdb-sheets-looker-studio)
2
4
3
5
A portable data-stack with:
4
6
- Custom Python ETL
@@ -7,37 +9,53 @@ A portable data-stack with:
7
9
- Google Sheets for Warehouse
8
10
- Data Studio for Visualization
9
11
10
-
#Draft
11
-
-requisites
12
-
- cuenta demo de piwikpro
13
-
- crear conectar service account. Add your service_account.json file in /reporting path
14
-
- crear un sheets vacío, darle acceso al mail del service account
15
-
- crear un data studio y conectarlo al sheets (hablar de caveats si se pierde la hoja)
16
-
- duckdb / Docker Desktop
17
-
- rename .env.example to .env and fill in your values
18
-
19
-
##Steps
20
-
-make build
21
-
- make run (generate data, save to duckdb, run dbt and export to sheets) -> container will keep running
22
-
- connect data studio to sheets
23
-
24
-
# if you want to query data
12
+
## 🙋🏻♂️ Pre-requisites
13
+
-Docker Desktop
14
+
- PiwikPRO demo account.
15
+
- GCP Service account to work with Google Sheets. Add your service_account.json file in /reporting path.
16
+
- An Empty Spreadsheet that shares access to the email generated by the GCP Service account.
17
+
- A Looker Studio connected to the Spreadhsheet.
18
+
- DuckDB to run local queries.
19
+
- rename .env.example to .env and fill in your values if you want to test it!
20
+
21
+
##📝 Steps
22
+
-To get started with the Docker Image:
23
+
```sh
24
+
make build
25
+
```
26
+
- This will generate data, save to DuckDB, run dbt and export to sheets. Container will keep running:
25
27
26
-
Uncomment line 45, 46, 47 and replace the command value with:
28
+
```sh
29
+
make run
30
+
```
31
+
- Once the data is present on the Google Sheet, connect Looker Studio to it.
32
+
- If you want to query data. Uncomment line 45, 46, 47 and replace the command value with:
27
33
28
-
```sh
34
+
```sh
29
35
sh -c "python model_export.py && tail -f /dev/null
30
36
```
31
-
This will leave the container open so you can copy the files to query.
32
37
33
-
- make duckdb-files (if you want to query data)
34
-
- make duckdb (query duckdb)
38
+
- This will leave the container open so you can copy the files to query and use DuckDB querying engine:
39
+
40
+
```sh
41
+
make duckdb-files
42
+
make duckdb
43
+
```
35
44
36
45
```sql
37
46
show tables;
38
47
select * from attribution limit 5;
39
48
````
49
+
- If you want to remove all dependencies from the container, run:
50
+
51
+
```sh
52
+
make clean
53
+
```
54
+
55
+
- To run in Github Actions, create your Actions secrets for the .env file.
40
56
41
-
- make clean (removes all generated files)
57
+
### 😎 [Follow me on Linkedin](https://www.linkedin.com/in/alejandro-aboy/)
58
+
- Get tips, learnings and tricks for your Data career!
42
59
43
-
- To run in Github Actions, create your secrets for the .env file
60
+
### 📩 [Subscribe to The Pipe & The Line](https://thepipeandtheline.substack.com/?utm_source=github&utm_medium=referral)
61
+
- Join the Substack newsletter to get similar content to this one and more to improve your Data career!
0 commit comments