Python Shell dag and README.md Update

Updated the python shell dag script to show how the data was extracted to the source bucket and how it would be moved to the bronze tier from the source bucket. Also updated README.md file with project outline going forward.

Author: jibbs1703

README.md

@@ -16,15 +16,15 @@ In this project, raw, untransformed data resides in external databases and is in
 into a bronze tier S3 bucket. The pipeline works on the raw data, processing it, and subsequently storing it in
 the appropriate data lake tier as determined by business requirements. The tiers are represented as folders within
 a single S3 bucket for this project. However, each tier should be given a dedicated bucket (as it is in production
-environments). 
+environments).
 
 An orchestrator triggers the extraction of the ingested raw and untransformed data into the bronze tier S3 where 
 it is then moved on to the other tiers, getting enriched as it gets refined up the data lake tiers.
 
 An understanding of creating and assigning IAM roles is required as the AWS resources used are configured in such
 a way to interact with one another, i.e., a role that grants permission to let AWS Glue access the resources it needs.
 The degree of restrictions can be narrowed down based on the level of security needed. For this pipeline, I assigned 
-an IAM role to access AWS Glue resources and read/write to AWS S3 and Redshift. 
+an IAM role to access AWS Glue resources and read/write to AWS S3 and Redshift.
 
 ## Medallion Architecture
 
@@ -41,7 +41,7 @@ Each tier houses its own schemas and tables, which differ based on data update f
 This multi-layered approach ensures data integrity and optimizes its use for business needs.
 
 
-## 
+## Pipeline Steps
 
 ### Ingesting the Data From Source
 
@@ -65,4 +65,11 @@ data, into the bronze layer, in a separate bucket (usually more secure with more
 source bucket).
 
 
+### Creating the Silver Data Tier
+
+The silver data tier takes the data a step further in its refinement as the data passes through extensive cleanup and
+validation. The silver tier sees the datatype standardization, filling and/or removal of null values, creation of 
+desirable datatypes, detection and removal of duplicates, and to a certain degree, some data aggregation as some facts
+and dimension tables could be merged in the silver tier to allow downstream users utilize the data. 
+
 

dags_python_shell.py

@@ -8,7 +8,7 @@
 config.read('config.ini')
 
 # Extract tables from Source to Bronze Bucket
-def extract_to_bronze_tier(db_path=config['DATABASE']['PATH'],
+def extract_to_source(db_path=config['DATABASE']['PATH'],
                      tables=eval(config.get('DATABASE', 'TABLES'))):
 
     """ Extracts tables from a SQLite database and uploads them to an S3 bucket as CSV files.
@@ -36,10 +36,52 @@ def extract_to_bronze_tier(db_path=config['DATABASE']['PATH'],
         s3_conn = S3Buckets.credentials(config['AWS_ACCESS']['REGION'])
         s3_conn.upload_dataframe_to_s3(df = df,
                                        bucket_name=config['AWS_ACCESS']['PROJECT_BUCKET'],
-                                       object_name=f'bronze/bronze_{table}.csv')
+                                       object_name=f'source/{table}.csv')
         print(f"The {table} table was uploaded")
 
     return f'The tables were was uploaded to the S3 Bucket'
 
+
+# Extraction to Bronze Tier from Source
+def extract_category_to_bronze(table='category'):
+
+    # Connect to AWS S3 and upload Extracted Dataframe as CSV file
+    s3_conn = S3Buckets.credentials(config['AWS_ACCESS']['REGION'])
+    s3_conn.upload_dataframe_to_s3(df=df,
+                                   bucket_name=config['AWS_ACCESS']['PROJECT_BUCKET'],
+                                   object_name=f'source/{table}.csv')
+    print(f"The {table} table was uploaded")
+    pass
+
+def extract_date_to_bronze():
+    pass
+
+def extract_event_to_bronze():
+    pass
+
+def extract_listing_to_bronze():
+    pass
+
+def extract_sale_to_bronze():
+    pass
+
+def extract_user_to_bronze():
+    pass
+
+def extract_venue_to_bronze():
+    pass
+
+
+
+
 def transform_to_silver_tier():
+    pass
+
+def transform_cat_eve_ven():
+    pass
+
+def transform_date_list_sales():
+    pass
+
+def transform_date_list_users():
     pass