-
Notifications
You must be signed in to change notification settings - Fork 3
Requirements
You will need to create a root user account in AWS. To accomplish this, follow the instructions in the links below.
Root User Creation Step-by-Step
Once you create a root user, you can create a designated IAM user with the appropriate permissions.
Once you have created a root user and IAM user in AWS (with AdminstratorAccess permissions), go to the Security Credentials tab, then under the Access Keys section, create an access key for the IAM user. This should provide two values, 'Access Key', and 'Secret Access Key'
When you generate the access keys, you will be able to download the credentials as a csv file.
Be sure to install the AWS CLI. MacOS and Windows have an installer package that makes it straight-forward.
On a Linux machine, you can run,
sudo apt install awscli
Once you have aws cli installed on your local machine, run,
aws configure
This will prompt you for your Access Key and Secret Access Key, default region, and default output format. Once you have successfully set up the aws cli with aws configure you will be able to test it with some of the aws cli commands such as,
aws s3 ls
This will list all of the buckets you have in the s3 account that the access keys grant permissions to list.
In all of the activities, we will be using the same buckets, and the code directly pulled from this repository. If you want the code to work as cloned, be sure you have a bucket in AWS called, orangutan-orchard
Just like on S3, you will want to create a bucket in GCS called, orangutan-orchard
If you are creating your GCP account for the first time, you can call your project-id, orangutan-orchard. If you already have a GCP account, you can add a new project to GCP by selecting your project at the top ribbon, and clicking the 'new project' button.
When you create your service account in the steps below, be sure to associate the service account to orangutan-orchard
Service accounts in Google Cloud Platform (GCP) provide a way for applications and services to interact with GCP resources. This guide will walk you through the steps to create and manage a GCP Service Account.
- A Google Cloud Platform account
- Access to the Google Cloud Console
- Appropriate permissions to create and manage service accounts
Navigate to the Google Cloud Console by visiting https://console.cloud.google.com/.
Once logged in, select the project where you want to create the service account from the "Select a Project" dropdown. You will want to have a project called, orangutan-orchard for all of the activities used in the codebase and wiki's.
On the left-hand sidebar, scroll down to "IAM & Admin" and then click on "Service accounts."
- Click on the “Create Service Account” button at the top of the page.
- Fill in the necessary fields like "Service Account Name," "Service Account ID," and "Service Account Description."
- Click "Create."
- Choose the roles that the service account should have. This will determine what resources the service account can access (e.g.,
Compute Admin,Storage Admin, etc.) - Click "Continue."
- You can add additional users who can act as this service account.
- Click "Done" if you don't need to add additional users.
- Go back to the Service Accounts page.
- Click on the created service account.
- Navigate to the "Keys" tab.
- Click "Add Key" and choose the key type (typically JSON).
- This will download a key file. Store this file securely, as it provides access to your GCP resources. We will be copying and pasting the json file contents into airflow to set up gcp connections.
If you are using the Google Cloud SDK, you can activate the service account using the following command:
gcloud auth activate-service-account [ACCOUNT] --key-file=[PATH]Replace [ACCOUNT] with the email address of the service account and [PATH] with the path to the downloaded key file.
Be sure to grant the following permissions to your service account on GCP:
BigQuery Admin
BigQuery Data Editor
BigQuery Data Owner
BigQuery Job User
Storage Admin
Storage Object Admin
Storage Object Creator
Storage Transfer Admin
Remember to review your service accounts and their permissions regularly to ensure they follow the principle of least privilege. Probably a smart idea to set up some billing alerts after creating your service account to avoid unexpected cloud costs.
-
Installing on Windows: click install for windows, and run the installer. This will automatically put Docker Desktop on your machine (or searchable in windows home)
-
Installing on MacOS: if using an M1/M2 chip, be sure to select 'Apple Chip'
-
once installer finishes, docker desktop should be accessible on your machine (Search windows home, or look in launchpad on mac)
Make sure the correct versions of Docker and Docker Compose are installed:
docker --version
docker-compose --version
Note, the api keys take a few hours to start working (I've seen it take up to 12+ hours for api keys to start working upon account creation), and there are some api call limits set with the Open Weather Map API.
To retrieve API keys, you will need to create an account
Create Open Weather Map Account
Once you have created an account, click your profile name at the top right (to the left of the 'Support' button), and click, "My API Keys"
On this page, you will be able to generate an API key that will be used in Activity 1.
- Raspberry Pi 3 or 4
Amazon, MicroCenter, Facebook Marketplace
~$35-$110 USD in cost
- Arducam Camera
Amazon, MicroCenter
~$10-$25 USD in cost
- microSD card reader
BestBuy, Amazon, MicroCenter
~$10-$20 USD in cost
- MicroSD card
BestBuy, Amazon, MicroCenter
~$10-$50 USD in cost