From cc078ee5a34302b47983da4d3f8124b4d783d79d Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:10:47 -0400 Subject: [PATCH 1/7] Update development documentation for setup and workflow --- DEVELOPMENT.md | 159 +++++++++++++++++++++++++++---------------------- README.md | 88 ++++++++++++++++++--------- 2 files changed, 148 insertions(+), 99 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 3d6810a..c6a312e 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,119 +1,136 @@ -# Development +# Development Guide -## Setup +This guide helps you set up the development environment for the GameCI Versioning Backend. -Install firebase globally +## Prerequisites -```bash -npm i -g firebase-tools -``` +- [Node.js](https://nodejs.org/) v20+ (required for Firebase Functions) +- [Yarn](https://yarnpkg.com/) for package management +- [Firebase CLI](https://firebase.google.com/docs/cli) for local development and deployment -Install dependencies +> **Tip:** Use [nvm](https://github.com/nvm-sh/nvm), [n](https://github.com/tj/n), or [volta](https://volta.sh/) to manage Node.js versions. -```bash -npm install -``` +## Setup -Run everything locally +### 1. Install Firebase CLI ```bash -firebase serve +npm i -g firebase-tools ``` -## Deployment +### 2. Install Dependencies -___Note:__ for this you will need access to the project._ +Install dependencies in both root repository and `functions` directory: -Login to your account +```bash +yarn install +``` ```bash -firebase login +cd functions +yarn install ``` -Deploy everything +### 3. Run Locally ```bash -firebase deploy +firebase emulators:start ``` -## Additional local setup (optional) +This starts the Firebase emulators for Functions and Firestore. -#### Credentials +## Credentials Setup -To be able to use functions that use the Firebase AdminSDK you need to set credentials. +### Firebase Admin SDK -1. Download the service account file from firebase -2. Set the path to that file to an environment variable +To use Firebase Admin SDK locally: -__Linux / MacOS__ +1. Download a service account key from your Firebase project settings +2. Set the environment variable to the key file +**Linux / macOS**: ```bash -export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json" +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-file.json" ``` -__Windows (PowerShell)__ - -```ps -$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json" +**Windows (PowerShell)**: +```powershell +$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\service-account-file.json" ``` -_(for more information, please see the [docs](https://firebase.google.com/docs/admin/setup))_ +### Integration Environment Variables -#### Integrations - -To test specific functionality like the integrations you will also have to set the following environment variables - -__Discord__ +To test integrations locally, set these environment variables: +**Discord**: ```bash -export discord.token="my_discord_token" +export discord.token="your_discord_token" ``` -__Github__ - +**GitHub**: ```bash -export github.client-secret="my_github_app_client_secret"` -export github.private-key="my_github_app_private_key" +export github.client-secret="your_github_app_client_secret" +export github.private-key="your_github_app_private_key" ``` -__Internal__ - -_Internal token is used for self-authentication and for communication with th -[docker repo](https://github.com/Unity-CI/docker)._ - -``` -export internal.token="my_internal_token" +**Internal Token**: +```bash +export internal.token="your_internal_token" ``` -_(value can be any single-line string, as long as it's the same in the docker repo)_ +> The internal token is used for self-authentication and communication with the [docker repo](https://github.com/Unity-CI/docker). -## Local Commands +## Deployment -In order to run firebase locally simply use +> **Note:** You need project access to deploy. -``` -firebase serve -``` +1. Login to Firebase: + ```bash + firebase login + ``` -To only run one component, like `hosting`, `functions` or `firestore` you may use the `--only` flag. +2. Deploy everything: + ```bash + firebase deploy + ``` -``` -firebase serve --only functions -``` + Or deploy specific services: + ```bash + firebase deploy --only functions + ``` -If everything works, finally deploy the changes +## Firebase Configuration -``` -firebase deploy -``` - -## Updating env/config variables - -_Typically this is only needed when migrating to new firebase project or environment, or when security token needs to rotate._ +Update environment variables in Firebase (useful when rotating tokens or migrating environments): -``` -firebase functions:config:set discord.token="my_discord_token" -firebase functions:config:set github.client-secret="my_github_app_client_secret" -firebase functions:config:set discord.private-key="my_github_app_private_key" -firebase functions:config:set internal.token="my_internal_token" -``` +```bash +firebase functions:config:set discord.token="your_discord_token" +firebase functions:config:set github.client-secret="your_github_app_client_secret" +firebase functions:config:set github.private-key="your_github_app_private_key" +firebase functions:config:set internal.token="your_internal_token" +``` + +## Development Workflow + +1. Make changes to code in `functions/src/` +2. Run the emulator to test locally: + ```bash + firebase emulators:start + ``` +3. For hot reloading during development: + ```bash + # In one terminal + cd functions && yarn watch + + # In another terminal + firebase emulators:start + ``` +4. Test your changes +5. Deploy when ready + +## Troubleshooting + +- **Firebase Login Issues**: Make sure you have access to the Firebase project +- **Emulator Port Conflicts**: Check for services using ports 4000, 5001, 8080, or 9000 +- **Admin SDK Errors**: Verify your service account file has the correct permissions +- **Integration Issues**: Ensure environment variables are correctly set diff --git a/README.md b/README.md index 1685bf9..7ac3f43 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,79 @@ # GameCI Versioning Backend -## Unity version ingest - -TODO - Describe how it works - -## game-ci/docker version ingest +The GameCI Versioning Backend automates the tracking, scheduling, and building of Unity versions and Docker images for the GameCI ecosystem. It connects with GitHub Actions for CI/CD workflows and Discord for notifications. + +## System Overview + +```mermaid +graph TD + A[Unity Version Archive] -->|Scrape & Detect| B[Version Ingest] + B -->|Store| C[Firestore Database] + B -->|Notify| D[Discord] + C -->|Schedule| E[CI Job Scheduler] + E -->|Trigger| F[GitHub Actions Workflows] + F -->|Report Status| G[CI Build Reporters] + G -->|Update| C + H[Ingeminator] -->|Retry Failed Builds| F + C -->|Monitor| H +``` -TODO - Describe how it works +## Unity Version Ingest -## Scheduler +The backend regularly scrapes Unity version information: -Each CiJob starts its own workflow. +1. Uses the [`unity-changeset` package](https://github.com/mob-sakai/unity-changeset) from [mob-sakai](https://github.com/mob-sakai) to detect new Unity versions +2. Filters versions (only stable versions 2017+) +3. Stores version details in Firestore +4. Notifies maintainers via Discord +5. Schedules build jobs for new versions -Each Workflow generates multiple CiBuilds: one per baseOs-targetPlatform combination. -The CiJob workflow will report back a CiBuild for each combination. +## CI Job Workflow -For example: +Each Unity version generates CI jobs and builds with the following relationships: -```text -... - ubuntu--linuxIl2cpp - ubuntu--webgl - windows--webgl -... ``` +CiJob (e.g., Unity 2022.3.15f1) + ├── CiBuild: ubuntu-2022.3.15f1-webgl + ├── CiBuild: ubuntu-2022.3.15f1-android + ├── CiBuild: windows-2022.3.15f1-webgl + └── ... (other baseOS-version-targetPlatform combinations) +``` + +## Scheduler -An endpoint from this backend will listen to the reports and update the database. -Each CiBuild starts with the status "started" after it is being reported. +The scheduler coordinates building Docker images: -When the last CiBuild is set to "published", the CiJob for that version is also set to "completed". -Completed CiJobs are reported to Discord. +- First ensures base and hub images are built +- Monitors for failed jobs and triggers the Ingeminator to retry them +- Prioritizes jobs based on Unity version recency +- Limits concurrent jobs to prevent overloading GitHub Actions ## Ingeminator -TODO - Describe how it works +The Ingeminator ("repeater") handles the reliability of the build system: + +- Detects failed builds and reschedules them +- Implements an exponential backoff strategy for retries +- Alerts via Discord when builds reach maximum retry attempts +- Works with the scheduler to manage retry priorities ## Database Backup -The firestore database can be backed up with the following command: -`yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile ` +Backup the Firestore database: +```bash +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json" +yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS +``` + +Restore a backup: +```bash +yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS +``` + +## Development + +For instructions on setting up the development environment, see [DEVELOPMENT.md](./DEVELOPMENT.md). -Similarly, it can be used to restore a backup with: -`yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile ` +## Contributing -You likely would want to empty the database before restoring but you can also use flags like overwrite, merge, etc to control the restoration -rules. +We welcome contributions! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines. From bf35347a7757fe9860081d4ba8424fa895006a93 Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:34:07 -0400 Subject: [PATCH 2/7] Update environment variable naming conventions in development documentation --- DEVELOPMENT.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index c6a312e..c188a52 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -64,18 +64,18 @@ To test integrations locally, set these environment variables: **Discord**: ```bash -export discord.token="your_discord_token" +export DISCORD_TOKEN="your_discord_token" ``` **GitHub**: ```bash -export github.client-secret="your_github_app_client_secret" -export github.private-key="your_github_app_private_key" +export GITHUB_CLIENT_SECRET="your_github_app_client_secret" +export GITHUB_PRIVATE_KEY="your_github_app_private_key" ``` **Internal Token**: ```bash -export internal.token="your_internal_token" +export INTERNAL_TOKEN="your_internal_token" ``` > The internal token is used for self-authentication and communication with the [docker repo](https://github.com/Unity-CI/docker). @@ -110,6 +110,8 @@ firebase functions:config:set github.private-key="your_github_app_private_key" firebase functions:config:set internal.token="your_internal_token" ``` +> Note: Firebase Functions configuration uses dot notation (e.g., `internal.token`) when setting with the CLI, but when using environment variables locally, use uppercase without dots (e.g., `INTERNAL_TOKEN`). This is due to how Firebase handles different configuration methods. + ## Development Workflow 1. Make changes to code in `functions/src/` From c57d9921c7879d0b125d6e6ecafb6b55c998d6ee Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:39:23 -0400 Subject: [PATCH 3/7] Correct spelling and update service account file path in database backup instructions --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7ac3f43..e56d618 100644 --- a/README.md +++ b/README.md @@ -59,9 +59,9 @@ The Ingeminator ("repeater") handles the reliability of the build system: ## Database Backup -Backup the Firestore database: +Back up the Firestore database: ```bash -export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json" +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-file.json" yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS ``` From aa0beebc4a22e8e9c402c84e3a40d59ccf0bef10 Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:40:12 -0400 Subject: [PATCH 4/7] Update service account file path in documentation --- DEVELOPMENT.md | 4 ++-- README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index c188a52..e2bb03a 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -50,12 +50,12 @@ To use Firebase Admin SDK locally: **Linux / macOS**: ```bash -export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-file.json" +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json" ``` **Windows (PowerShell)**: ```powershell -$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\service-account-file.json" +$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\serviceAccountKey.json" ``` ### Integration Environment Variables diff --git a/README.md b/README.md index e56d618..4e561ef 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ The Ingeminator ("repeater") handles the reliability of the build system: Back up the Firestore database: ```bash -export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-file.json" +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json" yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS ``` From 09ab610cf4a27da92cc186f35c073ee7ffa16453 Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:42:33 -0400 Subject: [PATCH 5/7] Add Java Runtime Environment requirements and update emulator port configuration --- DEVELOPMENT.md | 34 +++++++++++++++++++++++++++++++++- firebase.json | 2 +- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index e2bb03a..0e5a7c4 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -7,6 +7,7 @@ This guide helps you set up the development environment for the GameCI Versionin - [Node.js](https://nodejs.org/) v20+ (required for Firebase Functions) - [Yarn](https://yarnpkg.com/) for package management - [Firebase CLI](https://firebase.google.com/docs/cli) for local development and deployment +- [Java Runtime Environment](https://www.java.com/) for Firebase emulators > **Tip:** Use [nvm](https://github.com/nvm-sh/nvm), [n](https://github.com/tj/n), or [volta](https://volta.sh/) to manage Node.js versions. @@ -39,6 +40,31 @@ firebase emulators:start This starts the Firebase emulators for Functions and Firestore. +#### Prerequisites for Emulators + +- **Java Runtime Environment**: Firebase emulators require Java to be installed and available on your system PATH +- **Firebase Login**: Run `firebase login` to authenticate the CLI + +#### Port Configuration + +On macOS, port 5000 (used by the Hosting emulator) might be in use by AirPlay Receiver. You can configure a different port in your `firebase.json`: + +```json +{ + "emulators": { + "hosting": { + "port": 5002 + }, + "functions": { + "port": 5001 + }, + "firestore": { + "port": 8080 + } + } +} +``` + ## Credentials Setup ### Firebase Admin SDK @@ -133,6 +159,12 @@ firebase functions:config:set internal.token="your_internal_token" ## Troubleshooting - **Firebase Login Issues**: Make sure you have access to the Firebase project -- **Emulator Port Conflicts**: Check for services using ports 4000, 5001, 8080, or 9000 +- **Emulator Port Conflicts**: + - Check for services using ports 4000, 5001, 8080, or 9000 + - On macOS, port 5000 is commonly used by AirPlay Receiver - configure a different port as shown in the [Port Configuration](#port-configuration) section +- **Java Not Found Error**: + - The Firebase emulators require Java to be installed + - On macOS, install Java using `brew install openjdk@17` or download from [java.com](https://www.java.com) + - Make sure Java is on your PATH: `java -version` should return the installed version - **Admin SDK Errors**: Verify your service account file has the correct permissions - **Integration Issues**: Ensure environment variables are correctly set diff --git a/firebase.json b/firebase.json index 73d5cb6..9584ca3 100644 --- a/firebase.json +++ b/firebase.json @@ -29,7 +29,7 @@ "port": 8080 }, "hosting": { - "port": 5000 + "port": 5002 }, "pubsub": { "port": 8085 From 0bf2450d870f96177feaffa408ef4b4f4f886717 Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:47:23 -0400 Subject: [PATCH 6/7] Update DEVELOPMENT.md to clarify Firebase emulator port configuration and avoid conflicts --- DEVELOPMENT.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 0e5a7c4..185537b 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -38,7 +38,7 @@ yarn install firebase emulators:start ``` -This starts the Firebase emulators for Functions and Firestore. +This starts the Firebase emulators for Functions and Firestore, with hosting on port 5002 to avoid common port conflicts. #### Prerequisites for Emulators @@ -47,7 +47,9 @@ This starts the Firebase emulators for Functions and Firestore. #### Port Configuration -On macOS, port 5000 (used by the Hosting emulator) might be in use by AirPlay Receiver. You can configure a different port in your `firebase.json`: +This project's `firebase.json` is already configured to use port 5002 for the hosting emulator instead of the default port 5000, which helps avoid conflicts with AirPlay Receiver on macOS. + +You're free to customize these ports for your local development environment as needed. If you encounter port conflicts, you can modify the port numbers in your `firebase.json`: ```json { @@ -65,6 +67,8 @@ On macOS, port 5000 (used by the Hosting emulator) might be in use by AirPlay Re } ``` +Modifying these ports only affects local development environment and won't impact deployment. + ## Credentials Setup ### Firebase Admin SDK @@ -161,7 +165,9 @@ firebase functions:config:set internal.token="your_internal_token" - **Firebase Login Issues**: Make sure you have access to the Firebase project - **Emulator Port Conflicts**: - Check for services using ports 4000, 5001, 8080, or 9000 - - On macOS, port 5000 is commonly used by AirPlay Receiver - configure a different port as shown in the [Port Configuration](#port-configuration) section + - This project uses port 5002 for hosting to avoid conflicts with AirPlay Receiver on macOS + - Feel free to change any port in your local `firebase.json` if you encounter conflicts + - See the [Port Configuration](#port-configuration) section for details - **Java Not Found Error**: - The Firebase emulators require Java to be installed - On macOS, install Java using `brew install openjdk@17` or download from [java.com](https://www.java.com) From 3c36a64de286bb384702600384456fed916a3606 Mon Sep 17 00:00:00 2001 From: Gabriel Le Breton Date: Fri, 25 Apr 2025 10:51:19 -0400 Subject: [PATCH 7/7] Update DEVELOPMENT.md to include steps for building functions code and setting up credentials --- DEVELOPMENT.md | 88 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 81 insertions(+), 7 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 185537b..0f00c70 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -32,7 +32,69 @@ cd functions yarn install ``` -### 3. Run Locally +### 3. Build Functions Code + +Before running the emulators, you need to build the functions code: + +```bash +cd functions +yarn build +``` + +This will compile the TypeScript code into JavaScript in the `lib` directory, which the emulator needs to run the functions. + +### 4. Set Up Credentials + +> **Note**: For basic local development and testing, you can skip this step initially. You'll see some warnings in the emulator output, but most functionality will still work. If you need to test integrations or need full functionality, follow these steps. + +#### Firebase Admin SDK + +To use Firebase Admin SDK locally: + +1. Download a service account key from your Firebase project settings: + - Go to [Firebase Console](https://console.firebase.google.com/) + - Select your project + - Go to Project Settings > Service accounts + - Click "Generate new private key" + - Save the JSON file securely + +2. Set the environment variable to the key file: + +**Linux / macOS**: +```bash +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json" +``` + +**Windows (PowerShell)**: +```powershell +$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\serviceAccountKey.json" +``` + +> Note: Without this setup, you'll see warnings about being unable to fetch Admin SDK configuration, but the emulators will still run in a limited capacity. + +#### Integration Environment Variables + +To test integrations locally, set these environment variables: + +**Discord**: +```bash +export DISCORD_TOKEN="your_discord_token" +``` + +**GitHub**: +```bash +export GITHUB_CLIENT_SECRET="your_github_app_client_secret" +export GITHUB_PRIVATE_KEY="your_github_app_private_key" +``` + +**Internal Token**: +```bash +export INTERNAL_TOKEN="your_internal_token" +``` + +> The internal token is used for self-authentication and communication with the [docker repo](https://github.com/Unity-CI/docker). + +### 5. Run Locally ```bash firebase emulators:start @@ -145,11 +207,15 @@ firebase functions:config:set internal.token="your_internal_token" ## Development Workflow 1. Make changes to code in `functions/src/` -2. Run the emulator to test locally: +2. Build the functions code: + ```bash + cd functions && yarn build + ``` +3. Run the emulator to test locally: ```bash firebase emulators:start ``` -3. For hot reloading during development: +4. For hot reloading during development: ```bash # In one terminal cd functions && yarn watch @@ -157,8 +223,8 @@ firebase functions:config:set internal.token="your_internal_token" # In another terminal firebase emulators:start ``` -4. Test your changes -5. Deploy when ready +5. Test your changes +6. Deploy when ready ## Troubleshooting @@ -172,5 +238,13 @@ firebase functions:config:set internal.token="your_internal_token" - The Firebase emulators require Java to be installed - On macOS, install Java using `brew install openjdk@17` or download from [java.com](https://www.java.com) - Make sure Java is on your PATH: `java -version` should return the installed version -- **Admin SDK Errors**: Verify your service account file has the correct permissions -- **Integration Issues**: Ensure environment variables are correctly set +- **Missing Functions Library Error** (`functions/lib/index.js does not exist`): + - This indicates that the TypeScript code hasn't been compiled + - Run `cd functions && yarn build` to compile the code + - If that doesn't work, check for TypeScript compilation errors in the build output +- **Admin SDK Configuration Errors**: + - Set up the `GOOGLE_APPLICATION_CREDENTIALS` environment variable as described in the [Set Up Credentials](#4-set-up-credentials) section + - For testing, you can often ignore this warning as the emulators will still run with limited functionality +- **Integration Issues**: + - Ensure all required environment variables are correctly set + - For local development without integration testing, you can often proceed without setting these variables