Update development documentation for setup and workflow

DEVELOPMENT.md

@@ -1,119 +1,170 @@
-# Development
+# Development Guide
 
-## Setup
+This guide helps you set up the development environment for the GameCI Versioning Backend.
 
-Install firebase globally
+## Prerequisites
 
-```bash
-npm i -g firebase-tools
-```
+- [Node.js](https://nodejs.org/) v20+ (required for Firebase Functions)
+- [Yarn](https://yarnpkg.com/) for package management
+- [Firebase CLI](https://firebase.google.com/docs/cli) for local development and deployment
+- [Java Runtime Environment](https://www.java.com/) for Firebase emulators
 
-Install dependencies
+> **Tip:** Use [nvm](https://github.com/nvm-sh/nvm), [n](https://github.com/tj/n), or [volta](https://volta.sh/) to manage Node.js versions.
 
-```bash
-npm install
-```
+## Setup
 
-Run everything locally
+### 1. Install Firebase CLI
 
 ```bash
-firebase serve
+npm i -g firebase-tools
 ```
 
-## Deployment
+### 2. Install Dependencies
 
-___Note:__ for this you will need access to the project._
-
-Login to your account
+Install dependencies in both root repository and `functions` directory:
 
 ```bash
-firebase login
+yarn install
 ```
 
-Deploy everything
-
 ```bash
-firebase deploy
+cd functions
+yarn install
 ```
 
-## Additional local setup (optional)
+### 3. Run Locally
 
-#### Credentials
+```bash
+firebase emulators:start
+```
 
-To be able to use functions that use the Firebase AdminSDK you need to set credentials.
+This starts the Firebase emulators for Functions and Firestore.
 
-1. Download the service account file from firebase
-2. Set the path to that file to an environment variable
+#### Prerequisites for Emulators
 
-__Linux / MacOS__
+- **Java Runtime Environment**: Firebase emulators require Java to be installed and available on your system PATH
+- **Firebase Login**: Run `firebase login` to authenticate the CLI
 
-```bash
-export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"
-```
+#### Port Configuration
 
-__Windows (PowerShell)__
+On macOS, port 5000 (used by the Hosting emulator) might be in use by AirPlay Receiver. You can configure a different port in your `firebase.json`:
 
-```ps
-$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"
+```json
+{
+  "emulators": {
+    "hosting": {
+      "port": 5002
+    },
+    "functions": {
+      "port": 5001
+    },
+    "firestore": {
+      "port": 8080
+    }
+  }
+}
 ```
 
-_(for more information, please see the [docs](https://firebase.google.com/docs/admin/setup))_
+## Credentials Setup
 
-#### Integrations
+### Firebase Admin SDK
 
-To test specific functionality like the integrations you will also have to set the following environment variables
+To use Firebase Admin SDK locally:
 
-__Discord__
+1. Download a service account key from your Firebase project settings
+2. Set the environment variable to the key file
 
+**Linux / macOS**:
 ```bash
-export discord.token="my_discord_token"
+export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json"
 ```
 
-__Github__
-
-```bash
-export github.client-secret="my_github_app_client_secret"`
-export github.private-key="my_github_app_private_key"
+**Windows (PowerShell)**:
+```powershell
+$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\serviceAccountKey.json"
 ```
 
-__Internal__
+### Integration Environment Variables
 
-_Internal token is used for self-authentication and for communication with th
-[docker repo](https://github.com/Unity-CI/docker)._
+To test integrations locally, set these environment variables:
 
-```
-export internal.token="my_internal_token"
+**Discord**:
+```bash
+export DISCORD_TOKEN="your_discord_token"
 ```
 
-_(value can be any single-line string, as long as it's the same in the docker repo)_
+**GitHub**:
+```bash
+export GITHUB_CLIENT_SECRET="your_github_app_client_secret"
+export GITHUB_PRIVATE_KEY="your_github_app_private_key"
+```
 
-## Local Commands
+**Internal Token**:
+```bash
+export INTERNAL_TOKEN="your_internal_token"
+```
 
-In order to run firebase locally simply use
+> The internal token is used for self-authentication and communication with the [docker repo](https://github.com/Unity-CI/docker).
 
-```
-firebase serve
-```
+## Deployment
 
-To only run one component, like `hosting`, `functions` or `firestore` you may use the `--only` flag.
+> **Note:** You need project access to deploy.
 
-```
-firebase serve --only functions
-```
+1. Login to Firebase:
+   ```bash
+   firebase login
+   ```
 
-If everything works, finally deploy the changes
+2. Deploy everything:
+   ```bash
+   firebase deploy
+   ```
 
-```
-firebase deploy
-```
+   Or deploy specific services:
+   ```bash
+   firebase deploy --only functions
+   ```
 
-## Updating env/config variables 
+## Firebase Configuration
 
-_Typically this is only needed when migrating to new firebase project or environment, or when security token needs to rotate._
+Update environment variables in Firebase (useful when rotating tokens or migrating environments):
 
-```
-firebase functions:config:set discord.token="my_discord_token"
-firebase functions:config:set github.client-secret="my_github_app_client_secret"
-firebase functions:config:set discord.private-key="my_github_app_private_key"
-firebase functions:config:set internal.token="my_internal_token"
-```
+```bash
+firebase functions:config:set discord.token="your_discord_token"
+firebase functions:config:set github.client-secret="your_github_app_client_secret"
+firebase functions:config:set github.private-key="your_github_app_private_key"
+firebase functions:config:set internal.token="your_internal_token"
+```
+
+> Note: Firebase Functions configuration uses dot notation (e.g., `internal.token`) when setting with the CLI, but when using environment variables locally, use uppercase without dots (e.g., `INTERNAL_TOKEN`). This is due to how Firebase handles different configuration methods.
+
+## Development Workflow
+
+1. Make changes to code in `functions/src/`
+2. Run the emulator to test locally:
+   ```bash
+   firebase emulators:start
+   ```
+3. For hot reloading during development:
+   ```bash
+   # In one terminal
+   cd functions && yarn watch
+
+   # In another terminal  
+   firebase emulators:start
+   ```
+4. Test your changes
+5. Deploy when ready
+
+## Troubleshooting
+
+- **Firebase Login Issues**: Make sure you have access to the Firebase project
+- **Emulator Port Conflicts**: 
+  - Check for services using ports 4000, 5001, 8080, or 9000
+  - On macOS, port 5000 is commonly used by AirPlay Receiver - configure a different port as shown in the [Port Configuration](#port-configuration) section
+- **Java Not Found Error**: 
+  - The Firebase emulators require Java to be installed
+  - On macOS, install Java using `brew install openjdk@17` or download from [java.com](https://www.java.com)
+  - Make sure Java is on your PATH: `java -version` should return the installed version
+- **Admin SDK Errors**: Verify your service account file has the correct permissions
+- **Integration Issues**: Ensure environment variables are correctly set

README.md

@@ -1,47 +1,79 @@
 # GameCI Versioning Backend
 
-## Unity version ingest
-
-TODO - Describe how it works
-
-## game-ci/docker version ingest
+The GameCI Versioning Backend automates the tracking, scheduling, and building of Unity versions and Docker images for the GameCI ecosystem. It connects with GitHub Actions for CI/CD workflows and Discord for notifications.
+
+## System Overview
+
+```mermaid
+graph TD
+    A[Unity Version Archive] -->|Scrape & Detect| B[Version Ingest]
+    B -->|Store| C[Firestore Database]
+    B -->|Notify| D[Discord]
+    C -->|Schedule| E[CI Job Scheduler]
+    E -->|Trigger| F[GitHub Actions Workflows]
+    F -->|Report Status| G[CI Build Reporters]
+    G -->|Update| C
+    H[Ingeminator] -->|Retry Failed Builds| F
+    C -->|Monitor| H
+```
 
-TODO - Describe how it works
+## Unity Version Ingest
 
-## Scheduler
+The backend regularly scrapes Unity version information:
 
-Each CiJob starts its own workflow.
+1. Uses the [`unity-changeset` package](https://github.com/mob-sakai/unity-changeset) from [mob-sakai](https://github.com/mob-sakai) to detect new Unity versions
+2. Filters versions (only stable versions 2017+)
+3. Stores version details in Firestore
+4. Notifies maintainers via Discord
+5. Schedules build jobs for new versions
 
-Each Workflow generates multiple CiBuilds: one per baseOs-targetPlatform combination.
-The CiJob workflow will report back a CiBuild for each combination.
+## CI Job Workflow
 
-For example:
+Each Unity version generates CI jobs and builds with the following relationships:
 
-```text
-...
-  ubuntu-<version>-linuxIl2cpp
-  ubuntu-<version>-webgl
-  windows-<version>-webgl
-...
 ```
+CiJob (e.g., Unity 2022.3.15f1)
+  ├── CiBuild: ubuntu-2022.3.15f1-webgl
+  ├── CiBuild: ubuntu-2022.3.15f1-android
+  ├── CiBuild: windows-2022.3.15f1-webgl
+  └── ... (other baseOS-version-targetPlatform combinations)
+```
+
+## Scheduler
 
-An endpoint from this backend will listen to the reports and update the database.
-Each CiBuild starts with the status "started" after it is being reported.
+The scheduler coordinates building Docker images:
 
-When the last CiBuild is set to "published", the CiJob for that version is also set to "completed".
-Completed CiJobs are reported to Discord.
+- First ensures base and hub images are built
+- Monitors for failed jobs and triggers the Ingeminator to retry them
+- Prioritizes jobs based on Unity version recency
+- Limits concurrent jobs to prevent overloading GitHub Actions
 
 ## Ingeminator
 
-TODO - Describe how it works
+The Ingeminator ("repeater") handles the reliability of the build system:
+
+- Detects failed builds and reschedules them
+- Implements an exponential backoff strategy for retries
+- Alerts via Discord when builds reach maximum retry attempts
+- Works with the scheduler to manage retry priorities
 
 ## Database Backup
 
-The firestore database can be backed up with the following command:
-`yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
+Back up the Firestore database:
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS="/path/to/serviceAccountKey.json"
+yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
+```
+
+Restore a backup:
+```bash
+yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
+```
+
+## Development
+
+For instructions on setting up the development environment, see [DEVELOPMENT.md](./DEVELOPMENT.md).
 
-Similarly, it can be used to restore a backup with:
-`yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
+## Contributing
 
-You likely would want to empty the database before restoring but you can also use flags like overwrite, merge, etc to control the restoration
-rules.
+We welcome contributions! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.

firebase.json

@@ -29,7 +29,7 @@
       "port": 8080
     },
     "hosting": {
-      "port": 5000
+      "port": 5002
     },
     "pubsub": {
       "port": 8085