Complete full stack environment for dev & CI (#502)

* Diagram of primary models & relationships

* Add helpful commands to README, begin quickstart

* Simplify local dev by renaming local.yml to the default name

* Use env variable to specify API URL for backend

* Add compose config for CI workflows

* Bring back backend tests, fix most of them, disable a few of them

* Create sample data after migrations run

Author: mihow

.envs/.ci/.django

@@ -0,0 +1,18 @@
+# General
+# ------------------------------------------------------------------------------
+USE_DOCKER=yes
+DJANGO_SETTINGS_MODULE="config.settings.local"
+
+# Redis
+# ------------------------------------------------------------------------------
+REDIS_URL=redis://redis:6379/0
+
+DJANGO_CSRF_TRUSTED_ORIGINS=http://localhost:3000,
+
+MINIO_ENDPOINT=http://minio:9000
+MINIO_ROOT_USER=amistorage
+MINIO_ROOT_PASSWORD=amistorage
+MINIO_DEFAULT_BUCKET=ami
+MINIO_STORAGE_USE_HTTPS=False
+MINIO_TEST_BUCKET=ami-test
+MINIO_BROWSER_REDIRECT_URL=http://minio:9001

.envs/.ci/.postgres

@@ -0,0 +1,5 @@
+POSTGRES_HOST=postgres
+POSTGRES_PORT=5432
+POSTGRES_DB=ami-ci
+POSTGRES_USER=4JXkOnTAeDmDyIapSRrGEE
+POSTGRES_PASSWORD=d4xojpnJU3OzPQ0apSCLP1oHR1TYvyMzAlF5KpE9HFL6MPlnbDibwI

.envs/.local/.django

@@ -15,10 +15,6 @@ REDIS_URL=redis://redis:6379/0
 CELERY_FLOWER_USER=QSocnxapfMvzLqJXSsXtnEZqRkBtsmKT
 CELERY_FLOWER_PASSWORD=BEQgmCtgyrFieKNoGTsux9YIye0I7P5Q7vEgfJD2C4jxmtHDetFaE2jhS7K7rxaf
 
-# Monitoring
-NEW_RELIC_CONFIG_FILE=newrelic.ini
-NEW_RELIC_ENVIRONMENT=development
-
 DJANGO_CSRF_TRUSTED_ORIGINS=http://localhost:3000,
 
 MINIO_ENDPOINT=http://minio:9000

.envs/.local/.postgres

@@ -1,5 +1,3 @@
-# PostgreSQL
-# ------------------------------------------------------------------------------
 POSTGRES_HOST=postgres
 POSTGRES_PORT=5432
 POSTGRES_DB=ami

.github/workflows/test.backend.yml

@@ -33,21 +33,20 @@ jobs:
       - name: Run pre-commit
         uses: pre-commit/action@v3.0.1
 
-  # With no caching at all the entire ci process takes 4m 30s to complete!
-  # test:
-  #   runs-on: ubuntu-latest
-  #   steps:
-  #     - name: Checkout Code Repository
-  #       uses: actions/checkout@v4
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Code Repository
+        uses: actions/checkout@v4
 
-  #     - name: Build the Stack
-  #       run: docker-compose -f local.yml build
+      - name: Build the Stack
+        run: docker compose -f docker-compose.ci.yml build
 
-  #     - name: Run DB Migrations
-  #       run: docker-compose -f local.yml run --rm django python manage.py migrate
+      - name: Run DB Migrations
+        run: docker compose -f docker-compose.ci.yml run --rm django python manage.py migrate
 
-  #     - name: Run Django Tests
-  #       run: docker-compose -f local.yml run --rm django python manage.py test
+      - name: Run Django Tests
+        run: docker compose -f docker-compose.ci.yml run --rm django python manage.py test
 
-  #     - name: Tear down the Stack
-  #       run: docker-compose -f local.yml down
+      - name: Tear down the Stack
+        run: docker compose -f docker-compose.ci.yml down

README.md

@@ -1,101 +1,122 @@
 # Automated Monitoring of Insects ML Platform
 
-Platform for processing and reviewing images from automated insect monitoring stations.
+Platform for processing and reviewing images from automated insect monitoring stations. Intended for collaborating on multi-deployment projects, maintaining metadata and orchestrating multiple machine learning pipelines for analysis.
 
-[![Built with Cookiecutter Django](https://img.shields.io/badge/built%20with-Cookiecutter%20Django-ff69b4.svg?logo=cookiecutter)](https://github.com/cookiecutter/cookiecutter-django/)
 [![Black code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
 
-License: MIT
+## Quick Start
 
-## Settings
+The platform uses Docker Compose to run all backend services. To run all services locally, install Docker and run the following command:
 
-Moved to [settings](http://cookiecutter-django.readthedocs.io/en/latest/settings.html).
+    $ docker compose up
 
-## Basic Commands
+Explore the API
 
-### Setting Up Your Users
+- Rest Framework: http://localhost:8000/api/v2/
+- Access the Django admin: http://localhost:8000/admin/
+- OpenAPI / Swagger: http://localhost:8000/api/v2/docs/
 
-- To create a **normal user account**, just go to Sign Up and fill out the form. Once you submit it, you'll see a "Verify Your E-mail Address" page. Go to your console to see a simulated email verification message. Copy the link into your browser. Now the user's email should be verified and ready to go.
+Install and run the frontend:
 
-- To create a **superuser account**, use this command:
+```bash
+# Enter into the ui directory
+cd ui
+# Install Node Version Manager
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
+# Install required Node.js version
+nvm install
+# Install Yarn dependencies
+yarn install
+# Start the frontend
+yarn start
+```
 
-      $ python manage.py createsuperuser
+Visit http://localhost:3000/
 
-For convenience, you can keep your normal user logged in on Chrome and your superuser logged in on Firefox (or similar), so that you can see how the site behaves for both kinds of users.
+_TODO! Make a pre-built frontend available in the Docker compose stack._
 
-### Type checks
+## Development
 
-Running type checks with mypy:
+### Frontend
 
-    $ mypy ami
+#### Dependencies
 
-### Test coverage
+- [Node.js](https://nodejs.org/en/download/)
+- [Yarn](https://yarnpkg.com/getting-started/install)
 
-To run the tests, check your test coverage, and generate an HTML coverage report:
+#### Configuration
 
-    $ coverage run -m pytest
-    $ coverage html
-    $ open htmlcov/index.html
+By default this will try to connect to http://localhost:8000 for the backend API. Use the env var `API_PROXY_TARGET` to change this. You can create multiple `.env` files in the `ui/` directory for different environments or configurations. For example, use `yarn start --mode staging` to load `.env.staging` and point the `API_PROXY_TARGET` to a remote backend.
 
-#### Running tests with pytest
+### Backend
 
-    $ pytest
+#### Dependencies
 
-### Live reloading and Sass CSS compilation
+- [Docker](https://docs.docker.com/get-docker/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
 
-Moved to [Live reloading and SASS compilation](https://cookiecutter-django.readthedocs.io/en/latest/developing-locally.html#sass-compilation-live-reloading).
 
-### Celery
+#### Helpful Commands
 
-This app comes with Celery.
+##### Watch the logs of Django & the backend workers
 
-To run a celery worker:
+   docker compose logs -f django celeryworker
+
+##### Watch the logs of all services:
+
+    docker compose logs -f
+
+#####  Create a super user account:
+
+    docker compose exec django python manage.py createsuperuser
 
-```bash
-cd ami
-celery -A config.celery_app worker -l info
-```
 
-Please note: For Celery's import magic to work, it is important _where_ the celery commands are run. If you are in the same folder with _manage.py_, you should be right.
 
-To run [periodic tasks](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html), you'll need to start the celery beat scheduler service. You can start it as a standalone process:
+##### Run tests
 
 ```bash
-cd ami
-celery -A config.celery_app beat
+docker compose run --rm django python manage.py test
 ```
 
-or you can embed the beat service inside a worker with the `-B` option (not recommended for production use):
+##### Run tests with a specific pattern in the test name
 
 ```bash
-cd ami
-celery -A config.celery_app worker -B -l info
+docker compose run --rm django python manage.py test -k pattern
 ```
 
-### Email Server
+##### Launch the Django shell:
 
-In development, it is often nice to be able to see emails that are being sent from your application. For that reason local SMTP server [MailHog](https://github.com/mailhog/MailHog) with a web interface is available as docker container.
+    docker-compose exec django python manage.py shell
 
-Container mailhog will start automatically when you will run all docker containers.
-Please check [cookiecutter-django Docker documentation](http://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html) for more details how to start all containers.
+    >>> from ami.main.models import SourceImage, Occurrence
+    >>> SourceImage.objects.all(project__name='myproject')
 
-With MailHog running, to view messages that are sent by your application, open your browser and go to `http://127.0.0.1:8025`
+##### Install backend dependencies locally for IDE support (Intellisense, etc):
 
-### Sentry
-
-Sentry is an error logging aggregator service. You can sign up for a free account at <https://sentry.io/signup/?code=cookiecutter> or download and host it yourself.
-The system is set up with reasonable defaults, including 404 logging and integration with the WSGI application.
+```bash
+python -m venv venv
+source venv/bin/activate
+pip install -r requirements/local.txt
+```
 
-You must set the DSN url in production.
+##### Generate OpenAPI schema
 
-## Deployment
+```bash
+docker compose run --rm django python manage.py spectacular --api-version 'api' --format openapi --file ami-openapi-schema.yaml
+```
 
-The following details how to deploy this application.
+##### Generate TypeScript types from OpenAPI schema
 
-### Docker
+```bash
+docker run --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -i /local/ami-openapi-schema.yaml -g typescript-axios -o /local/ui/src/api-schema.d.ts
+```
 
-See detailed [cookiecutter-django Docker documentation](http://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html).
+##### Generate diagram graph of Django models & relationships (Graphviz required)
 
+```bash
+docker compose run --rm django python manage.py graph_models -a -o models.dot --dot
+dot -Tsvg  models.dot > models.svg
+```
 
 ## Project Data Storage
 
@@ -118,3 +139,31 @@ Bucket: ami
 - Upload some test images to a subfolder in the `ami` bucket (one subfolder per deployment)
 - Give the bucket or folder anonymous access using the "Anonymous access" button in the Minio web interface.
 - You _can_ test private buckets and presigned URLs, but you will need to add an entry to your local /etc/hosts file to map the `minio` hostname to localhost.
+
+## Email
+
+The local environment uses the `console` email backend. To view emails sent by the platform, check the console output (run the `docker compose logs -f django celeryworker` command).
+
+## Database
+
+The local environment uses a local PostgreSQL database in a Docker container.
+
+### Backup and Restore
+
+    docker compose run --rm postgres backup
+
+### Reset the database
+
+    docker compose run --rm django python manage.py reset_db
+
+### Show backups
+
+    docker compose run --rm postgres backups
+
+### Restore a backup
+
+    docker compose run --rm postgres restore <backup_file_name>
+
+### Load fixtures with test data
+
+    docker compose run --rm django python manage.py migrate

ami/jobs/tests.py

@@ -3,15 +3,25 @@
 
 from ami.base.serializers import reverse_with_params
 from ami.jobs.models import Job, JobProgress, JobState
-from ami.main.models import Project
+from ami.main.models import Project, SourceImageCollection
+from ami.ml.models import Pipeline
 from ami.users.models import User
 
 # from rich import print
 
 
 class TestJobProgress(TestCase):
     def setUp(self):
-        self.project = Project.objects.create(name="Job test")
+        self.project = Project.objects.create(name="Test project")
+        self.source_image_collection = SourceImageCollection.objects.create(
+            name="Test collection",
+            project=self.project,
+        )
+        self.pipeline = Pipeline.objects.create(
+            name="Test ML pipeline",
+            description="Test ML pipeline",
+        )
+        self.pipeline.projects.add(self.project)
 
     def test_create_job(self):
         job = Job.objects.create(project=self.project, name="Test job")
@@ -20,7 +30,13 @@ def test_create_job(self):
         self.assertEqual(job.progress.stages, [])
 
     def test_create_job_with_delay(self):
-        job = Job.objects.create(project=self.project, name="Test job", delay=1)
+        job = Job.objects.create(
+            project=self.project,
+            name="Test job",
+            delay=1,
+            pipeline=self.pipeline,
+            source_image_collection=self.source_image_collection,
+        )
         self.assertEqual(job.progress.stages[0].key, "delay")
         self.assertEqual(job.progress.stages[0].progress, 0)
         self.assertEqual(job.progress.stages[0].status, JobState.CREATED)
@@ -45,7 +61,17 @@ class TestJobView(APITestCase):
 
     def setUp(self):
         self.project = Project.objects.create(name="Jobs Test Project")
-        self.job = Job.objects.create(project=self.project, name="Test job", delay=0)
+        self.job = Job.objects.create(
+            project=self.project,
+            name="Test job",
+            delay=0,
+            pipeline=Pipeline.objects.create(name="Test pipeline"),
+            source_image_collection=SourceImageCollection.objects.create(
+                name="Test collection",
+                project=self.project,
+            ),
+        )
+
         self.user = User.objects.create_user(  # type: ignore
             email="testuser@insectai.org",
             is_staff=True,
@@ -82,9 +108,12 @@ def test_create_job(self):
         # request = self.factory.post(jobs_create_url, {"project": self.project.pk, "name": "Test job 2"})
         self.client.force_authenticate(user=self.user)
         job_data = {
-            "project_id": self.project.pk,
+            "project_id": self.job.project.pk,
             "name": "Test job 2",
+            "pipeline_id": self.job.pipeline.pk,  # type: ignore
+            "collection_id": self.job.source_image_collection.pk,  # type: ignore
             "delay": 0,
+            "start_now": False,
         }
         resp = self.client.post(jobs_create_url, job_data)
         self.client.force_authenticate(user=None)
@@ -94,7 +123,8 @@ def test_create_job(self):
         self.assertEqual(data["name"], "Test job 2")
         # self.assertEqual(data["progress"]["status"], "CREATED")
         progress = JobProgress(**data["progress"])
-        self.assertEqual(progress.summary.status, JobState.CREATED)
+
+        self.assertEqual(progress.summary.status, JobState.SUCCESS)
 
     def test_run_job(self):
         jobs_run_url = reverse_with_params("api:job-run", args=[self.job.pk], params={"no_async": True})

ami/main/apps.py

@@ -1,7 +1,13 @@
 from django.apps import AppConfig
+from django.db.models.signals import post_migrate
 from django.utils.translation import gettext_lazy as _
 
 
-class UsersConfig(AppConfig):
+class MainConfig(AppConfig):
     name = "ami.main"
     verbose_name = _("Main")
+
+    def ready(self):
+        from tests.fixtures.signals import setup_complete_test_project
+
+        post_migrate.connect(setup_complete_test_project, sender=self)