First draft of CD workflow #108

cc-a · 2025-01-22T14:14:29Z

A deployment pipeline intended for use with the Beta launch:

Features:

The deployment workflow is run on pushes to develop and main. In order to pass the sha of the published docker image the deployment is run is called from the publish workflow with the image tag being passed as an input argument.
The deployment workflow may also be triggered manually in order to support manual redeployment and disaster recovery scenarios. In this case the image tag is provided when the workflow is triggered.
The docker image is provided via sha to ensure that the relevant Kubernetes deployments are updated with the new image.
The same workflow is run for both the develop and main branches but different values are provided for the secrets and environment variables via Github Actions environments. For manual deployments the environment is provided as an input value.
The ref of the commit to use for the Helm chart is set via environment variable to allow controlled updates of the helm chart being used in testing/production.

Developer Checklist

Developers should review and confirm each of these items before requesting review

Code meets acceptance criteria from issue
Unit tests are written and all pass
User Test Scripts (if required) are written and have been run through
Code documentation and related non-code documentation has all been updated

Reviewer Checklist

Reviewers should review and confirm each of these items before approval
If there are multiple reviewers, this section can be duplicated for each reviewer

Code meets acceptance criteria from issue
Unit tests are written and all pass
User Test Scripts (if required) are written and have been run through
Code documentation and related non-code documentation has all been updated
Migation has been created and tested

Testing

List user test scripts that need to be run

List any non-unit test scripts that need to be run

Steven-Eardley

These latest changes make sense to me - we're using the raw helm command so it's acting equivalently to our dev environments

.github/workflows/deploy.yml

Copilot

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

.github/workflows/deploy.yml

alexdewar

Seems sensible. Is the plan to use helm for development as well as production eventually?

alexdewar · 2025-04-08T14:01:26Z

.github/workflows/deploy.yml

+          -f ${{ vars.CHART_OVERRIDE_PATH }} \
+          -n invenio --install --create-namespace \
+          --set invenio.secret_key="${{ secrets.INVENIO_SECRET_KEY }}" \
+          --set invenio.security_login_salt="${{ secrets.INVENIO_SECURITY_LOGIN_SALT }}" \
+          --set invenio.csrf_secret_salt="${{ secrets.INVENIO_CSRF_SECRET_SALT }}" \
+          --set invenio.extraConfig.ICL_OAUTH_CLIENT_ID=${{ vars.ICL_OAUTH_CLIENT_ID }} \
+          --set invenio.extraConfig.ICL_OAUTH_CLIENT_SECRET="${{ secrets.ICL_OAUTH_CLIENT_SECRET }}" \
+          --set invenio.extraConfig.ICL_OAUTH_WELL_KNOWN_URL=${{ vars.ICL_OAUTH_WELL_KNOWN_URL }} \
+          --set rabbitmq.auth.password="${{ secrets.RABBITMQ_AUTH_PASSWORD }}" \
+          --set postgresql.auth.password="${{ secrets.POSTGRESQL_AUTH_PASSWORD }}" \
+          --set invenio.hostname=${{ vars.HOSTNAME }} \
+          --set web.image=${{ inputs.image_tag_with_sha }} \
+          --set worker.image=${{ inputs.image_tag_with_sha }} \
+          --set invenio.datacite.password="${{ secrets.DATACITE_PASSWORD }}" \
+          --set invenio.datacite.username=${{ vars.DATACITE_USERNAME }} \
+          --set invenio.extraConfig.INVENIO_DATACITE_PREFIX=${{ vars.DATACITE_PREFIX }} \
+          --set-string invenio.extraConfig.INVENIO_DATACITE_TEST_MODE=True \
+          --set invenio.extraConfig.INVENIO_MAIL_USERNAME=${{ vars.EMAIL_USERNAME }} \
+          --set invenio.extraConfig.INVENIO_MAIL_PASSWORD="${{ secrets.MAIL_PASSWORD }}" \
+          --set persistence.size=${{ vars.FILES_STORAGE_SIZE }} \
+          --set-string invenio.extraConfig.INVENIO_MAIL_USE_TLS=True \
+          --set-string invenio.extraConfig.INVENIO_MAIL_DEFAULT_SENDER=${{ vars.EMAIL_USERNAME }}


How about some more indentation for readability:

Suggested change

-f ${{ vars.CHART_OVERRIDE_PATH }} \

-n invenio --install --create-namespace \

--set invenio.secret_key="${{ secrets.INVENIO_SECRET_KEY }}" \

--set invenio.security_login_salt="${{ secrets.INVENIO_SECURITY_LOGIN_SALT }}" \

--set invenio.csrf_secret_salt="${{ secrets.INVENIO_CSRF_SECRET_SALT }}" \

--set invenio.extraConfig.ICL_OAUTH_CLIENT_ID=${{ vars.ICL_OAUTH_CLIENT_ID }} \

--set invenio.extraConfig.ICL_OAUTH_CLIENT_SECRET="${{ secrets.ICL_OAUTH_CLIENT_SECRET }}" \

--set invenio.extraConfig.ICL_OAUTH_WELL_KNOWN_URL=${{ vars.ICL_OAUTH_WELL_KNOWN_URL }} \

--set rabbitmq.auth.password="${{ secrets.RABBITMQ_AUTH_PASSWORD }}" \

--set postgresql.auth.password="${{ secrets.POSTGRESQL_AUTH_PASSWORD }}" \

--set invenio.hostname=${{ vars.HOSTNAME }} \

--set web.image=${{ inputs.image_tag_with_sha }} \

--set worker.image=${{ inputs.image_tag_with_sha }} \

--set invenio.datacite.password="${{ secrets.DATACITE_PASSWORD }}" \

--set invenio.datacite.username=${{ vars.DATACITE_USERNAME }} \

--set invenio.extraConfig.INVENIO_DATACITE_PREFIX=${{ vars.DATACITE_PREFIX }} \

--set-string invenio.extraConfig.INVENIO_DATACITE_TEST_MODE=True \

--set invenio.extraConfig.INVENIO_MAIL_USERNAME=${{ vars.EMAIL_USERNAME }} \

--set invenio.extraConfig.INVENIO_MAIL_PASSWORD="${{ secrets.MAIL_PASSWORD }}" \

--set persistence.size=${{ vars.FILES_STORAGE_SIZE }} \

--set-string invenio.extraConfig.INVENIO_MAIL_USE_TLS=True \

--set-string invenio.extraConfig.INVENIO_MAIL_DEFAULT_SENDER=${{ vars.EMAIL_USERNAME }}

-f ${{ vars.CHART_OVERRIDE_PATH }} \

-n invenio --install --create-namespace \

--set invenio.secret_key="${{ secrets.INVENIO_SECRET_KEY }}" \

--set invenio.security_login_salt="${{ secrets.INVENIO_SECURITY_LOGIN_SALT }}" \

--set invenio.csrf_secret_salt="${{ secrets.INVENIO_CSRF_SECRET_SALT }}" \

--set invenio.extraConfig.ICL_OAUTH_CLIENT_ID=${{ vars.ICL_OAUTH_CLIENT_ID }} \

--set invenio.extraConfig.ICL_OAUTH_CLIENT_SECRET="${{ secrets.ICL_OAUTH_CLIENT_SECRET }}" \

--set invenio.extraConfig.ICL_OAUTH_WELL_KNOWN_URL=${{ vars.ICL_OAUTH_WELL_KNOWN_URL }} \

--set rabbitmq.auth.password="${{ secrets.RABBITMQ_AUTH_PASSWORD }}" \

--set postgresql.auth.password="${{ secrets.POSTGRESQL_AUTH_PASSWORD }}" \

--set invenio.hostname=${{ vars.HOSTNAME }} \

--set web.image=${{ inputs.image_tag_with_sha }} \

--set worker.image=${{ inputs.image_tag_with_sha }} \

--set invenio.datacite.password="${{ secrets.DATACITE_PASSWORD }}" \

--set invenio.datacite.username=${{ vars.DATACITE_USERNAME }} \

--set invenio.extraConfig.INVENIO_DATACITE_PREFIX=${{ vars.DATACITE_PREFIX }} \

--set-string invenio.extraConfig.INVENIO_DATACITE_TEST_MODE=True \

--set invenio.extraConfig.INVENIO_MAIL_USERNAME=${{ vars.EMAIL_USERNAME }} \

--set invenio.extraConfig.INVENIO_MAIL_PASSWORD="${{ secrets.MAIL_PASSWORD }}" \

--set persistence.size=${{ vars.FILES_STORAGE_SIZE }} \

--set-string invenio.extraConfig.INVENIO_MAIL_USE_TLS=True \

--set-string invenio.extraConfig.INVENIO_MAIL_DEFAULT_SENDER=${{ vars.EMAIL_USERNAME }}

alexdewar · 2025-04-08T14:04:04Z

.github/workflows/ci.yml

@@ -8,7 +8,7 @@ jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4


I was thinking dependabot normally does these... but it seems we don't actually have dependabot enabled for this repo. Do we want to add it?

cc-a · 2025-04-28T09:50:10Z

Would be great if you could take a look when you get the chance @Steven-Eardley

Steven-Eardley · 2025-05-01T15:35:01Z

.github/workflows/deploy.yml

+          --set persistence.size=${{ vars.FILES_STORAGE_SIZE }} \
+          --set-string invenio.extraConfig.INVENIO_MAIL_USE_TLS=True \
+          --set-string invenio.extraConfig.INVENIO_MAIL_DEFAULT_SENDER=${{ vars.EMAIL_USERNAME }} \
+          --set worker.resources.requests.cpu=${{ vars.WORKER_REQUESTS_CPU }} \


I understand this is what Trevor asked for - my approach would probably be to set these in the values.override and not necessarily bring them all the way out to helm upgrade just because of the verbosity of this approach. But since it was a request, let's carry on

Steven-Eardley · 2025-05-01T15:35:41Z

.github/workflows/publish.yml

@@ -9,6 +9,8 @@ on:
 jobs:
  build-and-publish:
    runs-on: ubuntu-latest
+    outputs:
+      image_tag_with_sha: ${{ steps.export-data.outputs.image_tag_with_sha }}


handy for a proper rolling upgrade

copilot spellcheck Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

cc-a marked this pull request as draft January 22, 2025 14:14

cc-a had a problem deploying to develop March 27, 2025 17:39 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from d9abc71 to d7b02d2 Compare March 27, 2025 18:00

cc-a had a problem deploying to develop March 27, 2025 18:00 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from d7b02d2 to 4ee95ec Compare March 27, 2025 18:02

cc-a had a problem deploying to develop March 27, 2025 18:02 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from 4ee95ec to 6b2736b Compare March 27, 2025 18:08

cc-a had a problem deploying to develop March 27, 2025 18:08 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from 6b2736b to 3f1502d Compare March 27, 2025 20:44

cc-a had a problem deploying to develop March 27, 2025 20:44 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from 3f1502d to db82446 Compare March 27, 2025 20:51

cc-a had a problem deploying to develop March 27, 2025 20:51 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from db82446 to dd21b1a Compare March 28, 2025 09:44

cc-a had a problem deploying to develop March 28, 2025 09:44 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from dd21b1a to effbe70 Compare March 28, 2025 10:49

cc-a had a problem deploying to develop March 28, 2025 10:49 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from effbe70 to d17633b Compare March 28, 2025 10:52

cc-a had a problem deploying to develop March 28, 2025 10:52 — with GitHub Actions Failure

cc-a force-pushed the feature/actions_deploy branch from d17633b to 7fd059b Compare March 28, 2025 10:57

cc-a had a problem deploying to develop March 28, 2025 10:57 — with GitHub Actions Failure

cc-a had a problem deploying to develop March 28, 2025 10:59 — with GitHub Actions Failure

cc-a temporarily deployed to develop March 28, 2025 11:00 — with GitHub Actions Inactive

cc-a temporarily deployed to develop March 28, 2025 11:03 — with GitHub Actions Inactive

Steven-Eardley approved these changes Apr 1, 2025

View reviewed changes

Steven-Eardley marked this pull request as ready for review April 1, 2025 11:41

Steven-Eardley temporarily deployed to develop April 1, 2025 11:45 — with GitHub Actions Inactive

Steven-Eardley reviewed Apr 1, 2025

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

Steven-Eardley mentioned this pull request Apr 8, 2025

Deploying helm with CI mislabelling #129

Open

cc-a requested review from Steven-Eardley and jamesturner246 April 8, 2025 11:44

cc-a requested review from alexdewar, Sahil590 and Copilot April 8, 2025 11:44

Copilot AI reviewed Apr 8, 2025

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

alexdewar approved these changes Apr 8, 2025

View reviewed changes

cc-a linked an issue Apr 10, 2025 that may be closed by this pull request

CI/CD pipeline for automated deployment #17

Closed

cc-a force-pushed the feature/actions_deploy branch from 8aeedc0 to d2bafb6 Compare April 15, 2025 14:43

Steven-Eardley approved these changes May 1, 2025

View reviewed changes

Steven-Eardley and others added 9 commits May 1, 2025 17:35

CI workflows

cf3c5bc

First pass at fully configured worflow

883c31f

Try different helm action

05132b6

Add --create-namespace flag for helm upgrade

0247dbe

Call deploy workflow from publish workflow

39a56c5

Change remaining helm overrides to environment variables

cb5a4cf

Set environment from workflow ref_name

d8549f0

Allow configuration of worker pod resources

a2b9438

Update .github/workflows/deploy.yml

8628a8d

copilot spellcheck Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

cc-a force-pushed the feature/actions_deploy branch from 9aa2293 to 8628a8d Compare May 1, 2025 16:35

cc-a merged commit 511c56d into develop May 1, 2025
3 checks passed

cc-a deleted the feature/actions_deploy branch May 1, 2025 16:41

github-project-automation bot moved this to Done in Fair Data Repository May 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First draft of CD workflow #108

First draft of CD workflow #108

cc-a commented Jan 22, 2025 •

edited

Loading

Steven-Eardley left a comment

Copilot AI left a comment

alexdewar left a comment

alexdewar Apr 8, 2025

alexdewar Apr 8, 2025

cc-a commented Apr 28, 2025

Steven-Eardley May 1, 2025

Steven-Eardley May 1, 2025

First draft of CD workflow #108

First draft of CD workflow #108

Conversation

cc-a commented Jan 22, 2025 • edited Loading

Developer Checklist

Reviewer Checklist

Testing

Steven-Eardley left a comment

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

alexdewar left a comment

Choose a reason for hiding this comment

alexdewar Apr 8, 2025

Choose a reason for hiding this comment

alexdewar Apr 8, 2025

Choose a reason for hiding this comment

cc-a commented Apr 28, 2025

Steven-Eardley May 1, 2025

Choose a reason for hiding this comment

Steven-Eardley May 1, 2025

Choose a reason for hiding this comment

cc-a commented Jan 22, 2025 •

edited

Loading