Tech Debt: Upgrade to Airflow 3

[joshgamache]

Airflow 3 is available. We are on version 2.10.5 as of this ticket. As this is a major release, there may be multiple steps required by us in order to handle the upgrade. This ticket will be used to catch sub-tickets for what is needed.

• Analyze existing DAGs to ensure they won't be affected by breaking changes. Airflow has a tool to check for compatibility.
• Upgrade infrastructure to Airflow 3 - Documentation. There is also a new version of the helm chart, which we should upgrade to prior to the Airflow upgrade (the chart supports both versions of Airflow.).
• Review existing DAGs for potential enhancements using upgraded features of Airflow 3

1. Pre-update tasks

   • Update dag-trigger/airflow-dag-trigger.sh to use API-V2.

   • Update helm/cas-airflow/webserver_config.py to handle Auth/FAB changes.

   • Check backups

     • On the pgbackrest pod pgbackrest info to ensure there is a recent full backup.

   • Run DB clean commands on the Airflow pod

     • airflow db clean --clean-before-timestamp '$TIMESTAMP' where $TIMESTAMP is 2025-06-01 for dev/test and 2023-09-01 for prod

   • Check dag processing errors on the Airflow pod

     • airflow dags reserialize

   • Run ruff to do an upgrade check on all dags and correct any identified issues. See the upgrade docs

     • ruff check dag/ --select AIR301 --fix --preview to execute automated fixes
     • ruff check dag/ --select AIR301 --show-fixes --preview to show recommended fixes afterward
       • cas-airflow/dags/
       • cas-bciers/dags/
       • cas-cif/dags/
       • cas-ciip-portal/dags/
       • cas-data-warehouse/dags/
       • cas-ggircs/dags/
       • cas-metabase/dags/

   • Install apache-airflow-providers-standard package, see step 4

     • Update dags using PythonOperator, BatchPythonOperator, TriggerDagRunOperator to import from the new package (backwards compatible with Airflow 2)

   • The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.

   • The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.

   • Update airflow helm chart to 1.18.0

     • No breaking changes. Ensure that values.airflow.airflowVersion is set, as it now defaults to 3.0.2 in the chart values.

2. Update tasks

   • Spin down airflow-scheduler and airflow-webserver deployments.
   • On the pgbackrest pod, run pgbackrest backup to create a cold backup.
   • Push changes to helm values with values.airflow.airflowVersion set to latest 3.0.6
   • Deploy to OpenShift
   • On an airflow pod, run airflow config update to check configuration. Automated fixes will be applied with airflow config update --fix.
   • Ensure database migrations are correct with airflow db migrate.

[joshgamache]

Tracking pre-planning research here. Following steps from the Airflow upgrading docs.

Notes for migration steps

Step 1: Take care of prerequisites

• Make sure that you are on Airflow 2.7 or later. It is recommended to upgrade to latest 2.x and then to Airflow 3.
• Make sure that your Python version is in the supported list. Airflow 3.0.0 supports the following Python versions: Python 3.9, 3.10, 3.11 and 3.12.
• Ensure that you are not using any features or functionality that have been removed in Airflow 3.

Actions

• Check prerequisites
  • Airflow 2.10.5
  • Python 3.12.19
• Check for breaking changes
  • SubDAGs (See Dag List)
    • none present
  • Sequential Executor (config)
    • we use the KubernetesExecutor
  • CeleryKubernetesExecutor and LocalKubernetesExecutor (config)
    • we use the KubernetesExecutor
  • SLAs (Dags)
    • none present
  • Subdir (configs, scripts, ci/cd, cli)
    • not used
  • REST API (/api/v1) (integrations, custom calls)
    • ❗Used in dag-trigger/airflow-dag-trigger.sh
  • Certain Airflow context variables (dags, templates)
    • none used
  • catchup_by_default (dags)
    • not used
  • create_cron_data_intervals (dag schedules)
    • not used
  • SimpleAuth is default. Check if we use FABAuthManager. (config)
    • webserver_config.py uses fab, will need to make sure to import directly

[!TODO] Dags to check

• automated_email_test_dag_cronjob_error cas-airflow
• automated_email_test_dag_local_error cas-airflow
• cas_bciers_process_due_transfer_events cas-bciers
• cas_bciers_test_migrations cas-bciers
• cas_cif_deploy_db cas-cif
• cas_cif_insert_backup_timestamp cas-cif
• cas_cif_test_db_backups cas-cif
• cas_ciip_portal_acme_issue cas-ciip-portal
• cas_ciip_portal_acme_renewal cas-ciip-portal
• cas_ciip_portal_deploy_db cas-ciip-portal
• cas_data_warehouse_import_data_sources cas-data-warehouse
• cas_ggircs_acme_issue cas-ggircs
• cas_ggircs_acme_renewal cas-ggircs
• cas_ggircs_ciip_load_testing_data cas-ggircs
• cas_ggircs_deploy_db cas-ggircs
• cas_ggircs_load_db cas-ggircs
• cas_ggircs_swrs_eccc cas-ggircs
• cas_metabase_prod_test_restore cas-metabase
• fetch_and_save_dag_from_github cas-airflow
• walg_backup_ciip_full cas-airflow, cas-ciip-portal
• walg_backup_ggircs_full cas-airflow, cas-ggircs

Tasks

After investigation actions, the following will need to be changed in code:

• Update dag-trigger/airflow-dag-trigger.sh to use API-V2.
• Update helm/cas-airflow/webserver_config.py to handle Auth/FAB changes.

Step 2: Clean and back up your existing Airflow Instance

• It is highly recommended that you make a backup of your Airflow instance, specifically your Airflow metadata database before starting the migration process.
  • If you do not have a “hot backup” capability for your database, you should do it after shutting down your Airflow instances, so that the backup of your database will be consistent. For example, if you don’t turn off your Airflow instance, the backup of the database will not include all TaskInstances or DagRuns.
  • If you did not make a backup and your migration fails, you might end up in a half-migrated state. This can be caused by, for example, a broken network connection between your Airflow CLI and the database during the migration. Having a backup is an important precaution to avoid problems like this.
• A long running Airflow instance can accumulate a substantial amount of data that are no longer required (for example, old XCom data). Schema changes will be a part of the Airflow 3 upgrade process. These schema changes can take a long time if the database is large. For a faster, safer migration, we recommend that you clean up your Airflow meta-database before the upgrade. You can use the airflow db clean Airflow CLI command to trim your Airflow database.
• Ensure that there are no errors related to dag processing, such as AirflowDagDuplicatedIdException. You should be able to run airflow dags reserialize with no errors. If you have to resolve errors from dag processing, ensure you deploy your changes to your old instance prior to upgrade, and wait until your dags have all been reprocessed (and all errors gone) before you proceed with upgrade.

Actions

• Check backups
• Run DB clean commands
  • airflow db clean
• Check dag processing errors
  • airflow dags reserialize

Warnings flagged when running these commands in dev

• The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
• The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.