Skip to content

Conversation

gabrielwol
Copy link
Collaborator

@gabrielwol gabrielwol commented May 6, 2025

What this pull request accomplishes:

  1. Updated constraints/requirements files to Airflow 3
  2. Committed, Airflow-3ed and renamed some HERE dags (biggest change was to sql templates)
    • dags/queen_park_refresh.py
    • dags/rapidto_aggregation.py
    • dags/lake_shore_signal_timing_refresh.py
    • dags/congestion_aggregate_temp.py (this shows up as an edit in the diff, but it was actually a remove/add 😞)
  3. Removed long unused DAGs: check_rescu, check_miovision, pull_weather, pull_interventions_dag, citywide_tti_aggregate
  4. Taskflow-ify assets_pull DAG + rename to file traffic_signals_dag.py
  5. Other misc import changes

See below testing:

  • The DAG folder on this branch is passing ruff Airflor 3 checks without any warnings.
  • The /data/airflow/dags folder is only failing on airflow.sdk.get_current_context issue, which is solved only in Airflow 3; (could not test this change in import location).
(airflow_venv) airflow@ip:~/dev_scripts/_/dags$ git status
On branch 1211-fix-airflow-3-breaking-changes
Your branch is up to date with 'origin/1211-fix-airflow-3-breaking-changes'.

nothing to commit, working tree clean
(airflow_venv) airflow@ip:~/dev_scripts/_/dags$ ruff check . --select AIR301 --preview
All checks passed!
(airflow_venv) airflow@ip:~/dev_scripts/_/dags$ cd ~/dags
(airflow_venv) airflow@ip:~/dags$ ruff check . --select AIR301 --preview
replicators.py:49:27: AIR301 `airflow.operators.python.get_current_context` is removed in Airflow 3.0
   |
47 |             def check_tbl_not_empty(conn_id, tables, tbl_index, **context):
48 |                 #name mapped task
49 |                 context = get_current_context()
   |                           ^^^^^^^^^^^^^^^^^^^ AIR301
50 |                 context["dest_table_name"] = tables[tbl_index]
   |
   = help: Use `airflow.sdk.get_current_context` instead

vz_google_sheets.py:89:19: AIR301 `airflow.operators.python.get_current_context` is removed in Airflow 3.0
   |
87 |         #get name for mapped task 
88 |         from airflow.operators.python import get_current_context
89 |         context = get_current_context()
   |                   ^^^^^^^^^^^^^^^^^^^ AIR301
90 |         context["task_year"] = spreadsheet["year"]
   |
   = help: Use `airflow.sdk.get_current_context` instead

wys_pull.py:182:23: AIR301 `airflow.operators.python.get_current_context` is removed in Airflow 3.0
    |
180 |         def read_google_sheet(ward, **context):
181 |             #name mapped task
182 |             context = get_current_context()
    |                       ^^^^^^^^^^^^^^^^^^^ AIR301
183 |             context["ward_no"] = f"ward_{ward[3]}"
    |
    = help: Use `airflow.sdk.get_current_context` instead

Found 3 errors.

Issue(s) this solves:

What, in particular, needs to reviewed:

What needs to be done by a sysadmin after this PR is merged

E.g.: these tables need to be migrated/created in the production schema.

@gabrielwol gabrielwol linked an issue May 6, 2025 that may be closed by this pull request
@gabrielwol gabrielwol self-assigned this May 6, 2025
@gabrielwol gabrielwol marked this pull request as ready for review May 7, 2025 15:33
@gabrielwol gabrielwol force-pushed the 1211-fix-airflow-3-breaking-changes branch from 94cf6a3 to 59445cb Compare May 7, 2025 15:34
@gabrielwol gabrielwol force-pushed the 1211-fix-airflow-3-breaking-changes branch 2 times, most recently from ffb073e to c8523fe Compare May 27, 2025 20:27
@gabrielwol gabrielwol force-pushed the 1211-fix-airflow-3-breaking-changes branch from 359bdde to 11d094a Compare June 2, 2025 16:36
@gabrielwol gabrielwol force-pushed the 1211-fix-airflow-3-breaking-changes branch from 309324e to 61b0ce9 Compare July 11, 2025 13:14
@gabrielwol gabrielwol force-pushed the 1211-fix-airflow-3-breaking-changes branch from 0c94e6d to b6a0f6a Compare August 6, 2025 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow 3 - breaking changes

1 participant