Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion dags/miovision_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,20 @@ def miovision_check_dag():
Identify high volume movements missing from intersection_movements table and notify.
'''

check_missing_centreline_ids = SQLCheckOperatorWithReturnValue(
task_id="check_missing_centreline_ids",
sql="select-missing-centreline_ids.sql",
conn_id="miovision_api_bot"
)
check_monitor_intersection_movements.doc_md = '''
Identify centreline_ids missing/outdated in centreline_miovision table and notify.
'''

t_upstream_done >> [
check_distinct_intersection_uid,
check_open_anomalous_ranges,
check_monitor_intersection_movements
check_monitor_intersection_movements,
check_missing_centreline_ids
]

miovision_check_dag()
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
This query looks for intersection legs with no entry or outdated entry in `centreline_miovision`.

Exceptions:
- Leg is labelled as restricted in intersections table
- When leg does not exist in centreline, but is a valid movement, add a null entry in `centreline_miovision` to opt out of notifications.

Check notice on line 6 in volumes/miovision/sql/data_checks/select-missing-centreline_ids.sql

View workflow job for this annotation

GitHub Actions / SQLFluff Lint

SQLFluff

LT05: Line is too long (137 > 100).

Check notice on line 6 in volumes/miovision/sql/data_checks/select-missing-centreline_ids.sql

View workflow job for this annotation

GitHub Actions / SQLFluff Lint

SQLFluff

LT01: Unnecessary trailing whitespace.
*/

WITH valid_legs AS (
SELECT
intersection_uid,
UNNEST(ARRAY[
CASE WHEN e_leg_restricted IS NULL THEN 'E' END,
CASE WHEN n_leg_restricted IS NULL THEN 'N' END,
CASE WHEN s_leg_restricted IS NULL THEN 'S' END,
CASE WHEN w_leg_restricted IS NULL THEN 'W' END
]) AS leg
FROM miovision_api.active_intersections
),

missing AS (
SELECT
vl.intersection_uid,
ai.api_name,
ai.geom,
vl.leg,
CASE
WHEN
cl.centreline_id IS NULL
THEN 'Entry missing from `miovision_api.centreline_miovision`.'
WHEN
latest.centreline_id IS NULL
THEN 'Entry is outdated (no longer in `gis_core.centreline_latest`).'
END AS description
FROM valid_legs AS vl
JOIN miovision_api.active_intersections AS ai USING (intersection_uid)
LEFT JOIN miovision_api.centreline_miovision AS cl USING (intersection_uid, leg)
LEFT JOIN gis_core.centreline_latest AS latest USING (centreline_id)
WHERE
vl.leg IS NOT NULL
AND (
cl.intersection_uid IS NULL --not in table (allowing for nulls)
--entry exists, but is no longer valid
OR (cl.centreline_id IS NOT NULL AND latest.centreline_id IS NULL)
)
ORDER BY intersection_uid
)

SELECT
NOT(COUNT(*) > 0) AS _check,
CASE WHEN COUNT(*) = 1 THEN 'There is ' ELSE 'There are ' END || COUNT(*)
|| ' Miovision legs which are missing/outdated in `miovision_api.centreline_miovision`. '
|| 'See [readme](https://github.com/CityofToronto/bdit_data-sources/tree/master/volumes/miovision/sql/#centreline_miovision).' --noqa
AS summ,
array_agg(
'intersection_uid: `' || intersection_uid
|| '`, leg: `' || leg || '` '
|| description
) AS gaps
FROM missing
5 changes: 4 additions & 1 deletion volumes/miovision/sql/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,10 @@ leg| text | A segment that forms part of a miovision intersection, identified by
**Known issues:**
- 91: Lakeshore and Spadina - Two centrelines matched to West leg are actually legit. One is for the Gardiner off-ramp (Left turns only) and one is for Lakeshore (Thru + Right). They could be differentiated by movement.
- 78: Bloor and Kingsway is a 4 legged intersection, but the south leg is not in the centreline (private road / cemetery entrance).
- 68: Steeles and Jane, N leg is outside of TO.
- 68: Steeles and Jane, N leg is outside of TO (used null entry).
- 105: Bayview and Brickworks - North leg (entry into Brickworks) is not on centreline (used null entry).
- 125: Two north legs (highway ramps)
- 123,124,127: Queensway midblock cameras; legs would not make sense.

### `alerts`

Expand Down
Loading