Should we avoid having duplicative bicycle classifications in aggregate data?

[radumas]

There's bicycle TMCs (less good data, available since the start of the data) and approach-only (better quality, only available from 2021-07-11 on). Both are available in the "ATR-style" table miovision_api.volumes_15min, which could lead someone to aggregate them together if they joined on the classifications table and grouped by the class_type field (or only released that).

https://github.com/CityofToronto/bdit_data-sources/tree/master/volumes/miovision#classifications

There is work on better documenting data quality issues in the anomalous ranges table, which could include this.

Some potential solutions:

1. Purge the TMC data entirely from the 15min table
2. Create a view that uses TMC data before the 2021-07-11 and then approach data after
3. Create a view that somehow incorporates the anomalous ranges table
4. Something else...

[Nate-Wessel]

I think there is value in both the TMCs and the approach counts. While Miovision has told us the approach counts are preferred, I don't think it's obvious from looking at the data (I did that deep dive to QA bike volumes, recall) that one is actually better than the other. The numbers are pretty far apart in absolute values though the trends are definitely correlated.

That sharp discontinuity between the numbers would make option #2 nonviable IMO. If you're looking for a long-term analysis, I would look only at one or the other. Indeed that's the approach I take for the bike index which uses both 2 and 10 classifications separately (and equally) over the times that we have them. Changing this table would require me to base the index on the raw volumes table.

The best solution here I think is just to be vigilant. Just because this slipped through a review process once doesn't mean it always will. Indeed, it's now fresh in our minds.