-
Notifications
You must be signed in to change notification settings - Fork 3
Description
Description:
The update_table_from_sql method in src/dremio/_mixins/_table.py currently merges and upserts data from a SQL query into the target table, but does not delete records from the target table that are missing in the source data. This means stale rows can persist after updates.
Proposed improvement:
After the MERGE operation, add a DELETE statement to remove records from the target table that do not exist in the source (e.g., via a temporary view based on the source SQL). Example logic:
SQL
DELETE FROM {target_table_fqn}
WHERE primary_key NOT IN (
SELECT primary_key FROM {temp_view_fqn}
)
The method should:
Create a temp view for the source SQL if needed
Merge/upsert via MERGE INTO
Delete missing records as shown above
Drop the temp view
Benefits:
This will ensure the target table accurately reflects the source data, supporting full sync scenarios.