Skip to content

Add deletion of missing records in update_table_from_sql #36

@holgerzer

Description

@holgerzer

Description:
The update_table_from_sql method in src/dremio/_mixins/_table.py currently merges and upserts data from a SQL query into the target table, but does not delete records from the target table that are missing in the source data. This means stale rows can persist after updates.

Proposed improvement:
After the MERGE operation, add a DELETE statement to remove records from the target table that do not exist in the source (e.g., via a temporary view based on the source SQL). Example logic:

SQL
DELETE FROM {target_table_fqn}
WHERE primary_key NOT IN (
SELECT primary_key FROM {temp_view_fqn}
)
The method should:

Create a temp view for the source SQL if needed
Merge/upsert via MERGE INTO
Delete missing records as shown above
Drop the temp view
Benefits:
This will ensure the target table accurately reflects the source data, supporting full sync scenarios.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions