note: technically this is about two issues. since they are closely related, i chose to file a combined bug report. tell me if they should be split.
Describe the bug
for microbatch, if concurrent_batches: true, then per-batch optimize and set tblproperties statements can collide and fail the batch(es):
- post-run
optimize, can opt out via databricks_skip_optimize: true (docs)
- setting
tblproperties (example), which triggers alter table ... set tblproperties (...)
Steps To Reproduce
- for the
optimize issue
- run a
microbatch model with concurrent_batches: true
- for the
tblproperties issue
- run a
microbatch model with concurrent_batches: true and a tblproperties config
- in our case: in-yaml
+tblproperties: delta.columnMapping.mode: name
Expected behavior
those statements should run only once per model (not batch), after the final batch
System information
The output of dbt --version:
dbt-core 1.11.8
dbt-databricks 1.11.7
The operating system you're using:
macOS 26.4
The output of python --version:
3.12
Additional context
- as workaround, you can opt out of the auto-optimize and avoid the tblproperties config, and run both of them as global post-hooks instead
- concurrent batches were enabled here
note: technically this is about two issues. since they are closely related, i chose to file a combined bug report. tell me if they should be split.
Describe the bug
for
microbatch, ifconcurrent_batches: true, then per-batchoptimizeandset tblpropertiesstatements can collide and fail the batch(es):optimize, can opt out viadatabricks_skip_optimize: true(docs)tblproperties(example), which triggersalter table ... set tblproperties (...)Steps To Reproduce
optimizeissuemicrobatchmodel withconcurrent_batches: truetblpropertiesissuemicrobatchmodel withconcurrent_batches: trueand atblpropertiesconfig+tblproperties: delta.columnMapping.mode: nameExpected behavior
those statements should run only once per model (not batch), after the final batch
System information
The output of
dbt --version:The operating system you're using:
macOS 26.4
The output of
python --version:3.12
Additional context