Skip to content

microbatch: run 'optimize' and 'set tblproperties' after last batch only #1443

@data-blade

Description

@data-blade

note: technically this is about two issues. since they are closely related, i chose to file a combined bug report. tell me if they should be split.

Describe the bug

for microbatch, if concurrent_batches: true, then per-batch optimize and set tblproperties statements can collide and fail the batch(es):

  • post-run optimize, can opt out via databricks_skip_optimize: true (docs)
  • setting tblproperties (example), which triggers alter table ... set tblproperties (...)

Steps To Reproduce

  • for the optimize issue
    • run a microbatch model with concurrent_batches: true
  • for the tblproperties issue
    • run a microbatch model with concurrent_batches: true and a tblproperties config
    • in our case: in-yaml +tblproperties: delta.columnMapping.mode: name

Expected behavior

those statements should run only once per model (not batch), after the final batch

System information

The output of dbt --version:

dbt-core 1.11.8
dbt-databricks 1.11.7

The operating system you're using:
macOS 26.4

The output of python --version:
3.12

Additional context

  • as workaround, you can opt out of the auto-optimize and avoid the tblproperties config, and run both of them as global post-hooks instead
  • concurrent batches were enabled here

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions