Fusion Diaries: first stats from State-Aware Orchestration, Cloud CLI creds update, complex column types fixes (2025-07-08)

[joellabes]

Just the one release this week (🇺🇸🎇), but it was a big one:

• 104 issues closed from the internal and public repos
• 1 jumbo-sized beta released
• Unblocked our internal data team’s largest project to successfully run all of its jobs on Fusion, giving the first concrete stats on how State-Aware Orchestration reduces warehouse spend.

Read on for the details of the latest beta, a first glance at incremental recompiles outside of the extension, my GitHub Codespaces question, a chance to talk to Elias this week, and how much we saved when we turned on State-Aware Orchestration. Plus, of course, the meme.

🆕 What’s been released this week

dbt Fusion engine releases

Run dbt system update to get the latest and greatest version of the Fusion engine - don’t let me see you filing bugs against beta 11!

2.0.0-beta.34 (Monday 7 July):

• support execution_project for BigQuery in profiles.yml
• compile Databricks sources containing complex data types
• correctly merge grants using the +select syntax
• support arbitrary keys in dbt list --output json
• better error handling when manifest download fails

and more:

• handle missing relation for get_columns_in_relation
• unit tests won't error on unused complex column types
• LSP autocomplete seeds' column names
• Add support to compile union by name + a ton of other SQL comprehension fixes
• Respect DBT_TARGET env var as an alternative to --target
• dbt.date_spine no longer prevents static analysis
• Skip cache when validating snapshot schema shape
• and even more

VS Code Extension, dbt Studio, LSP

• Laid foundations for more contextual autocomplete suggestions, by using Fusion's SQL parser in more places
• Fixed an issue in dbt Studio that sometimes caused the LSP to crash
• Reduced baseline memory consumption of LSP during compilation

dbt-autofix and JSON Schema

• Add logic to fix more files with invalid YAML that don’t raise errors in dbt Core but do so in Fusion
• Resolve package deprecations (dbt-autofix deprecations --include-packages) for local experimentation. Remember that these fixes will be overwritten the next time you run dbt deps.
• The JSON logs now include dbt’s deprecation class name (e.g. CustomKeyInConfigDeprecation)

🚧 Work in progress

• Continuing to knock out user-reported bugs
• Implementing a robust caching system across compilations usable in both the LSP and CLI context. We expect this to dramatically decrease subsequent compile times on large scale projects.
• Big VS Code extension improvements which should land throughout the coming month:
  • Cache some introspective queries to increase LSP performance
  • Context-aware column names in autocomplete suggestions
  • Preview a model even if an unrelated model has a compiler error
  • Maybe: update a file’s status and warnings as you type, instead of only on save
  • Improving memory usage in the LSP when calculating column-level lineage for large projects
• Continuing to help make Hub-hosted packages compatible with the Fusion engine.
• Redshift support

🐉 Here be dragons

• An update to everyone clamouring for Cloud CLI credentials support: we hear you! We’re working on ways to meet performance and security requirements - check out the GitHub issue for more. Remember that you can create a local profiles.yml by running dbt init in the meantime.
• If you’re running the dbt Fusion engine inside of a GitHub codespace, is it working for you? I’ve seen a couple of reports of the VS Code extension not working even though the LSP reports itself as active. Let me know in the comments (or make a new issue) if you have a repro case.

🐁 Stuff you should click

• @bthomson22 wrote about the warehouse spend reductions we’ve achieved in our primary dbt project this year. One relevant point for this audience: we're saving ~9% just by turning on State-Aware Orchestration in its default settings, on top of the ~16% we'd already found by using the cost management dashboard 📉
• @eliasdefaria is hosting a Fusion lunch and learn with Joe Reis on Discord this Thursday at 11am Mountain Time.

🏁 Made it to the meme

[sophieml]

Preview a model even if an unrelated model has a compiler error

Should we expect to see this land in dbt Studio as well? Or more generally, is it safe to assume feature parity between the VSCode extension and dbt Studio?

[joellabes]

Great question!

Short answer: you can expect to see this in both dbt Studio and the VSCE, but it's because we'll fix it in both places. The LSP is common between both tools, but the LSP isn't the part that needed fixed here.

Long answer: the LSP server is responsible for giving those tools information about the current state of the project (e.g. "there's a CTE called my_cte starting at line 5 character 1 and going until line 28 character 12", or "models/my_first_model.sql has an error on line 11 because it references a column that doesn't exist").

But the LSP doesn't run queries - it leaves that to the main Fusion binary.

dbt Studio and the VSCE are independently responsible for deciding what to do with the information the LSP gives (showing red squigglies, offering a Preview CTE lens, having a Problems tab...). They're also responsible for invoking the main dbt Fusion binary when they need to run other dbt commands, because the LSP can't execute SQL.

So there's three changes we have to make:

• The main fusion binary will support --inline and --select in the same command (select makes it possible to ignore unrelated issues)
• The VSCE will start including --inline and --select in Preview commands
• dbt Studio will start including --inline and --select in Preview commands

[randypitcherii]

Hoooo buddy I’ve been waiting for this one.

Time to relearn how to install dbtf and give it a go on my databricks project 💪