Future-proof authoring layer: a guide for package maintainers #401
dataders
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Future-proof authoring layer: a guide for package maintainers
Front Matter
Purpose
Enable dbt package maintainers to ensure their packages are compliant with the new, strict JSON schema that ships with both dbt Core
1.10.0
and the new dbt Fusion engine.Goals
After reading, package maintainers will:
Have Questions?
We're here to help! You can:
dbt‑autofix
ordbt-fusion
#package-maintainers
or#dbt-fusion-engine
on dbt SlackWhy are we here?
Key Terms & Background
The best introductory context is the "What is the dbt language?" section of the May 2025 dbt Core roadmap post. You should read that first—there's a lot more context in the linked GitHub discussions and issues.
authoring layer: how you define a dbt project using YAML and SQL
JSON Schema: the format for defining what constitutes valid YAML and JSON. A JSON Schema is what puts red squiggles on your
dbt_project.yml
when there are typos.TL;DR
User experience suffers when dbt cannot correctly and efficiently validate that your project's YAML is valid. We need to clearly define what valid dbt YAML is to ensure the long-term success of the authoring layer for both dbt Core and the dbt Fusion engine.
We cannot continue a world in which:
The standardization of valid dbt YAML happens through a few JSON Schemas.
what’s changing & when
Differences Between Core and Fusion
The new authoring layer is the same across both engines, but it's being implemented differently in Core versus Fusion.
In Core, starting with version
1.10
, the old YAML syntax is still understood but triggers deprecation warnings. These warnings may eventually become errors in future releases.Fusion, however, was built from scratch and only supports the new authoring layer. It has no understanding of the old YAML syntax. Currently, Fusion will warn users about "old YAML" but simply ignores it after displaying the warning. This is a much more severe warning than Core's deprecation warning. This rephrase is somewhat hyperbolic but gets the point across
The result is that there's strong incentives to get packages on the new authoring layer for:
Upcoming Milestones
Fusion will begin treating non-compliant YAML as errors rather than warnings around August 15. This change is beneficial as it prevents unexpected consequences when running packages. However, this date is subject to change pending our impact analysis. There are a few more changes to be made, most of the work is already done (>80%)
Impact on Packages and Their Maintainers
Packages are one of dbt's "killer features" because users can re-use models and macros trusted by the community. This wouldn't be possible without the incredible folks who maintain them!
Unfortunately, package support for Fusion currently needs work—and that's where you come in!
Today, when users run
dbt compile
, they often first see a long list of deprecation warnings related to code they don't own but package maintainers do.That's why we need your help, dear package maintainers! Please ensure your package complies with the new authoring layer so users can have the best experience possible.
How to support the changes
how to fix deprecations, automatically
dbt-autofix README has a table of the things that need to change
The predominant command you invoke to make your package compliant is
dbt-autofix deprecations
. It will fix all the issues it can find in the current directory’s dbt project. Though note that you’ll also have to fix the YAML of anyintegration_test/ project/
if you want to test your project against Fusion (which we encourage!).Today (before you’ve started the compatibility work)
If you have users already asking you about Fusion warnings, but haven’t yet started the work, tell users to:
--include-packages
flagupgrade to latest package version
Any release you ship will not be compatible for dbt Core versions less than 1.10. Accordingly, you should advise users to upgrade to the latest version of your package is you haven’t already. This will make it much easier for them to upgrade the compatible version that you ship.
Short‑term workaround
Tell users to add this after every
dbt deps
run (locally or in CI):It modifies all the packages that have just been cloned from the Hub into the
dbt_packages/
directory. This will resolve most warnings and erros in packages if they have yet to be compliant with the new authoring layer. However, the fixes will be overwritten the next time dbt deps is invokedInstall & upgrade
dbt‑autofix
dbt‑autofix
is an official CLI that rewrites YAML to match the latest JSON schema.We ship fresh binaries on every schema bump, so keep it updated.
ℹ️
uv
installs a tiny, isolated Python runtime so no virtual‑env juggling is required.Suggested release checklist for maintainers
dbt‑autofix deprecations
, commit, and fix any manual items left in the diff.More info
JSON Schema reference
Almost certainly, dbt-autofix should have everything you need, but if you need to see the actual JSON Schema, or are just curious, you can do so. They form the basis of the dbt-autofix behavior
Here’s one example:
They follow the below template format where:
RESOURCE
is eitherdbt-yaml-files
ordbt-project
VERSION
is the fusion versionv2.0.0-beta.34
, buthttps://public.cdn.getdbt.com/fs/latest.json
gives you the latest versionhttps://public.cdn.getdbt.com/fs/schemas/fs-schema-{RESOURCE}-{VERSION}.json
dbt‑autofix
uses those same schemas plus a set of codemods. Run with:Need to automate across many repos?
Wrap
dbt‑autofix
in a GH Actions workflow that opens a PR with the generated diff.Beta Was this translation helpful? Give feedback.
All reactions