Skip to content

Conversation

RRap0so
Copy link

@RRap0so RRap0so commented Sep 30, 2025

Implements BigQuery-specific relation naming macros to enable concurrent dbt execution without relation name collisions, matching functionality available in PostgreSQL adapter.

resolves #1359

Problem

The BigQuery adapter lacks standardized relation naming macros that are available in other adapters like PostgreSQL. This creates several issues:

  1. Concurrent execution conflicts: When multiple dbt instances run simultaneously (CI/CD pipelines, parallel development environments), temporary relations can have naming conflicts since there's
    no collision prevention mechanism
  2. Adapter inconsistency: PostgreSQL adapter provides make_temp_relation, make_intermediate_relation, make_backup_relation, and make_relation_with_suffix macros, but BigQuery does not, creating
    inconsistent experiences across adapters
  3. Limited materialization patterns: Complex materializations that require standardized temporary, intermediate, and backup relation naming cannot rely on consistent macro interfaces

Before this PR, BigQuery users had no standardized way to generate unique temporary relation names during concurrent dbt executions, leading to potential relation conflicts and failed runs.

Solution

Implemented four BigQuery-specific relation naming macros in adapters.sql that mirror the PostgreSQL adapter's functionality:

  • bigquery__make_relation_with_suffix: Core macro that handles identifier truncation to respect BigQuery's 1024 character limit and optional datetime stamping
  • bigquery__make_temp_relation: Creates temporary relations with unique datetime suffixes for collision prevention
  • bigquery__make_intermediate_relation: Creates intermediate relations without datetime suffixes
  • bigquery__make_backup_relation: Creates backup relations with configurable types (e.g., table → view)

Bonus

PostgreSql adapter wasn't tested, this commit adds tests for both to ensure nothing breaks.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

Copy link

cla-bot bot commented Sep 30, 2025

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @RRap0so

@RRap0so RRap0so marked this pull request as ready for review September 30, 2025 09:03
@RRap0so RRap0so requested a review from a team as a code owner September 30, 2025 09:03
@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Oct 3, 2025
@RRap0so RRap0so marked this pull request as draft October 3, 2025 04:45
Changes the datetime format from %j%H%M%S (9 digits) to %H%M%S%f (12 digits)
to provide microsecond-level uniqueness for temp table names during concurrent
dbt execution, matching PostgreSQL adapter implementation.

- Update bigquery__make_relation_with_suffix to use %H%M%S%f format
- Add functional tests for BigQuery and PostgreSQL to verify 12-digit format
@RRap0so RRap0so force-pushed the RRap0so/make-bigquery-temp-tbls-unique branch from 65eb85c to 07884d1 Compare October 3, 2025 05:33
@RRap0so RRap0so marked this pull request as ready for review October 3, 2025 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add BigQuery relation naming macros for consistency with PostgreSQL adapter

1 participant