Skip to content

[BUG] BigQuery external table pseudo column causing "No column found" error #877

@the-kevin-kong

Description

@the-kevin-kong

Describe the bug
By default, BigQuery external tables come with a hidden _FILE_NAME pseudo column [docs] that returns the source file of the record. Since this is not part of the information schema, it triggers an error in dbt Fusion.

dbt0227: No column _FILE_NAME found.

What version of dbt Fusion is this bug in? (find out by running dbt --version)
dbt-fusion 2.0.0-preview.41

Is this a discrepancy between the dbt Fusion Engine and dbt Core? Check one.

  • YES
  • NO

To Reproduce

  1. Create an external table in BigQuery
    CREATE EXTERNAL TABLE my_dataset.my_external_table (
        column1 STRING,
        column2 STRING,
        column3 STRING
    )
    OPTIONS (
        format = 'CSV',
        uris = ['gs://your-gcs-bucket/path/to/your/files/*'],
        field_delimiter = ',',
        skip_leading_rows = 1 -- If your CSV has a header row
    );
  2. Create a model that queries the external table, including the _FILE_NAME pseudo column
    select _FILE_NAME from {{ source('my_dataset', 'my_external_table') }}
  3. Run dbtf compile

Expected behavior
_FILE_NAME should be an acceptable field for external tables despite not being part of the information schema. Perhaps there can be an exception for _FILE_NAME when table_type = 'EXTERNAL'. Or a blanket exception for specific pseudo column fields.

Another pseudo column is _PARTITIONTIME which is used for partitioned tables. Similarly, Snowflake has METADATA columns.

Operating System and CPU Type (please complete the following information):

  • Windows
  • X86

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions