-
Notifications
You must be signed in to change notification settings - Fork 59
Description
Describe the bug
By default, BigQuery external tables come with a hidden _FILE_NAME
pseudo column [docs] that returns the source file of the record. Since this is not part of the information schema, it triggers an error in dbt Fusion.
dbt0227: No column _FILE_NAME found.
What version of dbt Fusion is this bug in? (find out by running dbt --version
)
dbt-fusion 2.0.0-preview.41
Is this a discrepancy between the dbt Fusion Engine and dbt Core? Check one.
- YES
- NO
To Reproduce
- Create an external table in BigQuery
CREATE EXTERNAL TABLE my_dataset.my_external_table ( column1 STRING, column2 STRING, column3 STRING ) OPTIONS ( format = 'CSV', uris = ['gs://your-gcs-bucket/path/to/your/files/*'], field_delimiter = ',', skip_leading_rows = 1 -- If your CSV has a header row );
- Create a model that queries the external table, including the
_FILE_NAME
pseudo columnselect _FILE_NAME from {{ source('my_dataset', 'my_external_table') }}
- Run
dbtf compile
Expected behavior
_FILE_NAME
should be an acceptable field for external tables despite not being part of the information schema. Perhaps there can be an exception for _FILE_NAME
when table_type = 'EXTERNAL'
. Or a blanket exception for specific pseudo column fields.
Another pseudo column is _PARTITIONTIME
which is used for partitioned tables. Similarly, Snowflake has METADATA columns.
Operating System and CPU Type (please complete the following information):
- Windows
- X86