[DOCS-11631] BigQuery Cost Allocation #30722

rtrieu · 2025-07-25T20:08:41Z

What does this PR do? What is the motivation?

Adds a new Cost Allocation landing page
Reorganizes the Container Cost Allocation section to accommodate new BigQuery page and existing providers
Adds a new BigQuery Cost Allocation page

Merge instructions

Merge readiness:

Ready for merge

For Datadog employees:

Your branch name MUST follow the <name>/<description> convention and include the forward slash (/). Without this format, your pull request will not pass CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.

If your branch doesn't follow this format, rename it or create a new branch and PR.

[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.

Additional notes

github-actions · 2025-07-25T20:09:01Z

📝 Documentation Team Review Required

This pull request requires approval from the @DataDog/documentation team before it can be merged.

Please ensure your changes follow our documentation guidelines and wait for a team member to review and approve your changes.

github-actions · 2025-07-25T20:12:06Z

Preview links (active after the `build_preview` check completes)

New or renamed files

Renamed files

https://docs-staging.datadoghq.com/rtrieu/kaylawan2/docs-11631-bigquery-cost-allocation/cloud_cost_management/cost_allocation/container_cost_allocation

elihschiff

Something about the ordering and flow feels off to me

What do about the Allocating costs section starting with

Compute allocation
Compute
Query-level tag extraction
the untitled text under Allocating costs
Understanding idle costs
Storage
Not sure if I missed anything

Great work, this is really good. I have a number of small comments but overall this is really great work and hits all the key points I would have wanted to know about

elihschiff · 2025-07-31T20:59:44Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+[1]: https://app.datadoghq.com/dashboard/ecm-es8-agw/bigquery-allocation
+[2]: https://app.datadoghq.com/cost/setup 


These are both hard coded for the us1 site, maybe somebody from the docs team knows if there is a better way to do this?

See here for an example

documentation/content/en/cloud_cost_management/custom_allocation_rules.md

Line 104 in 330f9ac

[1]: /metrics/#querying-metrics

you can basically drop everything up to /cost/setup

Got it, thank you!

elihschiff · 2025-07-31T21:00:36Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+
+| Feature | Requirements |
+|---|---|
+| Retrieve tags from labels of a query | Supported without monitoring or reservations |


Maybe say set up GCP CCM costs or something like that
So it is clear you have to have CCM GCP costs set up but you dont need anything else

Not everyone on this page might be a ccm customer

elihschiff · 2025-07-31T21:01:59Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+2. Enable BigQuery monitoring in your Google Cloud project.
+3. For reservation cost allocation, configure BigQuery reservations in your project.


What does a user need to explicitly do to set up the bigquery logs? We should either list out the setup exactly or link to a page in the docs and say set this up

elihschiff · 2025-07-31T21:03:23Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+Datadog Cloud Cost Management (CCM) automatically allocates the costs of your Google BigQuery resources to individual queries and workloads. Use cost metrics enriched with tags from queries, projects, and reservations to visualize BigQuery workload costs in the context of your entire cloud bill.
+
+## Resources
+CCM allocates costs for BigQuery resources for both types of Analysis costs (on-demand and reservation-based). CCM displays costs for resources including query-level analysis, storage, and data transfer on the [**BigQuery** dashboard][1].


Can we maybe add a link here to the official bigquery docs on their various pricing models
Not everyone using CCM might know the difference between the two

elihschiff · 2025-07-31T21:05:27Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+- `total_slot_usage`: The total slot-seconds used across all queries in the project's reservations
+- `total_project_reservation_cost`: The total cost of the reservations in a given project for the time period
+
+Any difference between the total billed reservation cost and the sum of allocated query costs is categorized as a project's `cluster_idle` cost, representing unused reservation capacity. 


Lets be more explicit here. It is tagged with allocated_spend_type:cluster_idle and maybe also mention usage as well somewhere here for contrast

elihschiff · 2025-07-31T21:13:18Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+
+CCM extracts the following tags from BigQuery query logs via the Data Observability platform:
+
+| Tag | Description |


Any other tags we can add? What about the tag that has the raw query id, also maybe mention the tag for the region and project id as well because those are super important for bigquery

Those 2 they might know already but customers in general can never find the tags they need, so I dont think it is a bad thing to be a little extra verbose here with the top tags they would care about

elihschiff · 2025-07-31T21:15:10Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+
+{{< partial name="whats-next/whats-next.html" >}}
+
+[1]: https://app.datadoghq.com/dashboard/ecm-es8-agw/bigquery-allocation


Lets make sure we do not merge this PR until this dashboard is pushed to production and the link is corrected to the version that works in everyone's account. Otherwise this is just a link to the internal test version of the dashboard

elihschiff · 2025-07-31T21:16:18Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+
+Datadog Cloud Cost Management (CCM) automatically allocates the costs of your Google BigQuery resources to individual queries and workloads. Use cost metrics enriched with tags from queries, projects, and reservations to visualize BigQuery workload costs in the context of your entire cloud bill.
+
+## Resources


I am not sure what this section is adding? Maybe it can be combined with the overview if there is anything important, otherwise I feel like the rest of the doc covers this.
just my opinion, feel free to ignore me

elihschiff · 2025-07-31T21:20:46Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+| Active | Includes any table or table partition that has been modified in the last 90 days |
+| Long-term | Includes any table or table partition that has not been modified for 90 consecutive days. The price of storage for that table automatically drops by approximately 50%. There is no difference in performance, durability, or availability between active and long-term storage |


What tag key is this? It is good to in general try and provide a key+value so a user can find it themselves

Its the google_usage_type tag! I will add that and also add the allocated_spend_type tag for usage and cluster_idle above!

elihschiff · 2025-07-31T21:23:16Z

content/en/cloud_cost_management/cost_allocation/_index.md

+    text: "BigQuery Cost Allocation"
+---
+
+## Overview


I have no idea where this should go but can you add a section on bigquery labels? This is a feature bigquery provides where you can add a label to your bigquery queries and then the labels show up in CCM
It is super powerful for our users and not many know about it

It is documented here https://cloud.google.com/bigquery/docs/adding-labels

I'll add a section for labels! This is definitely a useful feature 👍

congx4 · 2025-08-01T16:22:37Z

content/en/cloud_cost_management/cost_allocation/bigquery.md

+| `project_id` | GCP project ID where the BigQuery resource or job is located |
+| `google_location` | The specific Google Cloud region or zone where BigQuery resources are deployed (e.g., us-central1, europe-west1, asia-southeast1) |
+| `resource_name` | Full Google Cloud resource identifier |


are project_id and resource_name directly from bills? But I agree these tags are useful for customers to understand the cost allocation.

Yeah, they are directly from the bills -> CCM doesn't add them. I will split these into a subsection to clarify 👍

rtrieu added 2 commits July 25, 2025 13:00

[DOCS-11631] BigQuery Cost Allocation

2add607

landing page content

d49ef0c

rtrieu added the WORK IN PROGRESS No review needed, it's a wip ;) label Jul 25, 2025

rtrieu requested review from a team as code owners July 25, 2025 20:08

github-actions bot added the Architecture Everything related to the Doc backend label Jul 25, 2025

rtrieu and others added 15 commits July 25, 2025 13:14

remove duplicate info

c70e515

Update bigquery.md

e262f81

Update bigquery.md

2bc33b5

Update bigquery.md

bf9be64

Update bigquery.md

9b33597

Update bigquery.md

e2630b9

Update bigquery.md

8277f73

Update bigquery.md

758e02a

Update bigquery.md

0f8556c

Update bigquery.md

c99f1b9

Update bigquery.md

0237fca

Update bigquery.md

a54a9ab

correct url

e48fcd2

Update bigquery.md

8ad33fe

Update bigquery.md

7702178

elihschiff reviewed Jul 31, 2025

View reviewed changes

kaylawan2 added 3 commits July 31, 2025 18:24

some pr fixed

28665e4

Update bigquery.md

9f14e0b

Update bigquery.md

a179e66

congx4 reviewed Aug 1, 2025

View reviewed changes

kaylawan2 added 2 commits August 1, 2025 13:58

Update bigquery.md

3818a0f

Update bigquery.md

7614992

kaylawan2 added 2 commits August 1, 2025 14:08

Update bigquery.md

187f9e8

Update bigquery.md

1dbb799

		[1]: https://app.datadoghq.com/dashboard/ecm-es8-agw/bigquery-allocation
		[2]: https://app.datadoghq.com/cost/setup

		2. Enable BigQuery monitoring in your Google Cloud project.
		3. For reservation cost allocation, configure BigQuery reservations in your project.


		CCM extracts the following tags from BigQuery query logs via the Data Observability platform:

		\| Tag \| Description \|


		{{< partial name="whats-next/whats-next.html" >}}

		[1]: https://app.datadoghq.com/dashboard/ecm-es8-agw/bigquery-allocation


		Datadog Cloud Cost Management (CCM) automatically allocates the costs of your Google BigQuery resources to individual queries and workloads. Use cost metrics enriched with tags from queries, projects, and reservations to visualize BigQuery workload costs in the context of your entire cloud bill.

		## Resources

		\| Active \| Includes any table or table partition that has been modified in the last 90 days \|
		\| Long-term \| Includes any table or table partition that has not been modified for 90 consecutive days. The price of storage for that table automatically drops by approximately 50%. There is no difference in performance, durability, or availability between active and long-term storage \|

[DOCS-11631] BigQuery Cost Allocation #30722

Are you sure you want to change the base?

[DOCS-11631] BigQuery Cost Allocation #30722

Conversation

rtrieu commented Jul 25, 2025

What does this PR do? What is the motivation?

Merge instructions

Additional notes

Uh oh!

github-actions bot commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Documentation Team Review Required

Uh oh!

github-actions bot commented Jul 25, 2025

Preview links (active after the build_preview check completes)

New or renamed files

Renamed files

Uh oh!

elihschiff left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jul 25, 2025 •

edited

Loading

Preview links (active after the `build_preview` check completes)