Implement TiDB database monitoring #20826

takaidohigasi · 2025-07-23T02:33:40Z

What does this PR do?

extend part of MySQL database monitoring capability to TiDB.
#20811

note: wait event etc. does not work for TiDB

Motivation

database monitoring is very useful tool and we will use it with other metrics with MCP server.
If we have query baseline information in datadog, we are very happy.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Discussion and Some Review Point

Overview

Feature	MySQL Table	TiDB Table	Key Differences
Statement Stats	performance_schema.events_statements_summary_by_digest	information_schema.cluster_statements_summary	TiDB aggregates across all nodes, different column names
Current Activity	performance_schema.events_statements_current	information_schema.cluster_processlist	There isn't information for wait event for TiDB, so always N/A for TiDB wait_event
Execution Plans	Real-time EXPLAIN	Pre-collected in PLAN column(refer to information_schema.cluster_statements_summary PLAN)	TiDB stores plans with statements
Variables	performance_schema.global_variables	SHOW VARIABLES only	TiDB has no performance_schema

metric collection adjustments

Skips incompatible metrics for TiDB:
* InnoDB metrics - TiDB uses TiKV storage engine, no information_schema.innodb_* tables
* MySQL replication metrics - TiDB uses Raft consensus, no SHOW SLAVE STATUS
* MyISAM key cache metrics - TiDB doesn't support MyISAM, no Key_* status variables
* Binary log metrics - TiDB has different binlog implementation
Performance schema check returns False immediately for TiDB (TiDB has no performance_schema database)

compatibility with existing MySQL

explain (EventType: dbm-samples)
- to show explain result from GUI without error, we output both MySQL compatible one and TiDB specific one
- MySQL output explain with the following output, and we create normalized_plan from TiDB explan plan (_convert_tidb_plan_to_mysql_format) in plan.definition. access_type should be MySQL compatible one (e.g. const, ref, range, index... not Point_Get, Batch_Point_Get) for current datadog server implementation, so we are converting access_type for now.
- we also TiDB execution plan in query_block.tidb_execution_tree and query_block. tidb_original_text_plan. which we can confirm on RawJson in GUI

: MySQL output explain
{
  "query_block": {
    "select_id": 1,
    "cost_info": {
      "query_cost": "0.00"
    },
    "table": {
      "table_name": "items",
      "access_type": "const",
      "rows_examined_per_scan": 1,
      "rows_produced_per_join": 1,
      "key": "PRIMARY(id)",
      "used_key_parts": [
        "PRIMARY(id)"
      ],
      "operator_info": "table:items, clustered index:PRIMARY(id)",
      "execution_info": "time:551.7µs, loops:2, Get:{num_rpc:1, total_time:508.9µs}, time_detail: {total_process_time: 33.7µs, total_wait_time: 38.2µs, total_kv_read_wall_time: 74.5µs, tikv_wall_time: 99µs}, scan_detail: {total_process_keys: 1, total_process_keys_size: 227, total_keys: 1, get_snapshot_time: 5.9µs, rocksdb: {block: {cache_hit_count: 8}}}",
      "operator_id": "Point_Get_1"
    },
    "tidb_execution_tree": {
      "id": "Point_Get_1",
      "taskType": "root",
      "estRows": "1",
      "operatorInfo": "table:items, clustered index:PRIMARY(id)",
      "actRows": "1",
      "executionInfo": "time:551.7µs, loops:2, Get:{num_rpc:1, total_time:508.9µs}, time_detail: {total_process_time: 33.7µs, total_wait_time: 38.2µs, total_kv_read_wall_time: 74.5µs, tikv_wall_time: 99µs}, scan_detail: {total_process_keys: 1, total_process_keys_size: 227, total_keys: 1, get_snapshot_time: 5.9µs, rocksdb: {block: {cache_hit_count: 8}}}"
    },
    "tidb_original_text_plan": "\tid         \ttask\testRows\toperator info                           \tactRows\texecution info                                                                                                                                                                                                                                                                                                                                   \tmemory\tdisk\n\tPoint_Get_1\troot\t1      \ttable:items, clustered index:PRIMARY(id)\t1      \ttime:551.7µs, loops:2, Get:{num_rpc:1, total_time:508.9µs}, time_detail: {total_process_time: 33.7µs, total_wait_time: 38.2µs, total_kv_read_wall_time: 74.5µs, tikv_wall_time: 99µs}, scan_detail: {total_process_keys: 1, total_process_keys_size: 227, total_keys: 1, get_snapshot_time: 5.9µs, rocksdb: {block: {cache_hit_count: 8}}}\tN/A   \tN/A"
  }
}

TiDB also have EXPLAIN format=tidb_json, but we need to original query for it and cost higher, so I use statement_summary plan information. note that which is not realtime, and detected plan for the period. (default 30min)

https://docs.pingcap.com/tidb/stable/sql-statement-explain/

takaidohigasi · 2025-07-23T02:35:22Z

Run python ddev/src/ddev/utils/scripts/check_pr.py changelog --diff-file /tmp/diff --pr-file "$GITHUB_EVENT_PATH"  --repo "core"
Package "mysql" has changes that require a changelog. Please run `ddev release changelog new` to add it.
Error: Package "mysql" has changes that require a changelog. Please run `ddev release changelog new` to add it.
Error: Process completed with exit code 1.

github-actions · 2025-07-23T07:53:50Z

⚠️ Major version bump
The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

codecov · 2025-07-23T08:15:35Z

Codecov Report

❌ Patch coverage is 81.68215% with 355 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.04%. Comparing base (4840571) to head (dadcf97).
⚠️ Report is 43 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	`?`
cassandra	`?`
confluent_platform	`?`
hive	`?`
hivemq	`?`
hudi	`?`
ignite	`?`
jboss_wildfly	`?`
kafka	`?`
mysql	`87.12% <81.68%> (-2.25%)`	⬇️
presto	`?`
solr	`?`
tomcat	`?`
weblogic	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

takaidohigasi · 2025-07-23T13:45:59Z

please tell me if it's better not to devide files for tidb.

takaidohigasi · 2025-07-24T04:51:47Z

sorry, I want to add some function so I changed state to draft once.

iadjivon

Hi there, thanks for this PR. I have added some suggestions here. Let us know if you have any questions.

iadjivon · 2025-07-25T18:08:47Z

mysql/README.md

@@ -68,6 +68,24 @@ mysql> GRANT PROCESS ON *.* TO 'datadog'@'%';
 Query OK, 0 rows affected (0.00 sec)
 ```

+##### TiDB-specific setup
+
+For TiDB databases, the user setup is similar but with some differences:


Suggested change

For TiDB databases, the user setup is similar but with some differences:

For TiDB databases, the user setup is similar but with some differences:

Can you clarify what the set up is similar to? Is it to other Databases?

I'll add similart to other databases like MySQL, MariaDB, and so on.

clarified in
48bec2a

iadjivon · 2025-07-25T18:09:24Z

mysql/README.md

@@ -141,6 +159,27 @@ For a full list of available configuration options, see the [sample `mysql.d/con

 To collect `extra_performance_metrics`, your MySQL server must have `performance_schema` enabled - otherwise set `extra_performance_metrics` to `false`. For more information on `performance_schema`, see [MySQL Performance Schema Quick Start][9].

+##### TiDB Configuration


Suggested change

##### TiDB Configuration

##### TiDB configuration

fixed in
48bec2a

iadjivon · 2025-07-25T18:12:22Z

mysql/README.md

@@ -551,6 +598,24 @@ The check does not collect all metrics by default. Set the following boolean con
 | ---------------------- | ----------- |
 | mysql.info.schema.size | GAUGE       |

+#### TiDB limitations
+
+When using this integration with TiDB, be aware of the following limitations:


Suggested change

When using this integration with TiDB, be aware of the following limitations:

When using this integration with TiDB, be aware of the following limitations:

Which integration is this referencing? If it is the InnoDB integration, would this change work:

When using he InnoDB integration with TiDB, be aware of the following limitations:

this might not be satisfy your request. but fixed for some.

48bec2a

In fact I don't mention for the specific integration...
thanks for the review

iadjivon · 2025-07-25T18:15:06Z

mysql/README.md

+For Database Monitoring features:
+- Query samples and explain plans are collected from `cluster_statements_summary` with some approximations
+- Wait events are not available as TiDB doesn't track them in the same way as MySQL
+- Some query metrics are approximated (e.g., rows examined is estimated from keys processed)


Suggested change

- Some query metrics are approximated (e.g., rows examined is estimated from keys processed)

- Some query metrics are approximated (for example, rows examined is estimated from keys processed)

thanks, applied in
48bec2a

iadjivon · 2025-07-25T18:15:34Z

mysql/README.md

@@ -571,6 +636,15 @@ See [service_checks.json][22] for a list of service checks provided by this inte
 - [Database user lacks privileges][29]
 - [How to collect metrics with a SQL Stored Procedure?][30]

+### TiDB-specific troubleshooting
+
+**Missing metrics**: If you see warnings about missing InnoDB or performance_schema metrics when monitoring TiDB:


Suggested change

**Missing metrics**: If you see warnings about missing InnoDB or performance_schema metrics when monitoring TiDB:

**Missing metrics**: If you see warnings about missing InnoDB or `performance_schema` metrics when monitoring TiDB:

takaidohigasi · 2025-07-25T21:44:31Z

please tell me if it's better not to devide files for tidb.
→ I merged the file

takaidohigasi · 2025-07-28T07:19:42Z

mysql/datadog_checks/mysql/statement_samples.py

+            self._log.debug("Failed to parse TiDB plan to JSON: %s", e)
+            return json.dumps({"raw_plan": plan_text, "parse_error": str(e)})
+
+    def _convert_tidb_plan_to_mysql_format(self, tidb_plan_json):


I will fix some to this function. It turned out this does not cover some cases.

- Map all TiDB operator types to MySQL-compatible access types - Add support for additional TiDB operators: - Batch_Point_Get → const (multiple point lookups) - TableRowIDScan → ref (rowid lookup) - TableScan/IndexScan → ALL/index (generic scans) - IndexLookUp → ref (non-unique index lookup) - Improve mapping accuracy for better MySQL compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

takaidohigasi · 2025-07-30T09:25:36Z

@iadjivon I'm ready again. Would you please review the doc again?

temporal-github-worker-1 bot added agent/review-requested ecosystems/review-requested product/review-requested labels Jul 23, 2025

datadog-agent-integrations-bot bot added the integration/mysql label Jul 23, 2025

takaidohigasi mentioned this pull request Jul 23, 2025

enable TiDB for Database Monitoring #20811

Open

takaidohigasi changed the title ~~Implement tidb database monitoring~~ Implement TiDB database monitoring Jul 23, 2025

takaidohigasi force-pushed the implement-tidb-database-monitoring branch 3 times, most recently from 04ae784 to 27e20c0 Compare July 23, 2025 08:11

takaidohigasi force-pushed the implement-tidb-database-monitoring branch 2 times, most recently from ec97801 to 1a9eab2 Compare July 23, 2025 10:01

takaidohigasi marked this pull request as ready for review July 23, 2025 10:01

takaidohigasi requested review from a team as code owners July 23, 2025 10:01

datadog-agent-integrations-bot bot added team/agent-integrations team/database-monitoring-agent labels Jul 23, 2025

takaidohigasi marked this pull request as draft July 24, 2025 04:51

takaidohigasi force-pushed the implement-tidb-database-monitoring branch from 6fe482f to cdb5120 Compare July 24, 2025 22:11

temporal-github-worker-1 bot added the docs/review-requested label Jul 25, 2025

datadog-agent-integrations-bot bot added the documentation label Jul 25, 2025

takaidohigasi force-pushed the implement-tidb-database-monitoring branch from a119005 to 8a29ca0 Compare July 25, 2025 01:14

This comment was marked as resolved.

Sign in to view

Add TiDB database monitoring support

a8c9bbd

takaidohigasi force-pushed the implement-tidb-database-monitoring branch from dcfe488 to a8c9bbd Compare July 25, 2025 05:19

takaidohigasi marked this pull request as ready for review July 25, 2025 05:51

takaidohigasi requested review from a team as code owners July 25, 2025 05:51

datadog-agent-integrations-bot bot added team/documentation team/database-monitoring labels Jul 25, 2025

iadjivon requested changes Jul 25, 2025

View reviewed changes

temporal-github-worker-1 bot added docs/requested-changes and removed docs/review-requested labels Jul 25, 2025

improve documentation for TiDB

48bec2a

takaidohigasi commented Jul 28, 2025

View reviewed changes

takaidohigasi requested a review from iadjivon July 28, 2025 07:30

iadjivon added the do-not-merge/WIP label Jul 28, 2025

takaidohigasi force-pushed the implement-tidb-database-monitoring branch 3 times, most recently from bd3ecca to 5ea69a1 Compare July 30, 2025 02:28

takaidohigasi force-pushed the implement-tidb-database-monitoring branch from e72fbe8 to dadcf97 Compare July 30, 2025 08:27

	For TiDB databases, the user setup is similar but with some differences:
	For TiDB databases, the user setup is similar but with some differences:

		@@ -141,6 +159,27 @@ For a full list of available configuration options, see the [sample `mysql.d/con

		To collect `extra_performance_metrics`, your MySQL server must have `performance_schema` enabled - otherwise set `extra_performance_metrics` to `false`. For more information on `performance_schema`, see [MySQL Performance Schema Quick Start][9].

		##### TiDB Configuration

	When using this integration with TiDB, be aware of the following limitations:
	When using this integration with TiDB, be aware of the following limitations:

	- Some query metrics are approximated (e.g., rows examined is estimated from keys processed)
	- Some query metrics are approximated (for example, rows examined is estimated from keys processed)

	Missing metrics: If you see warnings about missing InnoDB or performance_schema metrics when monitoring TiDB:
	Missing metrics: If you see warnings about missing InnoDB or `performance_schema` metrics when monitoring TiDB:

Implement TiDB database monitoring #20826

Are you sure you want to change the base?

Implement TiDB database monitoring #20826

Conversation

takaidohigasi commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Discussion and Some Review Point

Overview

metric collection adjustments

compatibility with existing MySQL

Uh oh!

takaidohigasi commented Jul 23, 2025

Uh oh!

github-actions bot commented Jul 23, 2025

Uh oh!

codecov bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

takaidohigasi commented Jul 23, 2025

Uh oh!

takaidohigasi commented Jul 24, 2025

Uh oh!

This comment was marked as resolved.

iadjivon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

takaidohigasi commented Jul 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

takaidohigasi commented Jul 30, 2025

Uh oh!

Uh oh!

takaidohigasi commented Jul 23, 2025 •

edited

Loading

codecov bot commented Jul 23, 2025 •

edited

Loading