Skip to content

Commit 99b0bd1

Browse files
author
Docsite Preview Bot
committed
Preview PR pingcap/docs#20597 and this preview is triggered from commit pingcap/docs@cb0050d
1 parent 0540f38 commit 99b0bd1

File tree

4 files changed

+30
-31
lines changed

4 files changed

+30
-31
lines changed

markdown-pages/en/tidb/master/ticdc/monitor-ticdc.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -15,25 +15,25 @@ cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://ro
1515

1616
## Metrics for TiCDC in the new architecture
1717

18-
The monitoring dashboard **TiCDC-New-Arch** for [TiCDC New Architecture](/ticdc/ticdc-architecture.md) is not yet integrated into TiUP. To view the relevant monitoring information on Grafana, you need to manually import the TiCDC monitoring metrics file:
18+
The monitoring dashboard **TiCDC-New-Arch** for [TiCDC New Architecture](/ticdc/ticdc-architecture.md) is not managed by TiUP yet. To view the related monitoring data on Grafana, you need to manually import the TiCDC monitoring metrics file:
1919

2020
1. Download the monitoring metrics file for TiCDC in the new architecture:
2121

2222
```shell
2323
wget https://raw.githubusercontent.com/pingcap/ticdc/refs/heads/release-8.5/metrics/grafana/ticdc_new_arch.json
2424
```
2525

26-
2. Import the downloaded metrics file on the Grafana page.
26+
2. Import the downloaded metrics file on Grafana:
2727

2828
![Import Metrics File](/media/ticdc/ticdc-new-arch-import-grafana.png)
2929

3030
The monitoring dashboard for TiCDC new architecture mainly includes the following sections:
3131

32-
- [**Summary**](#summary)The summary information of the TiCDC cluster
33-
- [**Server**](#server)The summary information of TiKV nodes and TiCDC nodes in the TiDB cluster
34-
- [**Log Puller**](#log-puller)The detailed information of the TiCDC Log Puller module
35-
- [**Event Store**](#event-store)The detailed information of the TiCDC Event Store module
36-
- [**Sink**](#sink)The detailed information of the TiCDC Sink module
32+
- [**Summary**](#summary): The summary information of the TiCDC cluster
33+
- [**Server**](#server): The summary information of TiKV nodes and TiCDC nodes in the TiDB cluster
34+
- [**Log Puller**](#log-puller): The detailed information of the TiCDC Log Puller module
35+
- [**Event Store**](#event-store): The detailed information of the TiCDC Event Store module
36+
- [**Sink**](#sink): The detailed information of the TiCDC Sink module
3737

3838
### Summary
3939

@@ -62,12 +62,12 @@ The following is an example of the **Server** panel:
6262
The description of each metric in the **Server** panel is as follows:
6363

6464
- Uptime: The time for which TiKV nodes and TiCDC nodes have been running
65-
- Goroutine Count: The number of Goroutines in TiCDC nodes
65+
- Goroutine Count: The number of Goroutines on TiCDC nodes
6666
- Open FD Count: The number of file handles opened by TiCDC nodes
6767
- CPU Usage: The CPU usage of TiCDC nodes
6868
- Memory Usage: The memory usage of TiCDC nodes
69-
- Ownership History: The historical record of Owner nodes in the TiCDC cluster
70-
- PD Leader History: The historical record of PD Leader nodes in the upstream TiDB cluster
69+
- Ownership History: The historical records of Owner nodes in the TiCDC cluster
70+
- PD Leader History: The historical records of PD Leader nodes in the upstream TiDB cluster
7171

7272
### Log Puller
7373

@@ -99,11 +99,11 @@ The description of each metric in the **Event Store** panel is as follows:
9999
- Input Event Count/s: The number of events that Event Store processes per second
100100
- Input Bytes/s: The amount of data that Event Store processes per second
101101
- Write Requests/s: The number of write requests that Event Store executes per second
102-
- Write Worker Busy Ratio: The ratio of IO time to total runtime for Event Store write threads
102+
- Write Worker Busy Ratio: The ratio of I/O time to total runtime for Event Store write threads
103103
- Compressed Rows/s: The number of rows compressed per second in Event Store (triggered only when row size exceeds the threshold)
104104
- Write Duration: The time consumed by Event Store write operations
105105
- Write Batch Size: The batch size of a single write operation
106-
- Write Batch Event Count: The number of rows included in a single write batch
106+
- Write Batch Event Count: The number of row change events included in a single write batch
107107
- Data Size On Disk: The total data size that Event Store occupies on disk
108108
- Data Size In Memory: The total data size that Event Store occupies in memory
109109
- Scan Requests/s: The number of scan requests that Event Store executes per second
@@ -118,9 +118,9 @@ The following is an example of the **Sink** panel:
118118
The description of each metric in the **Sink** panel is as follows:
119119

120120
- Output Row Batch Count: The average number of rows per DML batch written by the Sink module
121-
- Output Row Count(per second): The number of DML rows written to downstream per second
121+
- Output Row Count (per second): The number of DML rows written to downstream per second
122122
- Output DDL Executing Duration: The time consumed by executing DDL events for the changefeed on the current node
123-
- Sink Error Count / m: The number of error messages reported per minute by the Sink module
123+
- Sink Error Count / m: The number of errors reported per minute by the Sink module
124124
- Output DDL Count / Minutes: The number of DDLs executed per minute for the changefeed on the current node
125125

126126
## Metrics for TiCDC in the classic architecture
@@ -182,7 +182,7 @@ The following is an example of the **Changefeed** panel:
182182

183183
- Changefeed catch-up ETA: The estimated time needed for the replication task to catch up with the upstream cluster data. When the upstream write speed is faster than the TiCDC replication speed, the metric might be extremely large. Because TiCDC replication speed is subject to many factors, this metric is for reference only and might not be the actual replication time.
184184

185-
## Events
185+
### Events
186186

187187
The following is an example of the **Events** panel:
188188

markdown-pages/en/tidb/master/ticdc/ticdc-architecture.md

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,27 +5,27 @@ summary: Introduces the features, architectural design, deployment guide, and no
55

66
# TiCDC New Architecture
77

8-
Starting from TiCDC v8.5.4-release.1, TiCDC introduces a new architecture that improves the performance, scalability, and stability of real-time data replication while reducing resource costs. This new architecture redesigns TiCDC core components and optimizes its data processing workflows, offering the following advantages:
8+
Starting from [TiCDC v8.5.4-release.1](https://github.com/pingcap/ticdc/releases/tag/v8.5.4-release.1), TiCDC introduces a new architecture that improves the performance, scalability, and stability of real-time data replication while reducing resource costs. This new architecture redesigns TiCDC core components and optimizes its data processing workflows, offering the following advantages:
99

1010
- **Higher single-node performance**: a single node can replicate up to 500,000 tables, achieving replication throughput of up to 190 MiB/s on a single node in wide table scenarios.
1111
- **Enhanced scalability**: cluster replication capability scales almost linearly. A single cluster can expand to over 100 nodes, support more than 10,000 changefeeds, and replicate millions of tables within a single changefeed.
1212
- **Improved stability**: changefeed latency is reduced and performance is more stable in scenarios with high traffic, frequent DDL operations, and cluster scaling events. Resource isolation and priority scheduling reduce interference between multiple changefeed tasks.
13-
- **Lower resource costs**: with improved resource utilization and reduced redundancy, CPU and memory resource usage is reduced by up to 50% in typical scenarios.
13+
- **Lower resource costs**: with improved resource utilization and reduced redundancy, CPU and memory resource usage can decrease by up to 50% in typical scenarios.
1414

1515
## Architectural design
1616

1717
![TiCDC New Architecture](/media/ticdc/ticdc-new-arch-1.png)
1818

1919
The TiCDC new architecture consists of two core components: Log Service and Downstream Adapter.
2020

21-
- Log Service: as the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disk. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter.
21+
- Log Service: as the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disks. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter.
2222
- Downstream Adapter: as the downstream data replication adaptation layer, Downstream Adapter handles user-initiated changefeed operations. It schedules and generates related replication tasks, fetches data from the Log Service, and replicates the fetched data to downstream systems.
2323

2424
By separating the architecture into stateful and stateless components, the TiCDC new architecture significantly improves system scalability, reliability, and flexibility. Log Service, as the stateful component, focuses on data acquisition, sorting, and storage. Decoupling it from changefeed processing logic enables data sharing across multiple changefeeds, effectively improving resource utilization and reducing system overhead. Downstream Adapter, as the stateless component, uses a lightweight scheduling mechanism that allows quick migration of replication tasks between instances. It can dynamically adjust the splitting and merging of replication tasks based on workload changes, ensuring low-latency replication in various scenarios.
2525

2626
## Comparison between the classic and new architectures
2727

28-
The new architecture is designed to address common issues during continuous system scaling, such as performance bottlenecks, insufficient stability, and limited scalability. Compared with the classic architecture, the new architecture achieves significant optimizations in the following key dimensions:
28+
The new architecture is designed to address common issues during continuous system scaling, such as performance bottlenecks, insufficient stability, and limited scalability. Compared with the [classic architecture](/ticdc/ticdc-classic-architecture.md), the new architecture achieves significant optimizations in the following key dimensions:
2929

3030
| Feature | TiCDC classic architecture | TiCDC new architecture |
3131
| ------------------------ | ---------------------------------------- | ---------------------------------------- |
@@ -46,8 +46,9 @@ The new architecture is designed to address common issues during continuous syst
4646
If your workload meets any of the following conditions, it is recommended to switch from the [classic TiCDC architecture](/ticdc/ticdc-classic-architecture.md) to the new architecture for better performance and stability:
4747

4848
- Bottlenecks in incremental scan performance: incremental scan tasks take an excessively long time to complete, leading to continuously increasing replication latency.
49+
- Ultra-high traffic scenarios: the total changefeed traffic exceeds 700 MiB/s.
4950
- Single tables with high-throughput writes in MySQL sink: the target table has **only one primary key or non-null unique key**.
50-
- Large-scale table replication: the number of tables to be replicated exceeds 100,000.
51+
- Large-scale table replication: the number of tables to be replicated exceeds 100,000.
5152
- Frequent DDL operations causing latency: frequent execution of DDL statements significantly increases replication latency.
5253

5354
## New features
@@ -61,12 +62,6 @@ When this feature is enabled, TiCDC automatically splits and distributes tables
6162

6263
## Compatibility
6364

64-
### Handling of server-level errors
65-
66-
In the classic TiCDC architecture, when a server-level error (such as `ErrEtcdSessionDone`) occurs, TiCDC automatically restarts the main thread without exiting the process.
67-
68-
In the new architecture, when a server-level error occurs, the TiCDC process exits directly and relies on orchestration tools such as TiUP or TiDB Operator to automatically restart the TiCDC instance.
69-
7065
### DDL progress tracking table
7166

7267
In the TiCDC classic architecture, DDL replication operations are strictly serial, thus the replication progress can be tracked only using the changefeed's `CheckpointTs`. In the new architecture, however, TiCDC replicates DDL changes for different tables in parallel whenever possible to improve DDL replication efficiency. To accurately record the DDL replication progress of each table in a downstream MySQL-compatible database, the TiCDC new architecture creates a table named `tidb_cdc.ddl_ts_v1` in the downstream database, specifically storing the DDL replication progress information of the changefeed.
@@ -114,6 +109,8 @@ You can deploy the TiCDC new architecture using TiUP or TiDB Operator.
114109
<SimpleTab>
115110
<div label="TiUP">
116111

112+
To deploy the TiCDC new architecture using TiUP, take the following steps:
113+
117114
1. If your TiDB cluster does not have TiCDC nodes yet, refer to [Scale out a TiCDC cluster](/scale-tidb-using-tiup.md#scale-out-a-ticdc-cluster) to add new TiCDC nodes in the cluster. Otherwise, skip this step.
118115

119116
2. Download the TiCDC binary package for the new architecture.
@@ -161,6 +158,8 @@ You can deploy the TiCDC new architecture using TiUP or TiDB Operator.
161158
</div>
162159
<div label="TiDB Operator">
163160

161+
To deploy the TiCDC new architecture using TiDB Operator, take the following steps:
162+
164163
- If your TiDB cluster does not include a TiCDC component, refer to [Add TiCDC to an existing TiDB cluster](https://docs.pingcap.com/tidb-in-kubernetes/stable/deploy-ticdc/#add-ticdc-to-an-existing-tidb-cluster) to add new TiCDC nodes. When doing so, specify the TiCDC image version as the new architecture version in the cluster configuration file.
165164

166165
For example:
@@ -240,6 +239,6 @@ For more command usage methods and details, see [Manage Changefeeds](/ticdc/ticd
240239

241240
## Monitoring
242241

243-
Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard in Grafana, you need to manually import the [TiCDC monitoring metrics file](https://github.com/pingcap/ticdc/blob/master/metrics/grafana/ticdc_new_arch.json).
242+
Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard on Grafana, you need to manually import the [TiCDC monitoring metrics file](https://github.com/pingcap/ticdc/blob/master/metrics/grafana/ticdc_new_arch.json).
244243

245244
For detailed descriptions of each monitoring metric, see [Metrics for TiCDC in the new architecture](/ticdc/monitor-ticdc.md#metrics-for-ticdc-in-the-new-architecture).

markdown-pages/en/tidb/master/ticdc/ticdc-classic-architecture.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ This document describes the classic architecture and working principles of TiCDC
1010
> **Note:**
1111
>
1212
> - This document applies to TiCDC versions earlier than v8.5.4-release.1.
13-
> - Starting from v8.5.4-release.1, TiCDC introduces a new architecture that improves real-time data replication performance, scalability, and stability while reducing resource costs. For more information, see [TiCDC New Architecture](/ticdc/ticdc-classic-architecture.md).
13+
> - Starting from v8.5.4-release.1, TiCDC introduces a new architecture that improves real-time data replication performance, scalability, and stability while reducing resource costs. For more information, see [TiCDC New Architecture](/ticdc/ticdc-architecture.md).
1414
1515
## TiCDC classic architecture
1616

markdown-pages/en/tidb/master/ticdc/ticdc-server-config.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,11 @@ The following are descriptions of options available in a `cdc server` command:
3030

3131
The following describes the configuration file specified by the `config` option in the `cdc server` command. You can find the default configuration file in [`pkg/cmd/util/ticdc.toml`](https://github.com/pingcap/tiflow/blob/master/pkg/cmd/util/ticdc.toml).
3232

33-
### `newarch` <span class="version-mark">New in v9.0.0</span>
33+
### `newarch` <span class="version-mark">New in v8.5.4-release.1</span>
3434

35-
- Controls whether to enable the [TiCDC new architecture](/ticdc/ticdc-new-arch.md).
35+
- Controls whether to enable the [TiCDC new architecture](/ticdc/ticdc-architecture.md).
36+
- Default value: `false`, indicating that the [TiCDC classic architecture](/ticdc/ticdc-classic-architecture.md) is used.
3637
- When it is set to `true`, the TiCDC new architecture is enabled.
37-
- By default, `newarch` is not specified, indicating that the classic architecture is used. `newarch` applies only to the new architecture. If `newarch` is added to the configuration file of the TiCDC classic architecture, it might cause parsing failures.
3838

3939
<!-- The configuration method of the following parameters is the same as that of CLI parameters, but the CLI parameters have higher priorities. -->
4040

0 commit comments

Comments
 (0)