diff --git a/TOC.md b/TOC.md index d7641e79c4ab4..2065601e434f4 100644 --- a/TOC.md +++ b/TOC.md @@ -192,7 +192,9 @@ - [Integrate with Confluent and Snowflake](/ticdc/integrate-confluent-using-ticdc.md) - [Integrate with Apache Kafka and Apache Flink](/replicate-data-to-kafka.md) - Reference - - [TiCDC Architecture](/ticdc/ticdc-architecture.md) + - TiCDC Architecture + - [TiCDC New Architecture](/ticdc/ticdc-architecture.md) + - [TiCDC Classic Architecture](/ticdc/ticdc-classic-architecture.md) - [TiCDC Data Replication Capabilities](/ticdc/ticdc-data-replication-capabilities.md) - [TiCDC Server Configurations](/ticdc/ticdc-server-config.md) - [TiCDC Changefeed Configurations](/ticdc/ticdc-changefeed-config.md) diff --git a/br/backup-and-restore-overview.md b/br/backup-and-restore-overview.md index 26a5e1eb0aa50..825b886e88e63 100644 --- a/br/backup-and-restore-overview.md +++ b/br/backup-and-restore-overview.md @@ -116,7 +116,7 @@ Backup and restore might go wrong when some TiDB features are enabled or disable | New collation | [#352](https://github.com/pingcap/br/issues/352) | Make sure that the value of the `new_collation_enabled` variable in the `mysql.tidb` table during restore is consistent with that during backup. Otherwise, inconsistent data index might occur and checksum might fail to pass. For more information, see [FAQ - Why does BR report `new_collations_enabled_on_first_bootstrap` mismatch?](/faq/backup-and-restore-faq.md#why-is-new_collation_enabled-mismatch-reported-during-restore). | | Global temporary tables | | Make sure that you are using v5.3.0 or a later version of BR to back up and restore data. Otherwise, an error occurs in the definition of the backed global temporary tables. | | TiDB Lightning Physical Import| | If the upstream database uses the physical import mode of TiDB Lightning, data cannot be backed up in log backup. It is recommended to perform a full backup after the data import. For more information, see [When the upstream database imports data using TiDB Lightning in the physical import mode, the log backup feature becomes unavailable. Why?](/faq/backup-and-restore-faq.md#when-the-upstream-database-imports-data-using-tidb-lightning-in-the-physical-import-mode-the-log-backup-feature-becomes-unavailable-why).| -| TiCDC | | BR v8.2.0 and later: if the target cluster to be restored has a changefeed and the changefeed [CheckpointTS](/ticdc/ticdc-architecture.md#checkpointts) is earlier than the BackupTS, BR does not perform the restoration. BR versions before v8.2.0: if the target cluster to be restored has any active TiCDC changefeeds, BR does not perform the restoration. | +| TiCDC | | BR v8.2.0 and later: if the target cluster to be restored has a changefeed and the changefeed [CheckpointTS](/ticdc/ticdc-classic-architecture.md#checkpointts) is earlier than the BackupTS, BR does not perform the restoration. BR versions before v8.2.0: if the target cluster to be restored has any active TiCDC changefeeds, BR does not perform the restoration. | | Vector search | | Make sure that you are using v8.4.0 or a later version of BR to back up and restore data. Restoring tables with [vector data types](/vector-search/vector-search-data-types.md) to TiDB clusters earlier than v8.4.0 is not supported. | ### Version compatibility diff --git a/media/ticdc/ticdc-new-arch-1.png b/media/ticdc/ticdc-new-arch-1.png new file mode 100644 index 0000000000000..06b55db0c1ebc Binary files /dev/null and b/media/ticdc/ticdc-new-arch-1.png differ diff --git a/media/ticdc/ticdc-new-arch-2.png b/media/ticdc/ticdc-new-arch-2.png new file mode 100644 index 0000000000000..16fd0f343fd25 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-2.png differ diff --git a/media/ticdc/ticdc-new-arch-import-grafana.png b/media/ticdc/ticdc-new-arch-import-grafana.png new file mode 100644 index 0000000000000..694bfac3c8c30 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-import-grafana.png differ diff --git a/media/ticdc/ticdc-new-arch-metric-event-store.png b/media/ticdc/ticdc-new-arch-metric-event-store.png new file mode 100644 index 0000000000000..61fb59cbfcdc1 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-metric-event-store.png differ diff --git a/media/ticdc/ticdc-new-arch-metric-log-puller.png b/media/ticdc/ticdc-new-arch-metric-log-puller.png new file mode 100644 index 0000000000000..4f8e343ede8ed Binary files /dev/null and b/media/ticdc/ticdc-new-arch-metric-log-puller.png differ diff --git a/media/ticdc/ticdc-new-arch-metric-server.png b/media/ticdc/ticdc-new-arch-metric-server.png new file mode 100644 index 0000000000000..87fcae2fd9844 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-metric-server.png differ diff --git a/media/ticdc/ticdc-new-arch-metric-sink.png b/media/ticdc/ticdc-new-arch-metric-sink.png new file mode 100644 index 0000000000000..8baf832c4b9f0 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-metric-sink.png differ diff --git a/media/ticdc/ticdc-new-arch-metric-summary.png b/media/ticdc/ticdc-new-arch-metric-summary.png new file mode 100644 index 0000000000000..c3f7d1d1a3512 Binary files /dev/null and b/media/ticdc/ticdc-new-arch-metric-summary.png differ diff --git a/releases/release-8.2.0.md b/releases/release-8.2.0.md index 3337ff533e793..7f1fa9109dc8f 100644 --- a/releases/release-8.2.0.md +++ b/releases/release-8.2.0.md @@ -165,7 +165,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.2/quick-start-with- * When using [`IMPORT INTO`](/sql-statements/sql-statement-import-into.md) to import a CSV file, if you specify the `SPLIT_FILE` parameter to split a large CSV file into multiple small CSV files to improve concurrency and import performance, you need to explicitly specify the line terminator `LINES_TERMINATED_BY`. The values can be `\r`, `\n` or `\r\n`. Failure to specify a line terminator might result in an exception when parsing the CSV file data. [#37338](https://github.com/pingcap/tidb/issues/37338) @[lance6716](https://github.com/lance6716) -* Before BR v8.2.0, performing [BR data restore](/br/backup-and-restore-overview.md) on a cluster with TiCDC replication tasks is not supported. Starting from v8.2.0, BR relaxes the restrictions on data restoration for TiCDC: if the BackupTS (the backup time) of the data to be restored is earlier than the changefeed [`CheckpointTS`](/ticdc/ticdc-architecture.md#checkpointts) (the timestamp that indicates the current replication progress), BR can proceed with the data restore normally. Considering that `BackupTS` is usually much earlier, it can be assumed that in most scenarios, BR supports restoring data for a cluster with TiCDC replication tasks. [#53131](https://github.com/pingcap/tidb/issues/53131) @[YuJuncen](https://github.com/YuJuncen) +* Before BR v8.2.0, performing [BR data restore](/br/backup-and-restore-overview.md) on a cluster with TiCDC replication tasks is not supported. Starting from v8.2.0, BR relaxes the restrictions on data restoration for TiCDC: if the BackupTS (the backup time) of the data to be restored is earlier than the changefeed [`CheckpointTS`](/ticdc/ticdc-classic-architecture.md#checkpointts) (the timestamp that indicates the current replication progress), BR can proceed with the data restore normally. Considering that `BackupTS` is usually much earlier, it can be assumed that in most scenarios, BR supports restoring data for a cluster with TiCDC replication tasks. [#53131](https://github.com/pingcap/tidb/issues/53131) @[YuJuncen](https://github.com/YuJuncen) ### MySQL compatibility @@ -263,7 +263,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.2/quick-start-with- + Backup & Restore (BR) - Optimize the backup feature, improving backup performance and stability during node restarts, cluster scaling-out, and network jitter when backing up large numbers of tables [#52534](https://github.com/pingcap/tidb/issues/52534) @[3pointer](https://github.com/3pointer) - - Implement fine-grained checks of TiCDC changefeed during data restore. If the changefeed [`CheckpointTS`](/ticdc/ticdc-architecture.md#checkpointts) is later than the data backup time, the restore operations are not affected, thereby reducing unnecessary wait times and improving user experience [#53131](https://github.com/pingcap/tidb/issues/53131) @[YuJuncen](https://github.com/YuJuncen) + - Implement fine-grained checks of TiCDC changefeed during data restore. If the changefeed [`CheckpointTS`](/ticdc/ticdc-classic-architecture.md#checkpointts) is later than the data backup time, the restore operations are not affected, thereby reducing unnecessary wait times and improving user experience [#53131](https://github.com/pingcap/tidb/issues/53131) @[YuJuncen](https://github.com/YuJuncen) - Add several commonly used parameters to the [`BACKUP`](/sql-statements/sql-statement-backup.md) statement and the [`RESTORE`](/sql-statements/sql-statement-restore.md) statement, such as `CHECKSUM_CONCURRENCY` [#53040](https://github.com/pingcap/tidb/issues/53040) @[RidRisR](https://github.com/RidRisR) - Except for the `br log restore` subcommand, all other `br log` subcommands support skipping the loading of the TiDB `domain` data structure to reduce memory consumption [#52088](https://github.com/pingcap/tidb/issues/52088) @[Leavrth](https://github.com/Leavrth) - Support encryption of temporary files generated during log backup [#15083](https://github.com/tikv/tikv/issues/15083) @[YuJuncen](https://github.com/YuJuncen) diff --git a/ticdc/monitor-ticdc.md b/ticdc/monitor-ticdc.md index 08f355e63f8bf..7f22a0a5994ff 100644 --- a/ticdc/monitor-ticdc.md +++ b/ticdc/monitor-ticdc.md @@ -5,7 +5,7 @@ summary: Learn some key metrics displayed on the Grafana TiCDC dashboard. # TiCDC Monitoring Metrics Details -If you use TiUP to deploy the TiDB cluster, you can see a sub-dashboard for TiCDC in the monitoring system which is deployed at the same time. You can get an overview of TiCDC's current status from the TiCDC dashboard, where the key metrics are displayed. This document provides a detailed description of these key metrics. +You can get an overview of TiCDC's current status from the TiCDC dashboard, where the key metrics are displayed. This document provides a detailed description of these key metrics. The metric description in this document is based on the following replication task example, which replicates data to MySQL using the default configuration. @@ -13,9 +13,119 @@ The metric description in this document is based on the following replication ta cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --changefeed-id="simple-replication-task" ``` -The TiCDC dashboard contains four monitoring panels. See the following screenshot: +## Metrics for TiCDC in the new architecture -![TiCDC Dashboard - Overview](/media/ticdc/ticdc-dashboard-overview.png) +The monitoring dashboard **TiCDC-New-Arch** for [TiCDC New Architecture](/ticdc/ticdc-architecture.md) is not managed by TiUP yet. To view the related monitoring data on Grafana, you need to manually import the TiCDC monitoring metrics file: + +1. Download the monitoring metrics file for TiCDC in the new architecture: + + ```shell + wget https://raw.githubusercontent.com/pingcap/ticdc/refs/heads/release-8.5/metrics/grafana/ticdc_new_arch.json + ``` + +2. Import the downloaded metrics file on Grafana: + + ![Import Metrics File](/media/ticdc/ticdc-new-arch-import-grafana.png) + +The monitoring dashboard for TiCDC new architecture mainly includes the following sections: + +- [**Summary**](#summary): The summary information of the TiCDC cluster +- [**Server**](#server): The summary information of TiKV nodes and TiCDC nodes in the TiDB cluster +- [**Log Puller**](#log-puller): The detailed information of the TiCDC Log Puller module +- [**Event Store**](#event-store): The detailed information of the TiCDC Event Store module +- [**Sink**](#sink): The detailed information of the TiCDC Sink module + +### Summary + +The following is an example of the **Summary** panel: + +![Summary](/media/ticdc/ticdc-new-arch-metric-summary.png) + +The description of each metric in the **Summary** panel is as follows: + +- Changefeed Checkpoint Lag: The lag of a replication task between downstream and upstream +- Changefeed ResolvedTs Lag: The lag between the internal processing progress of TiCDC nodes and the upstream database +- Upstream Write Bytes/s: The write throughput of the upstream database +- TiCDC Input Bytes/s: The amount of data that TiCDC receives from the upstream per second +- Sink Event Row Count/s: The number of rows that TiCDC writes to the downstream per second +- Sink Write Bytes/s: The amount of data that TiCDC writes to the downstream per second +- The Status of Changefeeds: The status of each changefeed +- Table Dispatcher Count: The number of dispatchers corresponding to each changefeed +- Memory Quota: The memory quota and usage of the Event Collector; excessive usage might cause throttling + +### Server + +The following is an example of the **Server** panel: + +![Server](/media/ticdc/ticdc-new-arch-metric-server.png) + +The description of each metric in the **Server** panel is as follows: + +- Uptime: The time for which TiKV nodes and TiCDC nodes have been running +- Goroutine Count: The number of Goroutines on TiCDC nodes +- Open FD Count: The number of file handles opened by TiCDC nodes +- CPU Usage: The CPU usage of TiCDC nodes +- Memory Usage: The memory usage of TiCDC nodes +- Ownership History: The historical records of Owner nodes in the TiCDC cluster +- PD Leader History: The historical records of PD Leader nodes in the upstream TiDB cluster + +### Log Puller + +The following is an example of the **Log Puller** panel: + +![Log Puller](/media/ticdc/ticdc-new-arch-metric-log-puller.png) + +The description of each metric in the **Log Puller** panel is as follows: + +- Input Events/s: The number of events that TiCDC receives per second +- Unresolved Region Request Count: The number of Region incremental scan requests that TiCDC has sent but not yet completed +- Region Request Finish Scan Duration: The time consumed by Region incremental scans +- Subscribed Region Count: The total number of subscribed Regions +- Memory Quota: The memory quota and usage of Log Puller; excessive usage might cause throttling +- Resolved Ts Batch Size (Regions): The number of Regions included in a single Resolved Ts event + +### Event Store + +The following is an example of the **Event Store** panel: + +![Event Store](/media/ticdc/ticdc-new-arch-metric-event-store.png) + +The description of each metric in the **Event Store** panel is as follows: + +- Resolved Ts Lag: The lag between Event Store processing progress and the upstream database +- Register Dispatcher StartTs Lag: The lag between dispatcher registration StartTs and the current time +- Subscriptions Resolved Ts Lag: The lag between subscription processing progress and the upstream database +- Subscriptions Data GC Lag: The lag between subscription data GC progress and the current time +- Input Event Count/s: The number of events that Event Store processes per second +- Input Bytes/s: The amount of data that Event Store processes per second +- Write Requests/s: The number of write requests that Event Store executes per second +- Write Worker Busy Ratio: The ratio of I/O time to total runtime for Event Store write threads +- Compressed Rows/s: The number of rows compressed per second in Event Store (triggered only when row size exceeds the threshold) +- Write Duration: The time consumed by Event Store write operations +- Write Batch Size: The batch size of a single write operation +- Write Batch Event Count: The number of row change events included in a single write batch +- Data Size On Disk: The total data size that Event Store occupies on disk +- Data Size In Memory: The total data size that Event Store occupies in memory +- Scan Requests/s: The number of scan requests that Event Store executes per second +- Scan Bytes/s: The amount of data that Event Store scans per second + +### Sink + +The following is an example of the **Sink** panel: + +![Sink](/media/ticdc/ticdc-new-arch-metric-sink.png) + +The description of each metric in the **Sink** panel is as follows: + +- Output Row Batch Count: The average number of rows per DML batch written by the Sink module +- Output Row Count (per second): The number of DML rows written to downstream per second +- Output DDL Executing Duration: The time consumed by executing DDL events for the changefeed on the current node +- Sink Error Count / m: The number of errors reported per minute by the Sink module +- Output DDL Count / Minutes: The number of DDLs executed per minute for the changefeed on the current node + +## Metrics for TiCDC in the classic architecture + +If you use TiUP to deploy the TiDB cluster, you can see a sub-dashboard for TiCDC in the [classic architecture](/ticdc/ticdc-classic-architecture.md) on Grafana, which is deployed at the same time as TiDB. The description of each panel is as follows: @@ -24,7 +134,7 @@ The description of each panel is as follows: - [**Events**](#events): The detail information about the data flow within the TiCDC cluster - [**TiKV**](#tikv): TiKV information related to TiCDC -## Server +### Server The following is an example of the **Server** panel: @@ -40,7 +150,7 @@ The description of each metric in the **Server** panel is as follows: - CPU usage: The CPU usage of TiCDC nodes - Memory usage: The memory usage of TiCDC nodes -## Changefeed +### Changefeed The following is an example of the **Changefeed** panel: @@ -72,7 +182,7 @@ The following is an example of the **Changefeed** panel: - Changefeed catch-up ETA: The estimated time needed for the replication task to catch up with the upstream cluster data. When the upstream write speed is faster than the TiCDC replication speed, the metric might be extremely large. Because TiCDC replication speed is subject to many factors, this metric is for reference only and might not be the actual replication time. -## Events +### Events The following is an example of the **Events** panel: @@ -102,7 +212,7 @@ The description of each metric in the **Events** panel is as follows: - KV client dispatch events/s: The number of events that the KV client module dispatches among the TiCDC nodes - KV client batch resolved size: The batch size of resolved timestamp messages that TiKV sends to TiCDC -## TiKV +### TiKV The following is an example of the **TiKV** panel: diff --git a/ticdc/ticdc-architecture.md b/ticdc/ticdc-architecture.md index 880c8750ce850..6dd0a66b7b5da 100644 --- a/ticdc/ticdc-architecture.md +++ b/ticdc/ticdc-architecture.md @@ -1,182 +1,244 @@ --- -title: Architecture and Principles of TiCDC -summary: Learn the architecture and working principles of TiCDC. +title: TiCDC New Architecture +summary: Introduces the features, architectural design, deployment guide, and notes of the TiCDC new architecture. --- -# Architecture and Principles of TiCDC +# TiCDC New Architecture -## TiCDC architecture +Starting from [TiCDC v8.5.4-release.1](https://github.com/pingcap/ticdc/releases/tag/v8.5.4-release.1), TiCDC introduces a new architecture that improves the performance, scalability, and stability of real-time data replication while reducing resource costs. This new architecture redesigns TiCDC core components and optimizes its data processing workflows, offering the following advantages: -Consisting of multiple TiCDC nodes, a TiCDC cluster uses a distributed and stateless architecture. The design of TiCDC and its components is as follows: +- **Higher single-node performance**: a single node can replicate up to 500,000 tables, achieving replication throughput of up to 190 MiB/s on a single node in wide table scenarios. +- **Enhanced scalability**: cluster replication capability scales almost linearly. A single cluster can expand to over 100 nodes, support more than 10,000 changefeeds, and replicate millions of tables within a single changefeed. +- **Improved stability**: changefeed latency is reduced and performance is more stable in scenarios with high traffic, frequent DDL operations, and cluster scaling events. Resource isolation and priority scheduling reduce interference between multiple changefeed tasks. +- **Lower resource costs**: with improved resource utilization and reduced redundancy, CPU and memory resource usage can decrease by up to 50% in typical scenarios. -![TiCDC architecture](/media/ticdc/ticdc-architecture-1.jpg) +## Architectural design -## TiCDC components +![TiCDC New Architecture](/media/ticdc/ticdc-new-arch-1.png) -In the preceding diagram, a TiCDC cluster consists of multiple nodes running TiCDC instances. Each TiCDC instance carries a Capture process. One of the Capture processes is elected as the owner Capture, which is responsible for scheduling workload, replicating DDL statements, and performing management tasks. +The TiCDC new architecture consists of two core components: Log Service and Downstream Adapter. -Each Capture process contains one or multiple Processor threads for replicating data from tables in the upstream TiDB. Because table is the minimum unit of data replication in TiCDC, a Processor is composed of multiple table pipelines. +- Log Service: as the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disks. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter. +- Downstream Adapter: as the downstream data replication adaptation layer, Downstream Adapter handles user-initiated changefeed operations. It schedules and generates related replication tasks, fetches data from the Log Service, and replicates the fetched data to downstream systems. -Each pipeline contains the following components: Puller, Sorter, Mounter, and Sink. +By separating the architecture into stateful and stateless components, the TiCDC new architecture significantly improves system scalability, reliability, and flexibility. Log Service, as the stateful component, focuses on data acquisition, sorting, and storage. Decoupling it from changefeed processing logic enables data sharing across multiple changefeeds, effectively improving resource utilization and reducing system overhead. Downstream Adapter, as the stateless component, uses a lightweight scheduling mechanism that allows quick migration of replication tasks between instances. It can dynamically adjust the splitting and merging of replication tasks based on workload changes, ensuring low-latency replication in various scenarios. -![TiCDC architecture](/media/ticdc/ticdc-architecture-2.jpg) +## Comparison between the classic and new architectures -These components work in serial with each other to complete the replication process, including pulling data, sorting data, loading data, and replicating data from the upstream to the downstream. The components are described as follows: +The new architecture is designed to address common issues during continuous system scaling, such as performance bottlenecks, insufficient stability, and limited scalability. Compared with the [classic architecture](/ticdc/ticdc-classic-architecture.md), the new architecture achieves significant optimizations in the following key dimensions: -- Puller: pulls DDL and row changes from TiKV nodes. -- Sorter: sorts the changes received from TiKV nodes in ascending order of timestamps. -- Mounter: converts the changes into a format that TiCDC sink can process based on the schema information. -- Sink: replicates the changes to the downstream system. +| Feature | TiCDC classic architecture | TiCDC new architecture | +| ------------------------ | ---------------------------------------- | ---------------------------------------- | +| **Processing logic driver** | Timer-driven | Event-driven | +| **Task triggering mechanism** | Timer-triggered main loop that checks tasks every 50 ms, with limited processing performance | Event-driven, triggered by events such as DML changes, DDL changes, and changefeed operations. Events in the queue are processed as quickly as possible without waiting for the fixed 50 ms interval, reducing additional latency | +| **Task scheduling method** | Each changefeed runs a main loop that polls for tasks | Events are queued and processed concurrently by multiple threads | +| **Task processing efficiency** | Each task goes through multiple cycles, creating performance bottlenecks | Events are processed immediately without waiting for fixed intervals, reducing latency | +| **Resource consumption** | Frequent checks on inactive tables waste CPU resources | Consumer threads only process queued events, avoiding the consumption of checking inactive tasks | +| **Complexity** | O(n), performance degrades as the number of tables increases | O(1), not affected by the number of tables, improving efficiency | +| **CPU utilization** | Each changefeed can only use one logical CPU | Each changefeed can fully utilize the parallel processing capabilities of multi-core CPUs | +| **Scalability** | Poor scalability (limited by the number of CPUs) | Strong scalability through multi-threaded processing and event queues | +| **Changefeed interference** | The owner node might cause interference between changefeeds | Event-driven mode avoids interference between changefeeds | -To realize high availability, each TiCDC cluster runs multiple TiCDC nodes. These nodes regularly report their status to the etcd cluster in PD, and elect one of the nodes as the owner of the TiCDC cluster. The owner node schedules data based on the status stored in etcd and writes the scheduling results to etcd. The Processor completes tasks according to the status in etcd. If the node running the Processor fails, the cluster schedules tables to other nodes. If the owner node fails, the Capture processes in other nodes will elect a new owner. See the following figure: +![Comparison between the TiCDC classic and new architectures](/media/ticdc/ticdc-new-arch-2.png) -![TiCDC architecture](/media/ticdc/ticdc-architecture-3.PNG) +## Choose between the classic and new architectures -## Changefeeds and tasks +If your workload meets any of the following conditions, it is recommended to switch from the [classic TiCDC architecture](/ticdc/ticdc-classic-architecture.md) to the new architecture for better performance and stability: -Changefeed and Task in TiCDC are two logical concepts. The specific description is as follows: +- Bottlenecks in incremental scan performance: incremental scan tasks take an excessively long time to complete, leading to continuously increasing replication latency. +- Ultra-high traffic scenarios: the total changefeed traffic exceeds 700 MiB/s. +- Single tables with high-throughput writes in MySQL sink: the target table has **only one primary key or non-null unique key**. +- Large-scale table replication: the number of tables to be replicated exceeds 100,000. +- Frequent DDL operations causing latency: frequent execution of DDL statements significantly increases replication latency. -- Changefeed: Represents a replication task. It carries the information about the tables to be replicated and the downstream. -- Task: After TiCDC receives a replication task, it splits this task into several subtasks. Such a subtask is called Task. These Tasks are assigned to the Capture processes of the TiCDC nodes for processing. +## New features -For example: +The new architecture supports **table-level task splitting** for MySQL sinks. You can enable this feature by setting `scheduler.enable-table-across-nodes = true` in the changefeed configuration. -``` -cdc cli changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1" -cat changefeed.toml -...... -[sink] -dispatchers = [ - {matcher = ['test1.tab1', 'test2.tab2'], topic = "{schema}_{table}"}, - {matcher = ['test3.tab3', 'test4.tab4'], topic = "{schema}_{table}"}, -] -``` +When this feature is enabled, TiCDC automatically splits and distributes tables with **only one primary key or non-null unique key** across multiple nodes for parallel replication if those tables meet any of the following conditions. This improves replication efficiency and resource utilization: + +- The table Region count exceeds the configured threshold (`100000` by default, adjustable via `scheduler.region-threshold`). +- The table write traffic exceeds the configured threshold (disabled by default, configurable via `scheduler.write-key-threshold`). + +## Compatibility -For a detailed description of the parameters in the preceding `cdc cli changefeed create` command, see [TiCDC Changefeed Configuration Parameters](/ticdc/ticdc-changefeed-config.md). +### DDL progress tracking table -The preceding `cdc cli changefeed create` command creates a changefeed task that replicates `test1.tab1`, `test1.tab2`, `test3.tab3`, and `test4.tab4` to the Kafka cluster. The processing flow after TiCDC receives this command is as follows: +In the TiCDC classic architecture, DDL replication operations are strictly serial, thus the replication progress can be tracked only using the changefeed's `CheckpointTs`. In the new architecture, however, TiCDC replicates DDL changes for different tables in parallel whenever possible to improve DDL replication efficiency. To accurately record the DDL replication progress of each table in a downstream MySQL-compatible database, the TiCDC new architecture creates a table named `tidb_cdc.ddl_ts_v1` in the downstream database, specifically storing the DDL replication progress information of the changefeed. -1. TiCDC sends this task to the owner Capture process. -2. The owner Capture process saves information about this changefeed task in etcd in PD. -3. The owner Capture process splits the changefeed task into several Tasks and notifies other Capture processes of the Tasks to be completed. -4. The Capture processes start pulling data from TiKV nodes, process the data, and complete replication. +### Changes in DDL replication behavior -The following is the TiCDC architecture diagram with Changefeed and Task included: +- The classic TiCDC architecture does not support DDLs that swap table names (for example, `RENAME TABLE a TO c, b TO a, c TO b;`). The new architecture supports such DDLs. -![TiCDC architecture](/media/ticdc/ticdc-architecture-6.jpg) +- The new architecture unifies and simplifies the filtering rules for `RENAME` DDLs. -In the preceding diagram, a changefeed is created to replicate four tables to downstream. This changefeed is split into three Tasks, which are sent to the three Capture processes respectively in the TiCDC cluster. After TiCDC processes the data, the data is replicated to the downstream system. + - In the classic architecture, the filtering logic is as follows: -TiCDC supports replicating data to MySQL, TiDB, and Kafka databases. The preceding diagram only illustrates the process of data transfer at the changefeed level. The following sections describe in detail how TiCDC processes data, using Task1 that replicates table `table1` as an example. + - Single-table renaming: a DDL statement is replicated only if the old table name matches the filter rule. + - Multi-table renaming: a DDL statement is replicated only if both old and new table names match the filter rules. -![TiCDC architecture](/media/ticdc/ticdc-architecture-5.jpg) + - In the new architecture, for both single-table and multi-table renaming, a DDL statement is replicated as long as old table names in the statement match the filter rules. -1. Push data: When a data change occurs, TiKV pushes data to the Puller module. -2. Scan incremental data: The Puller module pulls data from TiKV when it finds the data changes received not continuous. -3. Sort data: The Sorter module sorts the data received from TiKV based on the timestamps and sends the sorted data to the Mounter module. -4. Mount data: After receiving the data changes, the Mounter module loads the data in a format that TiCDC sink can understand. -5. Replicate data: The Sink module replicates the data changes to the downstream. + Take the following filter rule as an example: -The upstream of TiCDC is the distributed relational database TiDB that supports transactions. When TiCDC replicates data, it should ensure the consistency of data and that of transactions when replicating multiple tables, which is a great challenge. The following sections introduce the key technologies and concepts used by TiCDC to address this challenge. + ```toml + [filter] + rules = ['test.t*'] + ``` -## Key concepts of TiCDC + - In the classic architecture: for single-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1`, the old table name `test.t1` matches the rule, so it will be replicated. For a multi-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;`, because the new table name `ignore.t1` does not match the rule, it will not be replicated. + - In the new TiCDC architecture: because the old table names in both `RENAME TABLE test.t1 TO ignore.t1` and `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` match the rules, both DDL statements will be replicated. -For the downstream relational databases, TiCDC ensures the consistency of transactions in a single table and eventual transaction consistency in multiple tables. In addition, TiCDC ensures that any data change that has occurred in the upstream TiDB cluster can be replicated to the downstream at least once. +## Limitations -### Architecture-related concepts +The new TiCDC architecture incorporates all functionalities of the classic architecture. However, some features have not yet been fully tested. To ensure system stability, it is **NOT** recommended to use the following features in core production environments: -- Capture: The process that runs the TiCDC node. Multiple Capture processes constitute a TiCDC cluster. Each Capture process is responsible for replicating data changes in TiKV, including receiving and actively pulling data changes, and replicating the data to the downstream. -- Capture Owner: The owner Capture among multiple Capture processes. Only one owner role exists in a TiCDC cluster at a time. The Capture Owner is responsible for scheduling data within the cluster. -- Processor: The logical thread inside Capture. Each Processor is responsible for processing the data of one or more tables in the same replication stream. A Capture node can run multiple Processors. -- Changefeed: A task that replicates data from an upstream TiDB cluster to a downstream system. A changefeed contains multiple Tasks, and each Task is processed by a Capture node. +- [Syncpoint](/ticdc/ticdc-upstream-downstream-check.md) +- [Redo Log](/ticdc/ticdc-sink-to-mysql.md#eventually-consistent-replication-in-disaster-scenarios) +- [Pulsar Sink](/ticdc/ticdc-sink-to-pulsar.md) +- [Storage Sink](/ticdc/ticdc-sink-to-cloud-storage.md) -### Timestamp-related concepts +In addition, the new TiCDC architecture currently does not support splitting large transactions into multiple batches for downstream replication. As a result, there is still a risk of OOM when processing extremely large transactions. Make sure to evaluate and mitigate this risk appropriately before using the new architecture. -TiCDC introduces a series of timestamps (TS) to indicate the status of data replication. These timestamps are used to ensure that data is replicated to the downstream at least once and that the consistency of data is guaranteed. +## Upgrade guide -#### ResolvedTS +The TiCDC new architecture can only be deployed in TiDB clusters of v7.5.0 or later versions. Before deployment, make sure your TiDB cluster meets this version requirement. -This timestamp exists in both TiKV and TiCDC. +You can deploy the TiCDC new architecture using TiUP or TiDB Operator. -- ResolvedTS in TiKV: Represents the start time of the earliest transaction in a Region leader, that is, `ResolvedTS` = max(`ResolvedTS`, min(`StartTS`)). Because a TiDB cluster contains multiple TiKV nodes, the minimum ResolvedTS of the Region leader on all TiKV nodes is called the global ResolvedTS. The TiDB cluster ensures that all transactions before the global ResolvedTS are committed. Alternatively, you can assume that there are no uncommitted transactions before this timestamp. + +
-- ResolvedTS in TiCDC: +To deploy the TiCDC new architecture using TiUP, take the following steps: - - table ResolvedTS: Each table has a table-level ResolvedTS, which indicates all data changes in the table that are smaller than the Resolved TS have been received. To make it simple, this timestamp is the same as the minimum value of the ResolvedTS of all Regions corresponding to this table on the TiKV node. - - global ResolvedTS: The minimum ResolvedTS of all Processors on all TiCDC nodes. Because each TiCDC node has one or more Processors, each Processor corresponds to multiple table pipelines. +1. If your TiDB cluster does not have TiCDC nodes yet, refer to [Scale out a TiCDC cluster](/scale-tidb-using-tiup.md#scale-out-a-ticdc-cluster) to add new TiCDC nodes in the cluster. Otherwise, skip this step. - For TiCDC, the ResolvedTS sent by TiKV is a special event in the format of ``. In general, the ResolvedTS satisfies the following constraints: +2. Download the TiCDC binary package for the new architecture. + The download link follows this format: `https://tiup-mirrors.pingcap.com/cdc-${version}-${os}-${arch}.tar.gz`, where `${version}` is the TiCDC version, `${os}` is your operating system, and `${arch}` is the platform the component runs on (`amd64` or `arm64`). + + For example, to download the binary package of TiCDC v8.5.4-release.1 for Linux (x86-64), run the following command: + + ```shell + wget https://tiup-mirrors.pingcap.com/cdc-v8.5.4-release.1-linux-amd64.tar.gz ``` - table ResolvedTS >= global ResolvedTS + +3. If your TiDB cluster has running changefeeds, refer to [Pause a replication task](/ticdc/ticdc-manage-changefeed.md#pause-a-replication-task) to pause all replication tasks of the changefeeds. + + ```shell + # The default server port of TiCDC is 8300. + cdc cli changefeed pause --server=http://:8300 --changefeed-id ``` -#### CheckpointTS +4. Patch the downloaded TiCDC binary file to your TiDB cluster using the [`tiup cluster patch`](/tiup/tiup-component-cluster-patch.md) command: -This timestamp exists only in TiCDC. It means that the data changes that occur before this timestamp have been replicated to the downstream system. + ```shell + tiup cluster patch ./cdc-v8.5.4-release.1-linux-amd64.tar.gz -R cdc + ``` -- table CheckpointTS: Because TiCDC replicates data in tables, the table checkpointTS indicates all data changes that occur before CheckpointTS have been replicated at the table level. -- processor CheckpointTS: Indicates the minimum table CheckpointTS on a Processor. -- global CheckpointTS: Indicates the minimum CheckpointTS among all Processors. +5. Update the TiCDC configuration using the [`tiup cluster edit-config`](/tiup/tiup-component-cluster-edit-config.md) command to enable the new architecture: -Generally, a checkpointTS satisfies the following constraint: + ```shell + tiup cluster edit-config + ``` -``` -table CheckpointTS >= global CheckpointTS -``` + ```yaml + server_configs: + cdc: + newarch: true + ``` -Because TiCDC only replicates data smaller than the global ResolvedTS to the downstream, the complete constraint is as follows: +6. Refer to [Resume replication task](/ticdc/ticdc-manage-changefeed.md#resume-a-replication-task) to resume all replication tasks: -``` -table ResolvedTS >= global ResolvedTS >= table CheckpointTS >= global CheckpointTS -``` + ```shell + # The default server port of TiCDC is 8300. + cdc cli changefeed resume --server=http://:8300 --changefeed-id + ``` + +
+
+ +To deploy the TiCDC new architecture using TiDB Operator, take the following steps: + +- If your TiDB cluster does not include a TiCDC component, refer to [Add TiCDC to an existing TiDB cluster](https://docs.pingcap.com/tidb-in-kubernetes/stable/deploy-ticdc/#add-ticdc-to-an-existing-tidb-cluster) to add new TiCDC nodes. When doing so, specify the TiCDC image version as the new architecture version in the cluster configuration file. + + For example: + + ```yaml + spec: + ticdc: + baseImage: pingcap/ticdc + version: v8.5.4-release.1 + replicas: 3 + config: + newarch = true + ``` + +- If your TiDB cluster already includes a TiCDC component, take the following steps: -After data changes and transactions are committed, the ResolvedTS on the TiKV node will continue to advance, and the Puller module on the TiCDC node keeps receiving data pushed by TiKV. The Puller module also decides whether to scan incremental data based on the data changes it has received, which ensures that all data changes are sent to the TiCDC node. + 1. If your TiDB cluster has running changefeeds, pause all replication tasks of the changefeeds: -The Sorter module sorts data received by the Puller module in ascending order according to the timestamp. This process ensures data consistency at the table level. Next, the Mounter module assembles the data changes from the upstream into a format that the Sink module can consume, and sends it to the Sink module. The Sink module replicates the data changes between the CheckpointTS and the ResolvedTS to the downstream in the order of the timestamp, and advances the checkpointTS after the downstream receives the data changes. + ```shell + kubectl exec -it ${pod_name} -n ${namespace} -- sh + ``` -The preceding sections only cover data changes of DML statements and do not include DDL statements. The following sections introduce the timestamp related to DDL statements. + ```shell + # The default server port of TiCDC deployed via TiDB Operator is 8301. + /cdc cli changefeed pause --server=http://127.0.0.1:8301 --changefeed-id + ``` -#### Barrier TS + 2. Update the TiCDC image version in the cluster configuration file to the new architecture version: -Barrier TS is generated when there are DDL change events or a Syncpoint is used. + ```shell + kubectl edit tc ${cluster_name} -n ${namespace} + ``` -- DDL change events: Barrier TS ensures that all changes before the DDL statement are replicated to the downstream. After this DDL statement is executed and replicated, TiCDC starts replicating other data changes. Because DDL statements are processed by the Capture Owner, the Barrier TS corresponding to a DDL statement is only generated by the owner node. -- Syncpoint: When you enable the Syncpoint feature of TiCDC, a Barrier TS is generated by TiCDC according to the `sync-point-interval` you specified. When all table changes before this Barrier TS are replicated, TiCDC inserts the current global CheckpointTS as the primary TS to the table recording tsMap in downstream. Then TiCDC continues data replication. + ```yaml + spec: + ticdc: + baseImage: pingcap/ticdc + version: v8.5.4-release.1 + replicas: 3 + ``` -After a Barrier TS is generated, TiCDC ensures that only data changes that occur before this Barrier TS are replicated to downstream. Before these data changes are replicated to downstream, the replication task does not proceed. The owner TiCDC checks whether all target data has been replicated by continuously comparing the global CheckpointTS and the Barrier TS. If the global CheckpointTS equals to the Barrier TS, TiCDC continues replication after performing a designated operation (such as executing a DDL statement or recording the global CheckpointTS downstream). Otherwise, TiCDC waits for all data changes that occur before the Barrier TS to be replicated to the downstream. + ```shell + kubectl apply -f ${cluster_name} -n ${namespace} + ``` -## Major processes + 3. Resume all replication tasks: -This section describes the major processes of TiCDC to help you better understand its working principles. + ```shell + kubectl exec -it ${pod_name} -n ${namespace} -- sh + ``` -Note that the following processes occur only within TiCDC and are transparent to users. Therefore, you do not need to care about which TiCDC node you are starting. + ```shell + # The default server port of TiCDC deployed via TiDB Operator is 8301. + /cdc cli changefeed resume --server=http://127.0.0.1:8301 --changefeed-id + ``` -### Start TiCDC +
+
-- For a TiCDC node that is not an owner, it works as follows: +## Use the new architecture - 1. Starts the Capture process. - 2. Starts the Processor. - 3. Receives the Task scheduling command executed by the Owner. - 4. Starts or stops tablePipeline according to the scheduling command. +After deploying the TiCDC nodes with the new architecture, you can continue using the same commands as in the classic architecture. There is no need to learn new commands or modify the commands used in the classic architecture. -- For an owner TiCDC node, it works as follows: +For example, to create a replication task in a new architecture TiCDC node, run the following command: + +```shell +cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --changefeed-id="simple-replication-task" +``` + +To query details about a specific replication task, run the following command: + +```shell +cdc cli changefeed query -s --server=http://127.0.0.1:8300 --changefeed-id=simple-replication-task +``` - 1. Starts the Capture process. - 2. The node is elected as the Owner and the corresponding thread is started. - 3. Reads the changefeed information. - 4. Starts the changefeed management process. - 5. Reads the schema information in TiKV according to the changefeed configuration and the latest CheckpointTS to determine the tables to be replicated. - 6. Reads the list of tables currently replicated by each Processor and distributes the tables to be added. - 7. Updates the replication progress. +For more command usage methods and details, see [Manage Changefeeds](/ticdc/ticdc-manage-changefeed.md). -### Stop TiCDC +## Monitoring -Usually, you stop a TiCDC node when you need to upgrade it or perform some planned maintenance operations. The process of stopping a TiCDC node is as follows: +Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard on Grafana, you need to manually import the [TiCDC monitoring metrics file](https://github.com/pingcap/ticdc/blob/master/metrics/grafana/ticdc_new_arch.json). -1. The node receives the command to stop itself. -2. The node sets its service status to unavailable. -3. The node stops receiving new replication tasks. -4. The node notifies the Owner node to transfer its data replication tasks to other nodes. -5. The node stops after the replication tasks are transferred to other nodes. +For detailed descriptions of each monitoring metric, see [Metrics for TiCDC in the new architecture](/ticdc/monitor-ticdc.md#metrics-for-ticdc-in-the-new-architecture). \ No newline at end of file diff --git a/ticdc/ticdc-classic-architecture.md b/ticdc/ticdc-classic-architecture.md new file mode 100644 index 0000000000000..1709274ee16f4 --- /dev/null +++ b/ticdc/ticdc-classic-architecture.md @@ -0,0 +1,189 @@ +--- +title: TiCDC Classic Architecture +summary: Learn the classic architecture and working principles of TiCDC. +--- + +# TiCDC Classic Architecture + +This document describes the classic architecture and working principles of TiCDC. + +> **Note:** +> +> - This document applies to TiCDC versions earlier than v8.5.4-release.1. +> - Starting from v8.5.4-release.1, TiCDC introduces a new architecture that improves real-time data replication performance, scalability, and stability while reducing resource costs. For more information, see [TiCDC New Architecture](/ticdc/ticdc-architecture.md). + +## TiCDC classic architecture + +Consisting of multiple TiCDC nodes, a TiCDC cluster uses a distributed and stateless architecture. The design of TiCDC and its components is as follows: + +![TiCDC architecture](/media/ticdc/ticdc-architecture-1.jpg) + +## TiCDC components + +In the preceding diagram, a TiCDC cluster consists of multiple nodes running TiCDC instances. Each TiCDC instance carries a Capture process. One of the Capture processes is elected as the owner Capture, which is responsible for scheduling workload, replicating DDL statements, and performing management tasks. + +Each Capture process contains one or multiple Processor threads for replicating data from tables in the upstream TiDB. Because table is the minimum unit of data replication in TiCDC, a Processor is composed of multiple table pipelines. + +Each pipeline contains the following components: Puller, Sorter, Mounter, and Sink. + +![TiCDC architecture](/media/ticdc/ticdc-architecture-2.jpg) + +These components work in serial with each other to complete the replication process, including pulling data, sorting data, loading data, and replicating data from the upstream to the downstream. The components are described as follows: + +- Puller: pulls DDL and row changes from TiKV nodes. +- Sorter: sorts the changes received from TiKV nodes in ascending order of timestamps. +- Mounter: converts the changes into a format that TiCDC sink can process based on the schema information. +- Sink: replicates the changes to the downstream system. + +To realize high availability, each TiCDC cluster runs multiple TiCDC nodes. These nodes regularly report their status to the etcd cluster in PD, and elect one of the nodes as the owner of the TiCDC cluster. The owner node schedules data based on the status stored in etcd and writes the scheduling results to etcd. The Processor completes tasks according to the status in etcd. If the node running the Processor fails, the cluster schedules tables to other nodes. If the owner node fails, the Capture processes in other nodes will elect a new owner. See the following figure: + +![TiCDC architecture](/media/ticdc/ticdc-architecture-3.PNG) + +## Changefeeds and tasks + +Changefeed and Task in TiCDC are two logical concepts. The specific description is as follows: + +- Changefeed: Represents a replication task. It carries the information about the tables to be replicated and the downstream. +- Task: After TiCDC receives a replication task, it splits this task into several subtasks. Such a subtask is called Task. These Tasks are assigned to the Capture processes of the TiCDC nodes for processing. + +For example: + +``` +cdc cli changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1" +cat changefeed.toml +...... +[sink] +dispatchers = [ + {matcher = ['test1.tab1', 'test2.tab2'], topic = "{schema}_{table}"}, + {matcher = ['test3.tab3', 'test4.tab4'], topic = "{schema}_{table}"}, +] +``` + +For a detailed description of the parameters in the preceding `cdc cli changefeed create` command, see [TiCDC Changefeed Configuration Parameters](/ticdc/ticdc-changefeed-config.md). + +The preceding `cdc cli changefeed create` command creates a changefeed task that replicates `test1.tab1`, `test1.tab2`, `test3.tab3`, and `test4.tab4` to the Kafka cluster. The processing flow after TiCDC receives this command is as follows: + +1. TiCDC sends this task to the owner Capture process. +2. The owner Capture process saves information about this changefeed task in etcd in PD. +3. The owner Capture process splits the changefeed task into several Tasks and notifies other Capture processes of the Tasks to be completed. +4. The Capture processes start pulling data from TiKV nodes, process the data, and complete replication. + +The following is the TiCDC architecture diagram with Changefeed and Task included: + +![TiCDC architecture](/media/ticdc/ticdc-architecture-6.jpg) + +In the preceding diagram, a changefeed is created to replicate four tables to downstream. This changefeed is split into three Tasks, which are sent to the three Capture processes respectively in the TiCDC cluster. After TiCDC processes the data, the data is replicated to the downstream system. + +TiCDC supports replicating data to MySQL, TiDB, and Kafka databases. The preceding diagram only illustrates the process of data transfer at the changefeed level. The following sections describe in detail how TiCDC processes data, using Task1 that replicates table `table1` as an example. + +![TiCDC architecture](/media/ticdc/ticdc-architecture-5.jpg) + +1. Push data: When a data change occurs, TiKV pushes data to the Puller module. +2. Scan incremental data: The Puller module pulls data from TiKV when it finds the data changes received not continuous. +3. Sort data: The Sorter module sorts the data received from TiKV based on the timestamps and sends the sorted data to the Mounter module. +4. Mount data: After receiving the data changes, the Mounter module loads the data in a format that TiCDC sink can understand. +5. Replicate data: The Sink module replicates the data changes to the downstream. + +The upstream of TiCDC is the distributed relational database TiDB that supports transactions. When TiCDC replicates data, it should ensure the consistency of data and that of transactions when replicating multiple tables, which is a great challenge. The following sections introduce the key technologies and concepts used by TiCDC to address this challenge. + +## Key concepts of TiCDC + +For the downstream relational databases, TiCDC ensures the consistency of transactions in a single table and eventual transaction consistency in multiple tables. In addition, TiCDC ensures that any data change that has occurred in the upstream TiDB cluster can be replicated to the downstream at least once. + +### Architecture-related concepts + +- Capture: The process that runs the TiCDC node. Multiple Capture processes constitute a TiCDC cluster. Each Capture process is responsible for replicating data changes in TiKV, including receiving and actively pulling data changes, and replicating the data to the downstream. +- Capture Owner: The owner Capture among multiple Capture processes. Only one owner role exists in a TiCDC cluster at a time. The Capture Owner is responsible for scheduling data within the cluster. +- Processor: The logical thread inside Capture. Each Processor is responsible for processing the data of one or more tables in the same replication stream. A Capture node can run multiple Processors. +- Changefeed: A task that replicates data from an upstream TiDB cluster to a downstream system. A changefeed contains multiple Tasks, and each Task is processed by a Capture node. + +### Timestamp-related concepts + +TiCDC introduces a series of timestamps (TS) to indicate the status of data replication. These timestamps are used to ensure that data is replicated to the downstream at least once and that the consistency of data is guaranteed. + +#### ResolvedTS + +This timestamp exists in both TiKV and TiCDC. + +- ResolvedTS in TiKV: Represents the start time of the earliest transaction in a Region leader, that is, `ResolvedTS` = max(`ResolvedTS`, min(`StartTS`)). Because a TiDB cluster contains multiple TiKV nodes, the minimum ResolvedTS of the Region leader on all TiKV nodes is called the global ResolvedTS. The TiDB cluster ensures that all transactions before the global ResolvedTS are committed. Alternatively, you can assume that there are no uncommitted transactions before this timestamp. + +- ResolvedTS in TiCDC: + + - table ResolvedTS: Each table has a table-level ResolvedTS, which indicates all data changes in the table that are smaller than the Resolved TS have been received. To make it simple, this timestamp is the same as the minimum value of the ResolvedTS of all Regions corresponding to this table on the TiKV node. + - global ResolvedTS: The minimum ResolvedTS of all Processors on all TiCDC nodes. Because each TiCDC node has one or more Processors, each Processor corresponds to multiple table pipelines. + + For TiCDC, the ResolvedTS sent by TiKV is a special event in the format of ``. In general, the ResolvedTS satisfies the following constraints: + + ``` + table ResolvedTS >= global ResolvedTS + ``` + +#### CheckpointTS + +This timestamp exists only in TiCDC. It means that the data changes that occur before this timestamp have been replicated to the downstream system. + +- table CheckpointTS: Because TiCDC replicates data in tables, the table checkpointTS indicates all data changes that occur before CheckpointTS have been replicated at the table level. +- processor CheckpointTS: Indicates the minimum table CheckpointTS on a Processor. +- global CheckpointTS: Indicates the minimum CheckpointTS among all Processors. + +Generally, a checkpointTS satisfies the following constraint: + +``` +table CheckpointTS >= global CheckpointTS +``` + +Because TiCDC only replicates data smaller than the global ResolvedTS to the downstream, the complete constraint is as follows: + +``` +table ResolvedTS >= global ResolvedTS >= table CheckpointTS >= global CheckpointTS +``` + +After data changes and transactions are committed, the ResolvedTS on the TiKV node will continue to advance, and the Puller module on the TiCDC node keeps receiving data pushed by TiKV. The Puller module also decides whether to scan incremental data based on the data changes it has received, which ensures that all data changes are sent to the TiCDC node. + +The Sorter module sorts data received by the Puller module in ascending order according to the timestamp. This process ensures data consistency at the table level. Next, the Mounter module assembles the data changes from the upstream into a format that the Sink module can consume, and sends it to the Sink module. The Sink module replicates the data changes between the CheckpointTS and the ResolvedTS to the downstream in the order of the timestamp, and advances the checkpointTS after the downstream receives the data changes. + +The preceding sections only cover data changes of DML statements and do not include DDL statements. The following sections introduce the timestamp related to DDL statements. + +#### Barrier TS + +Barrier TS is generated when there are DDL change events or a Syncpoint is used. + +- DDL change events: Barrier TS ensures that all changes before the DDL statement are replicated to the downstream. After this DDL statement is executed and replicated, TiCDC starts replicating other data changes. Because DDL statements are processed by the Capture Owner, the Barrier TS corresponding to a DDL statement is only generated by the owner node. +- Syncpoint: When you enable the Syncpoint feature of TiCDC, a Barrier TS is generated by TiCDC according to the `sync-point-interval` you specified. When all table changes before this Barrier TS are replicated, TiCDC inserts the current global CheckpointTS as the primary TS to the table recording tsMap in downstream. Then TiCDC continues data replication. + +After a Barrier TS is generated, TiCDC ensures that only data changes that occur before this Barrier TS are replicated to downstream. Before these data changes are replicated to downstream, the replication task does not proceed. The owner TiCDC checks whether all target data has been replicated by continuously comparing the global CheckpointTS and the Barrier TS. If the global CheckpointTS equals to the Barrier TS, TiCDC continues replication after performing a designated operation (such as executing a DDL statement or recording the global CheckpointTS downstream). Otherwise, TiCDC waits for all data changes that occur before the Barrier TS to be replicated to the downstream. + +## Major processes + +This section describes the major processes of TiCDC to help you better understand its working principles. + +Note that the following processes occur only within TiCDC and are transparent to users. Therefore, you do not need to care about which TiCDC node you are starting. + +### Start TiCDC + +- For a TiCDC node that is not an owner, it works as follows: + + 1. Starts the Capture process. + 2. Starts the Processor. + 3. Receives the Task scheduling command executed by the Owner. + 4. Starts or stops tablePipeline according to the scheduling command. + +- For an owner TiCDC node, it works as follows: + + 1. Starts the Capture process. + 2. The node is elected as the Owner and the corresponding thread is started. + 3. Reads the changefeed information. + 4. Starts the changefeed management process. + 5. Reads the schema information in TiKV according to the changefeed configuration and the latest CheckpointTS to determine the tables to be replicated. + 6. Reads the list of tables currently replicated by each Processor and distributes the tables to be added. + 7. Updates the replication progress. + +### Stop TiCDC + +Usually, you stop a TiCDC node when you need to upgrade it or perform some planned maintenance operations. The process of stopping a TiCDC node is as follows: + +1. The node receives the command to stop itself. +2. The node sets its service status to unavailable. +3. The node stops receiving new replication tasks. +4. The node notifies the Owner node to transfer its data replication tasks to other nodes. +5. The node stops after the replication tasks are transferred to other nodes. diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index 970ec1b37ac55..822aafa11cb0c 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -52,7 +52,7 @@ TiCDC has the following key capabilities: > > Since v6.2, you can use the sink URI parameter [`transaction-atomicity`](/ticdc/ticdc-sink-to-mysql.md#configure-sink-uri-for-mysql-or-tidb) to control whether to split single-table transactions. Splitting single-table transactions can greatly reduce the latency and memory consumption of replicating large transactions. -## TiCDC architecture +## TiCDC architecture overview TiCDC is an incremental data replication tool for TiDB, which is highly available through PD's etcd. The replication process consists of the following steps: @@ -70,6 +70,8 @@ The components in the architecture diagram are described as follows: - TiCDC: TiCDC nodes where TiCDC processes run. Each node runs a TiCDC process. Each process pulls data changes from one or more tables in TiKV nodes and replicates the changes to the downstream system through the sink component. - PD: The scheduling module in a TiDB cluster. This module is responsible for scheduling cluster data and usually consists of three PD nodes. PD provides high availability through the etcd cluster. In the etcd cluster, TiCDC stores its metadata, such as node status information and changefeed configurations. +In implementation, both the [new architecture](/ticdc/ticdc-architecture.md) and the [classic architecture](/ticdc/ticdc-classic-architecture.md) of TiCDC are built on the same incremental data replication model. Compared with the classic architecture, the new architecture refactors and optimizes task scheduling and replication mechanisms, significantly improving the performance, scalability, and stability of real-time data replication while reducing resource costs. + As shown in the architecture diagram, TiCDC supports replicating data to TiDB, MySQL, Kafka, and storage services. ## Valid index @@ -157,6 +159,6 @@ Currently, the following scenarios are not supported: - The [`CREATE SEQUENCE` DDL operation](/sql-statements/sql-statement-create-sequence.md) and the [`SEQUENCE` function](/sql-statements/sql-statement-create-sequence.md#sequence-function) in TiDB. When the upstream TiDB uses `SEQUENCE`, TiCDC ignores `SEQUENCE` DDL operations/functions performed upstream. However, DML operations using `SEQUENCE` functions can be correctly replicated. - Currently, performing [TiDB Lightning physical import](/tidb-lightning/tidb-lightning-physical-import-mode.md) on tables and databases that are being replicated by TiCDC is not supported. For more information, see [Why does replication using TiCDC stall or even stop after data restore using TiDB Lightning and BR from upstream](/ticdc/ticdc-faq.md#why-does-replication-using-ticdc-stall-or-even-stop-after-data-restore-using-tidb-lightning-physical-import-mode-and-br-from-upstream). - Before v8.2.0, BR does not support [restoring data](/br/backup-and-restore-overview.md) for a cluster with TiCDC replication tasks. For more information, see [Why does replication using TiCDC stall or even stop after data restore using TiDB Lightning and BR from upstream](/ticdc/ticdc-faq.md#why-does-replication-using-ticdc-stall-or-even-stop-after-data-restore-using-tidb-lightning-physical-import-mode-and-br-from-upstream). -- Starting from v8.2.0, BR relaxes the restrictions on data restoration for TiCDC: if the `BackupTS` (the backup time) of the data to be restored is earlier than the changefeed [`CheckpointTS`](/ticdc/ticdc-architecture.md#checkpointts) (the timestamp that indicates the current replication progress), BR can proceed with the data restoration normally. Considering that the `BackupTS` is usually much earlier, it can be assumed that in most scenarios, BR supports restoring data for a cluster with TiCDC replication tasks. +- Starting from v8.2.0, BR relaxes the restrictions on data restoration for TiCDC: if the `BackupTS` (the backup time) of the data to be restored is earlier than the changefeed [`CheckpointTS`](/ticdc/ticdc-classic-architecture.md#checkpointts) (the timestamp that indicates the current replication progress), BR can proceed with the data restoration normally. Considering that the `BackupTS` is usually much earlier, it can be assumed that in most scenarios, BR supports restoring data for a cluster with TiCDC replication tasks. TiCDC only partially supports scenarios involving large transactions in the upstream. For details, refer to the [TiCDC FAQ](/ticdc/ticdc-faq.md#does-ticdc-support-replicating-large-transactions-is-there-any-risk), where you can find details on whether TiCDC supports replicating large transactions and any associated risks. diff --git a/ticdc/ticdc-server-config.md b/ticdc/ticdc-server-config.md index b21dc707eef8b..4fa90db8e7051 100644 --- a/ticdc/ticdc-server-config.md +++ b/ticdc/ticdc-server-config.md @@ -30,6 +30,12 @@ The following are descriptions of options available in a `cdc server` command: The following describes the configuration file specified by the `config` option in the `cdc server` command. You can find the default configuration file in [`pkg/cmd/util/ticdc.toml`](https://github.com/pingcap/tiflow/blob/master/pkg/cmd/util/ticdc.toml). +### `newarch` New in v8.5.4-release.1 + +- Controls whether to enable the [TiCDC new architecture](/ticdc/ticdc-architecture.md). +- Default value: `false`, indicating that the [TiCDC classic architecture](/ticdc/ticdc-classic-architecture.md) is used. +- When it is set to `true`, the TiCDC new architecture is enabled. + ### `addr`