ticdc: add description about ticdc new architecture (#20597) #21937
Conversation
ti-chi-bot
added
area/ticdc
Summary of ChangesHello @ti-chi-bot, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the documentation surrounding TiCDC's architecture by introducing a detailed description of its new architecture, which promises improved performance, scalability, and stability. It clearly delineates between the new and classic architectures, providing guidance on their respective use cases, features, and operational considerations. The changes also include updates to monitoring, configuration, and various cross-references to ensure consistency and clarity throughout the documentation. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces documentation for the new TiCDC architecture by splitting the existing architecture document into 'new' and 'classic' versions. The changes are well-structured and provide a good overview of the new architecture. I've provided a few suggestions to fix some minor grammatical errors, an incorrect YAML example, and an inconsistent link to improve the documentation's accuracy and clarity.
| version: v8.5.4-release.1 | ||
| replicas: 3 | ||
| config: | ||
| newarch = true |
There was a problem hiding this comment.
The YAML syntax in this example is incorrect. The config field should be a map with a key-value pair. Please correct it to newarch: true.
| newarch = true | |
| newarch: true |
| ## Monitoring | ||
|
|
||
| Usually, you stop a TiCDC node when you need to upgrade it or perform some planned maintenance operations. The process of stopping a TiCDC node is as follows: | ||
| Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard on Grafana, you need to manually import the [TiCDC monitoring metrics file](https://github.com/pingcap/ticdc/blob/master/metrics/grafana/ticdc_new_arch.json). |
There was a problem hiding this comment.
The link to the monitoring metrics file is incorrect as it points to a GitHub page instead of the raw file, and it refers to the master branch which might not be appropriate for a release document. It's also inconsistent with the link in ticdc/monitor-ticdc.md. Please use the raw content link pointing to the specific release branch.
| Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard on Grafana, you need to manually import the [TiCDC monitoring metrics file](https://github.com/pingcap/ticdc/blob/master/metrics/grafana/ticdc_new_arch.json). | |
| Currently, the monitoring dashboard **TiCDC-New-Arch** for the TiCDC new architecture is not managed by TiUP yet. To view this dashboard on Grafana, you need to manually import the [TiCDC monitoring metrics file](https://raw.githubusercontent.com/pingcap/ticdc/refs/heads/release-8.5/metrics/grafana/ticdc_new_arch.json). |
| - Log Service: as the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disks. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter. | ||
| - Downstream Adapter: as the downstream data replication adaptation layer, Downstream Adapter handles user-initiated changefeed operations. It schedules and generates related replication tasks, fetches data from the Log Service, and replicates the fetched data to downstream systems. |
There was a problem hiding this comment.
For consistency with other lists in this document, it's better to capitalize the first letter of the description for each component.
| - Log Service: as the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disks. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter. | |
| - Downstream Adapter: as the downstream data replication adaptation layer, Downstream Adapter handles user-initiated changefeed operations. It schedules and generates related replication tasks, fetches data from the Log Service, and replicates the fetched data to downstream systems. | |
| - Log Service: As the core data service layer, Log Service fetches information such as row changes and DDL events from the upstream TiDB cluster, and then temporarily stores the change data on local disks. It also responds to data requests from the Downstream Adapter, periodically merging and sorting DML and DDL data and pushing the sorted data to the Downstream Adapter. | |
| - Downstream Adapter: As the downstream data replication adaptation layer, Downstream Adapter handles user-initiated changefeed operations. It schedules and generates related replication tasks, fetches data from the Log Service, and replicates the fetched data to downstream systems. |
|
|
||
| ## Key concepts of TiCDC | ||
| - In the classic architecture: for single-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1`, the old table name `test.t1` matches the rule, so it will be replicated. For a multi-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;`, because the new table name `ignore.t1` does not match the rule, it will not be replicated. | ||
| - In the new TiCDC architecture: because the old table names in both `RENAME TABLE test.t1 TO ignore.t1` and `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` match the rules, both DDL statements will be replicated. |
There was a problem hiding this comment.
Minor grammar fix: capitalize 'Because' at the start of the list item and use 'rule' (singular) as it refers to a single filter rule in the example.
| - In the new TiCDC architecture: because the old table names in both `RENAME TABLE test.t1 TO ignore.t1` and `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` match the rules, both DDL statements will be replicated. | |
| - In the new TiCDC architecture: Because the old table names in both `RENAME TABLE test.t1 TO ignore.t1` and `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` match the rule, both DDL statements will be replicated. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: qiancai The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
2 participants
This is an automated cherry-pick of #20597
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?