Export merge tree partition on replicated engine

[arthurpassos]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• Tiered Storage (2h)

[arthurpassos]

Very drafty for now. Created the PR so I can take notes of things I still need to implement / ideas. Also to get cicd report

[github-actions]

Workflow [PR], commit [8948a90]

[arthurpassos]

TODO list from the top of my head:

1. Keep parts alive (done, not tested)
2. Improve zk communication (batch calls mainly)
3. Cleanup the code
4. Setting for max_retry_count
5. Preserve context from the time of export partition call
6. Document the code and the feature
7. Check if replica is up to date enough
8. Local & per part exponential back-off?
9. exceptions directory in zookeeper (done)
10. system table for replicated partitions (done)
11. ability to kill an export?
12. Setting to overwrite existing zk task (done)

[arthurpassos]

Keep parts alive

The first thing that comes to mind is to create an in-memory list of references to those parts as soon as the replicas get notified about a partition export. All replicas will references to all parts, even if they are not responsible for exporting a given part. This should be restored upon bootup.

There is one small time window in which this could fail: if between the time the export was published in zookeeper and the replicas are notified the part is completely deleted from all replicas. It seems rare enough and we can probably live with it.