[multicluster] Throttling concurrent list/watch requests during multicluster startup

[zoumo]

Description

When initializing, the current multicluster setup initiates concurrent list/watch requests to all clusters. This approach can lead to significant strain on the controller's CPU, memory, and bandwidth resources, especially when dealing with a large dataset.

Issue

High resource utilization during startup may not only impact the performance of the multicluster system but can also lead to potential timeouts or slow responses from the clusters being managed. This is particularly problematic in environments where resource constraints are strict or during peak operational hours.

Proposed Solution

It would be beneficial to implement a rate-limiting feature within the multicluster architecture. Specifically, introducing a configurable maximum number of concurrent connections during the startup phase could help mitigate the load on the system's resources.

By rate-limiting the number of concurrent list/watch requests, we can reduce the immediate impact on the CPU, memory, and bandwidth, and ensure a more stable and reliable startup process.

Expected Benefits

• Reduced CPU and memory usage during the startup phase
• Lowered network bandwidth consumption
• Decreased risk of timeouts and improved response times from the clusters
• Better overall system stability and reliability

I believe this feature would be a valuable addition to the multicluster project and would appreciate consideration for its implementation.