perf(tree): worker pooling for storage in multiproof generation #18887
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Added configuration for maximum and minimum storage proof workers. - Implemented a worker pool for processing storage proof tasks, improving efficiency by reusing transactions. - Updated `ProofTaskManager` to handle storage proof tasks via a dedicated channel. - Enhanced metrics to track storage proof requests and fallback scenarios. - Adjusted existing tests to accommodate the new storage worker functionality.
There was a problem hiding this comment.
Pull Request Overview
This PR introduces worker pooling for storage proof generation to improve multiproof performance through parallel processing. The implementation adds dedicated storage workers with long-lived database transactions while maintaining backward compatibility.
- Introduces worker pool architecture with pre-spawned workers for storage proof computation
- Replaces
ProofTaskManagerinternal task handling with mixed worker pool and on-demand execution - Adds crossbeam channels for efficient worker communication and transaction budget allocation
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
crates/trie/parallel/src/proof_task_metrics.rs |
Adds metrics tracking for storage proof workers and fallback scenarios |
crates/trie/parallel/src/proof_task.rs |
Major refactor introducing worker pool architecture and new task routing logic |
crates/trie/parallel/src/proof.rs |
Updates test to use new constructor signature with storage worker count |
crates/trie/parallel/Cargo.toml |
Adds crossbeam-channel dependency for worker communication |
crates/engine/tree/src/tree/payload_processor/multiproof.rs |
Updates test configuration for new worker pool parameters |
crates/engine/tree/src/tree/payload_processor/mod.rs |
Integrates storage worker configuration into payload processor |
crates/engine/primitives/src/config.rs |
Adds configuration options for storage proof worker pool sizing |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
- Enhanced documentation for `StorageProofJob` to clarify its current unused status and potential for future type-safe design. - Updated comments in `ProofTaskManager` regarding the handling of on-demand tasks and the possibility of refactoring to a more type-safe enum. - Improved logging for worker pool disconnection scenarios, emphasizing fallback to on-demand execution.
…Metrics and ProofTaskTrieMetrics
…clarity - Updated comments in `ProofTaskManager` to enhance clarity regarding on-demand transaction handling and queue management. - Renamed `pending_on_demand` to `on_demand_queue` for better understanding of its purpose. - Adjusted the `new` function documentation to reflect the correct allocation of concurrency budget between storage workers and on-demand transactions. - Improved the `queue_proof_task` method to use the new queue name.
…ement - Removed the unused `OnDemandTask` enum and updated comments in `ProofTaskManager` to clarify the distinction between storage worker pool and on-demand execution. - Enhanced documentation to better describe the public interface and task submission process. - Improved clarity regarding transaction handling and execution paths for proof requests.
- Eliminated the `storage_proof_workers` field and related constants from `TreeConfig`. - Updated the default implementation and related methods to reflect the removal, streamlining the configuration structure.
yongkangc
requested review from
rkrasiuk,
Rjected,
shekhirin,
mediocregopher,
mattsse,
fgimenez and
gakonst
as code owners
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
- Improved comments in `ProofTaskManager` and related functions for better clarity on task management and processing. - Updated queue capacity calculation to use 4x buffering, reducing fallback to slower on-demand execution during burst loads. - Removed redundant variable assignments to streamline the code.
|
Benching at 3fb97c6 on reth4 mainnet 
|
mediocregopher
requested changes
Oct 7, 2025
shekhirin
reviewed
Oct 8, 2025
shekhirin
reviewed
Oct 8, 2025
shekhirin
reviewed
Oct 8, 2025
Co-authored-by: Alexey Shekhirin <5773434+shekhirin@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <5773434+shekhirin@users.noreply.github.com>
shekhirin
reviewed
Oct 8, 2025
shekhirin
approved these changes
Oct 8, 2025
yongkangc
force-pushed
the
yk/worker_pool_storage_2
branch
from
3cd7546 to
f823c6b
Compare
Bench on Base cc @mattsse |
yongkangc
force-pushed
the
yk/worker_pool_storage_2
branch
from
a8fef7e to
4ca404e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR add worker pooling to our existing proof generation. This PR starts by adding worker pooling to storage proofs, which allows us to reduce overhead from spawning the worker.
Core logic:
Issues closed:
ProofTaskManager#18735References:
Next Steps:
Benches:
Reth4 Mainnet
