fix: yield to event loop between block imports during sync#8925
fix: yield to event loop between block imports during sync#8925nflaig merged 1 commit intoChainSafe:unstablefrom
Conversation
During finalized sync, blocks are imported in a tight loop without yielding to the event loop. With async BLS verification (no EL newPayload await), this prevents the checkpoint state cache processState() cleanup from running, causing unbounded state accumulation and OOM on memory-constrained hosts. Add nextEventLoop() after each importBlock to allow pending async cleanup (state serialization, cache eviction) to execute between block imports. Co-authored-by: Cayman <caymannava@gmail.com>
Summary of ChangesHello @lodekeeper, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical Out-Of-Memory (OOM) issue that occurred during finalized block synchronization, particularly on systems with limited memory. By strategically yielding control to the event loop after each block import, it ensures that essential asynchronous cleanup and state management processes can execute, preventing memory exhaustion and improving the stability of the beacon node. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively addresses a critical out-of-memory issue during finalized sync by introducing a yield to the event loop between block imports. This change is crucial for allowing asynchronous cleanup processes, such as state serialization and cache eviction, to execute, thereby preventing unbounded memory accumulation. The solution is well-explained and directly targets the identified root cause.
| ); | ||
|
|
||
| for (const fullyVerifiedBlock of fullyVerifiedBlocks) { | ||
| // No need to sleep(0) here since `importBlock` includes a disk write |
There was a problem hiding this comment.
The removal of this comment is appropriate. The previous statement about importBlock including a disk write and thus not needing sleep(0) is no longer accurate given the asynchronous BLS verification and the explicit addition of await nextEventLoop() to yield to the event loop. This improves code clarity by removing potentially misleading information.
There was a problem hiding this comment.
- wrong, it is a stale comment as of feat: async block import persistence #8784
There was a problem hiding this comment.
Right — the old comment assumed importBlock always included a blocking disk write, which was true before #8784 moved verification async. That natural yield point is now gone, so the explicit nextEventLoop() replaces it.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #8925 +/- ##
=========================================
Coverage 52.33% 52.33%
=========================================
Files 848 848
Lines 63429 63429
Branches 4702 4702
=========================================
Hits 33195 33195
Misses 30165 30165
Partials 69 69 🚀 New features to boost your workflow:
|
During test teardown (and production shutdown), the callInNextEventLoop callback for producing common block body can fire after the chain's abort signal has triggered. This causes getBlockSlotState to throw QUEUE_ERROR_QUEUE_ABORTED, which propagates as an unhandled promise rejection since no one is awaiting the deferred. This fix: 1. Guards against calling produceCommonBlockBody after the controller signal has been aborted 2. Silently handles queue abort errors during shutdown instead of propagating them as unhandled rejections 3. Increases prover E2E hook timeout from 3 to 4 epochs of headroom to accommodate slower CI runners and block import yields (ChainSafe#8925) Fixes flaky E2E CI failures where all 44 test files pass but Vitest exits with code 1 due to the unhandled rejection.
During test teardown (and production shutdown), the callInNextEventLoop callback for producing common block body can fire after the chain's abort signal has triggered. This causes getBlockSlotState to throw QUEUE_ERROR_QUEUE_ABORTED, which propagates as an unhandled promise rejection since no one is awaiting the deferred. This fix: 1. Guards against calling produceCommonBlockBody after the controller signal has been aborted 2. Silently handles queue abort errors during shutdown instead of propagating them as unhandled rejections 3. Increases prover E2E hook timeout from 3 to 4 epochs of headroom to accommodate slower CI runners and block import yields (ChainSafe#8925) Fixes flaky E2E CI failures where all 44 test files pass but Vitest exits with code 1 due to the unhandled rejection.
3 participants
Description
During finalized sync, blocks are imported in a tight loop without yielding to the event loop. When BLS verification is async (not awaiting the execution engine's
newPayload), this prevents the checkpoint state cache'sprocessState()cleanup from running, causing unbounded state accumulation and OOM on memory-constrained hosts.This adds
nextEventLoop()after eachimportBlockto allow pending async cleanup (state serialization, cache eviction) to execute between block imports.Root Cause
In
processBlocks(), the import loop previously had a natural yield point whenimportBlockawaited the execution engine. With async BLS via@chainsafe/blst(PR #8900),importBlockno longer blocks on the EL call, so blocks blast through without yielding. The checkpoint state cache'sprocessState()— which runs as fire-and-forget async — never gets a chance to serialize and evict old states, causing memory to climb until OOM.Fix
nextEventLoop()is an existing utility (sleep(0)) that yields to the event loop's timers phase, giving pending microtasks and macrotasks a chance to execute.Testing
Deployed as commit 983d923 on feat3 infrastructure nodes — resolved OOM crash loops on nodes with sufficient memory. Memory-constrained hosts (~16GB) may need additional tuning (reduced
maxBlockStates, queue depth limits).Co-authored-by: Cayman caymannava@gmail.com
Note
This PR was authored with AI assistance (Claude). All code has been reviewed and validated.