-
Notifications
You must be signed in to change notification settings - Fork 118
feat(l1): log warning if no CL messages are received in a while #5256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Lines of code reportTotal lines added: Detailed view |
Log an error message if no messages are received from the consensus layer.
crates/networking/rpc/rpc.rs
Outdated
| loop { | ||
| let result = timeout(Duration::from_secs(30), timer_receiver.changed()).await; | ||
| if result.is_err() { | ||
| error!("No messages from the consensus layer. Is the consensus client running? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't sure whether to make this warn or error level. I figured error was more appropriate for end user scenarios, since Ethrex would always need to run alongside a consensus client, but it could get cumbersome in testing scenarios where the client is run with no consensus client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warn is better
Removed 'warn' from tracing imports in rpc.rs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a watchdog mechanism to detect and log warnings when no messages from the consensus layer are received within 30 seconds, helping users identify connectivity issues between Ethrex and the consensus client.
- Added a watch channel-based timer that resets on each auth RPC request
- Spawned a monitoring task that logs an error with troubleshooting hints after 30 seconds of inactivity
- Modified the auth RPC handler to send notifications through the watch channel
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } | ||
| }); | ||
|
|
||
| let authrpc_handler = async move |ctx, auth, body| { |
Copilot
AI
Nov 11, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The closure signature is incompatible with axum's handler requirements. The async move should come before the pipe characters. The correct syntax is move |ctx, auth, body| async move { ... } to create a closure that returns a future, rather than an async closure.
| let authrpc_handler = async move |ctx, auth, body| { | |
| let authrpc_handler = move |ctx, auth, body| async move { |
crates/networking/rpc/rpc.rs
Outdated
| error!("No messages from the consensus layer. Is the consensus client running? | ||
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." |
Copilot
AI
Nov 11, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Multi-line string literals should use proper formatting. Consider using a raw string literal or concatenation to avoid the leading whitespace on line 307 which will appear in the logged error message.
| error!("No messages from the consensus layer. Is the consensus client running? | |
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." | |
| error!( | |
| "No messages from the consensus layer. Is the consensus client running?\n\ | |
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." |
crates/networking/rpc/rpc.rs
Outdated
| if result.is_err() { | ||
| error!("No messages from the consensus layer. Is the consensus client running? | ||
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." | ||
| ); |
Copilot
AI
Nov 11, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The spawned task will panic if the sender is dropped. When timer_receiver.changed() returns an error (which occurs when all senders are dropped), the code logs an error but continues looping. This will cause continuous error logging. The loop should break when changed() returns Err, which indicates the channel is closed.
| if result.is_err() { | |
| error!("No messages from the consensus layer. Is the consensus client running? | |
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." | |
| ); | |
| match result { | |
| Ok(Ok(_)) => { | |
| // Successfully received a change, continue looping. | |
| } | |
| Ok(Err(_)) | Err(_) => { | |
| error!("No messages from the consensus layer. Is the consensus client running? | |
| Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match." | |
| ); | |
| break; | |
| } |
Motivation
Making it clearer when the client isn't working properly
Description
This PR adds an extra thread which keeps track of whether a message from the consensus layer hasn't been received in the past 30 seconds. This is done by having the auth RPC handler communicate with it when a new message is received to reset the countdown. If no messages have been received after 30 seconds, a message is printed to warn the user, along with some hints as to possible reasons why Ethrex isn't detecting the consensus client.
Closes #5234