Skip to content

Conversation

@SDartayet
Copy link
Contributor

@SDartayet SDartayet commented Nov 10, 2025

Motivation

Making it clearer when the client isn't working properly

Description

This PR adds an extra thread which keeps track of whether a message from the consensus layer hasn't been received in the past 30 seconds. This is done by having the auth RPC handler communicate with it when a new message is received to reset the countdown. If no messages have been received after 30 seconds, a message is printed to warn the user, along with some hints as to possible reasons why Ethrex isn't detecting the consensus client.

Closes #5234

@github-actions
Copy link

github-actions bot commented Nov 10, 2025

Lines of code report

Total lines added: 13
Total lines removed: 0
Total lines changed: 13

Detailed view
+-------------------------------------+-------+------+
| File                                | Lines | Diff |
+-------------------------------------+-------+------+
| ethrex/crates/networking/rpc/rpc.rs | 848   | +13  |
+-------------------------------------+-------+------+

@SDartayet SDartayet changed the title Added warning for no CL messages feat(l1) warn if no CL messages are received in a while Nov 10, 2025
Log an error message if no messages are received from the consensus layer.
loop {
let result = timeout(Duration::from_secs(30), timer_receiver.changed()).await;
if result.is_err() {
error!("No messages from the consensus layer. Is the consensus client running?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure whether to make this warn or error level. I figured error was more appropriate for end user scenarios, since Ethrex would always need to run alongside a consensus client, but it could get cumbersome in testing scenarios where the client is run with no consensus client.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warn is better

@SDartayet SDartayet marked this pull request as ready for review November 10, 2025 21:51
@SDartayet SDartayet requested a review from a team as a code owner November 10, 2025 21:51
@SDartayet SDartayet changed the title feat(l1) warn if no CL messages are received in a while feat(l1): log error if no CL messages are received in a while Nov 10, 2025
@github-actions github-actions bot added the L1 Ethereum client label Nov 10, 2025
Removed 'warn' from tracing imports in rpc.rs
@mpaulucci mpaulucci requested a review from Copilot November 11, 2025 14:28
Copilot finished reviewing on behalf of mpaulucci November 11, 2025 14:30
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a watchdog mechanism to detect and log warnings when no messages from the consensus layer are received within 30 seconds, helping users identify connectivity issues between Ethrex and the consensus client.

  • Added a watch channel-based timer that resets on each auth RPC request
  • Spawned a monitoring task that logs an error with troubleshooting hints after 30 seconds of inactivity
  • Modified the auth RPC handler to send notifications through the watch channel

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

}
});

let authrpc_handler = async move |ctx, auth, body| {
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The closure signature is incompatible with axum's handler requirements. The async move should come before the pipe characters. The correct syntax is move |ctx, auth, body| async move { ... } to create a closure that returns a future, rather than an async closure.

Suggested change
let authrpc_handler = async move |ctx, auth, body| {
let authrpc_handler = move |ctx, auth, body| async move {

Copilot uses AI. Check for mistakes.
Comment on lines 306 to 307
error!("No messages from the consensus layer. Is the consensus client running?
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-line string literals should use proper formatting. Consider using a raw string literal or concatenation to avoid the leading whitespace on line 307 which will appear in the logged error message.

Suggested change
error!("No messages from the consensus layer. Is the consensus client running?
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."
error!(
"No messages from the consensus layer. Is the consensus client running?\n\
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."

Copilot uses AI. Check for mistakes.
Comment on lines 305 to 308
if result.is_err() {
error!("No messages from the consensus layer. Is the consensus client running?
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."
);
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spawned task will panic if the sender is dropped. When timer_receiver.changed() returns an error (which occurs when all senders are dropped), the code logs an error but continues looping. This will cause continuous error logging. The loop should break when changed() returns Err, which indicates the channel is closed.

Suggested change
if result.is_err() {
error!("No messages from the consensus layer. Is the consensus client running?
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."
);
match result {
Ok(Ok(_)) => {
// Successfully received a change, continue looping.
}
Ok(Err(_)) | Err(_) => {
error!("No messages from the consensus layer. Is the consensus client running?
Check the auth JWT coincides with the one used by the CL and the auth RPC address and port used by it and Ethrex match."
);
break;
}

Copilot uses AI. Check for mistakes.
@SDartayet SDartayet changed the title feat(l1): log error if no CL messages are received in a while feat(l1): log warning if no CL messages are received in a while Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Print warning if we don't receive any message from CL in a while

3 participants