Implement Exponential Backoff for Transient Sync Errors #588
Conversation
…ith retry delay multiplier
|
👋 Thanks for assigning @tnull as a reviewer! |
| @@ -504,8 +506,23 @@ impl ChainSource { | |||
|
|
|||
| Err(e) => { | |||
| log_error!(logger, "Failed to synchronize chain listeners: {:?}", e); | |||
| tokio::time::sleep(Duration::from_secs(CHAIN_POLLING_INTERVAL_SECS)) | |||
| .await; | |||
| backoff_delay_multiplier += 1; | |||
There was a problem hiding this comment.
I don't think we should let the backoff delay grow infinitely. Please include a sane upper bound.
| } else { | ||
| log_error!( | ||
| logger, | ||
| "Failed to synchronize chain listeners: {:?}, e" |
There was a problem hiding this comment.
This doesn't compile. Also, as discussed elsewhere, we at least need to error out. But technically, we should keep retrying currently as otherwise we'd need to add panic here as permanently being unable to connect to the chain source is an unrecoverable error.
Yeah, just saw that, would check it out now.. |
There was a problem hiding this comment.
Please also make sure to always provide rationale for the change in the commit message, and format it properly (heading shouldn't be larger than one line). Feel free to refer to https://cbea.ms/git-commit/ on how to write good commit messages.
This PR attempts to improve retry in the bitcoind RPC synchronization loop from #587 by replacing the existing linear backoff with a proper exponential backoff strategy. We could also add a maximum delay to prevent excessively long waits, but I am not sure if it would really help.