-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: detect invalid buffered block when insert fails #10217
Conversation
crates/engine/tree/src/tree/mod.rs
Outdated
@@ -1391,11 +1391,14 @@ where | |||
} | |||
Err(err) => { | |||
debug!(target: "engine", ?err, "failed to connect buffered block to tree"); | |||
if let Err(fatal) = self.on_insert_block_error(err) { | |||
warn!(target: "engine", ?fatal, "fatal error occurred while connecting buffered blocks"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fatal is mislead here imo because fatal means that this block is invalid not that a fatal error happened?
could we rephrase this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Validation errors should be filtered out and handled in insert_block_error
, the type in the Err
is an InsertBlockFatalError
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, my point is how should "fatal" be interpreted if a user sees this on the terminal
fatal error occurred while connecting buffered blocks
this lacks context that this means only the insert routine was fatal, but I'd like to avoid logging "fatal" here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because we don't terminate the task here, right? so it's not fatal, but maybe it should?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because we don't terminate the task here, right? so it's not fatal, but maybe it should?
Yeah, I guess we should bubble these up, since it would be an error that we can't otherwise do anything with
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, no I fully understood it, this is indeed a fatal internal error but can don't have an easy way to terminate the service, since we don't really expect this case under normal ops, this is appropriate until we find a way to either terminate the task or emit an error message
crates/engine/tree/src/tree/mod.rs
Outdated
@@ -1391,11 +1391,14 @@ where | |||
} | |||
Err(err) => { | |||
debug!(target: "engine", ?err, "failed to connect buffered block to tree"); | |||
if let Err(fatal) = self.on_insert_block_error(err) { | |||
warn!(target: "engine", ?fatal, "fatal error occurred while connecting buffered blocks"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this impls display I believe?
warn!(target: "engine", ?fatal, "fatal error occurred while connecting buffered blocks"); | |
warn!(target: "engine", %fatal, "fatal error occurred while connecting buffered blocks"); |
crates/engine/tree/src/tree/mod.rs
Outdated
if let Err(fatal) = self.on_insert_block_error(err) { | ||
warn!(target: "engine", ?fatal, "fatal error occurred while inserting downloaded block"); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should consider adding an error variant to engineapievent so that we bubble up this info.
since this is indeed fatal, this makes sense for now
This progresses us further in the
Invalid Missing Ancestor Syncing ReOrg
test, and allows us to actually detect the invalid ancestor block. Instead of justdebug
logging on each insert block error, we now perform theon_insert_block_error
check whenever we get an error. This will inspect the error and insert it into the invalid headers cache if it was an actual invalid block.