Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TimedRunnable log NPE #4425

Merged
merged 2 commits into from
Jun 10, 2024
Merged

Conversation

AnonHxy
Copy link
Contributor

@AnonHxy AnonHxy commented Jun 9, 2024

Descriptions of the changes in this PR:

Motivation

When the ReadEntryProcessor runs slow, the TimedRunnable will print the warn log like below:

Runnable xx:xx took too long xx micros to execute.

public void run() {
taskPendingStats.registerSuccessfulEvent(MathUtils.elapsedNanos(initNanos), TimeUnit.NANOSECONDS);
long startNanos = MathUtils.nowInNano();
try {
this.runnable.run();
} finally {
long elapsedMicroSec = MathUtils.elapsedMicroSec(startNanos);
taskExecutionStats.registerSuccessfulEvent(elapsedMicroSec, TimeUnit.MICROSECONDS);
if (elapsedMicroSec >= warnTimeMicroSec) {
log.warn("Runnable {}:{} took too long {} micros to execute.", runnable, runnable.getClass(),
elapsedMicroSec);
}

However we found that NPE happened in this log:

WARN  [BookieReadThreadPool-OrderedExecutor-1-0:OrderedExecutor$TimedRunnable@206] - Runnable [!!!org.apache.bookkeeper.proto.ReadEntryProcessor@1dd094f3=>java.lang.NullPointerException:Cannot invoke "org.apache.bookkeeper.proto.BookieProtocol$ReadRequest.getLedgerId()" because "this.request" is null!!!]:class org.apache.bookkeeper.proto.ReadEntryProcessor took too long 1125904 micros to execute.

The root cause is that the ReadEntryProcessor will be recycled after calling run method, which means we shouldn't call any method of the ReadEntryProcessor after calling ReadEntryProcessor#run(). So line206 calling toString() method of runnable(ReadEntryProcessor) will cause NPE or any other wrong message.

Changes

Save runnable string name and class when create TimedRunnable and TimedCallable

Copy link
Contributor

@dlg99 dlg99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!
One note about perf implications of the change


TimedRunnable(Runnable runnable) {
this.runnable = runnable;
this.initNanos = MathUtils.nowInNano();
this.runnableString = runnable.toString();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toString() can be expensive; maybe skip this and only log the class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice suggestion. Updated @dlg99

@hezhangjian hezhangjian merged commit 43b5ccc into apache:master Jun 10, 2024
23 checks passed
hezhangjian pushed a commit that referenced this pull request Jun 10, 2024
lhotari pushed a commit that referenced this pull request Jun 13, 2024
(cherry picked from commit 43b5ccc)
Ghatage pushed a commit to sijie/bookkeeper that referenced this pull request Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants