Add diagnostics for stuck FTE scheduler #19879
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
losipiuk
added
the
no-release-notes
findepi
approved these changes
Nov 24, 2023
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
4 times, most recently
from
67fb915 to
c4ab707
Compare
|
Updated @findepi , @wweiss-starburst PTAL |
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
from
c4ab707 to
885f11a
Compare
findepi
reviewed
Nov 27, 2023
| @@ -675,7 +676,7 @@ private Optional<Throwable> closeAndAddSuppressed(Optional<Throwable> existingFa | |||
| private boolean processEvents() | |||
| { | |||
| try { | |||
| Event event = eventQueue.poll(1, MINUTES); | |||
| Event event = eventQueue.poll(EVENT_PROCESSING_ENFORCED_FREQUENCY.toMillis(), MILLISECONDS); | |||
There was a problem hiding this comment.
or call it EVENT_PROCESSING_ENFORCED_FREQUENCY_MILLIS
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
2 times, most recently
from
9100c98 to
903b8ea
Compare
|
Some updates - unfortunatelly we cannot just filter out empty SplitAssignmentEvents. |
losipiuk
commented
Nov 27, 2023
Comment on lines
1417
to
1673
| // we need to process event empty events here so stageExecution.taskDescriptorLoadingComplete() | ||
| // is called in event handler. Otherwise IdempotentSplitSource may be not called again | ||
| // if there is no other SplitAssignmentEvent for this stage in queue. |
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
from
903b8ea to
94bbc08
Compare
findepi
reviewed
Nov 28, 2023
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/faulttolerant/SplitAssigner.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/faulttolerant/SplitAssigner.java
Outdated
Show resolved
Hide resolved
findepi
approved these changes
Nov 28, 2023
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/exchange/ExchangeSourceOutputSelector.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/exchange/ExchangeSourceOutputSelector.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/exchange/ExchangeSourceOutputSelector.java
Show resolved
Hide resolved
findepi
reviewed
Nov 28, 2023
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
.../java/io/trino/execution/scheduler/faulttolerant/EventDrivenFaultTolerantQueryScheduler.java
Outdated
Show resolved
Hide resolved
Also add default implementations for EventListener methods which delegate to appropriate intermediate method according to class hierachy.
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
2 times, most recently
from
04b1d30 to
c6eb8c6
Compare
Add code which will dump log debug information in case FTE scheduler is not getting any events for 10 minutes. This is to track rare bug where we observe queries running with retry_policy set to FALSE stuck sometimes.
losipiuk
force-pushed
the
lo/fte-diagnostics
branch
from
c6eb8c6 to
f4d34fc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add code which will dump log debug information in case FTE scheduler is not getting any events for 10 minutes. This is to track rare bug where we observe queries running with retry_policy set to FALSE stuck sometimes.
TODO: