-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(dashboard): add backpressure to the relation dependency graph #18280
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kwannoel
force-pushed
the
kwannoel/dag-mv
branch
2 times, most recently
from
August 28, 2024 08:00
1e39e39
to
12567d1
Compare
kwannoel
force-pushed
the
kwannoel/dag-mv
branch
from
August 29, 2024 09:02
12567d1
to
8574c4b
Compare
kwannoel
force-pushed
the
kwannoel/dag-mv
branch
from
September 16, 2024 09:07
d7e2349
to
e5fc1b0
Compare
kwannoel
force-pushed
the
kwannoel/dag-mv
branch
from
September 17, 2024 04:10
e5fc1b0
to
a62b4e6
Compare
Co-authored-by: Eric Fu <eric@singularity-data.com>
Co-authored-by: Eric Fu <eric@singularity-data.com>
kwannoel
changed the title
feat(dashboard): add ddl backpressure graph
feat(dashboard): add relation backpressure graph
Sep 19, 2024
xxchan
changed the title
feat(dashboard): add relation backpressure graph
feat(dashboard): add backpressure to the relation dependency graph
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
reviewed
Sep 19, 2024
xxchan
approved these changes
Sep 19, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you. The UI looks good! (Didn't check backpressure computation logic.
github-actions bot
pushed a commit
that referenced
this pull request
Sep 20, 2024
…18280) Co-authored-by: Eric Fu <eric@singularity-data.com>
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Much of the code is borrowed from
FragmentGraph
andDependencyGraph
components. Thanks for @xxchan and @BugenZhao who worked on that previously.This PR adds a ddl level backpressure graph, to identify bottleneck MVs and Sinks in the stream graph.
From here, we can observe the specific fragment's backpressure graph by using the search box in the fragment graph page.
We also add a
resetBackPressure
button. This is for resetting embedded backpressures. Otherwise if there's a long idle period, followed by a sudden surge, the backpressure graph will be shown to be uncongested, since the average backpressure will include the idle period. By resetting the backpressure graph, we can restart the computation of average backpressure from the current point in time, and get a more accurate visualization.Implementation details
The graph has
relations
, identified byrelationId
as its vertices.And the edges are backpressures between the relations.
To render the vertices, we fetch all relation metadata from the meta-node.
The relation metadata includes the
relation_id, schema_name, relation_name
, as well as theparentIds
.relation_id
andparentIds
will be used to render the relation DAG, since they form the vertices of the graph.Next, to render the edges, we have to remap our fragment level backpressure edges to relation level backpressure edges.
First, we obtain a mapping of input
fragmentId
and outputfragmentId
to their relation's ids.An input
fragmentId
corresponds to a fragment which contains stream scan for instance.An output
fragmentId
corresponds to a materialize fragment.Next, our backpressure info contains a mapping of
fragmentId-fragmentId
tobackpressure values
.We can map output fragment id to its corresponding relation, and likewise for input fragment id.
If it doesn't exist, we know that the edge can be removed, since it doesn't correspond to a relation-relation edge, and is internal to the relation itself.
From this we obtain a new mapping of
relationId-relationId
tobackpressure values
. Then we can use this to construct our backpressure edges for the ddl backpressure graph.Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.