Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[POC] Query-level resource usages instrumentation with SearchPhaseResult #12449

Closed
wants to merge 1 commit into from
Closed

[POC] Query-level resource usages instrumentation with SearchPhaseResult #12449

wants to merge 1 commit into from

Conversation

ansjcy
Copy link
Member

@ansjcy ansjcy commented Feb 24, 2024 •
edited
Loading

Bare minimum changes to:

  • Instrument resource usages before a task finishes on a data node, more specifically, get resource usages at the last step (serialization) before a phase response is sent from query/fetch/.. phase.
  • Piggyback the resource usages data with phase results.
  • Gather resource usages for all shard search tasks on coordinator node (in query insights plugin) to get the query-level resource usage.

This draft only added instrumentation for query phase and fetch phase.

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ansjcy ansjcy marked this pull request as draft February 24, 2024 01:56
@ansjcy ansjcy changed the title instrument resource usage before task finishes on data nodes and gath… Query-level resource usages instrumentation Feb 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 05d268d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change 05d268d

Incompatible components

Skipped components

Compatible components

@ansjcy ansjcy mentioned this pull request Feb 24, 2024
Closed
ansjcy
ansjcy commented Feb 26, 2024
View reviewed changes
libs/core/src/main/java/org/opensearch/core/tasks/resourcetracker/TaskResourceUsage.java
@@ -90,7 +90,7 @@ public static TaskResourceUsage fromXContent(XContentParser parser) {

@Override
public String toString() {
return Strings.toString(MediaTypeRegistry.JSON, this, true, true);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for debugging purpoose only

@opensearch-trigger-bot
Copy link
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added the stalled Issues that have stalled label Mar 28, 2024
@ansjcy ansjcy mentioned this pull request Apr 9, 2024
Open
32 tasks
@mgodwan
Copy link
Member

mgodwan commented Apr 23, 2024

@ansjcy Are you planning to continue on this PR and targeting any upcoming release?
This has been stalled now, and if this is not planned to be continued, lets close this.

@opensearch-trigger-bot opensearch-trigger-bot bot removed the stalled Issues that have stalled label Apr 23, 2024
@opensearch-trigger-bot
Copy link
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added the stalled Issues that have stalled label May 23, 2024
@kotwanikunal
Copy link
Member

Closing this PR - it has been in a draft state since a while.
@ansjcy Please reopen if you are still working on it.

@ansjcy ansjcy changed the title Query-level resource usages instrumentation [POC] Query-level resource usages instrumentation with SearchPhaseResult Jun 12, 2024
@ansjcy ansjcy changed the title [POC] Query-level resource usages instrumentation with SearchPhaseResult [POC] Query-level resource usages instrumentation with SearchPhaseResult Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 will be requested when the pull request is marked ready for review jed326 is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk will be requested when the pull request is marked ready for review jainankitk is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi will be requested when the pull request is marked ready for review linuxpi is a code owner

Assignees
No one assigned
Labels
stalled Issues that have stalled
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ansjcy @mgodwan @kotwanikunal