[Metricbeat] improve efficiency of quering instance metadata in GCP metrics #30154

kwinstonix · 2022-02-02T10:58:04Z

Type of change

Enhancement

What does this PR do?

details in #30153

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [Metricbeat] gcp metrics module should improve efficiency of querying instance metatdata #30153

mergify · 2022-02-02T10:58:29Z

This pull request does not have a backport label. Could you fix it @kwinstonix? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-02-02T11:08:25Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-15T10:57:19.914+0000
Duration: 53 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1083
Skipped	90
Total	1173

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2022-02-02T21:55:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix_gcp_compute_metadata upstream/fix_gcp_compute_metadata
git merge upstream/master
git push upstream fix_gcp_compute_metadata

endorama

Some question and small nitpicks but it looks good. Thanks for the contribution!

x-pack/metricbeat/module/gcp/metrics/compute/metadata.go

go.mod

kwinstonix · 2022-03-05T06:03:12Z

@endorama The PR is ready.

endorama

A small nitpick but looks good; as soon the comment is resolved I'll merge this!

x-pack/metricbeat/module/gcp/metrics/compute/metadata.go

kwinstonix · 2022-03-10T09:52:24Z

A new commit is pushed. The PR has been rebased on main branch and need be squashed. Is that ok ?

mergify · 2022-03-10T20:33:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix_gcp_compute_metadata upstream/fix_gcp_compute_metadata
git merge upstream/main
git push upstream fix_gcp_compute_metadata

…etrics. (elastic#30153) * replace `instance.get` with `instance.aggregatedList` api. * remove side effect on metadataCollector instance in instanceMetadata function.

endorama

Thank you for the contribution! Looks good to me

endorama · 2022-03-15T10:56:40Z

/test

endorama · 2022-03-15T14:31:55Z

I'm removing the backport to 8.1.0 as this is an enhancement and not a bugfix. Target release is 8.2.0.

I asked @gpop63 to test it to have a couple of eyes more, will merge as soon as tests pass!

mergify · 2022-03-15T14:32:44Z

This pull request does not have a backport label. Could you fix it @kwinstonix? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

gpop63 · 2022-03-15T22:18:42Z

I tested getting metrics from a compute instance with these changes, and it worked for me.
"metricbeat":{"gcp":{"compute":{"events":4,"failures":0,"success":4}}}

endorama · 2022-03-16T09:00:15Z

Merging! @kwinstonix thank you again for the contribution, next step for us will be to investigate #30098

…etrics (elastic#30154)

…etrics (#30154)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 2, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 2, 2022

kwinstonix force-pushed the fix_gcp_compute_metadata branch from 16ffb3a to 36cab27 Compare February 2, 2022 11:03

mergify bot assigned kwinstonix Feb 8, 2022

endorama reviewed Feb 16, 2022

View reviewed changes

endorama added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Feb 16, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 16, 2022

endorama added the backport-v8.1.0 Automated backport with mergify label Feb 16, 2022

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Feb 16, 2022

masci assigned endorama and unassigned kwinstonix Feb 23, 2022

kwinstonix force-pushed the fix_gcp_compute_metadata branch 3 times, most recently from 4f25e83 to 05724d3 Compare March 5, 2022 06:01

endorama requested changes Mar 9, 2022

View reviewed changes

x-pack/metricbeat/module/gcp/metrics/compute/metadata.go Outdated Show resolved Hide resolved

kwinstonix force-pushed the fix_gcp_compute_metadata branch from bf5e11b to 7488a4a Compare March 10, 2022 09:49

kwinstonix added 2 commits March 11, 2022 09:42

[Metricbeat] improve efficiency of quering instance metadata in GCP m…

ee91bd4

…etrics. (elastic#30153) * replace `instance.get` with `instance.aggregatedList` api. * remove side effect on metadataCollector instance in instanceMetadata function.

fix: revert the change of cache ttl value

08a8c29

kwinstonix force-pushed the fix_gcp_compute_metadata branch from 7488a4a to 08a8c29 Compare March 11, 2022 01:42

endorama approved these changes Mar 15, 2022

View reviewed changes

endorama removed the backport-v8.1.0 Automated backport with mergify label Mar 15, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Mar 15, 2022

endorama merged commit 1cb6081 into elastic:main Mar 16, 2022

kush-elastic pushed a commit to kush-elastic/beats that referenced this pull request May 2, 2022

[Metricbeat] improve efficiency of quering instance metadata in GCP m…

ef6be87

…etrics (elastic#30154)

endorama mentioned this pull request Aug 18, 2022

GCP metadata instance cache is never refreshed but also never used #32734

Closed

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

[Metricbeat] improve efficiency of quering instance metadata in GCP m…

48cb3f5

…etrics (#30154)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] improve efficiency of quering instance metadata in GCP metrics #30154

[Metricbeat] improve efficiency of quering instance metadata in GCP metrics #30154

kwinstonix commented Feb 2, 2022 •

edited

Loading

mergify bot commented Feb 2, 2022

elasticmachine commented Feb 2, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Feb 2, 2022

endorama left a comment

kwinstonix commented Mar 5, 2022

endorama left a comment

kwinstonix commented Mar 10, 2022 •

edited

Loading

mergify bot commented Mar 10, 2022

endorama left a comment

endorama commented Mar 15, 2022

endorama commented Mar 15, 2022

mergify bot commented Mar 15, 2022

gpop63 commented Mar 15, 2022

endorama commented Mar 16, 2022

[Metricbeat] improve efficiency of quering instance metadata in GCP metrics #30154

[Metricbeat] improve efficiency of quering instance metadata in GCP metrics #30154

Conversation

kwinstonix commented Feb 2, 2022 • edited Loading

Type of change

What does this PR do?

Checklist

Related issues

mergify bot commented Feb 2, 2022

elasticmachine commented Feb 2, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Feb 2, 2022

endorama left a comment

Choose a reason for hiding this comment

kwinstonix commented Mar 5, 2022

endorama left a comment

Choose a reason for hiding this comment

kwinstonix commented Mar 10, 2022 • edited Loading

mergify bot commented Mar 10, 2022

endorama left a comment

Choose a reason for hiding this comment

endorama commented Mar 15, 2022

endorama commented Mar 15, 2022

mergify bot commented Mar 15, 2022

gpop63 commented Mar 15, 2022

endorama commented Mar 16, 2022

kwinstonix commented Feb 2, 2022 •

edited

Loading

elasticmachine commented Feb 2, 2022 •

edited by jenkins-beats-ci bot

Loading

kwinstonix commented Mar 10, 2022 •

edited

Loading