Improve stats api #27963

newly12 · 2021-09-16T07:24:22Z

Enhancement

What does this PR do?

make sysinfo.Host() call only once in process.go.

Why is it important?

sysinfo.Host() is called multiple times when beats' /stats API is called for metrics collection. several reasons for this change

it collects host info, but in process.go host info is never used, which is unnecessary waste, and we call it only to get memory usage later.
we have seen issues in our production with this piece of code, where for some reasons syscalls are slow/hung, and metrics collection per minute makes it worse. the profile screenshot attached below. some more info about our deployment, metricbeat(the beat caused issue) is deployed as daemonset running on every k8s node, we have resource limit for it but because of syscall stuck, the process went to D state, and (I think) cgroup limitation is not working(we saw load of the node was high and kept climbing, and almost down to the floor after kill the pod). the issue happened on a lot of nodes in multiple k8s clusters.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2021-09-16T07:29:16Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-19T01:28:09.607+0000
Duration: 160 min 38 sec
Commit: a2c536c

Test stats 🧪

Test	Results
Failed	0
Passed	53933
Skipped	5344
Total	59277

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

elasticmachine · 2021-09-21T07:30:28Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-09-21T07:30:48Z

Pinging @elastic/integrations (Team:Integrations)

mergify · 2021-09-22T13:02:54Z

This pull request does not have a backport label. Could you fix it @newly12? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

fearful-symmetry

This seems reasonable, since I don't think we're using any of the fields that are filled out when we call Host(). However, I don't think we actually need to use init() here. Can we add the Host object to the Stats object instead?

newly12 · 2021-09-24T01:02:21Z

I've removed init() function and move the sysinfo.Host() call to Stats.Init(), the error of sysinfo.Host() call was swallowed to during the init to keep as much the same behaviour as before, and also the error will be logged just once during init.

newly12 · 2021-09-28T00:59:36Z

@fearful-symmetry Any update?

mergify · 2021-10-04T14:02:40Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b improve_stats_api upstream/improve_stats_api
git merge upstream/master
git push upstream improve_stats_api

fearful-symmetry

One small change. Sorry for the delay, I was occupied all of last week.

fearful-symmetry · 2021-10-04T15:55:26Z

libbeat/metric/system/process/process.go

+	procStats.host, err = sysinfo.Host()
+	if err != nil {
+		procStats.logger.Warnf("Getting host details: %v", err)
+	}


I'm not sure I 100% trust the behavior of go-sysinfo here. If we get an error, we may want to just set procStats.host to nil ourselves.

sure. updated.

fearful-symmetry · 2021-10-06T16:17:24Z

LGTM. You'll need to merge to fix the changelog conflict, then we can wait for CI.

newly12 · 2021-10-07T13:34:36Z

resolved conflicts. can you please take another look?

mergify · 2021-10-07T15:20:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b improve_stats_api upstream/improve_stats_api
git merge upstream/master
git push upstream improve_stats_api

fearful-symmetry

LGTM

fearful-symmetry · 2021-10-07T16:41:34Z

Alright, now to see if we can get CI to play nice...

newly12 · 2021-10-08T02:00:11Z

thanks! looks like CI is not approved. can you help with that?

mergify · 2021-10-11T15:16:09Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b improve_stats_api upstream/improve_stats_api
git merge upstream/master
git push upstream improve_stats_api

fearful-symmetry · 2021-10-11T21:39:43Z

So, CI seems to be broken right now. You can try re-merging to fix conflicts, but you might have to do it again in a few hours.

…28288)

… truncated records (elastic#28297)

clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details

…zations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that.

* Runner and Fetcher unit tests * Fix header formatting * Tweak test

* format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies

* Move labels and annotations under kubernetes.namespace.

…e token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt.

* Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

…ession type capture (elastic#28325)

* Redis: remove deprecated fields

…uration (elastic#28282) * remove deprecated config * changelog

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

…annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil

…elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root.

newly12 · 2021-10-13T01:46:35Z

ok.. I've rebased and fixed conflicts. can you please check?

mergify · 2021-10-13T14:16:45Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b improve_stats_api upstream/improve_stats_api
git merge upstream/master
git push upstream improve_stats_api

fearful-symmetry · 2021-10-13T22:11:45Z

Alas, it's in conflict again. At least, CI should be working now. This is why I hate changelogs.

newly12 · 2021-10-14T01:57:30Z

@fearful-symmetry it is really annoying, hope it works this time..

fearful-symmetry · 2021-10-18T19:20:55Z

Yah, CI has been a mess recently, and there's only so much I can do. This time I it seems like there's an issue with libbeat/metric/system/process/process.go , if you run mage fmt in the root libbeat directory, it should fix it.

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (elastic#28236) * [cloud][docker] use the private docker namespace (elastic#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) * seccomp: allow clone3 syscall for x86 (elastic#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (elastic#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (elastic#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (elastic#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (elastic#28253) * Fix build tags for Go 1.17 (elastic#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (elastic#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325) * Redis: remove deprecated fields (elastic#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (elastic#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (elastic#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

newly12 and others added 2 commits September 16, 2021 15:10

singleton sysinfo host to avoid frequently collecting host info

47e0935

Merge branch 'elastic:master' into improve_stats_api

165b149

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 16, 2021

ruflin added the Team:Elastic-Agent Label for the Agent team label Sep 21, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 21, 2021

ruflin added the Team:Integrations Label for the Integrations team label Sep 21, 2021

ChrsMark requested a review from fearful-symmetry September 21, 2021 08:02

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021

fearful-symmetry requested changes Sep 23, 2021

View reviewed changes

add Host object to Stats object

9100f05

newly12 and others added 2 commits September 27, 2021 14:36

Merge branch 'elastic:master' into improve_stats_api

e0f8dae

update changelog

1694024

fearful-symmetry requested changes Oct 4, 2021

View reviewed changes

set procStats.host to nil if any error calling sysinfo.Host()

30e78c2

Merge branch 'master' into improve_stats_api

88e81a6

fearful-symmetry approved these changes Oct 7, 2021

View reviewed changes

v1v and others added 20 commits October 13, 2021 09:43

[cloud][docker] use the private docker namespace (elastic#28286)

80b9e28

[7.x] [DOCS] Update api_key example on elasticsearch output (elastic#…

358842b

…28288)

packetbeat/protos/dns: don't render missing A and AAAA addresses from…

f80d1ac

… truncated records (elastic#28297)

Osquerybeat: Runner and Fetcher unit tests (elastic#28290)

7491d08

* Runner and Fetcher unit tests * Fix header formatting * Tweak test

Move labels and annotations under kubernetes.namespace. (elastic#27917)

3e5fe14

* Move labels and annotations under kubernetes.namespace.

Remove GCP support from Functionbeat (elastic#28253)

056c4d7

Fix build tags for Go 1.17 (elastic#28338)

5b26c25

[Elastic Agent] Add ability to communicate with Kibana through servic…

b191b59

…e token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt.

x-pack/filebeat/module/cisco: loosen time parsing and add group and s…

c77da38

…ession type capture (elastic#28325)

Redis: remove deprecated fields (elastic#28246)

a8c1ecd

* Redis: remove deprecated fields

Disable generator tests temporarily (elastic#28362)

b278380

Windows/perfmon metricset - remove deprecated perfmon.counters config…

e6780a5

…uration (elastic#28282) * remove deprecated config * changelog

[Filebeat] - S3 Input - Add support for only iterating/accessing only… (

f23aff7

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

Breaking change for 8.0, namespace_annotations replaced by namespace.…

69f2780

…annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil

Merge remote-tracking branch 'upstream/master' into improve_stats_api

172c6ba

Merge remote-tracking branch 'upstream/master' into improve_stats_api

6ecb286

mage fmt

a2c536c

fearful-symmetry merged commit 48c9d79 into elastic:master Oct 20, 2021

mdelapenya mentioned this pull request Oct 20, 2021

[Agent] Elastic-Agent-Packaging-Linux is failed (there for e2e-testing pr support) which blocks merges - seems to be a Heartbeat container build problem #28570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve stats api #27963

Improve stats api #27963

newly12 commented Sep 16, 2021 •

edited

Loading

elasticmachine commented Sep 16, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Sep 21, 2021

elasticmachine commented Sep 21, 2021

mergify bot commented Sep 22, 2021

fearful-symmetry left a comment

newly12 commented Sep 24, 2021

newly12 commented Sep 28, 2021

mergify bot commented Oct 4, 2021

fearful-symmetry left a comment

fearful-symmetry Oct 4, 2021

newly12 Oct 6, 2021

fearful-symmetry commented Oct 6, 2021

newly12 commented Oct 7, 2021

mergify bot commented Oct 7, 2021

fearful-symmetry left a comment

fearful-symmetry commented Oct 7, 2021

newly12 commented Oct 8, 2021

mergify bot commented Oct 11, 2021

fearful-symmetry commented Oct 11, 2021

newly12 commented Oct 13, 2021

mergify bot commented Oct 13, 2021

fearful-symmetry commented Oct 13, 2021

newly12 commented Oct 14, 2021

fearful-symmetry commented Oct 18, 2021

Improve stats api #27963

Improve stats api #27963

Conversation

newly12 commented Sep 16, 2021 • edited Loading

Enhancement

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Sep 16, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Sep 21, 2021

elasticmachine commented Sep 21, 2021

mergify bot commented Sep 22, 2021

fearful-symmetry left a comment

Choose a reason for hiding this comment

newly12 commented Sep 24, 2021

newly12 commented Sep 28, 2021

mergify bot commented Oct 4, 2021

fearful-symmetry left a comment

Choose a reason for hiding this comment

fearful-symmetry Oct 4, 2021

Choose a reason for hiding this comment

newly12 Oct 6, 2021

Choose a reason for hiding this comment

fearful-symmetry commented Oct 6, 2021

newly12 commented Oct 7, 2021

mergify bot commented Oct 7, 2021

fearful-symmetry left a comment

Choose a reason for hiding this comment

fearful-symmetry commented Oct 7, 2021

newly12 commented Oct 8, 2021

mergify bot commented Oct 11, 2021

fearful-symmetry commented Oct 11, 2021

newly12 commented Oct 13, 2021

mergify bot commented Oct 13, 2021

fearful-symmetry commented Oct 13, 2021

newly12 commented Oct 14, 2021

fearful-symmetry commented Oct 18, 2021

newly12 commented Sep 16, 2021 •

edited

Loading

elasticmachine commented Sep 16, 2021 •

edited by jenkins-beats-ci bot

Loading