Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent panic in cgroups monitoring #21355

Merged
merged 1 commit into from
Sep 29, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Sep 28, 2020

What does this PR do?

Fixes a panic in Beats monitoring under Linux.

Gosigar's cgroups GetStatsForProcesses can return a nil Stats pointer and no error when the ["blkio", "cpu", "cpuacct", "memory"] subsystems are on the root cgroup.

Why is it important?

Beats snapshots from master are crashing under CentOS 7.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Logs

2020-09-28T16:53:01.616Z        INFO        [monitoring]        runtime/panic.go:969        Stopping metrics logging.
panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x338d528]
goroutine 1337 [running]:
github.com/elastic/beats/v7/libbeat/cmd/instance.reportBeatCgroups(0xc00012e001, 0x5c87a40, 0xc000117480)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/metrics.go:289 +0x328
github.com/elastic/beats/v7/libbeat/monitoring.(*Func).Visit(0xc00076da68, 0x5507d01, 0x5c87a40, 0xc000117480)
/go/src/github.com/elastic/beats/libbeat/monitoring/metrics.go:219 +0x46
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000662080, 0xc0000a7a01, 0x5c87a40, 0xc000117480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1af
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(0xc000662080, 0x5504601, 0x5c87a40, 0xc000117480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65 +0x48
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000090a80, 0xc000117401, 0x5c87a40, 0xc000117480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1af
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65
github.com/elastic/beats/v7/libbeat/monitoring.CollectFlatSnapshot(0xc000090a80, 0x7facd2890101, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/monitoring/snapshot.go:63 +0x16c
github.com/elastic/beats/v7/libbeat/monitoring/report/log.makeSnapshot(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:159
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop.func1(0xc000734cc0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:121 +0x2fc
panic(0x4d5e800, 0x8017a90)
/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/elastic/beats/v7/libbeat/cmd/instance.reportBeatCgroups(0xc00012e001, 0x5c87a40, 0xc000bfe480)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/metrics.go:289 +0x328
github.com/elastic/beats/v7/libbeat/monitoring.(*Func).Visit(0xc00076da68, 0x5507d01, 0x5c87a40, 0xc000bfe480)
/go/src/github.com/elastic/beats/libbeat/monitoring/metrics.go:219 +0x46
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000662080, 0xc0000a7a01, 0x5c87a40, 0xc000bfe480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1af
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(0xc000662080, 0x5504601, 0x5c87a40, 0xc000bfe480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65 +0x48
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000090a80, 0xc000bfe401, 0x5c87a40, 0xc000bfe480)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1af
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65
github.com/elastic/beats/v7/libbeat/monitoring.CollectFlatSnapshot(0xc000090a80, 0xc000070101, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/monitoring/snapshot.go:63 +0x16c
github.com/elastic/beats/v7/libbeat/monitoring/report/log.makeSnapshot(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:159
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc000734cc0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:135 +0x203
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc000734cc0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x50
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157
metricbeat.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Unit metricbeat.service entered failed state.
metricbeat.service failed.

Gosigar's cgroups GetStatsForProcesses can return a nil Stats pointer
and no error when the ["blkio", "cpu", "cpuacct", "memory"] subsystems
are on the root cgroup.

Related elastic#21113
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 28, 2020
@adriansr adriansr added bug review Team:Integrations Label for the Integrations team labels Sep 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 28, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 28, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Started by user Adrian Serrano, Replayed #1]

  • Start Time: 2020-09-28T21:30:23.447+0000

  • Duration: 76 min 3 sec

Test stats 🧪

Test Results
Failed 0
Passed 20291
Skipped 1858
Total 22149

Steps errors

Expand to view the steps failures

  • Name: make -C generator/_templates/beat test

    • Description: make -C generator/_templates/beat test

    • Duration: 6 min 25 sec

    • Start Time: 2020-09-28T22:01:48.266+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 2

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-28T22:09:06.730+0000

    • log

  • Name: Process JUnit reports with runbld

    • Description:

    • Duration: 0 min 16 sec

    • Start Time: 2020-09-28T22:46:09.534+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-28T22:46:04.000Z]  Git commit:        4484c46d9d
[2020-09-28T22:46:04.000Z]  Built:             Wed Sep 16 17:02:36 2020
[2020-09-28T22:46:04.000Z]  OS/Arch:           linux/amd64
[2020-09-28T22:46:04.000Z]  Experimental:      false
[2020-09-28T22:46:04.000Z] 
[2020-09-28T22:46:04.000Z] Server: Docker Engine - Community
[2020-09-28T22:46:04.000Z]  Engine:
[2020-09-28T22:46:04.000Z]   Version:          19.03.13
[2020-09-28T22:46:04.000Z]   API version:      1.40 (minimum version 1.12)
[2020-09-28T22:46:04.000Z]   Go version:       go1.13.15
[2020-09-28T22:46:04.000Z]   Git commit:       4484c46d9d
[2020-09-28T22:46:04.000Z]   Built:            Wed Sep 16 17:01:06 2020
[2020-09-28T22:46:04.000Z]   OS/Arch:          linux/amd64
[2020-09-28T22:46:04.000Z]   Experimental:     false
[2020-09-28T22:46:04.000Z]  containerd:
[2020-09-28T22:46:04.000Z]   Version:          1.3.7
[2020-09-28T22:46:04.000Z]   GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
[2020-09-28T22:46:04.000Z]  runc:
[2020-09-28T22:46:04.000Z]   Version:          1.0.0-rc10
[2020-09-28T22:46:04.000Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-09-28T22:46:04.000Z]  docker-init:
[2020-09-28T22:46:04.000Z]   Version:          0.18.0
[2020-09-28T22:46:04.000Z]   GitCommit:        fec3683
[2020-09-28T22:46:07.587Z] Post stage
[2020-09-28T22:46:07.602Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats
[2020-09-28T22:46:07.617Z] Archiving artifacts
[2020-09-28T22:46:07.813Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats
[2020-09-28T22:46:07.824Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/uncategorized-1601329881145
[2020-09-28T22:46:07.864Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/libbeat-stress-tests-1601330109143
[2020-09-28T22:46:07.901Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/auditbeat-macos-macosx-1601330172536
[2020-09-28T22:46:07.939Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/winlogbeat-crosscompile-1601330182374
[2020-09-28T22:46:07.975Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-elastic-agent-build-1601330183063
[2020-09-28T22:46:08.010Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/heartbeat-macos-macosx-1601330185610
[2020-09-28T22:46:08.043Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/auditbeat-crosscompile-1601330214236
[2020-09-28T22:46:08.079Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/journalbeat-unitTest-1601330215268
[2020-09-28T22:46:08.116Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/generator-macos-metricbeat-macosx-1601330215494
[2020-09-28T22:46:08.150Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1601330220689
[2020-09-28T22:46:08.186Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-functionbeat-build-1601330286897
[2020-09-28T22:46:08.218Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-macos-macosx-1601330303828
[2020-09-28T22:46:08.253Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-elastic-agent-macos-macosx-1601330329271
[2020-09-28T22:46:08.286Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-auditbeat-macos-macosx-1601330366213
[2020-09-28T22:46:08.317Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/libbeat-crosscompile-1601330390024
[2020-09-28T22:46:08.348Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/packetbeat-build-1601330390121
[2020-09-28T22:46:08.382Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/packetbeat-macos-macosx-1601330422104
[2020-09-28T22:46:08.414Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-unitTest-1601330458160
[2020-09-28T22:46:08.447Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-filebeat-macos-macosx-1601330465832
[2020-09-28T22:46:08.486Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1601330493631
[2020-09-28T22:46:08.521Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-functionbeat-macos-macosx-1601330546352
[2020-09-28T22:46:08.552Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/heartbeat-build-1601330562066
[2020-09-28T22:46:08.592Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1601330590436
[2020-09-28T22:46:08.630Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-auditbeat-build-1601330595617
[2020-09-28T22:46:08.670Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1601330616298
[2020-09-28T22:46:08.702Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-metricbeat-macos-macosx-1601330636431
[2020-09-28T22:46:08.736Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1601330661246
[2020-09-28T22:46:08.769Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1601330667914
[2020-09-28T22:46:08.802Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1601330674691
[2020-09-28T22:46:08.835Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/auditbeat-build-1601330700476
[2020-09-28T22:46:08.873Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-crosscompile-1601330723304
[2020-09-28T22:46:08.904Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-libbeat-build-1601330759393
[2020-09-28T22:46:08.937Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1601330762700
[2020-09-28T22:46:08.970Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/filebeat-windows-windows-2019-1601330792057
[2020-09-28T22:46:09.009Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1601330811088
[2020-09-28T22:46:09.043Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1601330824060
[2020-09-28T22:46:09.076Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/generator-metricbeat-test-1601330831929
[2020-09-28T22:46:09.106Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/filebeat-macos-macosx-1601330856098
[2020-09-28T22:46:09.146Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/generator-macos-beat-macosx-1601330902955
[2020-09-28T22:46:09.177Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/generator-beat-test-1601330905606
[2020-09-28T22:46:09.208Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1601330915417
[2020-09-28T22:46:09.244Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1601330987395
[2020-09-28T22:46:09.283Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/filebeat-build-1601331585880
[2020-09-28T22:46:09.325Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/libbeat-build-1601331816810
[2020-09-28T22:46:09.372Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-filebeat-build-1601331946178
[2020-09-28T22:46:09.413Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-goIntegTest-1601332180992
[2020-09-28T22:46:09.455Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1601332286682
[2020-09-28T22:46:09.488Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats/x-pack-metricbeat-build-1601332977623
[2020-09-28T22:46:09.816Z] + cat
[2020-09-28T22:46:09.817Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-09-28T22:46:09.817Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-28T22:46:16.404Z] runbld>>> runbld started
[2020-09-28T22:46:16.404Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-28T22:46:17.347Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-28T22:46:17.347Z] runbld>>> Matches in the system config:
[2020-09-28T22:46:17.347Z] runbld>>> - Matched ^elastic\+beats
[2020-09-28T22:46:17.347Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-28T22:46:18.734Z] runbld>>> Debug logging enabled.
[2020-09-28T22:46:18.734Z] runbld>>> Storing result
[2020-09-28T22:46:18.734Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-28T22:46:18.734Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200928224618-CAAA935D
[2020-09-28T22:46:18.734Z] runbld>>> Adding system facts.
[2020-09-28T22:46:19.677Z] runbld>>> Sending debug log to infra-root+runbld-debug@e***.co
[2020-09-28T22:46:20.630Z] runbld>>> Error: The source clone was not found in /var/lib/jenkins/workspace/Beats_beats_PR-21355/src/github.com/elastic/beats.  The most common cause is that Jenkins and runbld are configured with different working directories (referred to as 'basedir' in JJB and 'cwd' in runbld config).
[2020-09-28T22:46:25.929Z] ERROR: runbld post build action failed.
[2020-09-28T22:46:25.929Z] ERROR: script returned exit code 1
[2020-09-28T22:46:26.072Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21355
[2020-09-28T22:46:26.102Z] [INFO] getVaultSecret: Getting secrets
[2020-09-28T22:46:26.177Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-28T22:46:26.827Z] + chmod 755 generate-build-data.sh
[2020-09-28T22:46:26.827Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21355/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21355/runs/2 FAILURE 4563121
[2020-09-28T22:46:26.827Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21355/runs/2/steps/?limit=10000 -o steps-info.json
[2020-09-28T22:46:32.023Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21355/runs/2/tests/?status=FAILED -o tests-errors.json

@axw
Copy link
Member

axw commented Sep 29, 2020

Argh - sorry, and thanks for fixing.

@adriansr adriansr merged commit 27e8bd3 into elastic:master Sep 29, 2020
v1v added a commit to v1v/beats that referenced this pull request Sep 29, 2020
* upstream/master:
  feat: prepare release pipelines (elastic#21238)
  Add IP validation to Security module (elastic#21325)
  Fixes for new 7.10 rsa2elk datasets (elastic#21240)
  o365input: Restart after fatal error (elastic#21258)
  Fix panic in cgroups monitoring (elastic#21355)
  Handle multiple upstreams in ingress-controller (elastic#21215)
  [CI] Fix runbld when workspace does not exist (elastic#21350)
  [Filebeat] Fix checkpoint (elastic#21344)
  [CI] Archive build reasons (elastic#21347)
  Add dashboard for pubsub metricset in googlecloud module (elastic#21326)
  [Elastic Agent] Allow embedding of certificate (elastic#21179)
  Adds a default for failure_cache.min_ttl (elastic#21085)
  [libbeat] Disk queue implementation (elastic#21176)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug review Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants