System test for beat module, xpack code path #12646

ycombinator · 2019-06-23T06:22:18Z

Similar to #12506 but for the beat module.

This PR introduces a system test to exercise the xpack code path (xpack.enabled: true) of the Beat Metricbeat module.

Testing this PR

Build and activate the virtual env for system tests.

cd metricbeat
make python-env
. build/python-env/bin/activate

Make sure fields.yml is present.
```
make update
```
Build the test binary.
```
make metricbeat.test
```

Run the integration tests

INTEGRATION_TESTS=1  nosetests -v --nocapture tests/system/test_beat.py

Assert that there are no errors.

Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

elasticmachine · 2019-06-25T12:09:36Z

Pinging @elastic/stack-monitoring

cachedout · 2019-06-25T12:39:04Z

I receive an error when attempting to test this per the instructions:

======================================================================
ERROR: beat metricset tests [with metricset='stats']
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/build/python-env/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_beat.py", line 20, in test_metricsets
    self.check_metricset("beat", metricset, self.get_hosts(), self.FIELDS + ["service"])
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/metricbeat.py", line 107, in check_metricset
    self.assert_fields_are_documented(evt)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 687, in assert_fields_are_documented
    raise Exception("Key '{}' found in event is not documented!".format(key))
Exception: Key 'beat.type' found in event is not documented!

cachedout · 2019-06-25T12:48:55Z

The full output also includes some Docker errors. I have included all of stdout and stderror below:

Step 1/4 : FROM docker.elastic.co/beats/metricbeat:7.0.0
 ---> 715bb0c19f94
Step 2/4 : COPY healthcheck.sh /
 ---> Using cache
 ---> fa883cae6fde
Step 3/4 : HEALTHCHECK --interval=1s --retries=300 CMD sh /healthcheck.sh
 ---> Using cache
 ---> 80c26d06bd49
Step 4/4 : ENTRYPOINT [ "metricbeat", "-E", "http.enabled=true", "-E", "http.host=0.0.0.0" ]
 ---> Using cache
 ---> bf82cf59cfb6

Successfully built bf82cf59cfb6
Successfully tagged metricbeat_metricbeat:latest
Recreating metricbeat_metricbeat_1 ... 
�[1A�[2K
Recreating metricbeat_metricbeat_1 ... �[32mdone�[0m
�[1Bbeat metricset tests [with metricset='stats'] ... ERROR
beat metricset tests [with metricset='state'] ... ERROR
beat-xpack module tests ... ok
Killing metricbeat_metricbeat_1    ... 
�[1A�[2K
Killing metricbeat_metricbeat_1    ... �[32mdone�[0m
�[1B
======================================================================
ERROR: beat metricset tests [with metricset='stats']
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/build/python-env/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_beat.py", line 20, in test_metricsets
    self.check_metricset("beat", metricset, self.get_hosts(), self.FIELDS + ["service"])
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/metricbeat.py", line 107, in check_metricset
    self.assert_fields_are_documented(evt)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 687, in assert_fields_are_documented
    raise Exception("Key '{}' found in event is not documented!".format(key))
Exception: Key 'beat.type' found in event is not documented!
-------------------- >> begin captured logging << --------------------
compose.config.config: DEBUG: Using configuration files: ./docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/Users/mp/.docker/config.json', '/Users/mp/.dockercfg']
docker.utils.config: DEBUG: Found file at path: /Users/mp/.docker/config.json
docker.auth: DEBUG: Found 'auths' section
docker.auth: DEBUG: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
compose.service: INFO: Building metricbeat
docker.api.build: DEBUG: Looking for auth config
docker.api.build: DEBUG: Sending auth config (u'https://index.docker.io/v1/')
compose.parallel: DEBUG: Pending: set([<Service: metricbeat>])
compose.parallel: DEBUG: Starting producer thread for <Service: metricbeat>
compose.parallel: DEBUG: Pending: set([<Container: metricbeat_metricbeat_1 (0e5d75)>])
compose.parallel: DEBUG: Starting producer thread for <Container: metricbeat_metricbeat_1 (0e5d75)>
compose.service: DEBUG: Added config hash: 887b8caf3a9380c539865416405562a5a90b0cc1b0390e5b4b6ee53b7494d68c
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Finished processing: <Container: metricbeat_metricbeat_1 (0e5d75)>
compose.parallel: DEBUG: Pending: set([])
compose.parallel: DEBUG: Finished processing: <Service: metricbeat>
compose.parallel: DEBUG: Pending: set([])
compose.config.config: DEBUG: Using configuration files: ./docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/Users/mp/.docker/config.json', '/Users/mp/.dockercfg']
docker.utils.config: DEBUG: Found file at path: /Users/mp/.docker/config.json
docker.auth: DEBUG: Found 'auths' section
docker.auth: DEBUG: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: beat metricset tests [with metricset='state']
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/build/python-env/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_beat.py", line 20, in test_metricsets
    self.check_metricset("beat", metricset, self.get_hosts(), self.FIELDS + ["service"])
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/metricbeat.py", line 107, in check_metricset
    self.assert_fields_are_documented(evt)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 687, in assert_fields_are_documented
    raise Exception("Key '{}' found in event is not documented!".format(key))
Exception: Key 'beat.state.management.enabled' found in event is not documented!
-------------------- >> begin captured logging << --------------------
compose.config.config: DEBUG: Using configuration files: ./docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/Users/mp/.docker/config.json', '/Users/mp/.dockercfg']
docker.utils.config: DEBUG: Found file at path: /Users/mp/.docker/config.json
docker.auth: DEBUG: Found 'auths' section
docker.auth: DEBUG: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 3 tests in 5.527s

FAILED (errors=2)
[{u'service': {u'type': u'beat', u'name': u'beat', u'address': u'localhost:32771'}, u'beat': {u'stats': {u'uptime': {u'ms': 1489}, u'libbeat': {u'output': {u'read': {u'errors': 0, u'bytes': 0}, u'write': {u'errors': 0, u'bytes': 0}, u'type': u'elasticsearch', u'events': {u'toomany': 0, u'batches': 0, u'duplicates': 0, u'acked': 0, u'failed': 0, u'dropped': 0, u'active': 0, u'total': 0}}}}, u'type': u'metricbeat', u'id': u'ae8af2b5-06c0-495e-bbc6-f03fd2c19d59'}, u'@timestamp': u'2019-06-25T12:47:42.823Z', u'agent': {u'ephemeral_id': u'7a6e69b9-44f7-4a4d-b848-2d3ebbf89adf', u'version': u'8.0.0', u'type': u'metricbeat', u'hostname': u'Mikes-MacBook-Pro.local', u'id': u'12530363-f837-4639-9a2b-311c450b6f50'}, u'metricset': {u'name': u'stats'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'event': {u'duration': 5751016, u'module': u'beat', u'dataset': u'beat.stats'}}]
{u'service': {u'type': u'beat', u'name': u'beat', u'address': u'localhost:32771'}, u'beat': {u'stats': {u'uptime': {u'ms': 1489}, u'libbeat': {u'output': {u'read': {u'errors': 0, u'bytes': 0}, u'write': {u'errors': 0, u'bytes': 0}, u'type': u'elasticsearch', u'events': {u'toomany': 0, u'batches': 0, u'duplicates': 0, u'acked': 0, u'failed': 0, u'dropped': 0, u'active': 0, u'total': 0}}}}, u'type': u'metricbeat', u'id': u'ae8af2b5-06c0-495e-bbc6-f03fd2c19d59'}, u'@timestamp': u'2019-06-25T12:47:42.823Z', u'agent': {u'ephemeral_id': u'7a6e69b9-44f7-4a4d-b848-2d3ebbf89adf', u'version': u'8.0.0', u'type': u'metricbeat', u'hostname': u'Mikes-MacBook-Pro.local', u'id': u'12530363-f837-4639-9a2b-311c450b6f50'}, u'metricset': {u'name': u'stats'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'event': {u'duration': 5751016, u'module': u'beat', u'dataset': u'beat.stats'}}
['@timestamp', 'agent', 'metricset.name', 'metricset.host', 'metricset.module', 'metricset.rtt', 'host.name', 'service.name', 'event', 'ecs', 'beat', 'service']
[{u'service': {u'id': u'ae8af2b5-06c0-495e-bbc6-f03fd2c19d59', u'type': u'metricbeat', u'name': u'b8de4c3ebeae', u'address': u'localhost:32771'}, u'beat': {u'state': {u'queue': {u'name': u'mem'}, u'output': {u'name': u'elasticsearch'}, u'management': {u'enabled': False}, u'module': {u'count': 3}}}, u'@timestamp': u'2019-06-25T12:47:44.630Z', u'agent': {u'ephemeral_id': u'cea84bfe-f91a-4114-87b6-fbf544f74bdb', u'hostname': u'Mikes-MacBook-Pro.local', u'type': u'metricbeat', u'id': u'017d628f-0bce-4478-b443-b2f2caf4b360', u'version': u'8.0.0'}, u'event': {u'duration': 5444564, u'module': u'beat', u'dataset': u'beat.state'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'metricset': {u'name': u'state'}}]
{u'service': {u'id': u'ae8af2b5-06c0-495e-bbc6-f03fd2c19d59', u'type': u'metricbeat', u'name': u'b8de4c3ebeae', u'address': u'localhost:32771'}, u'beat': {u'state': {u'queue': {u'name': u'mem'}, u'output': {u'name': u'elasticsearch'}, u'management': {u'enabled': False}, u'module': {u'count': 3}}}, u'@timestamp': u'2019-06-25T12:47:44.630Z', u'agent': {u'ephemeral_id': u'cea84bfe-f91a-4114-87b6-fbf544f74bdb', u'hostname': u'Mikes-MacBook-Pro.local', u'type': u'metricbeat', u'id': u'017d628f-0bce-4478-b443-b2f2caf4b360', u'version': u'8.0.0'}, u'event': {u'duration': 5444564, u'module': u'beat', u'dataset': u'beat.state'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'metricset': {u'name': u'state'}}
['@timestamp', 'agent', 'metricset.name', 'metricset.host', 'metricset.module', 'metricset.rtt', 'host.name', 'service.name', 'event', 'ecs', 'beat', 'service']

ycombinator · 2019-06-25T12:59:06Z

Apologies, @cachedout, I was missing some test steps in the PR description. I've added them now. Would you mind re-testing?

ycombinator · 2019-06-25T13:02:51Z

Standby, @cachedout, there seems to be something else going on here. I'm not able to get my tests to pass consistently locally even with the updated steps. Let me investigate further.

ycombinator · 2019-06-25T13:11:28Z

@cachedout Okay, I figured it out. Apparently there's a file called fields.yml under metricbeat that's not checked into source control but must be present in order for the integration tests to run correctly. So I've added a step in the PR description to ensure that this file is present.

Sorry about the noise. This should be ready to test now.

ycombinator · 2019-06-25T13:12:21Z

jenkins, test this

cachedout

I can confirm that with the latest instructions this works as-intended and tests pass!

ycombinator added needs_backport PR is waiting to be backported to other branches. v7.3.0 v8.0.0 labels Jun 23, 2019

ycombinator changed the title ~~Mb beat xpack system test~~ System test for kibana module, xpack code path Jun 23, 2019

ycombinator changed the title ~~System test for kibana module, xpack code path~~ System test for beat module, xpack code path Jun 23, 2019

This was referenced Jun 23, 2019

Docs for monitoring Beats with Metricbeat #12647

Closed

Write system tests for stack modules' xpack code paths #12256

Closed

ycombinator and others added 23 commits June 25, 2019 08:07

Renaming anchor for beats common fields

77d9eb6

Removing extraneous file accidentally checked in

831d274

Adding beats module with stats metricset

d2777f4

Rename argument for clarity

b5f8cc6

Adding tests

52f9e7c

Adding tests + sample data

683ee1f

Removing debugging statements

f458ac6

Regenerating data.json file

527bf3b

Trying a different host name

264d565

Stubbing out xpack code path

dfd6ffa

Fixing service name in system test

dc5b758

Fleshing out TODOs

9aa8f69

Fixing index name

e0bf4fc

Renaming module name from beats to beat

92a71e1

Regenerating data.json file

48e53ee

Update metricbeat/module/beat/_meta/config.reference.yml

cc9763a

Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

Update metricbeat/module/beat/_meta/config-xpack.yml

1147020

Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

Update metricbeat/module/beat/_meta/config.yml

6c0feaa

Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

Update metricbeat/modules.d/beat-xpack.yml.disabled

8129f87

Update metricbeat/modules.d/beat.yml.disabled

fca2260

Renaming system test

875b71d

Trying to use the new testdata framework

dc7ba8c

Renaming service to avoid naming collision

1a2dc45

ycombinator added 9 commits June 25, 2019 08:08

Rebase misses

b047d93

Regenerating docs

eca882c

Regenerating docs

c84dfd6

Undoing accidental change during rebase

8582778

Removing files since new testdata framework cannot be used yet

76c86db

Regenerating generated files

cc96620

Fixing datatype (again!)

fa40e3c

Adding beat module system test for xpack.enabled: true

469ea46

Fixing formatting

98dd356

ycombinator requested review from ruflin and cachedout June 25, 2019 12:09

ycombinator added Metricbeat Metricbeat module Feature:Stack Monitoring labels Jun 25, 2019

ycombinator added the review label Jun 25, 2019

Regenerating generated files

fe2cf7b

ycombinator marked this pull request as ready for review June 25, 2019 12:10

ycombinator requested review from a team as code owners June 25, 2019 12:10

cachedout approved these changes Jun 26, 2019

View reviewed changes

ycombinator merged commit de1f8d6 into elastic:master Jun 26, 2019

ycombinator deleted the mb-beat-xpack-system-test branch June 26, 2019 12:12

ycombinator removed the needs_backport PR is waiting to be backported to other branches. label Jan 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System test for beat module, xpack code path #12646

System test for beat module, xpack code path #12646

ycombinator commented Jun 23, 2019 •

edited

Loading

elasticmachine commented Jun 25, 2019

cachedout commented Jun 25, 2019

cachedout commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

cachedout left a comment

System test for beat module, xpack code path #12646

System test for beat module, xpack code path #12646

Conversation

ycombinator commented Jun 23, 2019 • edited Loading

Testing this PR

elasticmachine commented Jun 25, 2019

cachedout commented Jun 25, 2019

cachedout commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

ycombinator commented Jun 25, 2019

cachedout left a comment

Choose a reason for hiding this comment

ycombinator commented Jun 23, 2019 •

edited

Loading