[elasticsearch] add static and pipeline tests #4122

klacabane · 2022-09-02T11:54:50Z

Summary

Follow up from #4033

Add basic tests in elasticsearch data streams:

pipeline and static tests for logs data streams
static tests for metrics data streams

Because these tests verify that ingested fields are documented, I had to define additional mappings for a subset of data streams otherwise tests would fail. These fields are not required for Stack Monitoring to function but we should consider adding them to their .monitoring-* counterpart to keep the mappings aligned.

I played with system tests but the CI job failed when creating the logs-generation container, let's investigate these in #4008. The system tests will be useful to exercise #4013

elasticmachine · 2022-09-02T12:06:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 24 min 8 sec

Test stats 🧪

Test	Results
Failed	0
Passed	43
Skipped	0
Total	43

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-09-02T16:26:59Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`5/5`)	💚
Files	100.0% (`9/9`)	💚
Classes	100.0% (`9/9`)	💚
Methods	77.064% (`84/109`)	❕
Lines	91.98% (`562/611`)	❕
Conditionals	100.0% (`0/0`)	💚

This reverts commit 075372a.

packages/elasticsearch/data_stream/slowlog/elasticsearch/ingest_pipeline/default.yml

crespocarlos · 2022-09-06T09:44:43Z

packages/elasticsearch/data_stream/audit/_dev/test/pipeline/test-audit-logs.log-expected.json

@@ -0,0 +1,201 @@
+{
+    "expected": [


Just a note: In real world, the expected value will contain fields added by the some filebeat processors configured here. One example is the ecs.version, which is not present here because the pipeline tests don't consider filebeat processors config.

About the ecs.version field, now looking at the hardcoded value, I wonder whether it's correct. ECS version is now on 8.4. Keeping this up-to-date will be tough

Yes we'll add system tests to get end-to-end coverage, I have these stashed but got blocked by CI failures

Is the ecs version supposed to be bound to the one defined in the dependencies ? There's maybe a way to pass the pinned version to the template

server/deprecation/slowlogs entries are delivered with the ecs.version already populated by ES so we don't need to do anything, for the others we could align with the dependency version. Thoughts ?

Is the ecs version supposed to be bound to the one defined in the dependencies ? There's maybe a way to pass the pinned version to the template

I don't know for sure what's the ecs version defined in the dependencies is used for. As far as I know it's only used to build the docs. If it's possible to extract it from there, it could be a good option.

Looking at other packages like netflow, the ecs.version is also hardcoded and the update process appears to be done manually #3868.

@jsoriano @mtojek Is there a guideline regarding the ecs.version property and should we keep updating that latest version available on each release ?

packages/elasticsearch/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

packages/elasticsearch/data_stream/gc/_dev/test/pipeline/test-gc-logs.log-expected.json

crespocarlos · 2022-09-06T09:54:43Z

packages/elasticsearch/data_stream/server/elasticsearch/ingest_pipeline/pipeline-json.yml

@@ -5,88 +5,42 @@ on_failure:
      field: error.message
      value: '{{ _ingest.on_failure_message }}'
 processors:
-  - json:
+  - rename:


Since now we're going with pure ecs formatted logs, I think we don't need to add these fields via filebeat processor. Pretty sure that at least ecs.version isn't needed anymore, since it's already present in the log entries

I've created #4187 as a follow up

crespocarlos

Looks good! Thanks for adding these tests 💪!

We should revisit the filebeat processors present in the logs.hbs.yml files later on. There are things there that are probably outdated or not needed anymore.

matschaffer · 2022-09-06T06:09:59Z

packages/elasticsearch/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

@@ -17,7 +18,7 @@ processors:
      if: ctx.first_char != '{'
  - pipeline:
      if: ctx.first_char == '{'
-      name: '{< IngestPipeline "pipeline-json" >}'
+      name: '{{ IngestPipeline "pipeline-json" }}'


Glad we caught that :)

matschaffer · 2022-09-08T05:51:17Z

Do you know what "methods" and "lines" aren't being covered from #4122 (comment) ? I haven't been able to find that info yet.

matschaffer · 2022-09-08T05:55:36Z

Where's the info on running this locally? I read through https://github.com/elastic/integrations/blob/main/docs/testing_and_validation.md but still just get this. Not sure what I missed...

  ~/elastic/integrations/packages/elasticsearch   elasticsearch-add-tests ?1                                                                                                                                                 14s ▼  14:53:31
❯ elastic-package test
Run test suite for the package
Run system tests for the package
--- Test results for package: elasticsearch - START ---
No test results
--- Test results for package: elasticsearch - END   ---
Done
Run asset tests for the package
Error: error running package asset tests: could not complete test run: can't install the package: can't install the package: could not install package; API status code = 404; response body = {"statusCode":404,"error":"Not Found","message":"[elasticsearch] package not found in registry"}

I can see the package on https://localhost:8080/search?package=elasticsearch&experimental=true though

crespocarlos · 2022-09-08T08:27:14Z

The fact that you can see the package but still get the error when you run the test is weird. Did you try recreating the elastic-package stack? I only got the error from your test after stopping the package-registry container, but then the URL didn't work either.

When I first learned how to run tests locally, I followed the steps on https://github.com/elastic/elastic-package#elastic-package-test, and more specifically to pipeline tests here https://github.com/elastic/elastic-package/blob/main/docs/howto/pipeline_testing.md#running-a-pipeline-test

crespocarlos · 2022-09-08T09:07:05Z

Do you know what "methods" and "lines" aren't being covered from #4122 (comment) ? I haven't been able to find that info yet.

We still have metrics data_streams without tests here. It's hard to tell exactly. Out of curiosity, I ran locally elastic-package test --test-coverage and I briefly tried to understand the xml files. Those metrics data_streams are there as not covered. It also analyses each processor on the ingest_pipeline and can tell whether the tests covered them or not.

klacabane · 2022-09-08T11:47:54Z

[elasticsearch] package not found in registry

You could have a stack running with an unsupported version ? the packages only support versions >= 8.5.0 and I've already seen this error trying to install it on versions lower than that

klacabane · 2022-09-08T13:27:20Z

Do you know what "methods" and "lines" aren't being covered

I'd say methods should be from the lack of system tests which counts as MISSING in the coverage. As for lines it could be the pipeline processors. I don't know where the -12 is coming from though since this change is adding more coverage and not changing the logic

matschaffer · 2022-09-09T06:01:16Z

You could have a stack running with an unsupported version ?

Ah... I bet it's this. For some reason elastic-package stack up -v -d --version 8.5.0-SNAPSHOT was failing for me so I tried just elastic-package stack up -v -d. Looks like that's 8.3.3 by default.

matschaffer · 2022-09-12T06:34:10Z

Yep, that was it!

❯ elastic-package test --test-coverage
Run test suite for the package
Run pipeline tests for the package
--- Test results for package: elasticsearch - START ---
...

matschaffer

This looks good and the tests pass for me now. Would be awesome if the message made it clearer when a version mismatch was afoot but that's definitely not the goal of this PR so LGTM!

packages/elasticsearch/_dev/deploy/docker/scripts/generate-logs.sh

klacabane · 2022-09-12T10:05:34Z

Would be awesome if the message made it clearer when a version mismatch was afoot

We could add the verification in the service's docker-compose and have the elasticsearch container depend on it

matschaffer · 2022-09-12T11:12:13Z

I think in this case it'd be the e-p stack kibana version we need to check though right? Basically what I'm thinking is that it'd be nice if instead of just a 404 we got some information that the package exists but isn't a version match.

klacabane added 6 commits September 1, 2022 21:16

audit system test

202c597

message is ecs field

71888ce

gc system test

12514ea

sample event

873eb29

deprecation logs test

d0a675d

server logs tests

5d5c99c

klacabane added 12 commits September 2, 2022 14:41

slowlog test

c05d853

readme update

ff712dd

format

d17f4f0

index metrics test

d31ad10

index_summary test

6418a97

node metrics test

d44388f

shard metrics test

2ef5bc5

readme

860f110

index_recovery test

3415d46

node_stats metrics test

d8d7935

remove pipeline test

bc52e07

update readme

3df8184

klacabane added 6 commits September 5, 2022 01:09

add pipeline tests for logs

aa0b6f0

add ccr sample_event for static test

0b596dd

deprecation pipeline test

3e052ab

remove system tests

0204203

cleanup

7272ba0

remove node_stats test

ccaf314

klacabane self-assigned this Sep 5, 2022

klacabane added Integration:elasticsearch Elasticsearch v8.5.0 Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services labels Sep 5, 2022

klacabane marked this pull request as ready for review September 5, 2022 12:36

klacabane requested a review from a team as a code owner September 5, 2022 12:36

klacabane changed the title ~~[elasticsearch] add tests~~ [elasticsearch] add static and pipeline tests Sep 5, 2022

klacabane added 2 commits September 6, 2022 01:42

comment container

075372a

Revert "comment container"

b210d39

This reverts commit 075372a.

crespocarlos reviewed Sep 6, 2022

View reviewed changes

packages/elasticsearch/data_stream/slowlog/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

crespocarlos reviewed Sep 6, 2022

View reviewed changes

packages/elasticsearch/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

crespocarlos reviewed Sep 6, 2022

View reviewed changes

packages/elasticsearch/data_stream/gc/_dev/test/pipeline/test-gc-logs.log-expected.json Show resolved Hide resolved

crespocarlos reviewed Sep 6, 2022

View reviewed changes

crespocarlos approved these changes Sep 6, 2022

View reviewed changes

klacabane added 5 commits September 6, 2022 14:48

fix gc logs timestamp

b87d393

fix slowlog timestamp processor

38c65c2

fix server logs timestamp processor

41ecbe5

fix deprecation timestamp processor

70f5e63

fix audit logs timestamp processor

2138e59

matschaffer reviewed Sep 6, 2022

View reviewed changes

matschaffer approved these changes Sep 12, 2022

View reviewed changes

packages/elasticsearch/_dev/deploy/docker/scripts/generate-logs.sh Show resolved Hide resolved

klacabane mentioned this pull request Sep 12, 2022

[Stack Monitoring] clean filebeat processors #4187

Closed

klacabane merged commit 027b248 into elastic:main Sep 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[elasticsearch] add static and pipeline tests #4122

[elasticsearch] add static and pipeline tests #4122

klacabane commented Sep 2, 2022 •

edited

Loading

elasticmachine commented Sep 2, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Sep 2, 2022 •

edited

Loading

crespocarlos Sep 6, 2022 •

edited

Loading

klacabane Sep 6, 2022

klacabane Sep 6, 2022

crespocarlos Sep 6, 2022

klacabane Sep 6, 2022

crespocarlos Sep 6, 2022

klacabane Sep 12, 2022

crespocarlos left a comment

matschaffer Sep 6, 2022

matschaffer commented Sep 8, 2022

matschaffer commented Sep 8, 2022

crespocarlos commented Sep 8, 2022 •

edited

Loading

crespocarlos commented Sep 8, 2022 •

edited

Loading

klacabane commented Sep 8, 2022

klacabane commented Sep 8, 2022 •

edited

Loading

matschaffer commented Sep 9, 2022

matschaffer commented Sep 12, 2022

matschaffer left a comment

klacabane commented Sep 12, 2022 •

edited

Loading

matschaffer commented Sep 12, 2022

[elasticsearch] add static and pipeline tests #4122

[elasticsearch] add static and pipeline tests #4122

Conversation

klacabane commented Sep 2, 2022 • edited Loading

Summary

elasticmachine commented Sep 2, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Sep 2, 2022 • edited Loading

🌐 Coverage report

crespocarlos Sep 6, 2022 • edited Loading

Choose a reason for hiding this comment

klacabane Sep 6, 2022

Choose a reason for hiding this comment

klacabane Sep 6, 2022

Choose a reason for hiding this comment

crespocarlos Sep 6, 2022

Choose a reason for hiding this comment

klacabane Sep 6, 2022

Choose a reason for hiding this comment

crespocarlos Sep 6, 2022

Choose a reason for hiding this comment

klacabane Sep 12, 2022

Choose a reason for hiding this comment

crespocarlos left a comment

Choose a reason for hiding this comment

matschaffer Sep 6, 2022

Choose a reason for hiding this comment

matschaffer commented Sep 8, 2022

matschaffer commented Sep 8, 2022

crespocarlos commented Sep 8, 2022 • edited Loading

crespocarlos commented Sep 8, 2022 • edited Loading

klacabane commented Sep 8, 2022

klacabane commented Sep 8, 2022 • edited Loading

matschaffer commented Sep 9, 2022

matschaffer commented Sep 12, 2022

matschaffer left a comment

Choose a reason for hiding this comment

klacabane commented Sep 12, 2022 • edited Loading

matschaffer commented Sep 12, 2022

klacabane commented Sep 2, 2022 •

edited

Loading

elasticmachine commented Sep 2, 2022 •

edited

Loading

elasticmachine commented Sep 2, 2022 •

edited

Loading

crespocarlos Sep 6, 2022 •

edited

Loading

crespocarlos commented Sep 8, 2022 •

edited

Loading

crespocarlos commented Sep 8, 2022 •

edited

Loading

klacabane commented Sep 8, 2022 •

edited

Loading

klacabane commented Sep 12, 2022 •

edited

Loading