Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[elasticsearch] add static and pipeline tests #4122

Merged
merged 31 commits into from
Sep 12, 2022
Merged

[elasticsearch] add static and pipeline tests #4122

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
202c597
audit system test
klacabane Sep 1, 2022
71888ce
message is ecs field
klacabane Sep 1, 2022
12514ea
gc system test
klacabane Sep 1, 2022
873eb29
sample event
klacabane Sep 1, 2022
d0a675d
deprecation logs test
klacabane Sep 1, 2022
5d5c99c
server logs tests
klacabane Sep 2, 2022
c05d853
slowlog test
klacabane Sep 2, 2022
ff712dd
readme update
klacabane Sep 2, 2022
d17f4f0
format
klacabane Sep 2, 2022
d31ad10
index metrics test
klacabane Sep 2, 2022
6418a97
index_summary test
klacabane Sep 2, 2022
d44388f
node metrics test
klacabane Sep 2, 2022
2ef5bc5
shard metrics test
klacabane Sep 2, 2022
860f110
readme
klacabane Sep 2, 2022
3415d46
index_recovery test
klacabane Sep 2, 2022
d8d7935
node_stats metrics test
klacabane Sep 2, 2022
bc52e07
remove pipeline test
klacabane Sep 2, 2022
3df8184
update readme
klacabane Sep 2, 2022
aa0b6f0
add pipeline tests for logs
klacabane Sep 4, 2022
0b596dd
add ccr sample_event for static test
klacabane Sep 5, 2022
3e052ab
deprecation pipeline test
klacabane Sep 5, 2022
0204203
remove system tests
klacabane Sep 5, 2022
7272ba0
cleanup
klacabane Sep 5, 2022
ccaf314
remove node_stats test
klacabane Sep 5, 2022
075372a
comment container
klacabane Sep 5, 2022
b210d39
Revert "comment container"
klacabane Sep 5, 2022
b87d393
fix gc logs timestamp
klacabane Sep 6, 2022
38c65c2
fix slowlog timestamp processor
klacabane Sep 6, 2022
41ecbe5
fix server logs timestamp processor
klacabane Sep 6, 2022
70f5e63
fix deprecation timestamp processor
klacabane Sep 6, 2022
2138e59
fix audit logs timestamp processor
klacabane Sep 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/elasticsearch/_dev/deploy/docker/scripts/generate-logs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,5 +125,5 @@ do
}
}'

sleep 5
sleep 10
klacabane marked this conversation as resolved.
Show resolved Hide resolved
done
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/audit/_dev/test/pipeline/test-audit-logs.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"type":"audit", "timestamp":"2022-09-04T22:54:53,028+0000", "cluster.uuid":"sh0FdC0tRUGgzD6U7OsO3g", "node.id":"rsRsMdvhREeQqLkk3twtqA", "event.type":"transport", "event.action":"access_granted", "authentication.type":"REALM", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"172.19.0.3:48524", "request.id":"sdzMxhL5Rga_wTaN7_pfsw", "action":"indices:admin/create", "request.name":"CreateIndexRequest", "indices":["test_2"]}
{"type":"audit", "timestamp":"2022-09-04T22:54:53,034+0000", "cluster.uuid":"sh0FdC0tRUGgzD6U7OsO3g", "node.id":"rsRsMdvhREeQqLkk3twtqA", "event.type":"rest", "event.action":"anonymous_access_denied", "origin.type":"rest", "origin.address":"172.19.0.3:48526", "url.path":"/test_3", "request.method":"PUT", "request.id":"kDWih8w0SC6mY7Q5ExEI2w", "opaque_id":"myApp1", "trace.id":"0af7651916cd43dd8448eb211c80319c"}
{"type":"audit", "timestamp":"2022-09-04T22:54:53,040+0000", "cluster.uuid":"sh0FdC0tRUGgzD6U7OsO3g", "node.id":"rsRsMdvhREeQqLkk3twtqA", "event.type":"transport", "event.action":"access_granted", "authentication.type":"REALM", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"172.19.0.3:48528", "request.id":"fTP-0rxJQyyZUNGIs4Hpdg", "action":"indices:admin/create", "request.name":"CreateIndexRequest", "indices":["testindex2"], "opaque_id":"myAppId", "trace.id":"0af7651916cd43dd8448eb211c80319c"}
201 changes: 201 additions & 0 deletions ...ages/elasticsearch/data_stream/audit/_dev/test/pipeline/test-audit-logs.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
{
"expected": [
Copy link
Contributor

@crespocarlos crespocarlos Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note: In real world, the expected value will contain fields added by the some filebeat processors configured here. One example is the ecs.version, which is not present here because the pipeline tests don't consider filebeat processors config.

About the ecs.version field, now looking at the hardcoded value, I wonder whether it's correct. ECS version is now on 8.4. Keeping this up-to-date will be tough

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we'll add system tests to get end-to-end coverage, I have these stashed but got blocked by CI failures

Is the ecs version supposed to be bound to the one defined in the dependencies ? There's maybe a way to pass the pinned version to the template

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

server/deprecation/slowlogs entries are delivered with the ecs.version already populated by ES so we don't need to do anything, for the others we could align with the dependency version. Thoughts ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the ecs version supposed to be bound to the one defined in the dependencies ? There's maybe a way to pass the pinned version to the template

I don't know for sure what's the ecs version defined in the dependencies is used for. As far as I know it's only used to build the docs. If it's possible to extract it from there, it could be a good option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at other packages like netflow, the ecs.version is also hardcoded and the update process appears to be done manually #3868.

@jsoriano @mtojek Is there a guideline regarding the ecs.version property and should we keep updating that latest version available on each release ?

{
"@timestamp": "2022-09-04T22:54:53.028Z",
"elasticsearch": {
"audit": {
"action": "indices:admin/create",
"authentication.type": "REALM",
"cluster": {},
"event": {},
"indices": [
"test_2"
],
"layer": "transport",
"origin": {},
"origin.type": "rest",
"request": {
"id": "sdzMxhL5Rga_wTaN7_pfsw"
},
"request.name": "CreateIndexRequest",
"user": {},
"user.realm": "reserved",
"user.roles": [
"superuser"
]
},
"cluster": {
"uuid": "sh0FdC0tRUGgzD6U7OsO3g"
},
"node": {
"id": "rsRsMdvhREeQqLkk3twtqA"
}
},
"event": {
"action": "access_granted",
"category": "database",
"ingested": "2022-09-04T23:00:22.485831147Z",
"kind": "event",
"outcome": "success"
},
"host": {
"id": "rsRsMdvhREeQqLkk3twtqA"
},
"http": {
"request": {
"id": "sdzMxhL5Rga_wTaN7_pfsw"
}
},
"log": {
"level": "info"
},
"message": "{\"type\":\"audit\", \"timestamp\":\"2022-09-04T22:54:53,028+0000\", \"cluster.uuid\":\"sh0FdC0tRUGgzD6U7OsO3g\", \"node.id\":\"rsRsMdvhREeQqLkk3twtqA\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"authentication.type\":\"REALM\", \"user.name\":\"elastic\", \"user.realm\":\"reserved\", \"user.roles\":[\"superuser\"], \"origin.type\":\"rest\", \"origin.address\":\"172.19.0.3:48524\", \"request.id\":\"sdzMxhL5Rga_wTaN7_pfsw\", \"action\":\"indices:admin/create\", \"request.name\":\"CreateIndexRequest\", \"indices\":[\"test_2\"]}",
"related": {
"user": [
"elastic"
]
},
"service": {
"type": "elasticsearch"
},
"source": {
"address": "172.19.0.3:48524",
"ip": "172.19.0.3",
"port": 48524
},
"user": {
"name": "elastic"
}
},
{
"@timestamp": "2022-09-04T22:54:53.034Z",
"elasticsearch": {
"audit": {
"cluster": {},
"event": {},
"layer": "rest",
"opaque_id": "myApp1",
"origin": {},
"origin.type": "rest",
"request": {
"id": "kDWih8w0SC6mY7Q5ExEI2w"
},
"trace": {},
"url": {}
},
"cluster": {
"uuid": "sh0FdC0tRUGgzD6U7OsO3g"
},
"node": {
"id": "rsRsMdvhREeQqLkk3twtqA"
}
},
"event": {
"action": "anonymous_access_denied",
"category": "database",
"ingested": "2022-09-04T23:00:22.485849152Z",
"kind": "event",
"outcome": "failure"
},
"host": {
"id": "rsRsMdvhREeQqLkk3twtqA"
},
"http": {
"request": {
"id": "kDWih8w0SC6mY7Q5ExEI2w",
"method": "PUT"
}
},
"log": {
"level": "info"
},
"message": "{\"type\":\"audit\", \"timestamp\":\"2022-09-04T22:54:53,034+0000\", \"cluster.uuid\":\"sh0FdC0tRUGgzD6U7OsO3g\", \"node.id\":\"rsRsMdvhREeQqLkk3twtqA\", \"event.type\":\"rest\", \"event.action\":\"anonymous_access_denied\", \"origin.type\":\"rest\", \"origin.address\":\"172.19.0.3:48526\", \"url.path\":\"/test_3\", \"request.method\":\"PUT\", \"request.id\":\"kDWih8w0SC6mY7Q5ExEI2w\", \"opaque_id\":\"myApp1\", \"trace.id\":\"0af7651916cd43dd8448eb211c80319c\"}",
"service": {
"type": "elasticsearch"
},
"source": {
"address": "172.19.0.3:48526",
"ip": "172.19.0.3",
"port": 48526
},
"trace": {
"id": "0af7651916cd43dd8448eb211c80319c"
},
"url": {
"original": "/test_3"
}
},
{
"@timestamp": "2022-09-04T22:54:53.040Z",
"elasticsearch": {
"audit": {
"action": "indices:admin/create",
"authentication.type": "REALM",
"cluster": {},
"event": {},
"indices": [
"testindex2"
],
"layer": "transport",
"opaque_id": "myAppId",
"origin": {},
"origin.type": "rest",
"request": {
"id": "fTP-0rxJQyyZUNGIs4Hpdg"
},
"request.name": "CreateIndexRequest",
"trace": {},
"user": {},
"user.realm": "reserved",
"user.roles": [
"superuser"
]
},
"cluster": {
"uuid": "sh0FdC0tRUGgzD6U7OsO3g"
},
"node": {
"id": "rsRsMdvhREeQqLkk3twtqA"
}
},
"event": {
"action": "access_granted",
"category": "database",
"ingested": "2022-09-04T23:00:22.485851956Z",
"kind": "event",
"outcome": "success"
},
"host": {
"id": "rsRsMdvhREeQqLkk3twtqA"
},
"http": {
"request": {
"id": "fTP-0rxJQyyZUNGIs4Hpdg"
}
},
"log": {
"level": "info"
},
"message": "{\"type\":\"audit\", \"timestamp\":\"2022-09-04T22:54:53,040+0000\", \"cluster.uuid\":\"sh0FdC0tRUGgzD6U7OsO3g\", \"node.id\":\"rsRsMdvhREeQqLkk3twtqA\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"authentication.type\":\"REALM\", \"user.name\":\"elastic\", \"user.realm\":\"reserved\", \"user.roles\":[\"superuser\"], \"origin.type\":\"rest\", \"origin.address\":\"172.19.0.3:48528\", \"request.id\":\"fTP-0rxJQyyZUNGIs4Hpdg\", \"action\":\"indices:admin/create\", \"request.name\":\"CreateIndexRequest\", \"indices\":[\"testindex2\"], \"opaque_id\":\"myAppId\", \"trace.id\":\"0af7651916cd43dd8448eb211c80319c\"}",
"related": {
"user": [
"elastic"
]
},
"service": {
"type": "elasticsearch"
},
"source": {
"address": "172.19.0.3:48528",
"ip": "172.19.0.3",
"port": 48528
},
"trace": {
"id": "0af7651916cd43dd8448eb211c80319c"
},
"user": {
"name": "elastic"
}
}
]
}
2 changes: 2 additions & 0 deletions packages/elasticsearch/data_stream/audit/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dynamic_fields:
event.ingested: ".*"
3 changes: 2 additions & 1 deletion packages/elasticsearch/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ processors:
- set:
copy_from: "@timestamp"
field: event.created
ignore_empty_value: true
- grok:
field: message
patterns:
Expand All @@ -17,7 +18,7 @@ processors:
if: ctx.first_char != '{'
- pipeline:
if: ctx.first_char == '{'
name: '{< IngestPipeline "pipeline-json" >}'
name: '{{ IngestPipeline "pipeline-json" }}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad we caught that :)

klacabane marked this conversation as resolved.
Show resolved Hide resolved
- set:
field: event.kind
value: event
Expand Down
20 changes: 20 additions & 0 deletions packages/elasticsearch/data_stream/audit/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
- external: ecs
name: ecs.version
- external: ecs
name: http
- external: ecs
Expand All @@ -14,3 +16,21 @@
name: user
- external: ecs
name: user.name
- external: ecs
name: http.request.id
- external: ecs
name: http.request.method
- external: ecs
name: log.file.path
- external: ecs
name: log.level
- external: ecs
name: service.type
- external: ecs
name: source.address
- external: ecs
name: source.port
- external: ecs
name: trace.id
- external: ecs
name: message
4 changes: 4 additions & 0 deletions packages/elasticsearch/data_stream/audit/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
- name: elasticsearch.audit
type: group
fields:
- name: authentication.type
type: keyword
- name: opaque_id
type: keyword
- name: layer
type: keyword
description: 'The layer from which this event originated: rest, transport or ip_filter'
Expand Down
6 changes: 6 additions & 0 deletions packages/elasticsearch/data_stream/audit/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
- name: input.type
type: keyword
- name: log.offset
type: long
- name: related.user
type: keyword
- name: elasticsearch
type: group
fields:
Expand Down
113 changes: 113 additions & 0 deletions packages/elasticsearch/data_stream/audit/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
{
"@timestamp": "2022-09-01T19:20:17.967Z",
"agent": {
"ephemeral_id": "ec83bfa3-8e61-430e-91ca-dc784bfa56c0",
"id": "97025ce1-28a3-4aeb-926b-ed68301fc4d2",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.5.0"
},
"data_stream": {
"dataset": "elasticsearch.audit",
"namespace": "ep",
"type": "logs"
},
"ecs": {
"version": "1.10.0"
},
"elastic_agent": {
"id": "97025ce1-28a3-4aeb-926b-ed68301fc4d2",
"snapshot": true,
"version": "8.5.0"
},
"elasticsearch": {
"audit": {
"action": "cluster:monitor/main",
"authentication.type": "REALM",
"cluster": {},
"event": {},
"layer": "transport",
"origin": {},
"origin.type": "rest",
"request": {
"id": "YCHBXylbRnSC3Vc8-f3sIA"
},
"request.name": "MainRequest",
"user": {},
"user.realm": "reserved",
"user.roles": [
"superuser"
]
},
"cluster": {
"uuid": "wkVNYOctQ8mbbp1EkrFjKw"
},
"node": {
"id": "VdwTr-luTomz8dDpOp2OJQ"
}
},
"event": {
"action": "access_granted",
"agent_id_status": "verified",
"category": "database",
"created": "2022-09-01T19:20:39.899Z",
"dataset": "elasticsearch.audit",
"ingested": "2022-09-01T19:20:43Z",
"kind": "event",
"outcome": "success"
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "VdwTr-luTomz8dDpOp2OJQ",
"ip": [
"172.21.0.7"
],
"mac": [
"02:42:ac:15:00:07"
],
"name": "docker-fleet-agent",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.47-linuxkit",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
"version": "20.04.4 LTS (Focal Fossa)"
}
},
"http": {
"request": {
"id": "YCHBXylbRnSC3Vc8-f3sIA"
}
},
"input": {
"type": "log"
},
"log": {
"file": {
"path": "/tmp/service_logs/elasticsearch_audit.json"
},
"level": "info",
"offset": 0
},
"message": "{\"type\":\"audit\", \"timestamp\":\"2022-09-01T19:20:17,967+0000\", \"cluster.uuid\":\"wkVNYOctQ8mbbp1EkrFjKw\", \"node.id\":\"VdwTr-luTomz8dDpOp2OJQ\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"authentication.type\":\"REALM\", \"user.name\":\"elastic\", \"user.realm\":\"reserved\", \"user.roles\":[\"superuser\"], \"origin.type\":\"rest\", \"origin.address\":\"127.0.0.1:53716\", \"request.id\":\"YCHBXylbRnSC3Vc8-f3sIA\", \"action\":\"cluster:monitor/main\", \"request.name\":\"MainRequest\"}",
"related": {
"user": [
"elastic"
]
},
"service": {
"type": "elasticsearch"
},
"source": {
"address": "127.0.0.1:53716",
"ip": "127.0.0.1",
"port": 53716
},
"user": {
"name": "elastic"
}
}
Loading