Skip to content

Commit

Permalink
Add example value composite type checking (elastic#966)
Browse files Browse the repository at this point in the history
# Conflicts:
#	generated/csv/fields.csv
  • Loading branch information
ebeahan committed Sep 23, 2020
1 parent e6ba4c4 commit a9da38f
Show file tree
Hide file tree
Showing 13 changed files with 120 additions and 99 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Thanks, you're awesome :-) -->
#### Added

* Introduced `--strict` flag to perform stricter schema validation when running the generator script. #937
* Added check under `--strict` that ensures composite types in example fields are quoted. #966

#### Improvements

Expand Down
1 change: 1 addition & 0 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ $ python/generator.py --strict
Strict mode requires the following conditions, else the script exits on an exception:

* Short descriptions must be less than or equal to 120 characters.
* Example values containing arrays or objects must be quoted to avoid unexpected YAML interpretation when the schema files or artifacts are relied on downstream.

The current artifacts generated and published in the ECS repo will always be created using strict mode. However, older ECS versions (pre `v1.5.0`) will cause
an exception if attempting to generate them using `--strict`. This is due to schema validation checks introduced after that version was released.
Expand Down
14 changes: 7 additions & 7 deletions docs/field-details.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1211,7 +1211,7 @@ Note: this field should contain an array of values.



example: `['RD', 'RA']`
example: `["RD", "RA"]`

| extended

Expand Down Expand Up @@ -1343,7 +1343,7 @@ Note: this field should contain an array of values.



example: `['10.10.10.10', '10.10.10.11']`
example: `["10.10.10.10", "10.10.10.11"]`

| extended

Expand Down Expand Up @@ -4205,7 +4205,7 @@ Note: this field should contain an array of values.



example: `['/usr/bin/ssh', '-l', 'user', '10.0.0.16']`
example: `["/usr/bin/ssh", "-l", "user", "10.0.0.16"]`

| extended

Expand Down Expand Up @@ -4718,7 +4718,7 @@ Note: this field should contain an array of values.



example: `['Star-Lord']`
example: `["Star-Lord"]`

| extended

Expand Down Expand Up @@ -5624,7 +5624,7 @@ Note: this field should contain an array of values.



example: `['MII...', 'MII...']`
example: `["MII...", "MII..."]`

| extended

Expand Down Expand Up @@ -5757,7 +5757,7 @@ Note: this field should contain an array of values.



example: `['TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', '...']`
example: `["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "..."]`

| extended

Expand Down Expand Up @@ -5838,7 +5838,7 @@ Note: this field should contain an array of values.



example: `['MII...', 'MII...']`
example: `["MII...", "MII..."]`

| extended

Expand Down
37 changes: 9 additions & 28 deletions generated/beats/fields.ecs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1013,9 +1013,7 @@
description: 'Array of 2 letter DNS header flags.
Expected values are: AA, TC, RD, RA, AD, CD, DO.'
example:
- RD
- RA
example: '["RD", "RA"]'
- name: id
level: extended
type: keyword
Expand Down Expand Up @@ -1096,9 +1094,7 @@
formats it can contain. Extracting all IP addresses seen in there to `dns.resolved_ip`
makes it possible to index them as IP addresses, and makes them easier to
visualize and query for.'
example:
- 10.10.10.10
- 10.10.10.11
example: '["10.10.10.10", "10.10.10.11"]'
- name: response_code
level: extended
type: keyword
Expand Down Expand Up @@ -3229,11 +3225,7 @@
the executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
- name: args_count
level: extended
type: long
Expand Down Expand Up @@ -3376,11 +3368,7 @@
the executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
default_field: false
- name: parent.args_count
level: extended
Expand Down Expand Up @@ -3884,8 +3872,7 @@
ignore_above: 1024
description: Name, organization, or pseudonym of the author or authors who created
the rule used to generate this event.
example:
- Star-Lord
example: '["Star-Lord"]'
default_field: false
- name: category
level: extended
Expand Down Expand Up @@ -4652,9 +4639,7 @@
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the client. This is usually mutually-exclusive of `client.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
default_field: false
- name: client.hash.md5
level: extended
Expand Down Expand Up @@ -4735,10 +4720,8 @@
type: keyword
ignore_above: 1024
description: Array of ciphers offered by the client during the client hello.
example:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- '...'
example: '["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"..."]'
default_field: false
- name: client.x509.alternative_names
level: extended
Expand Down Expand Up @@ -4955,9 +4938,7 @@
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the server. This is usually mutually-exclusive of `server.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
default_field: false
- name: server.hash.md5
level: extended
Expand Down
37 changes: 9 additions & 28 deletions generated/ecs/ecs_flat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1384,9 +1384,7 @@ dns.header_flags:
description: 'Array of 2 letter DNS header flags.
Expected values are: AA, TC, RD, RA, AD, CD, DO.'
example:
- RD
- RA
example: '["RD", "RA"]'
flat_name: dns.header_flags
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -1514,9 +1512,7 @@ dns.resolved_ip:
it can contain. Extracting all IP addresses seen in there to `dns.resolved_ip`
makes it possible to index them as IP addresses, and makes them easier to visualize
and query for.'
example:
- 10.10.10.10
- 10.10.10.11
example: '["10.10.10.10", "10.10.10.11"]'
flat_name: dns.resolved_ip
level: extended
name: resolved_ip
Expand Down Expand Up @@ -4777,11 +4773,7 @@ process.args:
executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -5007,11 +4999,7 @@ process.parent.args:
executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.parent.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -5778,8 +5766,7 @@ rule.author:
dashed_name: rule-author
description: Name, organization, or pseudonym of the author or authors who created
the rule used to generate this event.
example:
- Star-Lord
example: '["Star-Lord"]'
flat_name: rule.author
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -6998,9 +6985,7 @@ tls.client.certificate_chain:
description: Array of PEM-encoded certificates that make up the certificate chain
offered by the client. This is usually mutually-exclusive of `client.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.client.certificate_chain
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -7126,10 +7111,8 @@ tls.client.subject:
tls.client.supported_ciphers:
dashed_name: tls-client-supported-ciphers
description: Array of ciphers offered by the client during the client hello.
example:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- '...'
example: '["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"..."]'
flat_name: tls.client.supported_ciphers
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -7508,9 +7491,7 @@ tls.server.certificate_chain:
description: Array of PEM-encoded certificates that make up the certificate chain
offered by the server. This is usually mutually-exclusive of `server.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.server.certificate_chain
ignore_above: 1024
level: extended
Expand Down
37 changes: 9 additions & 28 deletions generated/ecs/ecs_nested.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1735,9 +1735,7 @@ dns:
description: 'Array of 2 letter DNS header flags.
Expected values are: AA, TC, RD, RA, AD, CD, DO.'
example:
- RD
- RA
example: '["RD", "RA"]'
flat_name: dns.header_flags
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -1866,9 +1864,7 @@ dns:
formats it can contain. Extracting all IP addresses seen in there to `dns.resolved_ip`
makes it possible to index them as IP addresses, and makes them easier to
visualize and query for.'
example:
- 10.10.10.10
- 10.10.10.11
example: '["10.10.10.10", "10.10.10.11"]'
flat_name: dns.resolved_ip
level: extended
name: resolved_ip
Expand Down Expand Up @@ -5824,11 +5820,7 @@ process:
the executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -6054,11 +6046,7 @@ process:
the executable.
May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.parent.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -6890,8 +6878,7 @@ rule:
dashed_name: rule-author
description: Name, organization, or pseudonym of the author or authors who created
the rule used to generate this event.
example:
- Star-Lord
example: '["Star-Lord"]'
flat_name: rule.author
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -8193,9 +8180,7 @@ tls:
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the client. This is usually mutually-exclusive of `client.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.client.certificate_chain
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -8324,10 +8309,8 @@ tls:
tls.client.supported_ciphers:
dashed_name: tls-client-supported-ciphers
description: Array of ciphers offered by the client during the client hello.
example:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- '...'
example: '["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"..."]'
flat_name: tls.client.supported_ciphers
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -8706,9 +8689,7 @@ tls:
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the server. This is usually mutually-exclusive of `server.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.server.certificate_chain
ignore_above: 1024
level: extended
Expand Down
4 changes: 3 additions & 1 deletion schemas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,9 @@ Supported keys to describe fields
Defaults to the main description when absent.
If the main description has multiple paragraphs, then a 'short' description
with no newlines is required.
- example (optional): A single value example of what can be expected in this field
- example (optional): A single value example of what can be expected in this field.
Example values that are composite types (array, object) should be quoted to avoid YAML interpretation
in ECS-generated artifacts and other downstream projects depending on the schema.
- multi\_fields (optional): Specify additional ways to index the field.
- index (optional): If `False`, means field is not indexed (overrides type)
- format: Field format that can be used in a Kibana index template.
Expand Down
4 changes: 2 additions & 2 deletions schemas/dns.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
Array of 2 letter DNS header flags.
Expected values are: AA, TC, RD, RA, AD, CD, DO.
example: [RD, RA]
example: "[\"RD\", \"RA\"]"
normalize:
- array

Expand Down Expand Up @@ -205,6 +205,6 @@
data formats it can contain. Extracting all IP addresses seen in there to
`dns.resolved_ip` makes it possible to index them as IP addresses, and
makes them easier to visualize and query for.
example: [10.10.10.10, 10.10.10.11]
example: '["10.10.10.10", "10.10.10.11"]'
normalize:
- array
2 changes: 1 addition & 1 deletion schemas/process.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@
Array of process arguments, starting with the absolute path to the executable.
May be filtered to protect sensitive information.
example: ["/usr/bin/ssh", "-l", "user", "10.0.0.16"]
example: "[\"/usr/bin/ssh\", \"-l\", \"user\", \"10.0.0.16\"]"
normalize:
- array

Expand Down
Loading

0 comments on commit a9da38f

Please sign in to comment.