Introduce and make use of the timestamp type #1388
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tobim
force-pushed
the
story/ch22707/timestamp-type
branch
from
f874a64
to
7316487
Compare
tobim
changed the title
tobim
changed the base branch from
master
to
story/ch22834/udt-type-extractor
mavam
reviewed
Feb 18, 2021
tobim
force-pushed
the
story/ch22707/timestamp-type
branch
from
7316487
to
f031c29
Compare
mavam
reviewed
Feb 18, 2021
mavam
reviewed
Feb 19, 2021
There was a problem hiding this comment.
The code looks clean. Green light.
Before approving, we still need to give it a spin locally.
Also, while we're at it: What happens with aliases of aliases? Say I have type foo = timestamp. Will the symmetric difference between the result of :foo: and :timestamp queries be the strict delta to :timestamp? In other words, will :foo include the results of :timestamp?
|
@tobim: one todo: please add an integration that uses something like |
dominiklohmann
requested changes
Feb 19, 2021
There was a problem hiding this comment.
The general functionality works. You missed some things, though:
❯ git grep '#timestamp' -- ':(exclude)CHANGELOG.md'
README.md:vast export json '#timestamp > 1 hour ago && (6.6.6.6 || 5353/udp)'
doc/cli/vast-explore.md:`--before`, or `--context`) apply relative to the #timestamp field of the
doc/cli/vast-export.md:vast export --max-events=100 --continuous json '#timestamp < 1 hour ago'
doc/cli/vast-export.md:`#timestamp` attribute will be exported, and only if the timestamp in that field
doc/cli/vast-import-json.md:`#timestamp` or `#skip`.
libvast/src/concept/parseable/vast/expression.cpp: VAST_WARN("#timestamp queries are deprecated and should be replaced with "
libvast/test/expression_evaluation.cpp:TEST(evaluation - attribute extractor - #timestamp) {
libvast/test/expression_evaluation.cpp: auto expr = make_conn_expr("#timestamp <= 2009-11-18T08:09");
libvast/test/expression_parseable.cpp: MESSAGE("now > #timestamp");
libvast/test/expression_parseable.cpp: str = "now > #timestamp";
libvast/test/expression_parseable.cpp: "#timestamp > 2018-07-04+12:00:00.0 && #timestamp < 2018-07-04+23:55:04.0"s,
libvast/test/meta_index.cpp: std::string q = "#timestamp == 1970-01-01+";
libvast/test/meta_index.cpp: std::string q = "#timestamp >= 1970-01-01+";
libvast/test/meta_index.cpp: q += " && #timestamp <= 1970-01-01+";
libvast/test/system/query_processor.cpp:constexpr std::string_view query_str = "#timestamp < 1 week ago";
pyvast/pyvast/test_vast.py: query = "#timestamp < 1 hour ago"
pyvast/pyvast/test_vast.py: query = ":addr == 192.168.1.104 && #timestamp < 1 hour ago"
scripts/generate-sysmon-schema.py: return "time #timestamp"
vast/integration/default_set.yaml: - command: '-N count "#timestamp >= 1970-01-01 && #type != \"vast.metrics\""'
vast/integration/default_set.yaml: - command: 'count "#timestamp >= 1970-01-01 && #type != \"vast.metrics\""'
vast/integration/default_set.yaml: - command: -N export ascii '#timestamp >= 2011-08-15T03:48'
vast/integration/misc/schema/argus.schema: StartTime: time #timestamp,
I simply changed the |
tobim
force-pushed
the
story/ch22707/timestamp-type
branch
from
368a869
to
8405ac4
Compare
tobim
force-pushed
the
story/ch22834/udt-type-extractor
branch
from
8d3d115
to
73d615c
Compare
mavam
approved these changes
Feb 19, 2021
There was a problem hiding this comment.
Looks good from my side. Not sure if @dominiklohmann has something to add.
| ; | ||
| return type_type(f, l, a); | ||
| // Bogus indentation from clang-format ¯\_(ツ)_/¯ |
|
My feedback is still the same @mavam: Please change these as well. |
dominiklohmann
approved these changes
Feb 22, 2021
The parser erroneously accepted builtin type names if they were prefixes of user defined type names. For example `timestamp` would get parsed as `time`, leaving `stamp` uncomsumed.
Co-authored-by: Matthias Vallentin <matthias@tenzir.com> Co-authored-by: Dominik Lohmann <mail@dominiklohmann.de>
The `attr_list` parser is now only called once for the final type instead of for each alternative directly. The placeholder parser is back in it's old place.
The rewriting of the `#timestamp` query in the parsing step would otherwise make it impossible to extract data that is already in a db with the type `time #timestamp`. A section in this code is marked DEPRECATED and should be removed after a reasonable grace period.
tobim
force-pushed
the
story/ch22707/timestamp-type
branch
from
a711d8e
to
c09ea02
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📔 Description
📝 Checklist
🎯 Review Instructions
By commit.