Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add native Sigma support #1379

Merged
merged 14 commits into from
Feb 19, 2021
Merged

Add native Sigma support #1379

merged 14 commits into from
Feb 19, 2021

Conversation

mavam
Copy link
Member

@mavam mavam commented Feb 14, 2021 •
edited
Loading

📔 Description

This PR adds native parsing of Sigma rules to VAST. It is now possible to use a Sigma rule at places where we use a normal VAST expression. For example, this now works:

vast export json < sigma/rule.yaml

📝 Checklist

  • Handle wildcards properly
  • All user-facing changes have changelog entries.
  • The changes are reflected on docs.tenzir.com/vast, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

  1. Look at the unit tests and the integration tests.
  2. Skim the changes file-by-file to get an overview about the architecture.

@mavam mavam mentioned this pull request Feb 14, 2021
Closed
5 tasks
@mavam mavam added the feature New functionality label Feb 14, 2021
@mavam mavam marked this pull request as ready for review February 16, 2021 10:47
@mavam mavam requested a review from tobim February 16, 2021 10:48
dominiklohmann
dominiklohmann reviewed Feb 17, 2021
View reviewed changes
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had an initial look at it. Will take another look tomorrow.

libvast/src/detail/sigma.cpp Show resolved Hide resolved
dominiklohmann
dominiklohmann requested changes Feb 18, 2021
View reviewed changes
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works fine in practice, but I've had a fair bit of questions on the code itself.

libvast/src/detail/sigma.cpp Show resolved Hide resolved
libvast/src/detail/sigma.cpp Show resolved Hide resolved
libvast/src/detail/sigma.cpp Show resolved Hide resolved
libvast/src/expression.cpp Show resolved Hide resolved
libvast/src/system/spawn_arguments.cpp Show resolved Hide resolved
libvast/vast/concept/parseable/vast/schema.hpp Outdated Show resolved Hide resolved
@mavam
Add preliminary Sigma rule parsing support
f303883
@mavam
Move normalization out of parse function
f250bb5
@mavam
Hook Sigma rule parsing into command system
19f3f81 
We now try to parse a query expression as Sigma rule first. If it fails,
then we go to the regular VAST expression parser.

We currently fail silently on the Sigma parsing; only a debug log entry
is emitted.
@mavam
Complete full detection parsing logic
4530d46 
The "1/all of X" syntax now works. We also apply the "re" modifier and
translate an exact string into a pattern.
@mavam
Add changelog entry
37ed10f
@mavam
Bring back symbol table parser
ea33961 
This data structure was removed in
c6c4840 because it was not used
anymore. Now we need it again.
@mavam
Add integration test
9681fcd
@mavam
Refactor functions for easier testing
58bdc51
@mavam
Fixups buglets and add lots of unit tests
5a5f320
@mavam
Add comment about seemingly short test
add41ef
@mavam
Add some preliminary escaping
9f4e7a3 
This is not complete and by far not tested enough. It just serves as a
starting point. Since VAST currently cannot handle patterns, there's no
point in going exhaustive for now.
@mavam
Add comment regarding regex anchors
2f98022
@mavam
Bail out properly if we cannot handle a modifier
44041fe
@mavam mavam enabled auto-merge February 18, 2021 20:26
@mavam mavam merged commit c8ded9f into master Feb 19, 2021
@mavam mavam deleted the topic/sigma branch February 19, 2021 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dominiklohmann dominiklohmann dominiklohmann approved these changes

@tobim tobim Awaiting requested review from tobim

Assignees
No one assigned
Labels
feature New functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mavam @dominiklohmann