Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect/dataset: delay set operation after signature full match #11662

Closed

Conversation

catenacyber
Copy link
Contributor

@catenacyber catenacyber commented Aug 27, 2024

Link to ticket: https://redmine.openinfosecfoundation.org/issues/
https://redmine.openinfosecfoundation.org/issues/5576

Describe changes:

  • detect/dataset: delay set operation after signature full match

SV_BRANCH=OISF/suricata-verify#2000

#11623 with rebase to get greener CI

Side note: the limitation described for flowvar in https://redmine.openinfosecfoundation.org/issues/7197 also applies here to dataset, and needs a bigger design...

Draft : is it okay to get this in with this limitation ? As it already improves some cases ?
And could we run postmatch clean stuff if there is no complete match...

The set operation of dataset keyword was done even if signature
did not fully match.

This patch changes the behavior of the dataset keyword to do a
match and a post match for the set operation. In the match, the
buffer data that needs to end up in the set is captured and in
post match the dataset is updated (if ever the signature is fully
matching).

Ticket: OISF#5576
const SigMatchData *smd = s->sm_arrays[DETECT_SM_LIST_POSTMATCH];
if (smd != NULL) {
while (1) {
(void)sigmatch_table[smd->type].Match(det_ctx, p, NULL, smd->ctx);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts about this hack ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detect-engine-state-02: FAILED: got exit code -11, expected 0

DetectFilestorePostMatch detect-filestore.c:214 does not like that hack

Copy link

codecov bot commented Aug 27, 2024

Codecov Report

Attention: Patch coverage is 84.05797% with 11 lines in your changes missing coverage. Please review.

Project coverage is 70.34%. Comparing base (304271e) to head (8f61e9e).
Report is 50 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (304271e) and HEAD (8f61e9e). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (304271e) HEAD (8f61e9e)
suricata-verify 1 0
fuzzcorpus 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #11662       +/-   ##
===========================================
- Coverage   82.61%   70.34%   -12.28%     
===========================================
  Files         919      919               
  Lines      248997   248856      -141     
===========================================
- Hits       205717   175049    -30668     
- Misses      43280    73807    +30527     
Flag Coverage Δ
fuzzcorpus ?
livemode 18.74% <72.46%> (+0.07%) ⬆️
pcap 44.11% <13.04%> (-0.03%) ⬇️
suricata-verify ?
unittests 58.99% <11.59%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@catenacyber
Copy link
Contributor Author

SV failures to investigate

===> detect-engine-state-02: FAILED: got exit code -11, expected 0
===> datasets-06-state-long: Sub test #1: FAIL : Shell command failed: {'args': 'cat ${TEST_DIR}/expected/state.csv | sort > expected.csv\ncat state.csv | sort | cmp - expected.csv\n'} -> b'- expected.csv differ: char 1, line 1\n'

@suricata-qa
Copy link

Information:

ERROR: QA failed on SURI_TLPW2_single_alerts_cmp.

ERROR: QA failed on SURI_TLPW2_autofp_alerts_cmp.

ERROR: QA failed on SURI_TLPR1_alerts_cmp.

ERROR: QA failed on IPS_AFP_drop_chk.

field baseline test %
IPS_AFP_stats_chk
.ips.accepted 367733210 342623210 93.17%
.ips.blocked 747360 25857360 3459.83%
.ips.drop_reason.flow_drop 680400 25736400 3782.54%
.ips.drop_reason.rules 66960 120960 180.65%
.flow.end.state.established 550799 604799 109.8%
.flow.end.state.closed 1048672 994672 94.85%
.flow.end.tcp_state.established 169560 223560 131.85%
.flow.end.tcp_state.closed 1048672 994672 94.85%
.detect.alert 185760 293760 158.14%
TREX_GENERIC_stats_chk
.detect.alert 86688 129368 149.23%

Pipeline 22274

@catenacyber
Copy link
Contributor Author

Continued in #11704

@catenacyber catenacyber closed this Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants