Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hyperscan MPM integration v6 #1965

Closed
wants to merge 3 commits into from
Closed

Hyperscan MPM integration v6 #1965

wants to merge 3 commits into from

Conversation

jviiret
Copy link
Contributor

@jviiret jviiret commented Mar 28, 2016

(Version 6: adds the hashlittle_safe() function in c0b7bd6 to avoid asan/valgrind warnings when hashing arbitrary strings, as discussed in v5. The previous PR was #1955.)

This PR adds support for using Intel's Hyperscan regex engine as an MPM algo, namely "hs".

It has a couple of notable features over a straightforward implementation:

  • Caseful/caseless matching is done by Hyperscan natively.
  • Offset and depth are also checked by Hyperscan using its extended parameter support (this meant that these parameters had to be passed through to the add functions in 87bd04a).
  • Hyperscan MPM structures are cached and deduped. We found during testing that some configurations generate many duplicate copies of the same MPM matcher, and this has a very significant memory cost. It might be that this can be addressed, or that deduplication can be hoisted up to apply generically to any MPM implementation.

More info on Hyperscan: https://01.org/hyperscan
Ticket: https://redmine.openinfosecfoundation.org/issues/1704

Justin Viiret added 3 commits March 29, 2016 10:01
util-hash-lookup3: Add hashlittle_safe() variant
c0b7bd6 
By default, hashlittle() will read off the end of the key, up to the
next four-byte boundary, although the data beyond the end of the key
doesn't affect the hash. This read causes uninitialized read warnings
from Valgrind and Address Sanitizer.

Here we add hashlittle_safe(), which avoids reading off the end of the
buffer (using the code inside the VALGRIND-guarded block in the original
hashlittle() implementation).
mpm: pass offset, depth args to add functions
a179045 
MpmAddPatternCI and MpmAddPatternCS had arguments for offset and depth,
but these were not being passed in by the caller.
mpm: add Hyperscan integration
4a0c2fe 
This adds an MPM implementation that uses the Hyperscan regex engine
library from Intel, accessible as the "hs" mpm-algo.
@jviiret jviiret mentioned this pull request Mar 28, 2016
Closed
@inliniac
Copy link
Contributor

Prscript still passes:

@inliniac inliniac mentioned this pull request Mar 29, 2016
Merged
@inliniac
Copy link
Contributor

The suppressions in here make my DrMemory test pass as well: 31ed704

@inliniac
Copy link
Contributor

Merged through #1968, thanks a lot Justin!

@inliniac inliniac closed this Mar 30, 2016
@zhailiansen
Copy link

Hello,why suricata does not suport hyperscan stream mode?Please give me a reply,thanks.

@worldpeace365
Copy link

The hs was used in suricata， when the http traffic up to 2Gbps it crashed with only one thread，or less traffic with 4 threads。the core dump is follow：

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./suricata --runmode workers -Q 4 -c suricata.yaml --set mpm-algo=hs'.
Program terminated with signal 11, Segmentation fault.
#0 doNormal16 (mode=CALLBACK_OUTPUT, do_accel=0 '\000', s=1,
end=0x7fa59872864e <Address 0x7fa59872864e out of bounds>, c_inout=, m=0x126eb600)
at /root/hyperscan/src/nfa/mcclellan.c:138
138 u8 cprime = m->remap[*c];
Missing separate debuginfos, use: debuginfo-install file-libs-5.11-31.el7.x86_64 glib2-2.42.2-5.el7.x86_64 glibc-2.17-196.tl2.3.x86_64 gmime-2.6.23-1.el7.x86_64 gpgme-1.3.2-5.el7.x86_64 libassuan-2.1.0-3.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libffi-3.0.13-16.el7.x86_64 libgcc-4.8.5-39.tl2.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libpcap-1.5.3-8.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libstdc++-4.8.5-39.tl2.1.x86_64 libyaml-0.1.4-11.el7_0.x86_64 luajit-2.0.4-3.el7.x86_64 lz4-1.7.5-2.tl2.x86_64 nspr-4.10.8-2.el7_1.x86_64 nss-3.19.1-19.el7_2.x86_64 nss-softokn-3.16.2.3-13.el7_1.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 nss-util-3.19.1-4.el7_1.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-19.tl2.1.x86_64 pcre-8.32-15.el7.x86_64 re2-20160401-2.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 yaml-cpp-0.5.1-2.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 doNormal16 (mode=CALLBACK_OUTPUT, do_accel=0 '\000', s=1,
end=0x7fa59872864e <Address 0x7fa59872864e out of bounds>, c_inout=, m=0x126eb600)
at /root/hyperscan/src/nfa/mcclellan.c:138
#1 mcclellanExec16_i (mode=CALLBACK_OUTPUT, c_final=0x0, single=0 '\000', ctxt=0x7fa4b03bbbc0,
cb=0x7fa4eae75490 , offAdj=0, len=60,
buf=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, qstate=0x0, state=,
m=0x126eb600) at /root/hyperscan/src/nfa/mcclellan.c:274
#2 nfaExecMcClellan16_Bi (single=0 '\000', context=0x7fa4b03bbbc0,
cb=0x7fa4eae75490 , length=60,
buffer=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, offset=0, n=0x126eb5c0)
at /root/hyperscan/src/nfa/mcclellan.c:763
#3 nfaExecMcClellan16_B (n=0x126eb5c0, offset=0,
buffer=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, length=60,
cb=0x7fa4eae75490 , context=0x7fa4b03bbbc0)
at /root/hyperscan/src/nfa/mcclellan.c:971
#4 0x00007fa4eae625bd in runAnchoredTableBlock (t=, scratch=0x7fa4b03bbbc0,
atable=) at /root/hyperscan/src/rose/block.c:67
#5 roseBlockAnchored (scratch=0x7fa4b03bbbc0, t=0x126d6580) at /root/hyperscan/src/rose/block.c:212
#6 roseBlockExec (t=, scratch=) at /root/hyperscan/src/rose/block.c:395
#7 0x00007fa4ead93f9e in rawBlockExec (scratch=0x7fa4b03bbbc0, rose=0x126d6580)
at /root/hyperscan/src/runtime.c:188
#8 hs_scan (db=, data=, length=2644, flags=,
scratch=0x7fa4b03bbbc0, onEvent=, userCtx=0x7fa4c5e6d290)
at /root/hyperscan/src/runtime.c:419
#9 0x00000000006bed9c in SCHSSearch (mpm_ctx=, mpm_thread_ctx=,
pmq=, buf=, buflen=) at util-mpm-hs.c:938
#10 0x000000000058c04a in StreamMpmFunc (cb_data=, data=,
data_len=) at detect-engine-payload.c:64
#11 0x000000000067af5c in StreamReassembleRawInline (progress_out=0x7fa4b03b8580, cb_data=0x7fa4c5e6db70,
Callback=0x58c010 , p=0x7fa4b032cf60, ssn=)
at stream-tcp-reassemble.c:1487
#12 StreamReassembleRaw (ssn=, p=p@entry=0x7fa4b028be90,
Callback=Callback@entry=0x58c010 , cb_data=cb_data@entry=0x7fa4c5e6db70,
progress_out=progress_out@entry=0x7fa4b03b8580,
respect_inspect_depth=respect_inspect_depth@entry=false) at stream-tcp-reassemble.c:1677
#13 0x000000000058c1e8 in PrefilterPktStream (det_ctx=0x7fa4b03b8530, p=0x7fa4b028be90, pectx=0x4749470)
at detect-engine-payload.c:83
#14 0x000000000058f711 in Prefilter (det_ctx=det_ctx@entry=0x7fa4b03b8530, sgh=0xd415110,
p=p@entry=0x7fa4b028be90, flags=) at detect-engine-prefilter.c:169
#15 0x0000000000557c33 in DetectRunPrefilterPkt (tv=0x9736360, scratch=0x7fa4c5e6dc70, p=0x7fa4b028be90,
det_ctx=0x7fa4b03b8530, de_ctx=0x470a9d0) at detect.c:734
#16 DetectRun (th_v=th_v@entry=0x9736360, de_ctx=, det_ctx=0x7fa4b03b8530,
p=p@entry=0x7fa4b028be90) at detect.c:132
#17 0x0000000000559757 in DetectRun (p=0x7fa4b028be90, det_ctx=, de_ctx=,
th_v=0x9736360) at detect.c:1810
#18 DetectNoFlow (p=, det_ctx=, de_ctx=, tv=)
at detect.c:1810
#19 Detect (tv=tv@entry=0x9736360, p=p@entry=0x7fa4b028be90, data=data@entry=0x7fa4b03b8530,
pq=pq@entry=0x0, postpq=postpq@entry=0x0) at detect.c:1870
#20 0x00000000005eef5b in FlowWorker (tv=0x9736360, p=0x7fa4b028be90, data=0x7fa4b02ab430,
preq=0x5e9bfc0, unused=) at flow-worker.c:346
#21 0x0000000000680e0b in TmThreadsSlotVarRun (tv=tv@entry=0x9736360, p=p@entry=0x7fa4b028be90,
slot=slot@entry=0x5e9d3a0) at tm-threads.c:143
#22 0x0000000000661e2c in TmThreadsSlotProcessPkt (p=0x7fa4b028be90, s=0x5e9d3a0, tv=0x9736360)
at tm-threads.h:147
#23 ReceiveCFWLoop () at source-cfw.c:378
#24 0x0000000000681ee2 in TmThreadsSlotPktAcqLoop (td=0x9736360) at tm-threads.c:346
#25 0x00007fa4e9636e25 in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
#26 0x00007fa4e8f4935d in clone () from /lib64/libc.so.6

@victorjulien
Copy link
Member

Please report bugs in our issue tracker https://redmine.openinfosecfoundation.org/projects/suricata

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jviiret @inliniac @zhailiansen @worldpeace365 @victorjulien