new MPM: ac-tile #9
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DEFAULT_MPM defaults to MPM_AC.
Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
|
Victor is back and he will be far better than me on reviewing this. Please submit again to his repo. |
|
Hi Eric, Ok. Can you push the regit-master changes to Victor's branch? That will make integration easier. Thanks, On Aug 29, 2013, at 4:40 AM, "Eric Leblond" <notifications@github.commailto:notifications@github.com> wrote: Victor is back and he will be far better than me on reviewing this. Please submit again to his repo. — |
|
Hi ken, Already done check PR OISF#504 on inliniac tree. Ken Steele notifications@github.com a écrit :
|
|
Thanks, From: Eric Leblond [mailto:notifications@github.com] Hi ken, Already done check PR OISF#504 on inliniac tree. Ken Steele <notifications@github.commailto:notifications@github.com> a écrit :
— |
regit
added a commit
that referenced
this pull request
Nov 17, 2014
Under serious load, it is possible that a app layer get changed on a flow when another packet of the Flow is still examined in Detect. The consequence is that it is possible to app layer to get updated and the rest of the detection run with the new app layer that don't have the same property (such as Tx handling function which are called on TLS session). This is mainly the case when alproto is fetched from Flow and then the Flow is unlocked and modifiable by another thread. When alstate is fetch later, we can have alstate not matching alproto and this causes crashes. To fix that this patch is using alindex to access to the original application layer during the detection. This means some function prototype have been update to use alindex instead of alproto. Also the FlowGet*AtIndex function are used with alindex param to access to the correct application layer. For reference, here's one the backtrace: (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00000000004310e3 in AppLayerParserSetTransactionInspectId (pstate=0x151743b10, ipproto=ipproto@entry=6 '\006', alproto=alproto@entry=4, alstate=alstate@entry=0x14d47b790, direction=direction@entry=4 '\004') at app-layer-parser.c:536 #2 0x000000000048e6d7 in DeStateUpdateInspectTransactionId (f=0x7ffefc68ba00, direction=4 '\004') at detect-engine-state.c:785 #3 0x000000000045c034 in SigMatchSignatures (th_v=0x3420fa50, de_ctx=0x23b6f00, det_ctx=0x13fd68ea0, p=<optimized out>) at detect.c:1589 #4 0x000000000045c9f3 in Detect (data=<optimized out>, p=<optimized out>, tv=<optimized out>, pq=<optimized out>, postpq=<optimized out>) at detect.c:1744 #5 Detect (tv=<optimized out>, p=<optimized out>, data=<optimized out>, pq=<optimized out>, postpq=<optimized out>) at detect.c:1716 #6 0x000000000053d24d in TmThreadsSlotVarRun (tv=0x3420fa50, p=0x13fd56920, slot=0x14d47b790, slot@entry=0x13db1ecc0) at tm-threads.c:575 #7 0x00000000005186ba in TmThreadsSlotProcessPkt (p=0x13fd56920, s=0x13db1ecc0, tv=0x3420fa50) at tm-threads.h:148 #8 AFPReadFromRing (ptv=ptv@entry=0x13b029bd0) at source-af-packet.c:875 #9 0x000000000051b5fd in ReceiveAFPLoop (tv=<optimized out>, data=0x13b029bd0, slot=<optimized out>) at source-af-packet.c:1215 #10 0x00000000005408db in TmThreadsSlotPktAcqLoop (td=0x3420fa50) at tm-threads.c:722 #11 0x00007ffff6920b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007ffff51e8a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? ()
regit
added a commit
that referenced
this pull request
Jun 16, 2015
This patch fixes a partial long duration lock up in Suricata. The problem arises when max_pending_packet is reached in worker mode. In that condition some capture threads get blocked in FlowGetFlowFromHash call. The following backtrace shows an example of the lock up. The first thread is waiting on the flow bucket mutex and the second one is remaining stuck at PacketPoolWait because there is almost no signalling in the used worker mode: (gdb) bt #0 0x00007f5442a4ed5c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f5442a4a3a9 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f5442a4a1cb in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00000000005346c0 in FlowGetFlowFromHash (tv=0x2c9f68c80, dtv=0x2cc63cc30, p=0x2cc62a700) at flow-hash.c:653 #4 0x000000000053122c in FlowHandlePacket (tv=0x2c9f68c80, dtv=0x2cc63cc30, p=0x2cc62a700) at flow.c:340 #5 0x0000000000461cb0 in DecodeTCP (tv=0x2c9f68c80, dtv=0x2cc63cc30, p=0x2cc62a700, pkt=0x7f542ec00bd4 "\312?\037L\277h\257p", len=32, pq=0x2c9f68f10) at decode-tcp.c:206 #6 0x000000000045db38 in DecodeIPV4 (tv=0x2c9f68c80, dtv=0x2cc63cc30, p=0x2cc62a700, pkt=0x7f542ec00bc0 "E", len=61, pq=0x2c9f68f10) at decode-ipv4.c:561 #7 0x0000000000459887 in DecodeEthernet (tv=0x2c9f68c80, dtv=0x2cc63cc30, p=0x2cc62a700, pkt=0x7f542ec00bb2 "", len=75, pq=0x2c9f68f10) at decode-ethernet.c:60 #8 0x00000000005a928d in DecodeAFP (tv=0x2c9f68c80, p=0x2cc62a700, data=0x2cc63cc30, pq=0x2c9f68f10, postpq=0x0) at source-af-packet.c:1872 #9 0x00000000005db191 in TmThreadsSlotVarRun (tv=0x2c9f68c80, p=0x2cc62a700, slot=0x2c9f68ed0) at tm-threads.c:132 #10 0x00000000005a08a5 in TmThreadsSlotProcessPkt (tv=0x2c9f68c80, s=0x2c9f68ed0, p=0x2cc62a700) at tm-threads.h:147 #11 0x00000000005a2982 in AFPReadFromRing (ptv=0x2cc62b710) at source-af-packet.c:874 #12 0x00000000005a40c2 in ReceiveAFPLoop (tv=0x2c9f68c80, data=0x2cc62b710, slot=0x2c9f68d90) at source-af-packet.c:1214 #13 0x00000000005dbae6 in TmThreadsSlotPktAcqLoop (td=0x2c9f68c80) at tm-threads.c:336 #14 0x00007f5442a47b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007f544130f95d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #16 0x0000000000000000 in ?? () (gdb) thread 5 [Switching to thread 5 (Thread 0x7f54325a6700 (LWP 9282))] #0 0x00007f5442a4c344 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) bt #0 0x00007f5442a4c344 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005d806e in PacketPoolWait () at tmqh-packetpool.c:152 #2 0x0000000000539443 in FlowForceReassemblyPseudoPacketGet (direction=1, f=0x83a7880, ssn=0x2dcea3680, dummy=0) at flow-timeout.c:257 #3 0x000000000053972f in FlowForceReassemblyForFlow (f=0x83a7880, server=2, client=1) at flow-timeout.c:377 #4 0x0000000000535197 in FlowManagerFlowTimedOut (f=0x83a7880, ts=0x7f54325a52a0) at flow-manager.c:246 #5 0x0000000000535231 in FlowManagerHashRowTimeout (f=0x83a7880, ts=0x7f54325a52a0, emergency=0, counters=0x7f54325a5280) at flow-manager.c:294 #6 0x00000000005354f8 in FlowTimeoutHash (ts=0x7f54325a52a0, try_cnt=0, hash_min=0, hash_max=1048576, counters=0x7f54325a5280) at flow-manager.c:389 #7 0x0000000000535e48 in FlowManager (th_v=0x2dd38e330, thread_data=0x2dd38de80) at flow-manager.c:612 #8 0x00000000005dc7a0 in TmThreadsManagement (td=0x2dd38e330) at tm-threads.c:600 #9 0x00007f5442a47b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #10 0x00007f544130f95d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #11 0x0000000000000000 in ?? () This problem is due to the fact that the return_stack condition is not signaled if a packet is returned to the thread own PacketPool. So if the FlowManager try to get a packet and has to wait for some to be available then it can get stuck on the condition for a long time.
regit
added a commit
that referenced
this pull request
Mar 2, 2016
This patch fixes the following leak:
Direct leak of 9982880 byte(s) in 2902 object(s) allocated from:
#0 0x4c253b in malloc ??:?
#1 0x10c39ac in MimeDecInitParser /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/util-decode-mime.c:2379
#2 0x6a0f91 in SMTPProcessRequest /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1085
#3 0x697658 in SMTPParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1185
#4 0x68fa7a in SMTPParseClientRecord /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1208
#5 0x6561c5 in AppLayerParserParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-parser.c:908
#6 0x53dc2e in AppLayerHandleTCPData /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer.c:444
#7 0xf8e0af in DoReassemble /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:2635
#8 0xf8c3f8 in StreamTcpReassembleAppLayer /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3028
#9 0xf94267 in StreamTcpReassembleHandleSegmentUpdateACK /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3404
#10 0xf9643d in StreamTcpReassembleHandleSegment /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3432
#11 0xf578b4 in HandleEstablishedPacketToClient /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2245
#12 0xeea3c7 in StreamTcpPacketStateEstablished /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2489
#13 0xec1d38 in StreamTcpPacket /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:4568
#14 0xeb0e16 in StreamTcp /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:5064
#15 0xff52a4 in TmThreadsSlotVarRun /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:130
#16 0xffdad1 in TmThreadsSlotVar /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:474
#17 0x7f7cd678d181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 (discriminator 2)
We come to this case when a SMTP session contains at least 2 mails
and then the ending of the first is not correctly detected. In that
case, switching to a new tx seems a good solution. This way we still
have partial logging.
regit
added a commit
that referenced
this pull request
Mar 3, 2016
This patch fixes the following leak:
Direct leak of 9982880 byte(s) in 2902 object(s) allocated from:
#0 0x4c253b in malloc ??:?
#1 0x10c39ac in MimeDecInitParser /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/util-decode-mime.c:2379
#2 0x6a0f91 in SMTPProcessRequest /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1085
#3 0x697658 in SMTPParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1185
#4 0x68fa7a in SMTPParseClientRecord /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1208
#5 0x6561c5 in AppLayerParserParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-parser.c:908
#6 0x53dc2e in AppLayerHandleTCPData /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer.c:444
#7 0xf8e0af in DoReassemble /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:2635
#8 0xf8c3f8 in StreamTcpReassembleAppLayer /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3028
#9 0xf94267 in StreamTcpReassembleHandleSegmentUpdateACK /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3404
#10 0xf9643d in StreamTcpReassembleHandleSegment /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3432
#11 0xf578b4 in HandleEstablishedPacketToClient /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2245
#12 0xeea3c7 in StreamTcpPacketStateEstablished /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2489
#13 0xec1d38 in StreamTcpPacket /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:4568
#14 0xeb0e16 in StreamTcp /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:5064
#15 0xff52a4 in TmThreadsSlotVarRun /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:130
#16 0xffdad1 in TmThreadsSlotVar /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:474
#17 0x7f7cd678d181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 (discriminator 2)
We come to this case when a SMTP session contains at least 2 mails
and then the ending of the first is not correctly detected. In that
case, switching to a new tx seems a good solution. This way we still
have partial logging.
regit
added a commit
that referenced
this pull request
Mar 21, 2016
This patch fixes some error handling in code generating JSON output for
email event.
This fixes:
Indirect leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394771c (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x371c)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 96 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bc7b (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7c7b)
Indirect leak of 82 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff923949924 (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x5924)
Indirect leak of 72 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bcda in json_object (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7cda)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 64 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf5a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f5a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf2a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f2a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
regit
added a commit
that referenced
this pull request
Mar 21, 2016
This patch fixes some error handling in code generating JSON output for
email event.
This fixes:
Indirect leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394771c (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x371c)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 96 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bc7b (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7c7b)
Indirect leak of 82 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff923949924 (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x5924)
Indirect leak of 72 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bcda in json_object (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7cda)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 64 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf5a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f5a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf2a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f2a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
regit
added a commit
that referenced
this pull request
Mar 24, 2016
This patch fixes some error handling in code generating JSON output for
email event.
This fixes:
Indirect leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394771c (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x371c)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 96 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bc7b (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7c7b)
Indirect leak of 82 byte(s) in 3 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff923949924 (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x5924)
Indirect leak of 72 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bcda in json_object (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7cda)
#2 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#3 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#4 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#5 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#6 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#7 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#8 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 64 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf5a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f5a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
Indirect leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x50c142 in malloc (/home/eric/git/oisf/src/.libs/lt-suricata+0x50c142)
#1 0x7ff92394bf2a in json_array (/usr/lib/x86_64-linux-gnu/libjansson.so.4+0x7f2a)
#2 0x1bd04b5 in JsonEmailLogJsonData /home/eric/git/oisf/src/output-json-email-common.c:296:27
#3 0x1bd3309 in JsonEmailLogJson /home/eric/git/oisf/src/output-json-email-common.c:376:19
#4 0x1bfe774 in JsonSmtpLogger /home/eric/git/oisf/src/output-json-smtp.c:103:9
#5 0x1c378ff in OutputTxLog /home/eric/git/oisf/src/output-tx.c:165:17
#6 0x1f94ef3 in TmThreadsSlotVarRun /home/eric/git/oisf/src/tm-threads.c:134:17
#7 0x1d33478 in TmThreadsSlotProcessPkt /home/eric/git/oisf/src/./tm-threads.h:150:9
#8 0x1d32dd4 in PcapFileCallbackLoop /home/eric/git/oisf/src/source-pcap-file.c:184:9
#9 0x7ff924199013 (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f013)
regit
pushed a commit
that referenced
this pull request
May 20, 2020
Fixes runs with --enable-debug-validation. The target did not init a
packet pool, so for a tunnel packet would try to get a packet from
an uninitialized pool. In non-debug mode, this silently works by
falling back to a packet from alloc.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff35a6801 in __GI_abort () at abort.c:79
#2 0x00007ffff359639a in __assert_fail_base (fmt=0x7ffff371d7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555557fe7260 "!(pool->initialized == 0)",
file=file@entry=0x555557fe7220 "tmqh-packetpool.c", line=line@entry=253, function=function@entry=0x555557fe7500 <__PRETTY_FUNCTION__.21181> "PacketPoolGetPacket") at assert.c:92
#3 0x00007ffff3596412 in __GI___assert_fail (assertion=0x555557fe7260 "!(pool->initialized == 0)", file=0x555557fe7220 "tmqh-packetpool.c", line=253,
function=0x555557fe7500 <__PRETTY_FUNCTION__.21181> "PacketPoolGetPacket") at assert.c:101
#4 0x00005555577e24be in PacketPoolGetPacket () at tmqh-packetpool.c:253
#5 0x0000555556914ecd in PacketGetFromQueueOrAlloc () at decode.c:183
#6 0x00005555569161e1 in PacketTunnelPktSetup (tv=0x555559863980 <tv>, dtv=0x614000068e40, parent=0x61e0000fc080, pkt=0x61e0000fc470 "LL", len=72, proto=DECODE_TUNNEL_IPV4) at decode.c:286
#7 0x00005555569de694 in DecodeIPv4inIPv6 (tv=0x555559863980 <tv>, dtv=0x614000068e40, p=0x61e0000fc080, pkt=0x61e0000fc470 "LL", plen=72) at decode-ipv6.c:59
#8 0x00005555569e60b5 in DecodeIPV6ExtHdrs (tv=0x555559863980 <tv>, dtv=0x614000068e40, p=0x61e0000fc080, pkt=0x61e0000fc470 "LL", len=112) at decode-ipv6.c:522
#9 0x00005555569e846f in DecodeIPV6 (tv=0x555559863980 <tv>, dtv=0x614000068e40, p=0x61e0000fc080, pkt=0x61e0000fc420 "cLL", len=255) at decode-ipv6.c:641
#10 0x0000555556a032f9 in DecodeRaw (tv=0x555559863980 <tv>, dtv=0x614000068e40, p=0x61e0000fc080, pkt=0x61e0000fc420 "cLL", len=255) at decode-raw.c:70
#11 0x0000555557659ba8 in DecodePcapFile (tv=0x555559863980 <tv>, p=0x61e0000fc080, data=0x614000068e40) at source-pcap-file.c:412
#12 0x0000555556573401 in LLVMFuzzerTestOneInput (data=0x613000000047 "\241\262\315\064", size=339) at tests/fuzz/fuzz_sigpcap.c:158
#13 0x0000555557a4dc66 in main (argc=2, argv=0x7fffffffdfa8) at tests/fuzz/onefile.c:51
That line:
BUG_ON(pool->initialized == 0);
regit
pushed a commit
that referenced
this pull request
May 20, 2020
In case of bad IPv4, TCP or UDP, the per packet ip4vars/tcpvars/udpvar
structures would not be cleaned up because the cleanup depends on the
'header' pointer being set, but the error handling would unset that.
This could mean these structures were already filled with values before
the error was detected. As packets were recycled, the next packet decoding
would use this unclean structure.
To make things worse these structures are part of unions. IPv4/IPv6 and
TCP/ICMPv4/ICMPv6 share the same memory location.
LibFuzzer+UBSAN found this both locally and in Oss-Fuzz:
decode-ipv6.c:654:9: runtime error: load of value 6, which is not a valid value for type 'bool'
#0 0x6146f0 in DecodeIPV6 /src/suricata/src/decode-ipv6.c:654:9
#1 0x617e96 in DecodeNull /src/suricata/src/decode-null.c:70:13
#2 0x9dd8a4 in DecodePcapFile /src/suricata/src/source-pcap-file.c:412:9
#3 0x4c8ed2 in LLVMFuzzerTestOneInput /src/suricata/src/tests/fuzz/fuzz_sigpcap.c:158:25
#4 0x457e51 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:556:15
#5 0x457575 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:470:3
#6 0x459917 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:698:19
#7 0x45a6a5 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:830:5
#8 0x448728 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:824:6
#9 0x472552 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10
#10 0x7ff0d097b82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#11 0x41bde8 in _start (/out/fuzz_sigpcap+0x41bde8)
Bug: OISF#3496
regit
pushed a commit
that referenced
this pull request
Oct 13, 2023
Make sure to first close all ports before freeing device mempools. Thread 1 "Suricata-Main" received signal SIGSEGV, Segmentation fault. 0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so (gdb) bt #0 0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so #1 0x00007ffff469a948 in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so #2 0x00007ffff45606aa in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so #3 0x00007ffff6d4ed8d in rte_eth_dev_close () from /usr/lib/x86_64-linux-gnu/librte_ethdev.so.20.0 #4 0x000000000055fc4c in DPDKCloseDevice (ldev=ldev@entry=0xe3a400) at util-dpdk.c:53 #5 0x000000000055f4eb in LiveDeviceListClean () at util-device.c:331 #6 0x00000000005511c8 in GlobalsDestroy (suri=<optimized out>) at suricata.c:381 #7 0x0000000000550a76 in SuricataMain (argc=<optimized out>, argv=<optimized out>) at suricata.c:3059 #8 0x00007ffff6a24083 in __libc_start_main (main=0x54cca0 <main>, argc=8, argv=0x7fffffffe4c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4b8) at ../csu/libc-start.c:308 #9 0x000000000054cbde in _start () Bug: OISF#5619.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change optimizes multi-pattern matching for Tilera and makes ac-tile the default pattern matcher on Tile. Some of the optimizations here, particularly alphabet compression, might be useful on other architectures.