-
Notifications
You must be signed in to change notification settings - Fork 60
/
README
243 lines (194 loc) · 9.69 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
libprotoident 2.0.15
---------------------------------------------------------------------------
Copyright (c) 2011-2020 The University of Waikato, Hamilton, New Zealand.
All rights reserved.
This code has been developed by the University of Waikato WAND
research group. For further information please see http://www.wand.net.nz/.
---------------------------------------------------------------------------
See the file COPYING and COPYING.LESSER for full licensing details for this
software.
Report and bugs, questions or comments to contact@wand.net.nz
NEW: You can now lodge bugs by filing an issue on the libprotoident github:
https://github.com/wanduow/libprotoident
Authors:
Shane Alcock
With contributions from:
Donald Neal
Aaron Murrihy
Paweł Foremski <pjf@iitis.pl>
Fabian Weisshaar <elnappo@nerdpol.io>
Jeroen Roovers
Jiri Havranek
Romain Fontugne
Jacob van Walraven
Introduction
============
Libprotoident is a library designed to perform application protocol
identification using a very limited form of deep packet inspection, i.e. using
the first four bytes of application payload sent in each direction. The
library provides a simple API that will enable programmers to develop their own
tools that utilise application protocol information and we have also included
some tools that can be used to perform simple analysis of traffic flows.
Required Libraries
==================
libtrace 4.0.1 or later
* available from https://github.com/LibtraceTeam/libtrace
libflowmanager 3.0.0 or later
* optional, but required to build the tools
* available from https://github.com/LibtraceTeam/libflowmanager
Installation
============
After having installed the required libraries, running the following series
of commands should install libprotoident
./bootstrap.sh (only if you've cloned the source from GitHub)
./configure
make
make install
By default, libprotoident installs to /usr/local - this can be changed by
appending the --prefix=<new location> option to ./configure.
The libprotoident tools are built by default - this can be changed by using the
--with-tools=no option with ./configure.
Protocols Supported
===================
A full list of supported protocols can be found at
https://github.com/wanduow/libprotoident/wiki/SupportedProtocols
Libprotoident also currently has rules for several "mystery" protocols. These
are patterns that commonly occur in our trace sets that we cannot tie to an
actual protocol. It would be nice to know what these protocols actually are -
if you have any suggestions please feel free to email us at contact@wand.net.nz.
In addition, a flow can be assigned into a "category" based on the protocol
determined by libprotoident, enabling broader analysis. For example,
BitTorrent, Gnutella and eMule all fall into the P2P category, whereas SMTP,
POP3 and IMAP are part of the Mail category.
Tools
=====
There are three tools included with libprotoident.
* lpi_protoident
Description:
This tool attempts to identify each individual flow within the provided
trace. Identification only occurs when the flow has concluded or
expired, so it is not very effective for real-time applications.
Usage:
lpi_protoident <input trace URI>
The input trace must be a valid libtrace URI.
See https://github.com/LibtraceTeam/libtrace/wiki/Supported-Trace-Formats
to learn more about libtrace URIs. Note that a URI may be a live
source, such as a network interface.
Output:
For each flow in the input trace, a single line is printed to stdout
describing the flow. The line contains the following fields separated
by spaces (in order):
* Application protocol (as reported by libprotoident)
* IP address of the first endpoint
* IP address of the second endpoint
* Port used by the first endpoint
* Port used by the second endpoint
* Transport protocol (6 = TCP, 17 = UDP)
* Unix timestamp when the flow began
* Unix timestamp when the flow ended
* Total bytes sent from first endpoint to second endpoint
* Total bytes sent from second endpoint to first endpoint
* First four bytes of payload sent from first endpoint (in hex)
* First four bytes of payload sent from first endpoint (ASCII)
* Size of first payload-bearing packet sent from first endpoint
* First four bytes of payload sent from second endpoint (in hex)
* First four bytes of payload sent from second endpoint (ASCII)
* Size of first payload-bearing packet sent from second endpoint
* lpi_find_unknown
Description:
This tool reports all the flows in a trace which libprotoident
was unable to identify. Identification only occurs when the flow has
concluded or expired, so it is not very effective for real-time
applications.
This is mainly intended as a tool to aid development of new protocol
identifiers.
Usage:
lpi_find_unknown <input trace URI>
The input trace must be a valid libtrace URI.
See https://github.com/LibtraceTeam/libtrace/wiki/Supported-Trace-Formats
to learn more about libtrace URIs. Note that a URI may be a live
source, such as a network interface.
Output:
For each unknown flow in the input trace, a single line is printed to
stdout describing the flow. The line contains the following fields
separated by spaces (in order):
* IP address of the first endpoint
* IP address of the second endpoint
* Port used by the first endpoint
* Port used by the second endpoint
* Transport protocol (6 = TCP, 17 = UDP)
* Unix timestamp when the flow began
* Total bytes sent from first endpoint to second endpoint
* Total bytes sent from second endpoint to first endpoint
* First four bytes of payload sent from first endpoint (in hex)
* First four bytes of payload sent from first endpoint (ASCII)
* Size of first payload-bearing packet sent from first endpoint
* First four bytes of payload sent from second endpoint (in hex)
* First four bytes of payload sent from second endpoint (ASCII)
* Size of first payload-bearing packet sent from second endpoint
* lpi_arff
Description:
This tool is similar to lpi_protoident except that it writes its
output in the ARFF format so that it is compatible with the Weka
machine learning software (http://www.cs.waikato.ac.nz/ml/weka/).
This tool was contributed by Paweł Foremski <pjf@iitis.pl>.
Usage:
lpi_arff <input trace URI>
The input trace must be a valid libtrace URI.
See https://github.com/LibtraceTeam/libtrace/wiki/Supported-Trace-Formats
to learn more about libtrace URIs. Note that a URI may be a live
source, such as a network interface.
Output:
The output begins with a series of lines describing each feature that
will be used to describe each flow. Following that,
for each flow in the input trace, a single line is printed to stdout
describing the flow. The line contains the following fields separated
by commas (in order):
* Application protocol (as reported by libprotoident)
* ID number for the application protocol
* Total number of packets sent from first endpoint to second endpoint
* Total number of bytes sent from first endpoint to second endpoint
* Total number of packets sent from second endpoint to first endpoint
* Total number of bytes sent from second endpoint to first endpoint
* Minimum payload size sent from first endpoint to second endpoint
* Mean payload size sent from first endpoint to second endpoint
* Maximum payload size sent from first endpoint to second endpoint
* Standard deviation of payload size sent from first endpoint to
second endpoint
* Minimum payload size sent from second endpoint to first endpoint
* Mean payload size sent from second endpoint to first endpoint
* Maximum payload size sent from second endpoint to first endpoint
* Standard deviation of payload size sent from second endpoint to
first endpoint
* Minimum packet interarrival time for packets sent from first
endpoint to second endpoint
* Mean packet interarrival time for packets sent from first
endpoint to second endpoint
* Maximum packet interarrival time for packets sent from first
endpoint to second endpoint
* Standard deviation of packet interarrival time for packets sent from
first endpoint to second endpoint
* Minimum packet interarrival time for packets sent from second
endpoint to first endpoint
* Mean packet interarrival time for packets sent from second
endpoint to first endpoint
* Maximum packet interarrival time for packets sent from second
endpoint to first endpoint
* Standard deviation of packet interarrival time for packets sent from
second endpoint to first endpoint
* Flow duration (in microseconds)
* Flow start time (as a Unix timestamp)
API
===
If you want to develop your own tools based on libprotoident, you'll need to
use the libprotoident API. The API is very simple and the best way to learn it
is to examine how the existing tools work. The source for the tools is
located in the tools/ directory.
The tools use libflowmanager to do the flow tracking, using an instance of a
FlowManager class. You will probably want to incorporate this into your own
tool. Usage of libprotoident itself is through functions beginning with 'lpi_'.
The libprotoident API functions themselves are documented in
lib/libprotoident.h if you need further guidance.
Further documentation of the API can also be found at
https://github.com/LibtraceTeam/libflowmanager
If all else fails, drop me a line at shane@alcock.co.nz