Support (BPF) filter in sniff() with offline parameter set

[p-l-]

Fixes #393
Also, fixes #355

filter= was not documented in sniff() (#355) because it is only passed to the socket. For the same reason, it was not supported with offline=, because no socket is used in that case.

Since there is no way (apart from writing a BPF filter parser in Scapy) to filter a PCAP, this patch uses tcpdump.

[nyov]

This works for me! Awesome work.
So I see the issue was only because I tested an offline pcap file. Oh well.

Couldn't handle gzipped pcap, but then that seems tcpdump's limitation.
Thanks!

[nyov]

FYI. Indent level here seems wrong.

[p-l-]

Nice catch!

[p-l-]

Thanks for the tests and review @nyov!

[guedou]

Looks perfect !

You could add the following unit tests to regression.uts. This option is a good opportunity to test sniff. What do you think ?

import tempfile
fd, filename = tempfile.mkstemp()
wrpcap(filename, Ether(dst="ff:ff:ff:ff:ff")/IP(dst=["1.2.3.4", "5.6.7.8"])/ICMP())
len(sniff(offline=filename, filter="host 5.6.7.8")) == 1

[p-l-]

@guedou @gpotter2 any idea on how I could easily install tcpdump in AppVeyor? So far I have disabled tests requiring tcpdump (in AppVeyor and in Travis-CI when tcpdump is not in $PATH).

[gpotter2]

I'm performing some tests to see how you could do that...

[p-l-]

Thanks @gpotter2. If/when you find a solution, could you submit a PR?
In the meantime, if the tests pass and are OK for you @guedou, I'd say good to merge.

[gpotter2]

@p-l-
On my windows computer, adding this:

  - ps: wget http://www.winpcap.org/windump/install/bin/windump_3_9_5/WinDump.exe -UseBasicParsing -OutFile C:\Windows\System32\Npcap\windump.exe

between
- choco install -y nmap
and
- refreshenv

in appveyor.yml did the trick... But i'm not sure that it will work on Appveyor...

[p-l-]

For some reason that does not work.

@gpotter2 have you been able to use sniff(offline=..., filter=... on Windows?

I suspect -w - is not working here (should be however, according to Windump manual).

[gpotter2]

Investigation is in progress:

• windump looks to be working
• the "magic" is empty when loading on Appveyor in PcapReader_metaclass.__call__, which cause the crash.
• The temp file is readable + writable

[p-l-]

So when I run the tests, the temporary file from which I read contains (base64 encoded here):

1MOyoQIABAAAAAAAAAAAAP//AABlAAAAz8WsVm8qCgAoAAAAKAAAAEUAACgAAQAAQAZ8zX8AAAF/
AAABABQAUAAAAAAAAAAAUAIgAJF8AADPxaxWXy0KABwAAAAcAAAARQAAHAABAABAEXzOfwAAAX8A
AAEANQA1AAgBcs/FrFb5MAoAHAAAABwAAABFAAAcAAEAAEABfN5/AAABfwAAAQgA9/8AAAAA

According to tcpdump on my Linux machine, that's correct:

$ echo 1MOyoQIABAAAAAAAAAAAAP//AABlAAAAz8WsVm8qCgAoAAAAKAAAAEUAACgAAQAAQAZ8zX8AAAF/AAABABQAUAAAAAAAAAAAUAIgAJF8AADPxaxWXy0KABwAAAAcAAAARQAAHAABAABAEXzOfwAAAX8AAAEANQA1AAgBcs/FrFb5MAoAHAAAABwAAABFAAAcAAEAAEABfN5/AAABfwAAAQgA9/8AAAAA \
>  | base64 -d | tcpdump -n -r -
reading from file -, link-type RAW (Raw IP)
15:16:47.666223 IP 127.0.0.1.20 > 127.0.0.1.80: Flags [S], seq 0, win 8192, length 0
15:16:47.666975 IP 127.0.0.1.53 > 127.0.0.1.53: [|domain]
15:16:47.667897 IP 127.0.0.1 > 127.0.0.1: ICMP echo request, id 0, seq 0, length 8

But when, in AppVeyor, I run with subprocess ['C:\\Windows\\System32\\Npcap\\windump.exe', '-r', 'c:\\users\\appveyor\\appdata\\local\\temp\\tmp8rdgva'], I get both stdout and stderr empty. So for some reason windump is not able to read the PCAP file. Is that because of the link-type (Raw IP)? What I don't understand is why I don't even get an error message.

[p-l-]

I have tried to add an Ether() layer to change the link-layer from RAW (Raw IP) to EN10MB (Ethernet), same results. So to sum up:

• Raw IP link layer is not the cause
• Using STDIO (-r - and/or -w -) is not the cause
• The file on the disk is correct (it's a PCAP file that can be read without any problem on Linux)
• Windump does not read the file (windump.exe -r <filename> produces no output or error)

Maybe Windump is not correctly installed or needs something else? But in that case, wouldn't it produce at least an error?

@gpotter2 if you don't mind, can you, using the base64 dump, reproduce the file on your environment and check if windump.exe -r <filename> works?

[gpotter2]

I've performed some tests on AppVeyor, and windump.exe does exists in the path... I guess we're getting it output badly...

[p-l-]

@gpotter2 I have no doubt it exists in the PATH, I have a clear error when it was not installed. I don't have access to a Windows machine for now, so if you can, can you do that test for me (see my previous comment)

[gpotter2]

Sure

[gpotter2]

@p-l- File created with linux:

echo 1MOyoQIABAAAAAAAAAAAAP//AABlAAAAz8WsVm8qCgAoAAAAKAAAAEUAACgAAQAAQAZ8zX8AAAF/AAABABQAUAAAAAAAAAAAUAIgAJF8AADPxaxWXy0KABwAAAAcAAAARQAAHAABAABAEXzOfwAAAX8AAAEANQA1AAgBcs/FrFb5MAoAHAAAABwAAABFAAAcAAEAAEABfN5/AAABfwAAAQgA9/8AAAAA | base64 -d > file.pcap

and executed on windows with:

Z:\>windump -r file.pcap
reading from file file.pcap, link-type RAW (Raw IP)
15:16:47.666223 IP ZALMAN.20 > ZALMAN.80: S 0:0(0) win 8192
15:16:47.666975 IP ZALMAN.53 > ZALMAN.53: [|domain]
15:16:47.667897 IP ZALMAN > ZALMAN: ICMP echo request, id 0, seq 0, length 8

(I share files between linux and windows)

[p-l-]

@gpotter2 thanks a lot. That's weird. Can you try something else for me?

python2 -c "from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', 'file.cap'], stdout=PIPE, stderr=PIPE).communicate()"

If that also works, the problem is probably that Windump is not correctly installed in AppVeyor (something is missing for example).

[gpotter2]

Weirdly, i cannot manage to select the file... I will always have:

Z:\>python -c "import subprocess; from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', 'file.cap'], stdout=PIPE, stderr=PIPE).communicate()"
('', 'windump: file.cap: No such file or directory\r\n')

Z:\>python -c "import subprocess; from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', 'Z:\file.cap'], stdout=PIPE, stderr=PIPE).communicate()"
('', 'windump: Z:/file.cap: No such file or directory\r\n')

Z:\>python -c "import subprocess; from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', './file.cap'], stdout=PIPE, stderr=PIPE).communicate()"
('', 'windump: Z:\file.cap: No such file or directory\r\n')

[p-l-]

@gpotter2 can you try "Z:\\file.cap"?

[gpotter2]

@p-l- Updated, sorry

C:\Users\gpotter\Desktop>python -c "import subprocess; from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', 'C:\Users\gpotter\Desktop\file.cap'], stdout=PIPE, stderr=PIPE).communicate()"
('15:16:47.666223 IP ZALMAN.20 > ZALMAN.80: S 0:0(0) win 8192\r\n15:16:47.666975 IP ZALMAN.53 > ZALMAN.53: [|domain]\r\n15:16:47.667897 IP ZALMAN > ZALMAN: ICMP echo request, id 0, seq 0, length 8\r\n', 'reading from file C:\\Users\\gpotter\\Desktop\\ile.pcap, link-type RAW (Raw IP)\r\n')

[gpotter2]

@p-l- You might be right:
I was assuming that @guedou's work aroung (installing winpcap-nmap) would install winpcap, but it's a modified version so as this says:
https://www.winpcap.org/windump/install/
The requirements won't be required i guess...

Edit: Nah adding double \ doesn't work

[p-l-]

@gpotter2 I think you should double each \, shouldn't you? so probably something like python -c "import subprocess; from subprocess import Popen, PIPE; print subprocess.Popen(['windump', '-r', 'C:\\Users\\gpotter\\Desktop\\file.cap'], stdout=PIPE, stderr=PIPE).communicate()"

[p-l-]

If that's because of Windump install I might just skip Windows tests requiring Windump for now...

[gpotter2]

@p-l- Well i've found a fix:

• The chocolatey script has been updated: replace choco install -y nmap with choco install -y winpcap
• Replace the older line PATH with: - set PATH=%APPVEYOR_BUILD_FOLDER%;%PATH%
• Replace

[...] -OutFile C:\Windows\System32\Npcap\windump.exe

by

[...] -OutFile C:\Windows\System32\windump.exe

(removed \Npcap)

It works... https://ci.appveyor.com/project/gpotter2/scapy

SideNote: Please kill all of your build in travis...

[p-l-]

@gpotter2 Cool! What is it?

[gpotter2]

@p-i- see my comment above.

Edit: My bad p-i- 😢

[p-i-]

@p-l- see @gpotter2 's comment above ;)

[p-l-]

Thanks @p-i- and sorry for the notification ;)

[p-l-]

Thanks @gpotter2! AppVeyor tests are now OK (and I have canceled the other builds --- it takes a long time to start new MacOS tests on Travis-CI, queue "OSX for open source" has been full for 8 hours, and counting).

[guedou]

Thanks for the unit tests ! They are really useful.

I am glad to see that you find another way to install wpcap.dll. That's something that we might want to document.

[nohadrweesh]

I'm afraid I can't understand it well,can you provide an example of using tcdump to filter captured packets?
Is it something like--->pkts=sniff(offline='file.pcap',tcdump='tcp') as I tried it but it didn't work

[gpotter2]

@nohadrweesh Please open a new issue if really needed, you should fix your sample as:
pkts=sniff(offline='file.pcap',filter='tcp')

Also, sniff doc is available with:
help(sniff)