Lost packets

[L0laapk3]

Hello

I've tried using this project to sniff targeted packets coming to a target mac address by setting the ESP's mac address to the targets mac. This works perfectly fine and I can already do a lot of relevant data analysis.

However, I live in a very noisy area, and I see about 90%-99% of relevant packets are not getting callbacked on. This in combination with that you need multiple successive packets to start making sense of the data makes for a very low success rate.
I am receiving about 30-200 beacon frames per second. I set my code up so these get filtered out in the first line of the callback function, but its not enough. I'm also running the cpu on 160MHz just in case that would make any difference.

I was wondering if you had come across any switch or any kind of direction I could go look to have the hardware (or software in a very early stage) filter these out.

Have you encountered this problem aswell where you are not receiving even close to all the packets? Or is that something that is only happening to me because of my noise.

Any suggestions for where to start with trying to fix this is appreciated, I am a bit hopeless.

[ernacktob]

To be honest I have not been able to experiment a lot, and since about 3 years I have not owned an ESP so I can't conduct tests anymore (when I was working on this project I was using a borrowed one).

First, I think ESP's SDK provided an official sniffer API, if I recall correctly. The drawback was that it would truncate the packets to the first 112 bytes or something. Have you tried using this API and check if you are able to get more relevant packets in this manner (albeit possibly truncated)? If that is the case, then it could be a problem in the mechanism I use to circumvent the API (which I unfortunately cannot guarantee to work because I just found it by some reverse-engineering and experimentation). Otherwise, it could be a more fundamental issue with the ESP itself.

If you get more relevant packets using the official sniffer API but you are still bogged down by truncation issues, you could also try to speed up the callback that processes the packets. If you are receiving too many noisy packets, the callback might perhaps get overloaded. I don't know what exactly you do in the callback, but if you haven't done so, you could implement filtering to quickly ignore packets that don't have the type you want, such as beacon frames, before doing any further processing. Perhaps you could also implement a queue system and do the processing in a separate task.

I don't know if there is a setting that would allow blocking all packets that are not directed specifically to the ESP's mac address (i.e. drop broadcast packets).

[ernacktob]

I found this API reference from Espressif (not sure if that matches the SDK version you are using): http://espressif.com/sites/default/files/documentation/2c-esp8266_non_os_sdk_api_reference_en.pdf

In the Sniffer Related APIs section (3.11) they define the wifi_promiscuous_set_mac function which supposedly allows setting your MAC address to filter for sniffing. You could also try using that.

I remember that I did not succeed to have my wifi_raw callback called if I use the official promiscuous mode. I was not able to figure out much of what the promiscuous API really does, but perhaps someone else has or is willing to do reverse-engineering on these functions. I believe it uses a different data path from the normal one in non-promiscuous mode.

If you're interested to go down that rabbit hole, I have some (incomplete) reverse-engineered code for symbols in the libpp library (in the esp8266_wifi_raw/disas/ and /disas/pp/ folders). You can have a look and use that as a starting ground...

[L0laapk3]

Hello

Thank you so much for your elaborate response.
In the API, it is stated that wifi_promiscuous_set_mac will include broadcast packets.
I cannot use the promiscuous mode as I need data from the packets past the truncation. I think its reasonably fair to assume that if the hardware was capable of getting the complete packet, the API would be able to do it so I won't explore that path any further.

As for the packet noise, I will have to experiment with it in a less noisy area before I can jump to any conclusions.

For data analysis, I have been able to capture packets in both directions from and to the target device by turning on station+softAP mode and setting station mac to target mac and softAP mac to router mac. The amount of packet loss is the same as station only mode.
However, turning on softAP with same mac address as router, makes some routers spam deauth packets. The deauth packets could also possibly be coming from the esp. I would have to effectively handicap the softAP to prevent it from actually sending any packets. I have not experimented with anything yet to prevent this. I will have to experiment with it myself first before I ask for help, I was planning on waiting to answer until I had done that but I felt a bit rude not responding.

Ps. If you wish, you can add me on discord :) L0laapk3#2010