Reconstruct OTA images from PCAP files #29

puddly · 2023-02-22T08:48:02Z

This is a (hacky and WIP) tool to reconstruct Zigbee OTA images from raw traffic. It combines blocks in any order from multiple independent PCAPs.

There's unfortunately no simple, pure-Python way to parse PCAP files so it relies on Wireshark's tshark executable to be available in the PATH. Sample usage:

$ zigpy ota reconstruct-from-pcaps --add-network-key aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99 --output-root ./extracted/ *.pcap
Constructing image type=0x298b, version=0x00000005, manuf_code=0x115f: 157424 bytes
2023-02-22 03:39:51.406 ubuntu zigpy_cli.ota ERROR Missing 48 bytes starting at offset 0x0000ADA0: filling with 0xAB
2023-02-22 03:39:51.406 ubuntu zigpy_cli.ota ERROR Missing 48 bytes starting at offset 0x000106B0: filling with 0xAB
Constructing image type=0x298b, version=0x00000009, manuf_code=0x115f: 163136 bytes

It will generate OTA files that correspond to every detected image:

$ ls extracted
ota_t0x298b_m0x115f_v0x00000005_partial.ota
ota_t0x298b_m0x115f_v0x00000009.ota

Hardware-specific images aren't currently supported.

CC: @TheJulianJES

pipiche38 · 2023-02-22T12:11:04Z

This looks interesting, I did try some time ago ( https://github.com/pipiche38/Capture-OTA-from-Wireshark ).
Some of the obstacles are:

Sniff do not report all packets
Some of the devices keep what have been already downloaded, so if the OTA is interrupted and you restart, it uploads only the missing packets.

MattWestb · 2023-02-22T14:19:26Z

Good morning our Puddly !!

I have my Dirigera with one GU10-WS2 and one E27-CWS3 plus 2 OnOff dimmer switches (E1743).
Im have one E1743 that have getting one OpenClose user data flashed and is hocked up with SWD so i was flashing one older firmware on it and paring with Dirigera.
And it was doing one node update from 0x0.

Instruction is in the update ;-))

Edit with filtered sniff:
IKEAOTA.pcapng.gz

TheJulianJES · 2023-02-22T16:12:00Z

Linking another somewhat similar project for reference: https://github.com/compujuckel/ZigbeeOtaExtractor

MattWestb · 2023-02-22T16:31:32Z

Using this as filter i only getting the response with the payload
((wpan.dst16 == 0x90d9) && (zbee_zcl_general.ota.cmd.srv_tx.id == 0x05)) (the node is 0x90d9 so only need changing it to your node if more is doing OTA).
The question is if one better format exporting the cap file for doing the puzeling easier ?

…ltiple

codecov-commenter · 2023-02-22T17:30:14Z

Codecov Report

Base: 2.61% // Head: 2.20% // Decreases project coverage by -0.42% ⚠️

Coverage data is based on head (c3d7c76) compared to base (125b537).
Patch coverage: 0.00% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@           Coverage Diff            @@
##             dev     #29      +/-   ##
========================================
- Coverage   2.61%   2.20%   -0.42%     
========================================
  Files          8       8              
  Lines        420     498      +78     
========================================
  Hits          11      11              
- Misses       409     487      +78

Impacted Files	Coverage Δ
zigpy_cli/ota.py	`0.00% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

puddly · 2023-02-22T17:34:53Z

@MattWestb Thanks! You can leave them unfiltered, the network key will automatically be extracted by tshark and only the relevant Zigbee packets will be used. Your captures worked fine for me without modification and the extracted .ota file passed GBL validation and CRC checks:

$ $ zigpy ota reconstruct-from-pcaps --output-root ./extracted/ ~/Downloads/E1743-244005.pcapng ~/Downloads/IKEAOTA.pcapng
Constructing image type=0x11c5, version=0x24040005, manuf_code=0x117c: 205676 bytes
$ zigpy ota info ./extracted/ota_t0x11c5_m0x117c_v0x24040005.ota
./extracted/ota_t0x11c5_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl.0: 0>, manufacturer_id=4476, image_type=4549, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_onoff_controller', image_size=205676, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4549), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID

Attached: ota_t0x11c5_m0x117c_v0x24040005.ota.zip

MattWestb · 2023-02-22T17:44:48Z

Great work !!!

I flashing the old firmware on the "OpenClose" and testing one ZHA RCP updating it with your file and reporting back :-)))

Then you have it reedy for testing (= not so technical user like my can installing it) i like using it for more OTA sniffs from Dirigera.

MattWestb · 2023-02-22T20:20:48Z

2023-02-22 21:12:55.647 DEBUG (MainThread) [zigpy.zcl] [0xA004:1:0x0019] Decoded ZCL frame: Ota:upgrade_end(status=<Status.SUCCESS: 0>, manufacturer_code=4476, image_type=4549, file_version=604241925)

So the signing is not broken and the device have flashing it OK and after restart its showing Firmware: 0x24040005 so ZHA can using the Puddly OTA files :-)))

Great work done !!

And large thanks for helping getting our system working better and getting more great futures.

MattWestb · 2023-02-22T20:24:11Z

By the was @puddly was you looking in the sniff how IKEA is doing the paring in Dirigera ?
Its very interesting then using commands ZHA and Z2M is not using for getting the device not going in to sleep and also configuring all things they needing.

MattWestb · 2023-02-22T20:47:42Z

May i request implanting of metadata patching that Julian have doing with commands and its also working zigpy/zigpy#1153 (comment) without braking the signing?
Shall being great having it in zigby-CLI as standard command (perhaps having override commands for manufacture, device ID, Hardware version and software version).

MattWestb · 2023-02-23T10:00:58Z

One warning for Windows users !!

If having wireshark installed you need updating the system PATH and the instruction that is floating around is working very well but its overwriting you current system PATH with only the new one and don NOT appending it = loosing the system ground PATH.
If have done very mush things stop working OK and if you have system installed on SSD and no restore point its only possible with copy the old register settings with shadow copy or restore factory default and all installed programs must being reinstalled.

So do one register backup before doing some experiment or adding the path thru system settings GUI.

MattWestb · 2023-02-24T19:37:36Z

Was trying installing with pip and using the branch or the commit as tag but it was not working.
Last try was pip install git+https://github.com/puddly/zigpy-cli@2674f45 but was missing the OTA.py so i was adding it manually in my test system and it looks working with 2 sniffs:

0x11C6 4550	TRADFRI SHORTCUT Button (E1812)

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1812-244005.pcapng
Constructing image type=0x11c6, version=0x24040005, manuf_code=0x117c: 205180 bytes


zigpy ota info .\extracted\ota_t0x11c6_m0x117c_v0x24040005.ota
extracted\ota_t0x11c6_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl: 0>, manufacturer_id=4476, image_type=4550, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_shortcut_button', image_size=205180, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4550), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID



0x11CA 4554	SYMFONISK Sound Controller (E1744)

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1744-244005.pcapng
Constructing image type=0x11ca, version=0x24040005, manuf_code=0x117c: 214692 bytes
2023-02-24 20:19:20.334 MW-E3222 zigpy_cli.ota ERROR Missing 63 bytes starting at offset 0x00006ACE: filling with 0xAB
2023-02-24 20:19:20.334 MW-E3222 zigpy_cli.ota ERROR Missing 63 bytes starting at offset 0x00010E75: filling with 0xAB

So i need "finding" 2 lost packages for getting Sxmfonisk to working but Shortcutt button looks being OK.

Can some one hinting doing the PIP right for installing on commits or forked branches ?

puddly · 2023-02-24T20:21:10Z

@MattWestb It should work with the branch name:

pip install git+https://github.com/puddly/zigpy-cli.git@puddly/ota-pcap-extraction

You may have to uninstall and reinstall the package to get it to actually "upgrade".

MattWestb · 2023-02-24T20:42:59Z

Thanks Puddly !!
I was using the -U that shall doing the same but perhaps somthing was going wrong.

In 2 hours i shall have the next sniff for testing :-)))

MattWestb · 2023-02-24T21:48:34Z

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1744-244005A.pcapng
Constructing image type=0x11ca, version=0x24040005, manuf_code=0x117c: 214692 bytes

zigpy ota info .\extracted\ota_t0x11ca_m0x117c_v0x24040005.ota
extracted\ota_t0x11ca_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl: 0>, manufacturer_id=4476, image_type=4554, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_dimmer', image_size=214692, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4554), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID

Tomorrow testing SWD flashing older version and updating them with ZHA for validating and if going well putting them in the IKEA OTA Matrix :-)))

MattWestb · 2023-02-27T10:28:42Z

I have made sniffs of paring and updating 6 device types that is having Dirigera updates and only missed 3 packages at all from 2 devices and was SWD flashing back old firmware for redoing it.
The best is getting the device connecting to one router and sniffing in between so getting the frames sniffed 2 times (End device > parent > coordinator) you i getting 100% higher chance not loosing some frames.

I think Starkvind have getting one update but i dont have device but i shall trying flashing the firmware on one "Billy module" and see if i can getting it starting OK.

Also we have getting the firmware from Symfonisk 2 and our Danish friends have making one updated PCB for there wall switch but the device is debug locked so we cant dumping the firmware or the userdata also they was baying some new E14 lights with MG21 module and they was also debug locked but was possible unlocking and getting one chip erase. Looks working OK flashing one Styrbar dump and then the extracted Symfonisk 2 OTA file on it and its starting with new extra end points but its not 100% tested.

So also all IKEA blinds is now running 24.4.5 :-)))

Great thanks to our Puddly !!!

Hedda · 2023-02-27T13:04:24Z

FYI, posted about a another similar tool here -> zigpy/zigpy#723

compujuckel written a small program for extracting and reassembling Zigbee OTA images file from Wireshark packet captures:

https://github.com/compujuckel/ZigbeeOtaExtractor

"Zigbee OTA Extractor - Tool to extract Zigbee OTA images from packet captures"

I originally read about that in this other thread -> Koenkk/zigbee2mqtt#14926

Hedda · 2023-02-27T13:08:12Z

compujuckel written a small program for extracting and reassembling Zigbee OTA images file from Wireshark packet captures:

https://github.com/compujuckel/ZigbeeOtaExtractor

"Zigbee OTA Extractor - Tool to extract Zigbee OTA images from packet captures"

I originally read about that in this other thread -> Koenkk/zigbee2mqtt#14926

By the way, note his prerequisites:

A Zigbee sniffer is required to get the packet capture. Follow this guide to get started.
If your device needs an install code, you'll have to derive a link key using AES-MMO first. For convenience, I've written a small tool to do that here.

https://jsfiddle.net/p7yroctn/

That was mentioned for Bosch Radiator Thermostat II device which is a device that can not be joined without its install code:

Koenkk/zigbee2mqtt#14926

MattWestb · 2023-02-27T13:34:03Z

If your device needs an install code, you'll have to derive a link key using AES-MMO first. For convenience, I've written a small tool to do that here.

I think his install code program is great so can getting the TC-Link Key for decryption packages !!
Can being somthing for Puddly :-))

MattWestb · 2023-02-27T13:37:54Z

Future request

Adding check of the OTA file then its finished assembled like with the info command so not need rerunning zigpy-CLI one time after having the file made and is getting all nice printed in the CLI with one command :-))

MattWestb · 2023-03-31T18:58:04Z

Thanks P ! !

MattWestb · 2023-04-03T13:49:18Z

I think one comment in readmy that if pairing the one device in the PCAP is the network key not needed the tshark is extracting it and using it for the file(s?).

puddly added 4 commits February 22, 2023 03:36

[WIP] Reconstruct OTA files out of PCAP captures

dde7cde

Rename command to ota reconstruct-from-pcaps

d3c4c42

Log an error when identical chunks with different values

c7974c2

Reduce key lengths a little

af71c57

Rename --network-key to --add-network-key and allow specifying mu…

8cddc9a

…ltiple

Always use default network keys

2674f45

mmattel mentioned this pull request Feb 24, 2023

[New device support]: Bosch Radiator Thermostat II Koenkk/zigbee2mqtt#14926

Closed

puddly added 2 commits February 27, 2023 11:44

Print the OTA image info of every dumped image

c1d7db9

Allow adding install codes with --add-install-code

c3d7c76

TheJulianJES mentioned this pull request Mar 9, 2023

Update firmware for Bosch BTH-RA to january 2023 release Koenkk/zigbee-OTA#279

Merged

Expand README

98221bd

puddly merged commit d380e1f into zigpy:dev Mar 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconstruct OTA images from PCAP files #29

Reconstruct OTA images from PCAP files #29

puddly commented Feb 22, 2023 •

edited

Loading

pipiche38 commented Feb 22, 2023

MattWestb commented Feb 22, 2023 •

edited

Loading

TheJulianJES commented Feb 22, 2023

MattWestb commented Feb 22, 2023 •

edited

Loading

codecov-commenter commented Feb 22, 2023 •

edited

Loading

puddly commented Feb 22, 2023

MattWestb commented Feb 22, 2023

MattWestb commented Feb 22, 2023

MattWestb commented Feb 22, 2023

MattWestb commented Feb 22, 2023

MattWestb commented Feb 23, 2023 •

edited

Loading

MattWestb commented Feb 24, 2023

puddly commented Feb 24, 2023

MattWestb commented Feb 24, 2023

MattWestb commented Feb 24, 2023

MattWestb commented Feb 27, 2023

Hedda commented Feb 27, 2023 •

edited

Loading

Hedda commented Feb 27, 2023 •

edited

Loading

MattWestb commented Feb 27, 2023

MattWestb commented Feb 27, 2023

MattWestb commented Mar 31, 2023

MattWestb commented Apr 3, 2023

Reconstruct OTA images from PCAP files #29

Reconstruct OTA images from PCAP files #29

Conversation

puddly commented Feb 22, 2023 • edited Loading

pipiche38 commented Feb 22, 2023

MattWestb commented Feb 22, 2023 • edited Loading

TheJulianJES commented Feb 22, 2023

MattWestb commented Feb 22, 2023 • edited Loading

codecov-commenter commented Feb 22, 2023 • edited Loading

Codecov Report

puddly commented Feb 22, 2023

MattWestb commented Feb 22, 2023

Great work !!!

MattWestb commented Feb 22, 2023

Great work done !!

MattWestb commented Feb 22, 2023

MattWestb commented Feb 22, 2023

MattWestb commented Feb 23, 2023 • edited Loading

One warning for Windows users !!

MattWestb commented Feb 24, 2023

puddly commented Feb 24, 2023

MattWestb commented Feb 24, 2023

MattWestb commented Feb 24, 2023

MattWestb commented Feb 27, 2023

Great thanks to our Puddly !!!

Hedda commented Feb 27, 2023 • edited Loading

Hedda commented Feb 27, 2023 • edited Loading

MattWestb commented Feb 27, 2023

MattWestb commented Feb 27, 2023

Future request

MattWestb commented Mar 31, 2023

Thanks P ! !

MattWestb commented Apr 3, 2023

puddly commented Feb 22, 2023 •

edited

Loading

MattWestb commented Feb 22, 2023 •

edited

Loading

MattWestb commented Feb 22, 2023 •

edited

Loading

codecov-commenter commented Feb 22, 2023 •

edited

Loading

MattWestb commented Feb 23, 2023 •

edited

Loading

Hedda commented Feb 27, 2023 •

edited

Loading

Hedda commented Feb 27, 2023 •

edited

Loading