Z-Stack_3.x.0 coordinator 20240315 feedback

[Koenkk]

Update 2024-07-13: The 20240710 firmware is now available for testing: The new firmware, based on the latest TI SDK, is finally available for testing: #505

Since some users are having issues with the 20230507 firmware which they didn't have with 20221226, I've compiled a new test firmware. This is a continuation of #474, #478 and #483

Download: 20240315_20240316.zip

Note: P7 and R7 firmwares will come later, they are still being tested.
Update 2024-04-02: available now
Update 2024-04-16: an issue was found in the P7 and R7 firmware, this has been fixed in 20240316

When providing feedback copy and comment the template below:

- Amount of devices in network:
- Bridge software (Zigbee2MQTT/ZHA):
- 20221226 worked fine: yes/no

If possible, provide herdsman debug log or even sniffing logs from the moment everything works fine until it fails.

• See https://www.zigbee2mqtt.io/guide/usage/debug.html on how to enable the herdsman debug logging. Note that this is only logged to STDOUT and not to log files.
• Sniffing Zigbee traffic

This firmware switches back to the 6.10 SDK (which 20221226 also used) + some improvements which should reduce NWK errors.

[chris-1243]

Let's "crash" my network for a try...

- Amount of devices in network: 28
- Bridge software (Zigbee2MQTT/ZHA): Z2M
- 20221226 worked fine: yes (partially, I just had some difficulties to pair some end devices)

I am running 20230922 which is also based on this SDK without any issues. I guess it should perform as it should. I will edit in a few days.

Many thanks for your hard work !!!

Edit: After a few days, this firmware is working fine for me like 20230922.

[popy2k14]

Also flashed 20240315 and see how it goes.
In my case i also was running 20230922 without any issues.

- Amount of devices in network: 132 (end devices: 89, routers: 43)
- Bridge software (Zigbee2MQTT/ZHA): Z2M 1.36.0
- 20221226 & 20230922 worked fine: yes

[popy2k14]

Running 20240315 since release now (~3 weeks).
Here are the results:

• Amount of devices in network: 132 (end devices: 89, routers: 43)
• Bridge software (Zigbee2MQTT/ZHA): Z2M 1.36.0
• 20221226 worked fine: yes
• 20230507 worked fine: no
• 20230922 worked fine: yes, best so far
• 20240315 worked fine: sometimes devices does not respond or respond with up to ~10 seconds delayed. But no crashes. After flashing some Aqara door/window sensors where offline and needed re-pair.

[popy2k14]

I am back to 20230922, which is the most stable for me

[backlink-market]

why no repeater version?

[chris-1243]

We need first a coordinator firmware which is stable and then @Koenkk may create a router version.

It would not be a good option to have an unstable router version and it would be more work to correct the possible errors. One thing after an other.

For the Sonoff Dongle-E, the router version is based on an old v8 stack (coordinator is on v13...) and it is working like a charm... New version may add more bugs than an older.

[Koenkk]

The current router fw also uses a newer SDK (6.30) than this coordinator FW (6.10), therefore I don't see a reason to release a new router firmware (as it is stable)

[TimB87]

Coming from 20230922 to this. The former worked with caveats for me, occasionally battery driven sensors or actors would act up (e.g. temp/humidity sensors, contact sensors and heat valves) and needed an reset/repair dance that I still haven't figured out. Sometimes it takes a good while to get them running correctly again..

Had to repair the bunch as usual after restarting my Zigbee2MQTT container (that didn't used to be normal, it only started for me with all the discussion around fw problems here. Maybe thats worth a sniffer log debugging session?), but it appears that the situation seems better. Haven't tried it yet but will have to eventually.

- Amount of devices in network: currently 69
- Bridge software (Zigbee2MQTT/ZHA): Zigbee2MQTT
- 20221226 worked fine: yes

[popy2k14]

Also for me some Aqara Door/Window was offline after flashing. Re-paired and no issue since. Will see how it goes.

[TimB87]

For me, this time, two out of five Sonoff sensors already went unavailable after reparing and everything appeared working, but in Home Assistant it also didn't show updated values:

Note the blue and yellow line.

Repaired and hit reconfigure from the z2m Interface made them update the value, now I need to see if they stay connected nicely.

[TimB87]

Quick update on my part: overall, it seems a lot more stable. Almost all sensors reconnect automatically after restarting z2m, only my thermostat valves seem to require a manual reconnect for some reason, which is a great step towards resolving the issue imo :)

[SargonofAssyria]

• Amount of devices in network: 94
• Bridge software (Zigbee2MQTT/ZHA): Zigbee2MQTT 1.36.0-dev
• 20221226 worked fine: yes/no yes

Previous 20231111 firmware works fine, but will let you know how this works out.

[SargonofAssyria]

Running fine with no errors for 1 week now.
Occasionally I see messages like "Accepting joining not in blocklist dev '0xxxxxxx'" for a number of devices, but they all work good.

[1igeza]

• Amount of devices in network: 63
• Bridge software: Zigbee2MQTT
• 20221226 worked fine: yes

Previous 20230922 firmware works fine. I will let you know if I see anything wrong.

[1igeza]

For me it works stable. I have 4 aqara door sensors and all working correctly without disconnections.

[popy2k14]

After re-pairing (after upgrade/flash) they are also stable for me. No drop off`s and the network is fast and snappy.

[jonnycastaway]

• Amount of devices in network: 84
• Bridge software (Zigbee2MQTT/ZHA): Zigbee2MQTT 1.36.0-dev commit: 1006725b
• 20221226 worked fine: yes

Let's test again :-)

[jonnycastaway]

The same here. The door/window sensors lose their connection completely randomly and in confusion.
Worked before without any problems.

[mrmaximas]

have you try to activate these door/window sensors some times?

[jonnycastaway]

What do you mean with activate?
Yes i tested pressing the little switch short but that didn't bring back the sensor.
To reactivate him i had to re-pair it (without deleting the sensor in z2m) with pressing the switch long

[mrmaximas]

@jonnycastaway I had the same issue today, a couple of sensors and buttons had to be re-paired again without deleting from z2m.

[jonnycastaway]

I never delete the sensors in z2m.
I always re-pair the sensors by long buttonpress on the sensor because my network ist always in pairing mode (there are hue presence sensors that fall of the network and try a reconnect by itself) and i cut the network down to my own sensors by the passing list.

[petergebruers]

• Amount of devices in network: End devices: 29 Router: 9
• Bridge software (Zigbee2MQTT/ZHA): 1.36.0 commit: 26ba0c20
• 20221226 worked fine: yes

Uptime running firmware 20240315 ... about 12 hours and no issues to report!

[KHV8]

Installed on 17 march

• Amount of devices in network: 129, mains:57, end:70, greenpower:2
• Bridge software (Zigbee2MQTT/ZHA): Zigbee2MQTT
• 20221226 worked fine: YES
• 20230922 worked fine: Yes

After FW upgrade everything looked OK, however 3 Aqara door sensors and 2 Aqara buttons did not work/report. Based on placement, best guess, is these devices were directly connected to the coordinator. A number of other Aqara devices worked.
I have re-paired the 5 devices, Will report back if other problems show up..

ERROR: 2 of the Aqara door sensors lost connection again, app 1½ hour after I re-paired them. I do not have any errors in the log. My loglevel was "error", hence no good input. As this is Sunday evening, I have flashed back to 20230922 to ensure the rest of the family will not object. Will test again next weekend.

[KHV8]

After downgrading to 20230922 everything have been stable for 12 hours, as it was stable before.

[snippem]

Came from 22-09-2023 and that sort of worked.
Had the same issues you describe sonoff sensors occasionally stopped working. Aqara sensors that are offline by whatever reason and laggy response here and there. All devices could be revived by either readding them or remove and readding them. So what you discribe for me does not sound as related to this firmware revision. Since yesterday I flashed the new firmware and till now no immediate issues. Response times are great and no sensors from aqara of sonoff are dropped from the network of became unresponsive.( Running z2m in a LXC container on proxmox version 1.36)

[mmithril]

I am not able to start the magic using the test firmware

zigbee2mqtt_1    | Using '/app/data' as data directory
zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:06:25: Loaded state from file /app/data/state.json
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:06:25: Logging to console and directory: '/app/data/log/2024-03-17.22-06-25' filename: log.txt
zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:06:25: Removing old log directory '/app/data/log/2024-03-17.22-02-06'
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:06:25: Starting Zigbee2MQTT version 1.36.0 (commit #86ed71c)
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:06:25: Starting zigbee-herdsman (0.35.1)
zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:06:25: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/app/data/coordinator_backup.json","databaseBackupPath":"/app/data/database.db.backup","databasePath":"/app/data/database.db","network":{"channelList":[26],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":49908},"serialPort":{"path":"/dev/ttyACM0"}}'
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:06:36: Error while starting zigbee-herdsman
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:06:36: Failed to start zigbee
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:06:36: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:06:36: Exiting...
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:06:36: Error: SRSP - AF - register after 6000ms
zigbee2mqtt_1    |     at Object.start (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:63:23)
zigbee2mqtt_1    |     at /app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:312:45
zigbee2mqtt_1    |     at Queue.execute (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:35:26)
zigbee2mqtt_1    |     at Znp.request (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:300:27)
zigbee2mqtt_1    |     at ZnpAdapterManager.registerEndpoints (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:442:32)
zigbee2mqtt_1    |     at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:100:9)
zigbee2mqtt_1    |     at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:132:29)
zigbee2mqtt_1    |     at Zigbee.start (/app/lib/zigbee.ts:62:27)
zigbee2mqtt_1    |     at Controller.start (/app/lib/controller.ts:109:27)
zigbee2mqtt_1    |     at start (/app/index.js:107:5)

It seems that the flashing was OK

$  python cc2538-bsl.py -ewv -p COM4 ../CC1352P2_CC2652P_launchpad_coordinator_20240315.hex
Opening port COM4, baud 500000
Reading data from ../CC1352P2_CC2652P_launchpad_coordinator_20240315.hex
Your firmware looks like an Intel Hex file
Connecting to target...
CC1350 PG2.1 (7x7mm): 352KB Flash, 20KB SRAM, CCFG.BL_CONFIG at 0x00057FD8
Primary IEEE Address: 00:12:4B:00:29:E7:2F:D0
    Performing mass erase
Erasing all main bank flash sectors
    Erase done
Writing 360448 bytes starting at address 0x00000000
Write 104 bytes at 0x00057F988
    Write done
Verifying by comparing CRC32 calculations.
    Verified (match: 0x4f20ea4b)

[mmithril]

After some retries the error message changes to

zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:08:35: Loaded state from file /app/data/state.json
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:08:35: Logging to console and directory: '/app/data/log/2024-03-17.22-08-35' filename: log.txt
zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:08:35: Removing old log directory '/app/data/log/2024-03-17.22-04-16'
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:08:35: Starting Zigbee2MQTT version 1.36.0 (commit #86ed71c)
zigbee2mqtt_1    | Zigbee2MQTT:info  2024-03-17 22:08:35: Starting zigbee-herdsman (0.35.1)
zigbee2mqtt_1    | Zigbee2MQTT:debug 2024-03-17 22:08:36: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/app/data/coordinator_backup.json","databaseBackupPath":"/app/data/database.db.backup","databasePath":"/app/data/database.db","network":{"channelList":[26],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":49908},"serialPort":{"path":"/dev/ttyACM0"}}'
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:08:46: Error while starting zigbee-herdsman
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:08:46: Failed to start zigbee
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:08:46: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:08:46: Exiting...
zigbee2mqtt_1    | Zigbee2MQTT:error 2024-03-17 22:08:46: Error: SRSP - ZDO - extFindGroup after 6000ms
zigbee2mqtt_1    |     at Object.start (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:63:23)
zigbee2mqtt_1    |     at /app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:312:45
zigbee2mqtt_1    |     at Queue.execute (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:35:26)
zigbee2mqtt_1    |     at Znp.request (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:300:27)
zigbee2mqtt_1    |     at ZnpAdapterManager.addToGroup (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:454:39)
zigbee2mqtt_1    |     at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:103:20)
zigbee2mqtt_1    |     at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:132:29)
zigbee2mqtt_1    |     at Zigbee.start (/app/lib/zigbee.ts:62:27)
zigbee2mqtt_1    |     at Controller.start (/app/lib/controller.ts:109:27)
zigbee2mqtt_1    |     at start (/app/index.js:107:5)

[mmithril]

I can start the network and do a short smoke test using the older firmware (CC1352P2_CC2652P_launchpad_coordinator_20230923.hex).

• Amount of devices in network: 26
• Bridge software: Zigbee2MQTT
• 20221226 worked but had route table full problems
• 20230507 had big stability issues
• tried 20230923 and can at least start but failed after one day
• now trying 20231203

[mrmaximas]

• Amount of devices in network:
  Total 76
  Router: 42
  End devices: 34
• Bridge software: Zigbee2MQTT dev latest
• 20221226 worked fine: yes
  Not possible to debug now, BUT 20240315 looks as VERY GOOD because the average LQI increased significantly
  

[pastukhov]

How do you make this graph?
Looks like Grafana. Which exporter do you use?

[mrmaximas]

@pastukhov using export from mqtt to influx then grafana:

1. https://flows.nodered.org/node/node-red-contrib-zigbee2mqtt
2. filter numeric values only
   ==
   let data = {}
   let deviceId = msg.changed.topic.split('/')[1];
   let keys = Object.keys(data)
   for (const [key, value] of Object.entries(msg.changed.payload)) {
   if (typeof value === 'number'){
   data[key] = value
   }
   };
   msg = {}
   msg.measurement = "mymadhome"
   msg.organisation = "IOT"
   msg.payload = [data,
   {
   "deviceId": deviceId
   }];
   return msg;
   ==
3. https://flows.nodered.org/node/node-red-contrib-influxdb
4. grafana

[SargonofAssyria]

Isn't this about coordinator 20240315 feedback?

[deviantintegral]

• Amount of devices in network: 106
• Bridge software (Zigbee2MQTT/ZHA): Zigbee2MQTT
• 20221226 worked fine: no

Running for 24 hours with no problems so far.

[deviantintegral]

Now been running a week with no issues due to the firmware that I can tell. I did have one device go offline after cutting power, but I think that's an issue in the adapter code and not the firmware: Koenkk/zigbee-herdsman#992

[deviantintegral]

Now over a month with no adapter crashes, which was the biggest issue with previous builds. However, I have noticed some issues with end devices when routers are powered off. For example, my Ikea air purifier was moved and unplugged for a few hours and some devices went offline.

Normally, I'd ignore this, but I have a https://www.zigbee2mqtt.io/devices/TS0203.html#tuya-ts0203 that was likely routing through a Philips bulb that was turned off. The sensor also has line-of-sight visibility to the coordinator and is about 15 feet away. I was sniffing the network, and can see the local time requests being sent, but z2m processes them or responds:

[alexruffell]

• Amount of devices in network: 94 (50% routers)
• Bridge software (Zigbee2MQTT/ZHA): ZHA
• 20221226: worked fine
• 20240315: network goes down every day

After updating the firmware 3 days ago, nothing came back up (I disable ZHA, update stick on another system, reconnect the stick, and re-enable ZHA) but a reboot of the system seemed to fix it. The past couple days I noticed that some sensors were not triggering the automations as expected, and I occasionally saw routing devices marked as "unavailable" on my Home Assistant dashboards. At least 2, if not 3, times in the last 2 days I found the coordinator marked as "initializing" which means that nothing was working and a quick reboot (instead of reloading that did not seem to work) fixed the issue. I just saw it happen again but waited and it initialized on its own after a long delay. I am guessing this means the stick is having occasional serial communication issues with the system, or maybe this is part of some mechanism to get it back up and running after something happens. My HA is installed in a VM on Proxmox with the USB 3.1 port passed through to HA - no, I do not have any USB 2.0 I can use, and yes I use an extension cable).

I don't have detailed ZHA logs but found many errors in the system logs. This one is zigbee related and appears a lot:

2024-03-18 17:19:16.594 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1100, in request_callback_rsp
return await callback_rsp
^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1097, in request_callback_rsp
async with async_timeout.timeout(timeout):
File "/usr/local/lib/python3.12/site-packages/async_timeout/__init__.py", line 141, in __aexit__
self._do_exit(exc_type)
File "/usr/local/lib/python3.12/site-packages/async_timeout/__init__.py", line 228, in _do_exit
raise asyncio.TimeoutError
TimeoutError

Another zigbee related error I see a lot:

2024-03-18 18:03:44.811 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1098, in request_callback_rsp
await self.request(request, timeout=timeout, **response_params)
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1033, in request
async with ctx:
File "/usr/local/lib/python3.12/site-packages/zigpy/datastructures.py", line 22, in __aenter__
await self.on_enter()
File "/usr/local/lib/python3.12/site-packages/zigpy/datastructures.py", line 111, in acquire
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1097, in request_callback_rsp
async with async_timeout.timeout(timeout):
File "/usr/local/lib/python3.12/site-packages/async_timeout/__init__.py", line 141, in __aexit__
self._do_exit(exc_type)
File "/usr/local/lib/python3.12/site-packages/async_timeout/__init__.py", line 228, in _do_exit
raise asyncio.TimeoutError
TimeoutError

Another one:

2024-03-18 18:03:53.739 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1098, in request_callback_rsp
await self.request(request, timeout=timeout, **response_params)
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1052, in request
self._uart.send(frame)
^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'send'

2024-03-18 18:03:53.743 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/sensor.py", line 263, in _refresh
await self.async_update()
File "/usr/src/homeassistant/homeassistant/components/zha/entity.py", line 256, in async_update
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/homeautomation.py", line 127, in async_update
result = await self.get_attributes(attrs, from_cache=False, only_cache=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 513, in _get_attributes
read, _ = await self.cluster.read_attributes(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/zcl/__init__.py", line 528, in read_attributes
result = await self.read_attributes_raw(to_read, manufacturer=manufacturer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/zcl/__init__.py", line 377, in request
return await self._endpoint.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/endpoint.py", line 253, in request
return await self.device.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/device.py", line 339, in request
await send_request()
File "/usr/local/lib/python3.12/site-packages/zigpy/application.py", line 841, in request
await self.send_packet(
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/zigbee/application.py", line 959, in send_packet
await self._send_request_raw(
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/zigbee/application.py", line 850, in _send_request_raw
response = await asyncio.shield(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1098, in request_callback_rsp
await self.request(request, timeout=timeout, **response_params)
File "/usr/local/lib/python3.12/site-packages/zigpy_znp/api.py", line 1052, in request
self._uart.send(frame)
^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'send'

All these errors appear in large numbers. Other than this, when I look at the list of devices, they always seem to be marked online so it seems like the system thinks they are online while there may be some communication / message delivery / etc issue going on.

Edit: March 20 - Morning after the comment above - my zigbee network is down even though in a table that shows online status only one router is supposedly down. Reinitializing the coordinator seemed to fix the problem at least partially (I have not checked everything as family needs to get ready). I will need to downgrade later today.

[alexruffell]

@Koenkk I have tried all versions and have had no luck with any, including 20230922. Some had more dramatic issues than others but I would typically end up with a non functional network. Other than my comments in previous posts, I did not keep track of specific issues per version tested. However, I recall frequent uart and/or timeout issues (probably related; same thing). Problematic firmwares also noticeably increased the occurrence of devices that randomly stopped working.

[popy2k14]

@alexruffell can you look/post at the kernel (dmesg) logs at the time of the issues? Maybe they are related to your Proxmox/VM serial Port passthrough?

[alexruffell]

@popy2k14 Ok, I will although right now I am back on the version that seems to work the best. Biggest issue is that I always discover it when I really need it to work (kids bedtime or wake-up routines... nope they aren't zigbee... I have yet to figure out their protocol). Anyhow, I did change machines (Lenovo TinyPC m920q w/ i7 to a Lenovo Tiny P360 w/ i7) and the problem followed. Also, I pass through a zwave stick and it doesn't seem to have issues. While not 100% conclusive, it does look like it is something specific to zigbee, no?

[popy2k14]

Yeah the protocol of kids are mostly not discovered yet :-)
Ok, when you changed hardware and the issue followed, it seems it's not the hardware.
Is it possible to try without any container (plain HA install)?

Sorry i have not so much experience with ZHA, i am using Z2M addon in HA.

[alexruffell]

Update...

I don't know whether this is very meaningful or not but looked like it was. Under the zigbee coordinator in ZHA I saw a hidden entity that exposed zigbee_coordinator_0 which was a large always increasing number that would revert to zero very often. Every time it reverted to zero, a table I have where I can see all the online and LQI status of my ~90 zigbee devices would show all LQIs at N/A while the table was repopulating and devices were slowly coming back online. While I am not sure, I understood this to be a UART communication breakdown or a stick crash of some sort.

One thing I noticed is that when power outages happen, some devices (looking at you Jasco) appear to go in a weird state (if they don't simply die due to the bad cap issue) making the network very unreliable and apparently causing the stick to crash repeatedly (I always have the issue, just gets worse with power outages) until I power cycle the entire home at the breaker panel. Is there some automated reset feature that starts over if problems are detected? Maybe this is more of a ZHA question but figured I'd ask.

To eliminate the possibility of a back zigbee coordinator, I migrated my mesh network to an identical twin I had but the issues did not change at all. Fed up, I purchased the SiLabs E version of the Sonoff dongle and migrated to that. Everything works like it should! The network is so much faster and 2 days in it does not seem to have any crashing issues. This latter test, assuming things continue to work, should eliminate Proxmox VM, USB, HA, ZHA, stick hw issues, etc. as potential culprits. This leaves me wondering why none of the firmware releases since 2022 appeared to work in my home setup. There must be some combination between the firmware and something specific to my setup causing the issue... but I really did not want to deal with it anymore.

I've read of many issues people using the SiLabs version have, so I may have just traded some issues with others but for now it seems to work better than it has in years. I also found a ton of parameters such as PHY_CCA_FAIL_COUNT which are exposed in ZHA that can help me find issues. This latter one should show when the stick fails to communicate due to interference. That would be so nice to have on the TI firmware too.

[rzhivotov]

Please build any new firmware for P7.

• Amount of devices in network: 44
• Bridge software: Zigbee2MQTT 1.36
• 20230507 worked fine: no

[theimo1221]

• Amount of devices in network: 219
• Bridge software: Zigbee2MQTT 1.36.0
• 20221226: worked fine
• 20230507 worked fine: no
• 20230922 worked fine: semi (regular timeouts and device responses aren't received, still they perform the commands)

Right after the update I noticed a Link-Quality improvement and much less error in the log 👍

Edit: 12 hours in --> System still running smoothly and average link-quality improved by roughly 25%:

[mrmaximas]

@theimo1221 how do you calculate the “average temperature in the hospital”, I meant the average lqi? I collect this data in to influxdb then show in grafana, but I still don’t understand how to create an averaged graph of all my 70+ devices, with the condition that the data arrives at different times and with different frequencies.

[theimo1221]

@mrmaximas Grouping at a 15-min Interval:

SELECT
  date_bin('15 min', date, '2000-1-1') AS min15_slot,
  AVG("linkQuality")
FROM
  hoffmation_schema."ZigbeeDeviceData"
WHERE
  $__timeFilter(date)
GROUP BY min15_slot
order By min15_slot asc

For Influx you might want to give time(15m) a try as a group-by clause

Those 15minutes equal my device-persistance interval within hoffmation:
https://github.com/Hoffmation/Hoffmation-Base/blob/3beda9c5b2db899286b855d1166330f0c458fbd9/src/server/devices/zigbee/BaseDevices/zigbeeDevice.ts#L18-L25

[mrmaximas]

@theimo1221 influxQL is a different.. I'm gonna go study it.

[theimo1221]

For Influx you might have to use Grafanas aggregation functions, as like you said, you only have data-points for updates.
In my case as I'm persisting the data controlled in a static time frame, I have a single datapoint in each 15min interval for everry single device.

[mogim0gi]

would be a lovely addon for home assistant to be able to collect this data (sorry not motivated enough to bang my head about grafana and influxQL)

[Tonio16]

• Amount of devices in network: 71
• Bridge software (Zigbee2MQTT/ZHA): zigbee2mqtt/jeesom/zigbeelinker
• 20221226 worked fine: yes

The new firmware, 20240315 works fine except with my aqara door sensors directly connected to my sonoff P dongle, i.e. they are not seen on the network anymore after they fall asleep for the first time. The ones connected to a router are fine.

Antoine

[popy2k14]

Did they drop off the network when the issue occurs?
I also have a bunch of those sensors and had also issues with those (dropping off the network).
All other devices was running fine with 20240315.

[Tonio16]

Hello

I am not sure to understand the meaning of drop here.
What I can say. I can paired then to my network but after few hours, they are shown as n/a.

Antoine

[dankocrnkovic]

I have 6 of them. Two of them I had to delete from Z2MQTT, removed battery, restart Z2MQTT addon, and then rejoin them. After that they are stable for 2 weeks now.

[Tonio16]

Hello
I just switch back to 20221226 and interessingly the aqara door detector that where not visible on after some hours/days if directly bind to the coordinator, reappeared. No need to pair them again. Normal?

Antoine

[zen2]

Notes about Aqara end devices:

• Aqara devices do not follow the Zigbee specification and use their own implementation of the protocol at the application layer. In other words, they are very picky with which devices they can talk to.
• Aqara devices don’t work well with: Legrand/Netatmo, Centralite, Ledvance/Osram, Lightify/Sylvania, Securifi, Orvibo, Iris, General Electric or SmartThings/Samsung routers
• Aqara devices work well with: Phillips Hue, Ikea Tradfri (newer), Tuya Zigbee 3.0 routers

So If an Aqara end device don't find its usual router for whatever reason it is possible that it drops from the network.
I use a lot of Aqara end devices and it appears that some drops from the network when:

• I switch off house main energy supply for electric maintenance
• I remove a router device
• I change a router device location

So I suspect that Aqara issues related with new firmwares test are related with routes establishment or mesh network stabilization.

[alexruffell]

@zen2 Is this still accurate with the new generation that can be identified via the P1, P2, T1, etc suffix. I had read that these new versions, announced years ago but still hard to find for some products, are now fully Zigbee 3.0 compliant. For that reason I never purchased any Aqara products and only recently when I found a few sensors of the new type online, I got them for testing.

[zen2]

You're right. I was speaking about aqara end devices previous to Zigbee 3.0 ones.
I think many of us are still using these old aqara devices because they are easy to find and are cheap.

BTW, is there a way to check which zigbee version a device is using ?

[RobLeighton67]

Updated 2022126 stick that had started misbehaving when running for a few week or two with 19 devices - inc Xiaomi,Freint, Sonoff, Ikea (all devices would disappear until I reloaded z2m or power cycled the host).
Running 20240315 for over a month - no issues.

[LeoeLeoeL]

If somebody is testing a firmware and has updated Z2M to 1.38, pay attention because problems could be due to the last release.
I'm experiencing all devices offline after 24 hours. Until 1.37.1 was ok.

[chris-1243]

If somebody is testing a firmware and has updated Z2M to 1.38, pay attention because problems could be due to the last release.

I'm experiencing all devices offline after 24 hours. Until 1.37.1 was ok.

I am running the latest version of Z2M with firmware 20240315 on my Sonoff P and everything is stable since the past 8 days

[SargonofAssyria]

I am running 1.38 with 20240315, and ALL is running fine.

[LeoeLeoeL]

Reverted to 1.37.1.
After 36 hours, same behaviour of 1.38. Everything OFFLINE.

[popy2k14]

No issues with 1.38 with 20230922.

On which system (hardware) and versions you are running Z2M?
Do you have kernel log when all is offline.

[LeoeLeoeL]

Sonoff Dongle P under Proxmox.reverted again to 1.38 but 44 devices remein OFFLINE.
I think the coordinator needs a powercycle to delete something.
No logs. Today I was in hurry to fix everything.

[NH-Networks]

Running Z-Stack_3.x.0 coordinator 20240315 on ZHA with Zigstar v4 since 2 days.. for now all stable and working.

[rccoleman]

I'm been running 20240315 on my Tubesz CC2652-P7 for a week or so with 1.38.0-dev commit: dbcae00 and I no longer see all the errors that I used to see when using 20230507. My screen used to be flooded with timeouts and errors whenever I would switch to the Z2M dashboard tab in my browser and now there's nothing. I have 126 Zigbee devices in my mesh.

[NH-Networks]

@Koenkk can we see this version as "the way to go".. it should be available as recommend if you ask me.

[GD-Dal]

With this firmware, I'm losing zigbee devices. And some devices are not able to pair again not matter how many interviews or factory resets I do.
Will try to revert back to the previous one

[stemer77]

Ok so after a week of using this fw I have only comparison to 20230507 where whole Zigbee network could fail in 2 days (devices were out of network and didn't go back). On fw 20230507 I had availability turned on and OTA updates checking on. On 20240316 I have availability
turned on and OTA off. I only experience a few devices timeout but when I turn on ZB group of lights, when one is off, every light turns on, but when I try to turn on the specific timed out light, it doesn't turn on. The lights usually come back to network after 1-2 minutes. I'm using UZG-01 over eth.

• Amount of devices in network: 127 (routers: 65, end devices: 62)
• Bridge software (Zigbee2MQTT/ZHA): 1.38.0
• 20221226 worked fine: didn't installed it

[Koenkk]

Some update from my side; this FW is based on an old 6.10 SDK, we couldn't update because there was an issue starting from 6.20 that led to crashes. This issue has now been fixed, so I'm evaluating newer SDKs at the moment.

[KHV8]

Cyablo, do not think this can be firmware related, with this number of errors. Look for wifi interference, use long USB extender cable, 1½m-2m. Look at the "last reported" updates. If some devices are reporting a lot, like several times a minute, then remove them.

[cyablo]

Trust me, I already tried all that and more. The zigbee dongle is connected to a USB hub to get it away from the raspberry z2m is running on and on a non overlapping channel with my wifi network, plus I'm not living in a crowded location, so there is zero wifi interference here. Also it almost exclusively affects aqara battery devices + sonoff/girier relays (no matter what location they are in). All other devies react totally fine (I have over 110 zigbee devices in my network.). The non reacting ikea bulb may be some collateral damage, but I've never had only one mains powered device drop off the network with older firmware.

[KHV8]

Have you checked that you have all routers in the coordinator list., with the coordinator checke. #474, look in first post.
I have a lot of Aqara battery devices, and they are stable. Only problem is they do not always roam to new routers if one is powered off (bulbs) or I remove a router from the network.

[cyablo]

Already tried that: MQTT publish: topic 'zigbee2mqtt/bridge/response/coordinator_check', payload '{"data":{"missing_routers":[]},"status":"ok"}'. I have zero switchable or removable zigbee devices (routers) that could be powered off by accident, all hard wired and ONLY switched by homematic or zigbee switches over node red or home assistant.

[KHV8]

OK, you have been fare in you search:-) 2 Observations, probably not relevant however could help with ideas.

One thing I have seen once is bulbs not behaving well. It was some EWOX bulbs. They were updating at least every 2 seconds, and with a lot of updates they made my zigbee network unstable, especially hitting battery devices. It was easy to determine, as they in the Z2M overview always was on the top if sorted by last updated. The company claimed they would not do this, if I used their coordinator, hence I solved the problem by re-locating them to the trash can.

Besides this I have one more observation, however this is not proven or validated in any way. I have quite a lot of Aqara plugs in the setup. Back in the days where I were running ZHA on a Conbee the Aqara battery devices was unstable. By try/error I concluded that if the Aqara battery devices were paired to a Aqara plug, then they would be more stable. I still do this, even I have no idea if it makes any difference. We are all people of habits :-)

Hope you will be able to get a stable network.

[Sn00kiT]

Hey there, just encountered a strange behaviour after updating. reverting back to the old stable does not help and flashing it with another browser/pc does also not help:

[2024-06-23 14:25:27] info: z2m: Logging to console, file (filename: log.log)
[2024-06-23 14:25:27] info: z2m: Starting Zigbee2MQTT version 1.38.0 (commit #unknown)
[2024-06-23 14:25:27] info: z2m: Starting zigbee-herdsman (0.49.2)
[2024-06-23 14:25:27] info: zh:zstack:znp: Opening TCP socket with 192.168.178.11:6638
[2024-06-23 14:25:27] info: zh:zstack:znp: Socket connected
[2024-06-23 14:25:27] info: zh:zstack:znp: Socket ready
[2024-06-23 14:25:27] info: zh:zstack:znp: Writing CC2530/CC2531 skip bootloader payload
[2024-06-23 14:25:28] info: zh:zstack:znp: Skip bootloader for CC2652/CC1352
[2024-06-23 14:25:46] error: z2m: Error while starting zigbee-herdsman
[2024-06-23 14:25:46] error: z2m: Failed to start zigbee
[2024-06-23 14:25:46] error: z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
[2024-06-23 14:25:46] error: z2m: Exiting...
[2024-06-23 14:25:46] error: z2m: Error: target adapter aps link key data table size insufficient (size=3)
at AdapterBackup.restoreBackup (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/adapter-backup.ts:330:31)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at ZnpAdapterManager.beginRestore (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:293:9)
at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:74:17)
at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:127:29)
at Zigbee.start (/app/lib/zigbee.ts:62:27)
at Controller.start (/app/lib/controller.ts:109:27)
at start (/app/index.js:107:5

Googleling for the Error: target adapter aps link key data table size insufficient (size=3) does only lead to dead ends.
I tried to use another adapater also, same failure. UZG and also SLZB07

[sjorge]

It comes with a full blown webui you can use, it is certainly very convenient:

It will briefly disconnect z2m over the web socket though when it's updating. So I just stop z2m, install new fw, start z2m.

Edit: Oops off by 1 number, you have a 07p7, which is the USB only version.

[snippem]

It comes with a full blown webui you can use, it is certainly very convenient:

It will briefly disconnect z2m over the web socket though when it's updating. So I just stop z2m, install new fw, start z2m.

Edit: Oops off by 1 number, you have a 07p7, which is the USB only version.

I have the usb one SLZB-07p7

[agriadsi]

Do you have a idea what i am doing wrong ?

Did you read and followed all steps in cc2538-bsl documentation? Especially:
https://github.com/JelmerT/cc2538-bsl#requirements
https://github.com/JelmerT/cc2538-bsl#dependencies

[rccoleman]

It comes with a full blown webui you can use, it is certainly very convenient:

This would be a really nice feature addition for the Tubeszb adapters. The web UI for them essentially provides the command to run from another machine to update the firmware, but that's non-trivial for "normal" people and it will fail if you don't manually stop Z2M.

[Sn00kiT]

The latest 20240710 Firmware fixed this issue

[Koenkk]

Update from my side, I'm now evaluating the 7.41 SDK since TI has fixed the issue which prevented this SDK from working on a device with PA (CC2652P and CC1352P). I need to do some tests on my own prod network after which I will release the FW for testing.

[samuele2723]

Sorry whats the right one for sonoff zongle P ?

CC2652R
CC2652R7
CC2652RB

thanks

[arteck]

Sorry whats the right one for sonoff zongle P ?

CC2652R CC2652R7 CC2652RB

thanks

https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_3.x.0/bin

[Koenkk]

The new firmware, based on the latest TI SDK, is finally available for testing: #505

[mogim0gi]

@Koenkk I have updated my coordinator to the newest version and its working fine, however I am now facing a problem. my Sonoff dongle P, flashed as router doesnt complete pairing:

I get a loop of error :

zh:controller: Interview failed for '0x00124b0029de0f87 with error 'Error: Interview failed because can not get active endpoints ('0x00124b0029de0f87')'

I have reflashed the router firmware to this (from the same firmware) : CC1352P2_CC2652P_launchpad_router_20221102.zip , it has been working for me before, could you please take a look into this ?

it appears online, but pairing never finishes, tried flashing a couple of times, no change.

[harvindhillon]

This issue is still there in the latest version firmware and Z2M, not sure which is the issue. These routers drop as well even after they are paired after a couple of days. Only way to recover then is to restart the Z2M container so it might be a Z2M issue instead of firmware.

[mogim0gi]

i reflashed my coordinator to 20221227 and it paired again, so its not Z2M issue.

[harvindhillon]

Yea it can pair fine even on the latest version if you restart the container and your coordinator. Usually pairs fairly quickly. However once it starts dropping and you are trying to pair, it doesnt until your restart everything.

I had the issue on 20221227 as well