Entire System Fails (after an unknown amount of time) (started 2 versions ago, I think).

[RJ-Make]

Checklist

• I have read and followed the above instructions

• I have checked the troubleshooting section and my problem is not described there.

• I have read the changelog and my problem was not mentioned there or the fix did not work.

Describe the issue

What is happening?
after an unknown amount of time the entire z-wave system fails to function.

What did you expect to happen instead?
System to remain functional.

This started, I believe 2 version back. The problem is I also updated HAOS, and CORE, so I'm unclear on the root cause.

EDIT ** Restarting the ADD-ON restores function.

Software versions

zwave-js-ui: 9.3.1
zwave-js: 12.2.3

Core 2023.11.1
Supervisor 2023.10.1
Operating System 11.1
Frontend 20231030.1

Device information

Manufacturer: Zooz
Model name: ... 700
Node ID: ... 001
Firmware 7.18.3

Checklist

• I made sure to provide a driver log on level debug.

• The log includes a re-interview of the problematic device (if applicable).

• The log includes the problematic interaction with the device (if applicable).

• I provided the node ID of the problematic device (if applicable).

Upload Logfile

zwave-js-ui-store.zip

[AlCalzone]

I think this is the same issue that was fixed by #6473 in the latest 12.3.0 release.
HA should update soon, there's already an open PR for the update.

[RJ-Make]

12.3 Did not resolve issue. Logs attached.

zwave-js-ui: 9.3.2
zwave-js: 12.3.0
home id: 3609363144
home hex: 0xd72282c8

zwave-js-ui-store(1).zip

[AlCalzone]

Can you roughly narrow it down to a timeframe where the issue occured?

[RJ-Make]

Unfortunately I can't. It's never the same span of working time.

That said, the attached logs should contain another failure point for which I had several overnight automation that didn't succeed. If I where to guess, this failure point time frame should be between 9pm and 4am.

zwave-js-ui-store(2).zip

EDIT: The strange thing with these network failures is there are no errors being raised by the failed automations. It's like HA doesn't even know the network is down.

[AlCalzone]

I had a hunch that the fix I included in 12.3.0 didn't take in some situations, so I've tried to reproduce the issue from what I see in your logs.
Unfortunately I wasn't able to - in my tests these situations were handled as expected. Not sure if I missed the actual issue in the huge amount of incoming reporting. When this happens again, can you please execute the following driver function in Z-Wave JS UI and immediately afterwards share the current log again?

driver.driverLog.print("Normal queue:");
driver.driverLog.sendQueue(driver.queue);
driver.driverLog.print("Immediate queue:");
driver.driverLog.sendQueue(driver.immediateQueue);

Aside from that, this is caused by your controller becoming unresponsive sometimes where it does not complete a command cycle. Z-Wave JS UI tries to recover, but that doesn't always seem to work.

Plus you've got a lot of incoming reports from sensors - this might have an effect on this happening as well. Node 121 alone sends 7 every 30 seconds. Node 74 often sends identical temperature readings after 60 seconds, etc... This kind of traffic can be detrimental for a Z-Wave network.
I suggest giving https://zwave-js.github.io/node-zwave-js/#/troubleshooting/network-health?id=optimizing-the-reporting-configuration a good read and applying it where you can.

[RJ-Make]

Ok, Will do.

Thanks

[RJ-Make]

I noticed this in the add-on logs.. Attached more logs... These logs are right after an Add-on restart. Right before that a switch was not working (I didn't test other devices, but generally these means the entire network is down).

2023-11-12 06:25:37.658 INFO APP: GET /assets/index-34ce4c5e.js 304 0.849 ms - -
2023-11-12 06:25:37.659 INFO APP: GET /registerSW.js 200 2.594 ms - 136
2023-11-12 06:25:37.665 INFO APP: GET /assets/index-d19296fd.css 200 4.696 ms - 649653
2023-11-12 06:25:37.760 INFO APP: GET /assets/Login-1826b0a1.css 200 2.269 ms - 5433
2023-11-12 06:25:37.761 INFO APP: GET /assets/Login-fed50131.js 200 3.684 ms - 4135
ValidationError: The 'X-Forwarded-For' header is set but the Express 'trust proxy' setting is false (default). This could indicate a misconfiguration which would prevent express-rate-limit from accurately identifying users. See https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/ for more information.
    at Object.xForwardedForHeader (/opt/node_modules/express-rate-limit/dist/index.cjs:166:13)
    at Object.wrappedValidations.<computed> [as xForwardedForHeader] (/opt/node_modules/express-rate-limit/dist/index.cjs:324:22)
    at Object.keyGenerator (/opt/node_modules/express-rate-limit/dist/index.cjs:579:20)
    at /opt/node_modules/express-rate-limit/dist/index.cjs:629:32
    at /opt/node_modules/express-rate-limit/dist/index.cjs:611:5 {
  code: 'ERR_ERL_UNEXPECTED_X_FORWARDED_FOR',
  help: 'https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/'
}

zwave-js-ui-store(3).zip

[AlCalzone]

This is unrelated

[RJ-Make]

driver.driverLog.print("Normal queue:");
driver.driverLog.sendQueue(driver.queue);
driver.driverLog.print("Immediate queue:");
driver.driverLog.sendQueue(driver.immediateQueue);

I came home today to find the network down again. Hopefully I've done this correctly..

zwave-js-ui-store(4).zip

[AlCalzone]

Ok that helps. Still need to figure out what happened exactly, but there's a command to a sleeping node blocking the send queue:

2023-11-17T22:57:24.774Z DRIVER   Send queue:                                                     (398 messages)
                                  · [REQ] SendDataBridge (Security2CCNonceReport) [Node 128, Asleep]
                                  · [REQ] GetBackgroundRSSI
                                  · [REQ] GetBackgroundRSSI
                                  · [REQ] GetBackgroundRSSI
                                  · [REQ] GetBackgroundRSSI
                                  · [REQ] GetBackgroundRSSI
                                  · [REQ] GetBackgroundRSSI
...
                                  · [REQ] SendDataBridge (Security2CCMessageEncapsulation) [Node 3, Alive]
...

[AlCalzone]

Side note: What kind of device is Node 128? It seems that every time it sends something, it needs to be re-synchronized.

[AlCalzone]

Okay, I think I know what's going on:

• Z-Wave JS sends an S2 nonce to Node 128
• The controller fails to call back, triggering Z-Wave JS's recovery
• This puts the S2 nonce back on the queue, but the wrong one, where it blocks due to the node being asleep

[AlCalzone]

This is now tracked in #6506

[AlCalzone]

and fixed in 12.3.1

[RJ-Make]

Side note: What kind of device is Node 128? It seems that every time it sends something, it needs to be re-synchronized.

It is a Zooz ZSE44 Temperature and Humidity Sensor. I have 4 of them I believe.

[AlCalzone]

Ok I remember reporting this to Zooz a while back - not sure if they fixed it in the newer firmwares.