Unstable network, seems to have routing issues

[tekman54190]

Hi all,

I need help regarding really strange issue on my Zwave network... I have multiples issues:

• Network unstability : since few months, I've some strange behaviour on the network. Sometimes I have latency, sometime drop of commands / reports.
• Strange messages in the logs: after analysis it seems that I have 1 node discovered as neighbour of some nodes but this node ID is not recognised by the controller ! I've also some discard message from a node ID not present on the network...

I performed some analysis since then but i'm reaching my knowledge limits of Zwave and node-zwave-js...

Let's start with context, I've 75 nodes in my network, almost all of them are live connected (fibaro Walli, fibaro dimmers, and some LED RGBW from fibaro) and 11 are powered by battery.

In the logs I can see this list of known nodes on the controller side:
2023-03-28T19:48:45.927Z CNTRLR received additional controller information: Z-Wave API version: 8 (legacy) Z-Wave chip type: ZW050x node type Controller controller role: primary controller is the SIS: true controller supports timers: false nodes in the network: 1, 4, 5, 6, 8, 22, 23, 28, 48, 50, 51, 54, 55, 5 9, 82, 90, 92, 93, 95, 96, 99, 100, 101, 102, 103, 108, 109, 110, 111, 112, 11 3, 114, 115, 116, 118, 120, 121, 122, 123, 124, 126, 129, 132, 133, 134, 135, 136, 137, 138, 141, 142, 146, 148, 152, 153, 155, 157, 163, 164, 166, 167, 170 , 171, 172, 173, 174, 175, 179, 180, 181, 185, 186, 187, 193, 196

As you can see I don't have a node ID 49 :) but this one is discovered by some of the nodes:
2023-03-27T20:11:04.215Z CNTRLR « [Node 022] node neighbors received: 8, 23, 28, 48, 49, 50, 51, 54, 59, 82, 92, 2023-03-27T20:13:10.482Z CNTRLR « [Node 023] node neighbors received: 4, 6, 8, 22, 28, 48, 49, 50, 51, 54, 55, 5 2023-03-27T20:15:15.610Z CNTRLR « [Node 028] node neighbors received: 5, 6, 8, 22, 23, 48, 49, 50, 51, 54, 55, 5 2023-03-27T20:16:47.173Z CNTRLR « [Node 048] node neighbors received: 8, 22, 23, 28, 29, 49, 50, 51, 54, 55, 59, 2023-03-27T20:18:28.850Z CNTRLR « [Node 050] node neighbors received: 4, 8, 22, 23, 28, 48, 49, 51, 54, 55, 59, 2023-03-27T20:20:13.118Z CNTRLR « [Node 059] node neighbors received: 4, 5, 6, 8, 22, 23, 28, 48, 49, 50, 51, 54 2023-03-27T20:23:40.966Z CNTRLR « [Node 099] node neighbors received: 6, 23, 28, 48, 49, 50, 51, 54, 55, 59, 82, 2023-03-27T20:25:24.083Z CNTRLR « [Node 101] node neighbors received: 6, 8, 22, 23, 27, 28, 48, 49, 50, 54, 59, 2023-03-27T20:28:51.542Z CNTRLR « [Node 103] node neighbors received: 4, 8, 22, 28, 48, 49, 50, 51, 55, 96, 118, 2023-03-27T20:43:16.542Z CNTRLR « [Node 152] node neighbors received: 4, 5, 6, 8, 22, 23, 28, 48, 49, 51, 54, 55 2023-03-27T20:46:30.477Z CNTRLR « [Node 170] node neighbors received: 4, 5, 6, 8, 22, 23, 27, 28, 48, 49, 50, 51 2023-03-27T20:51:51.611Z CNTRLR « [Node 185] node neighbors received: 5, 8, 22, 23, 28, 48, 49, 50, 51, 54, 55, 2023-03-27T20:53:43.048Z CNTRLR « [Node 187] node neighbors received: 4, 8, 28, 48, 49, 50, 51, 54, 55, 96, 102,

I tried to switch off the fuse to try isolate this node ID 49 without success for the moment...
In the meantime I've lot of errors like:
2023-03-28T20:03:26.677Z CNTRLR [Node 096] Route resolution failed 2023-03-28T20:03:31.386Z CNTRLR [Node 099] Route resolution failed 2023-03-28T20:03:36.071Z CNTRLR [Node 101] Route resolution failed 2023-03-28T20:03:38.813Z CNTRLR [Node 102] Route resolution failed 2023-03-28T20:03:45.298Z CNTRLR [Node 103] Route resolution failed 2023-03-28T20:03:45.662Z CNTRLR [Node 109] Route resolution failed 2023-03-28T20:03:52.293Z CNTRLR [Node 113] Route resolution failed 2023-03-28T20:03:58.779Z CNTRLR [Node 118] Route resolution failed 2023-03-28T20:04:03.493Z CNTRLR [Node 120] Route resolution failed 2023-03-28T20:04:08.192Z CNTRLR [Node 121] Route resolution failed 2023-03-28T20:04:10.918Z CNTRLR [Node 122] Route resolution failed 2023-03-28T20:04:15.623Z CNTRLR [Node 123] Route resolution failed 2023-03-28T20:04:22.281Z CNTRLR [Node 124] Route resolution failed 2023-03-28T20:04:24.998Z CNTRLR [Node 132] Route resolution failed 2023-03-28T20:04:29.690Z CNTRLR [Node 134] Route resolution failed 2023-03-28T20:04:34.389Z CNTRLR [Node 135] Route resolution failed 2023-03-28T20:04:39.096Z CNTRLR [Node 136] Route resolution failed 2023-03-28T20:04:41.824Z CNTRLR [Node 138] Route resolution failed 2023-03-28T20:04:45.467Z CNTRLR [Node 155] Route resolution failed 2023-03-28T20:04:49.761Z CNTRLR [Node 166] Route resolution failed 2023-03-28T20:04:53.791Z CNTRLR [Node 172] Route resolution failed 2023-03-28T20:04:56.514Z CNTRLR [Node 173] Route resolution failed 2023-03-28T20:05:01.202Z CNTRLR [Node 174] Route resolution failed 2023-03-28T20:05:05.886Z CNTRLR [Node 180] Route resolution failed 2023-03-28T20:05:08.591Z CNTRLR [Node 181] Route resolution failed 2023-03-28T20:05:12.890Z CNTRLR [Node 185] Route resolution failed 2023-03-28T20:05:19.746Z CNTRLR [Node 187] Route resolution failed
and some:
2023-03-28T20:08:10.043Z DRIVER Dropping message because it could not be deserialized: The command class Appli cation Status is not implemented (ZW0303) 2023-03-28T20:08:10.696Z DRIVER Dropping message because it could not be deserialized: The command class Appli cation Status is not implemented (ZW0303)

Of course, I performed healing of network without more success...

Is there somebody facing such kind of errors, or behaviour? I'd like to avoid destroying the complete network and readopt all the nodes it would cost me tremendous time...

Waiting for your advices :)

Thanks by advance and of course thanks for the development

[AlCalzone]

I performed healing of network without more success...

Can you share the log of healing just one of those nodes that have that ghost node as a neighbor?

I have to admit though that I've never seen Route resolution failed actually happen before, so I'm not 100% sure what to do about that.

[tekman54190]

Sure :) which level do you need? debug?

Thanks for helping

[AlCalzone]

debug please

[AlCalzone]

take your time though, I won't be able to look before tomorrow

[tekman54190]

Sure take your time, I can live with this status few more days :)

So I created a new log file including the fresh startup of the network, then healing of node 48 and healing of node 51.

Thanks ;)
zwavejs_2023-03-28.log

[tekman54190]

I 'm currently troubleshooting and I just saw there is another ghost node... ID: 27

So, we have same situation for node 27 and 49, really weird...