Netdata from BR disappears #7642

neye-zhaw · 2022-04-27T07:31:50Z

neye-zhaw
Apr 27, 2022

We have a network with three BRs which all register the NAT64 route. The NAT64 route of one of the BRs now disappears within a few hours of restarting it (its still listed in netdata show local). I have reported a similar issue with an older OT version before. Any ideas?

ot-br-posix commit: d77ab33

BR 6060

pi@clt-lab-t-6060:~ $ sudo ot-ctl rloc16
5c00
Done
pi@clt-lab-t-6060:~ $ sudo ot-ctl netdata show
Prefixes:
fd01:0:0:0::/64 paros med 5c00
fd01:0:0:0::/64 paros med e800
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
64:ff9b:0:0:0:0::/96 s med e800
fd6d:8cef:1e1d:0::/64 s med 5c00
fd6d:8cef:1e1d:0::/64 s med e800
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s e800
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
Done
pi@clt-lab-t-6060:~ $ sudo ot-ctl netdata show local
Prefixes:
fd01:0:0:0::/64 paros med 5c00
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
fd6d:8cef:1e1d:0::/64 s med 5c00
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
Done

BR 6092

pi@clt-lab-t-6092:~ $ sudo ot-ctl rloc16
e800
Done
pi@clt-lab-t-6092:~ $ sudo ot-ctl netdata show
Prefixes:
fd01:0:0:0::/64 paros med 5c00
fd01:0:0:0::/64 paros med e800
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
64:ff9b:0:0:0:0::/96 s med e800
fd6d:8cef:1e1d:0::/64 s med 5c00
fd6d:8cef:1e1d:0::/64 s med e800
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s e800
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
Done
pi@clt-lab-t-6092:~ $ sudo ot-ctl netdata show local
Prefixes:
fd01:0:0:0::/64 paros med e800
Routes:
64:ff9b:0:0:0:0::/96 s med e800
fd6d:8cef:1e1d:0::/64 s med e800
Services:
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s e800
Done

BR 6061 (this ones netdata is lost)

pi@clt-lab-t-6061:~ $ sudo ot-ctl rloc16
90f3
Done
pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show
Prefixes:
fd01:0:0:0::/64 paros med 5c00
fd01:0:0:0::/64 paros med e800
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
64:ff9b:0:0:0:0::/96 s med e800
fd6d:8cef:1e1d:0::/64 s med 5c00
fd6d:8cef:1e1d:0::/64 s med e800
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s e800
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
Done
pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show local
Prefixes:
fd01:0:0:0::/64 paros med 90f3
Routes:
64:ff9b:0:0:0:0::/96 s med 90f3
fd6d:8cef:1e1d:0::/64 s med 90f3
fd01:0:0:0::/64 s med 90f3
fd01:0:0:0::/64 s med 90f3
Services:
44970 5d fd7bd5a9ff20c73fb55053245a06972fd11f s 90f3
Done

Network setup script

#!/usr/bin/env bash

# Thread network setup extracted from HandleFormNetworkRequest():
# https://github.com/openthread/ot-br-posix/blob/main/src/web/web-service/wpan_service.cpp#L184

set -eEuo pipefail

# When ot-ctl returns anything other than the string "Done" it failed
run_check_ot-ctl() {
  echo $1
  ot-ctl $1 | grep -z "^Done[[:cntrl:]]*$"
}

# see FactoryReset() at
# https://github.com/openthread/ot-br-posix/blob/main/src/web/web-service/ot_client.cpp#L300
echo "performing factory reset"
ot-ctl factoryreset 2>/dev/null || true
sleep 4
ot-ctl version | grep -q "OPENTHREAD" || (echo "factory reset failed"; exit -1)

eui64=$(cat /sys/class/net/eth0/address | sed -E -e 's/://g' -e 's/(.{6})/\1ffff/')
run_check_ot-ctl "extaddr $eui64"

run_check_ot-ctl "txpower 8"

run_check_ot-ctl "networkname igi-ot-net"
run_check_ot-ctl "extpanid dada1234affe5678"
run_check_ot-ctl "panid 0xdada"
run_check_ot-ctl "networkkey fc1cbb6e94fb7590615f0cd66ee8ff9f"
run_check_ot-ctl "channel 23"
run_check_ot-ctl "pskc -p 123456"
run_check_ot-ctl "ifconfig up"
run_check_ot-ctl "thread start"

run_check_ot-ctl "prefix add fd01::/64 pasor"
run_check_ot-ctl "route add 64:ff9b::/96 s med"
run_check_ot-ctl "netdata register"

exit 0

Answered by jwhui

May 9, 2022

I agree that the duplicate prefixes are problematic. I actually submitting a fix for this issue several days ago in #7653. @neye-zhaw , can you see if #7653 helps address your issue?

View full answer

jwhui · 2022-04-27T12:51:42Z

jwhui
Apr 27, 2022
Maintainer

@neye-zhaw , can you provide more information on your network topology? How many router-capable devices do you have? How many devices are active routers?

One thing I noticed is that BR 6061 is a child (its RLOC16 is 90f3). OpenThread includes some code to delay registering its local network data with the Leader if it will attempt to become a router. I'm wondering if this logic is blocking BR 6061 from registering its network data.

10 replies

jwhui May 6, 2022
Maintainer

@neye-zhaw , can you provide the output of:

netdata show local
netdata show
netdata show -x

Thanks.

neye-zhaw May 9, 2022
Author

@jwhui

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show local
Prefixes:
fd01:0:0:0::/64 paros med b4e7
Routes:
64:ff9b:0:0:0:0::/96 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd6d:8cef:1e1d:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
fd01:0:0:0::/64 s med b4e7
Services:
44970 5d fd7bd5a9ff20c73fde140d02ed1ec707d11f s b4e7
Done

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show
Prefixes:
fd01:0:0:0::/64 paros med 5c00
fd01:0:0:0::/64 paros med 6000
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
64:ff9b:0:0:0:0::/96 s med 6000
fd6d:8cef:1e1d:0::/64 s med 5c00
fd6d:8cef:1e1d:0::/64 s med 6000
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s 6000
Done

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show -x
08040b02ecc303180040fd0100000000000005085c0033006000330007021340031600600064ff9b000000000000000001065c00006000000b0e8001010d095c007504b000000e100b2f81015d0d145c00fd7bd5a9ff20c73fdba678fbb81367f1d11f0d146000fd7bd5a9ff20c73f1d4cf74180cce995d11f03120040fd6d8cef1e1d000001065c0000600000030e0040fd2c90cd6599000107020240
Done

abtink May 9, 2022
Collaborator

Thanks @neye-zhaw. Can you please also paste netdata show local -x?

One thing I noticed is that there seems to be multiple instances of same route in local network data. I do not expect this to be possible though. The question is how the device ended up in such a state (apparently having multiple of same route prefix fd01:0:0:0::/64 s med b4e7)

I think this may be the root cause. If I recall correctly, the leader code integrating network data will reject it if the requested new network data from a node has duplicate entries. So that is why the registrations from this node are not accepted and not seen in partition (non-local) network data.

jwhui May 9, 2022
Maintainer

I agree that the duplicate prefixes are problematic. I actually submitting a fix for this issue several days ago in #7653. @neye-zhaw , can you see if #7653 helps address your issue?

Answer selected by jwhui

neye-zhaw May 11, 2022
Author

@abtink

netdata show local

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show local
Prefixes:
fd01:0:0:0::/64 paros med dc96
Routes:
64:ff9b:0:0:0:0::/96 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd6d:8cef:1e1d:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
fd01:0:0:0::/64 s med dc96
Services:
44970 5d fd7bd5a9ff20c73fde140d02ed1ec707d11f s dc96
Done

netdata show local -x

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show local -x
03100040fd010000000000000504dc963300031300600064ff9b00000000000000000103dc9600030f0040fd010000000000000103dc96000b1980015d0d14dc96fd7bd5a9ff20c73fde140d02ed1ec707d11f030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600030f0040fd6d8cef1e1d00000103dc9600030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600030f0040fd010000000000000103dc9600
Done

netdata show

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show
Prefixes:
fd01:0:0:0::/64 paros med 5c00
fd01:0:0:0::/64 paros med 2000
Routes:
64:ff9b:0:0:0:0::/96 s med 5c00
64:ff9b:0:0:0:0::/96 s med 2000
fd6d:8cef:1e1d:0::/64 s med 5c00
fd6d:8cef:1e1d:0::/64 s med 2000
Services:
44970 01 7504b000000e10 s 5c00
44970 5d fd7bd5a9ff20c73fdba678fbb81367f1d11f s 5c00
44970 5d fd7bd5a9ff20c73f1d4cf74180cce995d11f s 2000
Done

netdata show -x

pi@clt-lab-t-6061:~ $ sudo ot-ctl netdata show -x
08040b02ecc303180040fd0100000000000005085c0033002000330007021340031600600064ff9b000000000000000001065c00002000000b0e8001010d095c007504b000000e100b2f81015d0d145c00fd7bd5a9ff20c73fdba678fbb81367f1d11f0d142000fd7bd5a9ff20c73f1d4cf74180cce995d11f03120040fd6d8cef1e1d000001065c0000200000030e0040fd2c90cd6599000107020240
Done

Thanks for the answer. It will be some weeks/months before I can test that fix in that network.

neye-zhaw Jul 14, 2022
Author

I have been testing ot-br-posix commit e19e60b for over a month now. Seems to be munch better.

jwhui Jul 14, 2022
Maintainer

Great to hear! Thanks for the update.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenThread

Netdata from BR disappears #7642

{{title}}

Replies: 1 comment 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

OpenThread

Netdata from BR disappears #7642

neye-zhaw Apr 27, 2022

Replies: 1 comment · 10 replies

jwhui Apr 27, 2022 Maintainer

jwhui May 6, 2022 Maintainer

neye-zhaw May 9, 2022 Author

abtink May 9, 2022 Collaborator

jwhui May 9, 2022 Maintainer

neye-zhaw May 11, 2022 Author

neye-zhaw Jul 14, 2022 Author

jwhui Jul 14, 2022 Maintainer

neye-zhaw
Apr 27, 2022

Replies: 1 comment 10 replies

jwhui
Apr 27, 2022
Maintainer

jwhui May 6, 2022
Maintainer

neye-zhaw May 9, 2022
Author

abtink May 9, 2022
Collaborator

jwhui May 9, 2022
Maintainer

neye-zhaw May 11, 2022
Author

neye-zhaw Jul 14, 2022
Author

jwhui Jul 14, 2022
Maintainer