[FRR 8.2.2] BGP sessions are unable to establish due to interface IP not recognized

[yxieca]

Description

Recently, many PR test for master/202205 branch has been failing because some BGP sessions were down after config reload or config load_minigraph, or just after deploying minigraph.

The issue has similarity to FRRouting/frr#10404. However, the fix FRRouting/frr@2cf7651 didn't address the issue.
(Check experimental PR #12366)

Steps to reproduce the issue:

1. Create master/202205 PR.

Describe the results you received:

Test failed after deploying minigraph, or in any test cases involves config reload/load_minigraph.

Describe the results you expected:

BGP sessions always come up.

Output of show version:

SONiC Software Version: SONiC.master-12366.159811-344393d99
Distribution: Debian 11.5
Kernel: 5.10.0-12-2-amd64
Build commit: 344393d99
Build date: Wed Oct 12 19:16:01 UTC 2022
Built by: AzDevOps@sonic-build-workers-00284R

Output of show techsupport:

Oct 12 22:59:05.512453 vlab-03 ERR bgp#bgpd[50]: [VX6SM-8YE5W][EC 33554460] 10.0.0.33: nexthop_set failed, resetting connection - intf 0x0
Oct 12 22:59:05.512739 vlab-03 ERR bgp#bgpd[50]: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 10.0.0.33, fd 68
Oct 12 22:59:05.647274 vlab-03 INFO bgp#bgpd[50]: [V1CHF-JSGRR] %NOTIFICATION: sent to neighbor 10.0.0.33 5/0 (Neighbor Events Error/Unspecific) 0 bytes 
Oct 12 22:59:10.005396 vlab-03 ERR bgp#bgpd[50]: [VX6SM-8YE5W][EC 33554460] 10.0.0.33: nexthop_set failed, resetting connection - intf 0x0
Oct 12 22:59:10.005558 vlab-03 ERR bgp#bgpd[50]: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 10.0.0.33, fd 68
Oct 12 22:59:10.093347 vlab-03 INFO bgp#bgpd[50]: [V1CHF-JSGRR] %NOTIFICATION: sent to neighbor 10.0.0.33 5/0 (Neighbor Events Error/Unspecific) 0 bytes 
Oct 12 22:59:16.023509 vlab-03 ERR bgp#bgpd[50]: [VX6SM-8YE5W][EC 33554460] 10.0.0.33: nexthop_set failed, resetting connection - intf 0x0
Oct 12 22:59:16.023509 vlab-03 ERR bgp#bgpd[50]: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 10.0.0.33, fd 68
Oct 12 22:59:16.023509 vlab-03 INFO bgp#bgpd[50]: [V1CHF-JSGRR] %NOTIFICATION: sent to neighbor 10.0.0.33 5/0 (Neighbor Events Error/Unspecific) 0 bytes 
Oct 12 22:59:20.029164 vlab-03 ERR bgp#bgpd[50]: [VX6SM-8YE5W][EC 33554460] 10.0.0.33: nexthop_set failed, resetting connection - intf 0x0
Oct 12 22:59:20.029350 vlab-03 ERR bgp#bgpd[50]: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 10.0.0.33, fd 68

Additional information you deem important (e.g. issue happens only occasionally):

sonic_dump_vlab-03_20221012_230519.tar.gz

[yxieca]

Alternative repro steps: download/build vs image from recent PR build but without including PR #12381. #12381 seems to be a workaround for this issue that we might be taking for the short term.

Deploy a kvm t1 testbed, the run this bash script:

#!/bin/bash -e

declare -i i

while true; do
    i=i+1
    echo "=== Iteration $i ==="
    sudo config reload -y &>/dev/null
    sleep 120
    bgp=$(show ip bgp sum)
    down=$(echo ${bgp} | grep "Active\|Idle" || true)
    if [[ -n "${down}" ]]; then
        echo "=== Not all BGP sessions are up ==="
        show ip bgp sum
        break
    fi
    echo "=== Success. Sleep a bit before next iteration ==="
    sleep 120
done

[adyeung]

@hasan-brcm will help take a look. @yxieca I'd suggest to backout #12381 first to unblock the build test while we are working on the RC

[yxieca]

@hasan-brcm will help take a look. @yxieca I'd suggest to backout #12381 first to unblock the build test while we are working on the RC

back out? or are you suggesting take it first? Just FYI, #12381 is not a bullet proof work-around. It does reduce the chance of hitting the issue by at least 10x though.

[yxieca]

@adyeung @hasan-brcm please be advised that we've merged #12381, if you want to test a fix, it would be better to revert this #12381 in your test.

[hasan-brcm]

We had seen similar issues internally in our lab as well with frr 7.2 and 7.5 releases. The issue was if a netlink dump is triggered while updates are still happening, the received dump had missing data. The issues were ranging from ip-address missing (which is the case here) to invalid vrf binding. The issue seems mostly from kernel instead of frr. But let me dig more both from the kernel and frr if there is anything that can be fixed.

[StormLiangMS]

@hasan-brcm I noticed one thing when debug another issue, for bullseye, the network.service would be restarted for several times during the bootup, I'm not sure if that lead to the ip-address missing issue? For the old kernel, network.service will be stared very early during the system boot up. But I didn't figure out why network.service is restarted for several times during bootup.