unstable srsue traffic

[Waseem0808]

Issue Description

DL traffic is very unstable in SRSUE and is causing the connection release.

Setup Details

A linux machine where gNB runs thru UHD B200 SDR with 10 MHZ geobondar clock in FDD mode(band 3).
In the same linux machine, open5gs is also running.
An another linux machine, where srsUE runs.

gNB configuration:
gNB.txt
Note: The gNB config is modified so it serves for both SRSUE COTS UE.

srsUE configuration:
srsUe.txt

I am using iperf tool to pump DL traffic (UDP):
Command in Server (srsUE PC): iperf -i 60 -s -b 5M -u
Command in Client (gNB/open5gs PC): iperf -i 30 -t 600 -c 10.45.0.2 -b 5M -u

nb_ngap file:
nb_ngap.zip

Expected Behavior

The DL traffic runs without interpretation.

Actual Behaviour

The 5Mbps traffic runs for few minutes (around 5 minutes) and then connection suddenly releases because of high error rate.
The error rate suddenly jumps to quite high.
If I try higher data rates then connection release even much earlier.
I am adding few gNB snapshots along with time.

srsueSnapshot:

iperf snapshots:
Server:

Client:

Additional Information

The performance script is run first.
Enabling more logs is making connection to loose connection much faster.

[pgawlowicz]

Do you run both gnb and srsUE on the same PC?
did you execute this script? https://github.com/srsran/srsRAN_Project/blob/main/scripts/srsran_performance

[Waseem0808]

Do you run both gnb and srsUE on the same PC? did you execute this script? https://github.com/srsran/srsRAN_Project/blob/main/scripts/srsran_performance

No, the gNB and srsUE is run on two different linux machines.
Yes, I have run the performance script.

[pgawlowicz]

Could you share your logs?
does it work better with smaller BW? like 10mhz?

[Waseem0808]

I tried lowering the bandwidth from 20 MHZ to 10 MHZ but the situation remains the same.
I am adding the logs for a 20 MHZ test run.
Uploading nb.zip…

[Waseem0808]

It looks logs are not getting uploaded.
Please use this link to download the logs:
https://www.dropbox.com/scl/fo/kx9qkwihb934d4w368s99/AHRidDmVz4efQMWiobaJvSQ?rlkey=qbgylqz755h7mvr4kwvyl9c05&st=ln419tjr&dl=0

[pgawlowicz]

could you try to apply the patch from this comment to srsUE, recompile and try again?
#826 (comment)

[Waseem0808]

Thanks for the patch.
I am facing an issue with my setup now, I am having high error rate just after attach.
This issue seems after I was making srsue to connect to internet only thru tun_srsue, These are the changes I made in srsue PC:

1. Delete Existing Default Route
   The current default route is pointing to 192.168.12.1 via wlo1.
   sudo ip route del default via 192.168.12.1 dev wlo1

2. Add a New Default Route via tun_srsue
   Next, set the default route to tun_srsue.
   sudo ip route add default via 10.45.0.1 dev tun_srsue

3. Verify the Routing Table
   Check if the new default route is set correctly:
   ip route show
   Ensure that the default route is now via tun_srsue.

4. echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

I then reverted these changes
REVERTING TO ORIGINAL DEFAULT ROUTE:

sudo ip route del default via 10.45.0.1 dev tun_srsue
sudo ip route add default via 192.168.12.1 dev wlo1

My routing table before modifying routing table looks as after srsue attach:

root@srsue# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 tun_srsue
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 tun_srsue
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1

After I reverted the changes back, my routing table looks like:

root@srsue# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 tun_srsue
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlo1
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1

There is difference in routing table. I am not sure, How its impacting the traffic & how to revert it back. Please guide.

For reference, my routing table in gNB machine looks as:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 ogstun
10.53.1.0       0.0.0.0         255.255.255.0   U     0      0        0 br-1f7bd8df2c69
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 br-1f7bd8df2c69
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.19.1.0      0.0.0.0         255.255.255.0   U     0      0        0 br-c314e7707126
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1

[pgawlowicz]

you can just reboot the machine, the routing changes you did are not persistent.

[Waseem0808]

I rebooted both the PCs multiple times but still the same issue.
The srsue is release as soon as its attached.
I didn't made any change in my setup except the routing table change.
Earlier I used to have two tun_srsue interfaces in srsue PC after attach. Why there were two interfaces? Are we expecting two interfaces?
I am now having only 1 interface, Is it okay?

[pgawlowicz]

Could you share gnb and srsUE logs?

[Waseem0808]

I am able to fix the setup. I re-installed the srsran project and re-applied the FPGA.
I applied the suggested patch in srsue. The call is released only in few seconds.
I tested by setting time_adv_nsamples as 300 & auto in srsue config file.
I am adding both gnb, srsue logs & patch changes & screenshots for your reference.
Log/Patch changes are available here:
https://www.dropbox.com/scl/fo/ryuknp0uochhypjwmh11w/AFGqtVH56P0Qmmn3TUIRuz8?rlkey=3frcjwpl5zchd90ss8u7haxzc&st=6zp45fne&dl=0

[Waseem0808]

@pgawlowicz any update on this issue?

[pgawlowicz]

@Waseem0808 I apologize for the delay, Did you have any success with this issue?

[Waseem0808]

I downgraded the bandwidth from 20Mhz to 10Mhz. I also used srsUE patch and it looks better.

MCS | CHANEL | DL_RATE | Comments

5 | 10MHZ | 4.7Mbps | SRSUE, Stability  at least 10 min

6 | 10MHZ | 5.7Mbps | SRSUE, Stability  at least 10 min

7 | 10MHZ | 6.6Mbps | SRSUE, Stability  at least 10 min

8 | 10MHZ | 7.7Mbps | SRSUE, Stability  at least 10 min

9 | 10MHZ | 8.7Mbps | SRSUE, Stability  at least 10 min

I sometimes need to rerun the testcase multiple times for the mentioned stability.
I get this maximum dl_rate with the mentioned MCS's. I was wondering whether this is the right data rates corresponding to the configuration.
What more can be done to make the connection stable?
I am attaching config, corresponding to shared results.
gNB.txt

srsue.txt