Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iperf3 increases packets per seconds for UDP tests after some time #1367

Closed
FlyDrag opened this issue Aug 1, 2022 · 5 comments
Closed

iperf3 increases packets per seconds for UDP tests after some time #1367

FlyDrag opened this issue Aug 1, 2022 · 5 comments

Comments

@FlyDrag
Copy link

FlyDrag commented Aug 1, 2022

Context

  • Version of iperf3:
    client (sender): 3.11 (i assume that this is sender issue) - from FreeBSD packages
    server (receiver): 3.7 - from FreeBSD packages (upgraded to 3.11 - doesn't help)

  • Hardware:
    i82576 PCIe-x4 dual NIC.
    Intel Pentium G5420 CPU

  • Operating system (and distribution, if any):
    FreeBSD 13.1-RELEASE (sender)
    FreeBSD 12.0-RELEASE-p12 (receiver)

Bug Report

For some combinations of bandwidth and length iperf3 increases pps (keeping bitrate - so I assume that it decreases the packet size) after some time.

Test setup - two PCs connected by direct ethernet cable. 1 GBps interface speed. 9216 MTU size.

Test 1. length 9000, bw 950M - working fine

iperf3 -u -b 950M -l 9000 --dont-fragment -c fc2
Connecting to host fc2, port 5201
[  5] local 192.168.254.29 port 41050 connected to 192.168.254.30 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  13184
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  13195
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  13194
[  5]   3.00-4.00   sec   113 MBytes   950 Mbits/sec  13195
[  5]   4.00-5.00   sec   113 MBytes   950 Mbits/sec  13194
[  5]   5.00-6.00   sec   113 MBytes   950 Mbits/sec  13194
[  5]   6.00-7.00   sec   113 MBytes   950 Mbits/sec  13195
[  5]   7.00-8.00   sec   113 MBytes   950 Mbits/sec  13194
[  5]   8.00-9.00   sec   113 MBytes   950 Mbits/sec  13195
[  5]   9.00-10.00  sec   113 MBytes   950 Mbits/sec  13194
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   950 Mbits/sec  0.000 ms  0/131934 (0%)  sender
[  5]   0.00-10.00  sec  1.11 GBytes   950 Mbits/sec  0.106 ms  0/131933 (0%)  receiver

iperf Done.

Test2. length 1500, bw 950M - working fine

iperf3 -u -b 950M -l 1500 --dont-fragment -c fc2
Connecting to host fc2, port 5201
[  5] local 192.168.254.29 port 13599 connected to 192.168.254.30 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   950 Mbits/sec  79136
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  79137
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  79200
[  5]   3.00-4.00   sec   113 MBytes   950 Mbits/sec  79167
[  5]   4.00-5.00   sec   113 MBytes   950 Mbits/sec  79190
[  5]   5.00-6.00   sec   113 MBytes   950 Mbits/sec  79162
[  5]   6.00-7.00   sec   113 MBytes   949 Mbits/sec  79115
[  5]   7.00-8.00   sec   113 MBytes   950 Mbits/sec  79183
[  5]   8.00-9.00   sec   113 MBytes   951 Mbits/sec  79207
[  5]   9.00-10.00  sec   113 MBytes   949 Mbits/sec  79100
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   950 Mbits/sec  0.000 ms  0/791597 (0%)  sender
[  5]   0.00-10.00  sec  1.11 GBytes   950 Mbits/sec  0.016 ms  0/791597 (0%)  receiver

iperf Done.

Тест 3. length 1200, bw 950M - increased pps at second 5, 100% cpu load, packet loss. 100k datagrams at sec 4 but 500k datagrams sent at sec 5.

iperf3 -u -b 950M -l 1200 --dont-fragment -c fc2
Connecting to host fc2, port 5201
[  5] local 192.168.254.29 port 33674 connected to 192.168.254.30 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  98910
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  98958
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  98960
[  5]   3.00-4.00   sec   113 MBytes   950 Mbits/sec  98958
[  5]   4.00-5.00   sec   113 MBytes   945 Mbits/sec  508499
[  5]   5.00-6.00   sec   113 MBytes   944 Mbits/sec  379452
[  5]   6.00-7.00   sec   113 MBytes   944 Mbits/sec  534518
[  5]   7.00-8.00   sec   113 MBytes   944 Mbits/sec  379277
[  5]   8.00-9.00   sec   113 MBytes   944 Mbits/sec  379214
[  5]   9.00-10.00  sec   113 MBytes   944 Mbits/sec  379217
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.10 GBytes   947 Mbits/sec  0.000 ms  0/2955963 (0%)  sender
[  5]   0.00-10.23  sec  1.10 GBytes   925 Mbits/sec  0.015 ms  1969944/2955901 (67%)  receiver

iperf Done.

** Test 4.** The same, but test run for 4 seconds: Working fine.

iperf3 -u -b 950M -l 1200 --dont-fragment -c fc2 -t 4
Connecting to host fc2, port 5201
[  5] local 192.168.254.29 port 44539 connected to 192.168.254.30 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  98905
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  98968
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  98948
[  5]   3.00-4.00   sec   113 MBytes   950 Mbits/sec  98966
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-4.00   sec   453 MBytes   950 Mbits/sec  0.000 ms  0/395787 (0%)  sender
[  5]   0.00-4.03   sec   453 MBytes   944 Mbits/sec  0.013 ms  0/395780 (0%)  receiver

iperf Done.

It seems that iperf decreases packet size at second 5 of test.

** Test 5.** Another interface between this PCs. 1200 bytes works, 1100 - errors starts from second 4:

iperf3 -u -b 950M -l 1100 --dont-fragment -c receiver -t 4
Connecting to host receiver, port 5201
[  5] local 192.168.250.1 port 61085 connected to 192.168.250.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  107852
[  5]   1.00-2.00   sec   113 MBytes   951 Mbits/sec  108056
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  107950
[  5]   3.00-4.00   sec   113 MBytes   945 Mbits/sec  471221
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-4.00   sec   452 MBytes   949 Mbits/sec  0.000 ms  0/795079 (0%)  sender
[  5]   0.00-4.00   sec   450 MBytes   943 Mbits/sec  0.013 ms  351652/780450 (45%)  receiver

iperf Done.


iperf3 -u -b 950M -l 1100 --dont-fragment -c receiver -t 3
Connecting to host receiver, port 5201
[  5] local 192.168.250.1 port 29245 connected to 192.168.250.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   950 Mbits/sec  107950
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  107953
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  107943
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-3.00   sec   340 MBytes   950 Mbits/sec  0.000 ms  0/323846 (0%)  sender
[  5]   0.00-3.00   sec   337 MBytes   943 Mbits/sec  0.009 ms  0/321564 (0%)  receiver

iperf Done.

Test 6. bw 950M, length 1000: Problems at sec 2.

iperf3 -u -b 950M -l 1000 --dont-fragment -c receiver -t 3
Connecting to host receiver, port 5201
[  5] local 192.168.250.1 port 16975 connected to 192.168.250.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   950 Mbits/sec  118729
[  5]   1.00-2.00   sec   113 MBytes   946 Mbits/sec  274747
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec  397296
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-3.00   sec   338 MBytes   945 Mbits/sec  0.000 ms  0/790772 (0%)  sender
[  5]   0.00-3.00   sec   335 MBytes   938 Mbits/sec  0.010 ms  430546/782288 (55%)  receiver

iperf Done.
  • Expected Behavior

Keep PPS rate.

  • Actual Behavior

Increase PPS rate.

I'm really confused. I can't find any option to disable this behavior.
@TheRealDJ
Copy link
Contributor

All of your results that are good show zero packet loss (Lost Datagrams) at the receiver. They aren't getting there so they are retransmitted, hence the higher packet per second count. Using UDP there will never be any loss listed by the sender as the sender would have no idea any packets were lost.

@FlyDrag
Copy link
Author

FlyDrag commented Aug 2, 2022

TheRealDJ, Just to make everything clear. I passed my CCIE exams in 2004, so I have some :-) networking experience.

This is a UDP test. So, there is NO reason to retransmit packets.

I made some simple calculations. Iperf sent 3900567 packets for 10 seconds. netstat -s shows the same result. But, for 1100 bytes payload it is more than 3Gbps. Why iperf is sending more than 3 gigabits per second?

I understand that there should be some packet drops. 1100 bytes payload + 42 bytes overhead = 1142 bytes = 9136 bits. Plus 96 bits IPG = 9232 bits on wire.

9232 bits * 107946 frames = 996.5 Mbps on wire, this is really at line rate. BUT I thing there is NO reason to send 3 Gbps from iperf and display 72% packet loss.

netstat -s -p udp && iperf3 -u -b 950000000 -l 1100 --dont-fragment -c receiver -t 10 -w 1M && netstat -s -p udp
udp:
        10069605 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        886 dropped due to no socket
        884 broadcast/multicast datagrams undelivered
        74984 dropped due to full socket buffers
        0 not for hashed pcb
        9992851 delivered
        141127615 datagrams output
        0 times multicast source filter matched
Connecting to host receiver, port 5201
[  5] local 192.168.250.1 port 18349 connected to 192.168.250.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   950 Mbits/sec  107946
[  5]   1.00-2.00   sec   113 MBytes   950 Mbits/sec  107907
[  5]   2.00-3.00   sec   113 MBytes   950 Mbits/sec  107954
[  5]   3.00-4.00   sec   113 MBytes   946 Mbits/sec  476432
[  5]   4.00-5.00   sec   112 MBytes   943 Mbits/sec  606677
[  5]   5.00-6.00   sec   112 MBytes   943 Mbits/sec  549998
[  5]   6.00-7.00   sec   112 MBytes   943 Mbits/sec  635021
[  5]   7.00-8.00   sec   112 MBytes   943 Mbits/sec  518005
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec  395515
[  5]   9.00-10.00  sec   112 MBytes   943 Mbits/sec  395112
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec  0.000 ms  0/3900567 (0%)  sender
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  0.018 ms  2819329/3891371 (72%)  receiver

iperf Done.
udp:
        10069606 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        886 dropped due to no socket
        884 broadcast/multicast datagrams undelivered
        74984 dropped due to full socket buffers
        0 not for hashed pcb
        9992852 delivered
        145028183 datagrams output
        0 times multicast source filter matched

@davidBar-On
Copy link
Contributor

davidBar-On commented Aug 3, 2022 •
edited
Loading

The problem is probably because network buffers got full and ENOBUFS errno error is received when sending UDP packets. I am not sure how it is related to the packet length. Maybe this is because of the overhead required by the network stack to send each UDP packet (to get the same throughput, more shorter packet are requires than longer packets).

Per the code, the sent UDP packets are counted regardless of whether the packet was successfully sent or not. The failure does not terminate the test on "soft" error, when the error is ENOBUFS. Note that the number of bytes sent is calculated only according to the number of bytes successfully sent (although it doesn't mean they were not lost in the way to the server).

@jlearman
Copy link

Regardless, iperf3 shouldn't be trying to send faster than 950 Mb/s, but clearly (if we're to believe the "Total Datagrams" column) it is. I suspect some unusual error is occurring that is causing a different path through the iperf3 code, circumventing the transmission rate limit (bitrate). There's clearly a bug here.

@davidBar-On davidBar-On mentioned this issue Aug 19, 2022
Merged
@davidBar-On
Copy link
Contributor

davidBar-On commented Aug 20, 2022 •
edited
Loading

Regardless, iperf3 shouldn't be trying to send faster than 950 Mb/s ... There's clearly a bug here.

As the test results of this issue description show, iperf3 is not trying (and cannot) send faster then 950Mb/s. Since after 5 seconds sending most of the packets fail, iperf3 is continue to to try sending.

I agree that is a bug in the way iperf3 counts the failed packets, and that it sends packets with new numbering instead of retrying sending the same packet number. These issues cause the client to show that it sent many packets that practically were not sent. Worse is that the server shows a lot of lost packets, although practically they were not sent.

Submitter PR #1380 which should fix the above statistics issues.

@davidBar-On davidBar-On mentioned this issue Aug 20, 2022
Merged
@davidBar-On davidBar-On mentioned this issue Sep 20, 2022
Closed
bmah888 added a commit that referenced this issue Jul 21, 2023
@bmah888
Merge pull request #1380 from davidBar-On/issue-1367-do-not-count-UDP…
bd50531 
…-messages-that-were-not-sent

Do not count UDP messages that were not sent.

Fixes #1367
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@FlyDrag @jlearman @TheRealDJ @davidBar-On