Skip to content

Latest commit

 

History

History
2668 lines (2267 loc) · 146 KB

README.org

File metadata and controls

2668 lines (2267 loc) · 146 KB

Linux Branch: “iptfs”:

Linux TFS development environment

There’s a Makefile that will git clone the needed repos.

Usage Notes

Building

If starting from scratch run these 2 commands:

make setup
make

This will checkout a linux kernel source tree under linux, a tagged release of buildroot under buildroot, and a modified branch of iproute2 under iproute2. Then it will build the kernel and the rootfs.

Launching a “Host <-> Router <-> Router <-> Host” setup.

You can use the tests to bring up a working iptfs ipsec tunnel. You’ll need to have installed qemu, socat, python and optional but highly recommended tmux, additionally you’ll need to install the python requirements,

python3 -m venv venv
source venv/bin/activate
pip install -r python-requirements.txt

Then run a test adding the `–pause` flag, and it will pause before running the first test, but after having configured the hosts (h1, h2) and routers (r1, r2).

NOTE: If your `sudo` command forces a restricted PATH (secure_path) then the python virtual environment may not work. In this case `sudo bash` and then activate the virtual environment as root.

NOTE: SUDO: For best results add the following to your /etc/sudoers config. This will allow tmux to continue to work inside the sudo environment.

Defaults env_keep += "TMUX"
Defaults env_keep += "TMUX_PANE"
$ sudo -E pytest -s -v tests/simplenet --pause
[...]
== PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==

or

$ sudo -E bash
# tmux # optional but highly useful
# source venv/bin/activate
# pytest -s -v tests/simplenet --pause
[...]
== PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==

You can now log into the running setup in another terminal. Use socat to log into the console of the running qemu’d linux.

$ sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 \
    unix-connect:/tmp/unet-test/tests.simplenet.test_simplenet/r1/s/console2

You can use mucmd to simply enter the namespace of the running node (e.g., to ping from h1 to h2 over the iptfs tunnel use the following command).

$ sudo mucmd -d /tmp/unet-test/tests.simplenet.test_simplenet h1 ping 10.0.2.4
PING 10.0.2.4 (10.0.2.4) 56(84) bytes of data.
64 bytes from 10.0.2.4: icmp_seq=1 ttl=62 time=1.22 ms
64 bytes from 10.0.2.4: icmp_seq=2 ttl=62 time=1.40 ms
64 bytes from 10.0.2.4: icmp_seq=3 ttl=62 time=1.25 ms
...

Qemu

consoles

3 serial consoles are created using unix sockets:

  • ‘%RUNDIR%/s/console’
  • ‘%RUNDIR%/s/vcon0’
  • ‘%RUNDIR%/s/vcon1’

- These can be accessed with::

socat /dev/stdin,escape=0x1d,rawer unix-connect:%RUNDIR%/s/console

Where if the node is named r1 then %RUNDIR% is usually tmp/unet-root/r1 so:

sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 unix-connect:/tmp/unet-root/r1/s/console

If your socat doesn’t support rawer option replace with raw,echo=0,icanon=0.

GDB

$ sudo gdb linux/vmlinux
(gdb) target remote /tmp/unet-root/r1/s/gdbserver
...

or
(gdb)
target remote /tmp/unet-test/tests.simplenet.test_simplenet/r1/s/gdbserver
target remote /tmp/unet-test/tests.simplenet.test_simplenet/r2/s/gdbserver

target remote /tmp/unet-test/tests.errors.test_errors/r1/s/gdbserver
target remote /tmp/unet-test/tests.errors.test_errors/r2/s/gdbserver

target remote /tmp/unet-test/tests.frags.test_frags/r1/s/gdbserver
target remote /tmp/unet-test/tests.frags.test_frags/r2/s/gdbserver

target remote /tmp/unet-test/tests.phynic.test_simplenet/r1/s/gdbserver

target remote /tmp/unet-test/tests.stress.test_stress/r2/s/gdbserver

target remote /tmp/unet-test/tests.utpkt.test_utpkt/r1/s/gdbserver

target remote /tmp/unet-test/tests.verify.test_verify/r1/s/gdbserver

Building/Editing Notes

CCLS

I like to use ccls with LSP mode in emacs, which will allow you to jump to all references with semantics (i.e., it’s not just a tag search) as well as displaying documentation and tab completion of symbols etc..

In order for this to work seemlessly you need to do a couple steps:

  1. cd linux # kernel source directory
  2. run ./scripts/clang-tools/gen_compile_commands.py -d ../output-linux
  3. printf “%%compile_commands.json\n-D__IN_CCLS__\n” > .ccls

CI Usage

Testing.

The tests contained herein are used directly in the iptfs-linux github project in it’s CI for testing kernel changes.

Releases

To create a new release (for artifacts) of a kernel and rootfs on github add a tag. These artifacts are used by other external project (e.g., by munet to test munet VM capabilities).

This works in the iptfs-dev directory as well as in iptfs-linux as they both use this repo for CI.

Design Notes

Config

The most basic configuration is to select “iptfs” as the mode rather than “tunnel” when configuring an IPsec SA.

Additionally there are further configuration options based on the mode the iptfs tunnel should run in.

common configuration

iptfs-dont-fragment
boolean to disable fragmenting inner packets, defaults to false
iptfs-max-queue-size
The maximum queue size for pending packets to send.
iptfs-pkt-size
the size of the outer packet (outer ip, esp, iptfs, + inner ip packets), either a value or auto to use PMTU
iptfs-reorder-window-size
The number of packets to hold waiting for re-ordered packets to arrive before considering those missing packets dropped. Default for fixed-rate send is 1. Default for demand-rate is 3. If the other endpoint is in dont-fragment mode then this value can be set to 0.
iptfs-drop-time
The number microseconds to wait unti considering the next in sequence packet as lost.

fixed-rate fixed-sized configuration

iptfs-fixed-rate
a fixed rate to send outer packets
iptfs-inner-fixed-rate
alternate form of fixed rate to sepcify inner packets datarate
iptfs-max-delay
alternate config for max-queue-size, which is based on the fixed send rate.
iptfs-disable-congestion-control
disble congestion control, should only be used when the user is in full administrative control of all paths the tunnel may take.
iptfs-no-pad-only
dont’ send all pad packets (debug option)

demand-rate configuration

iptfs-initial-delay
amount of time in microseconds to wait after before servicing the output queue when the initial packet arrives (first in queue). This time allows for collecting more packets to take advantage of IPTFS packet aggregation.

From Steffen’s Mail

[…] look at:

net/xfrm/* net/ipv4/xfrm* net/ipv4/esp4* net/ipv6/xfrm* net/ipv6/esp6*

> Anything else you think might be useful too would be much appreciated of course.

I think TFS should be a new encapsulation mode. We currently have tunnel, transport and beet mode (and some odd ipv6 modes). Adding a tfs_tunnel mode to add all the TFS special stuff would be the way to go at a first glance. The modes are implemented in:

net/xfrm/xfrm_output.c net/xfrm/xfrm_input.c

Bugs

  • xfrmi_rcv_cb is looking up xfrm_state from our newly created skb from decaping iptfs, but it has not xfrm_state so we panic
    • Need to associate the xfrm_state with new skbs too.. is there a refcnt for this?

Miscellaneous Development Scratchpad

This is an area for development notes. Please don’t consider them current or relevant, they’re just a place to keep notes while developing IPTFS.

Coverage, missing

  • Large Packet Fragmenting Prior to ingress
  • No reordering window fast path file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(gro_cells, x, skb);
  • ECN and DSCP for IPv4 and IPv6 file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“dropping in-progress reassemble\n”);
  • drop timer, future saved in queue to output now file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“receiving ordered list of len %u\n”, count);

Reassembly

  • during reassembly:
    • seq < want, drops packet, can only happen when reorder-window == 0 code coverage shows no code here… file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_OLD_SEQ);
    • seq > want, (will need to push past window), next fragment file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“missed needed seq\n”);
    • seq == want but no fragment
      • all pad, ok file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_ALL_PAD_SKIP);
      • not pad, abort and use next fragment file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“missing expected fragment\n”);
    • all fragment into next packet
  • not during reassembly
    • fragment start (unexpected), handle next fragment in packet

Reordering

  • reorder past file:linux/net/xfrm/xfrm_iptfs.c::return __reorder_past(xtfs, inskb, freelist, fcount);
  • reorder future: dup of one we’ve received file:linux/net/xfrm/xfrm_iptfs.c::/* a dup of a future */
  • reoder far future (larger that window)
    • need something in saved window that also shifts out during middle of shifting file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“send slot0 during shift: %llu”, s0seq);
    • need something at end of window that shifts into slot0 (will happend from the above). file:linux/net/xfrm/xfrm_iptfs.c::extra_drops–; /* we aren’t dropping what’s in slot0 */
  • was_gso == true in ingress/output collect function
  • zero reorder window size file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(x, skb);
  • free list from iptfs_input_reorder file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“freeing list of len %u\n”, fcount);

Ingress (collecting)

  • input GSO packets

file:linux/net/xfrm/xfrm_iptfs.c::netdev_features_t features = netif_skb_features(skb);

Ingress (sending)

  • rehoming a fraglist

file:linux/net/xfrm/xfrm_iptfs.c::static struct sk_buff **iptfs_rehome_fraglist(struct sk_buff **nextp,

XXXX DO THIS NEXT

  • iptfs_should_fragment

file:linux/net/xfrm/xfrm_iptfs.c::iptfs_should_fragment(skb2, mtu, remaining)) { XXXX

Implementation Catalog

Utility

_proto _seq __esp_seq

State Management

xfrm_iptfs_get_rtt_and_delays xfrm_iptfs_init_state xfrm_iptfs_state_destroy xfrm_iptfs_user_init xfrm_iptfs_copy_to_user_state

Egress (receive) functions

skb_copy_bits_seq iptfs_alloc_header_skb iptfs_alloc_skb iptfs_pskb_extract_seq iptfs_input_save_runt __iptfs_iplen __iptfs_iphdrlen

Reassembly

iptfs_complete_inner_skb _iptfs_reassem_done iptfs_reassem_abort iptfs_reassem_done iptfs_reassem_cont iptfs_input_ordered

Reordering

__vec_shift __reorder_past __reorder_drop __reorder_this iptfs_set_window_drop_times __reorder_future_fits __reorder_future_shifts iptfs_input_reorder

Input

iptfs_drop_timer xfrm_iptfs_input

Ingress (sending) functions

From xfrm

iptfs_enqueue xfrm_iptfs_output_collect

Dequeue and send

iptfs_xfrm_output iptfs_output_prepare_skb iptfs_first_skb iptfs_rehome_fraglist iptfs_should_fragment iptfs_output_queued iptfs_delay_timer

header_len usages:

Initializing

AH (ah[46].c)

  • ah_init_state():
  • ah6_init_state():
if (x->props.flags & XFRM_STATE_ALIGN4)
    x->props.header_len = XFRM_ALIGN4(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
else
    x->props.header_len = XFRM_ALIGN8(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
if (x->props.mode == XFRM_MODE_TUNNEL)
     x->props.header_len += sizeof(struct iphdr);

     x->props.header_len += sizeof(struct ipv6hdr);

ESP (esp[46].c)

  • esp_init_state():
  • esp6_init_state():
/* TRANSPORT mode only has ESP header/trailer + crypto, no IP */
x->props.header_len = sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead);
if (x->props.mode == XFRM_MODE_TUNNEL) {
    /* v4 */
    x->props.header_len += sizeof(struct ipv6hdr);
    /* v6 */
    x->props.header_len += sizeof(struct iphdr);
    /* v4 */
} else if (x->props.mode == XFRM_MODE_BEET && x->sel.family != AF_INET6)
    x->props.header_len += IPV4_BEET_PHMAXLEN;
/* v6 */
else if (x->sel.family != AF_INET6)
    x->props.header_len += IPV4_BEET_PHMAXLEN + (sizeof(struct ipv6hdr) - sizeof(struct iphdr));
if (x->encap) { switch (encap->encap_type) {
    case UDP_ENCAP_ESPINUDP:
        x->props.header_len += sizeof(struct udphdr);
    case UDP_ENCAP_ESPINUDP_NON_IKE:
        x->props.header_len += sizeof(struct udphdr) + 2 * sizeof(u32);
    case TCP_ENCAP_ESPINTCP:
        /* only the length field, TCP encap is done by the socket */
        x->props.header_len += 2;
} }
align = ALIGN(crypto_aead_blocksize(aead), 4);
x->props.trailer_len = align + 1 + crypto_aead_authsize(aead);

ESP Offload (esp[46]_offload.c)

  • pcode:
static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x,
                                                   struct sk_buff *skb,
                                                   netdev_features_t features)
{
    skb->transport_header += x->props.header_len;
    ops = rcu_dereference(inet_offloads[xo->proto]);
    return ops->callbacks.gso_segment(skb, features);
}

static struct sk_buff *xfrm4_beet_gso_segment(struct xfrm_state *x,
                                              struct sk_buff *skb,
                                              netdev_features_t features)
{
    skb->transport_header += x->props.header_len;
    if (x->sel.family != AF_INET6) {
        if (proto == IPPROTO_BEETPH) {
            struct ip_beet_phdr *ph = (struct ip_beet_phdr *)skb->data;
            skb->transport_header += ph->hdrlen * 8;
            proto = ph->nexthdr;
        } else {
            skb->transport_header -= IPV4_BEET_PHMAXLEN;
        }
    } else {
        __be16 frag;
        skb->transport_header += ipv6_skip_exthdr(skb, 0, &proto, &frag);
        if (proto == IPPROTO_TCP)
            skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV4;
    }

    if (proto == IPPROTO_IPV6)
        skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP4;
    __skb_pull(skb, skb_transport_offset(skb));
    ops = rcu_dereference(inet_offloads[xo->proto]);
    return ops->callbacks.gso_segment(skb, features);
}

IPCOMP (ipcomp.c)

  • pcode:
static int ipcomp4_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
{
    x->props.header_len = 0;
    if (x->props.mode == XFRM_MODE_TUNNEL)
        x->props.header_len += sizeof(struct iphdr);
}

IPIP (xfrm4_tunnel.c)

  • pcode:
static int ipip_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
{
    x->props.header_len = sizeof(struct iphdr);
}

Using

xfrm_state:

  • pcode: xfrm_state_mtu
  • This is used as a pad-to target maybe along with configured pad-to size

the padding added to the trailer will be (this value - skb->len)

u32 xfrm_state_mtu(struct xfrm_state *x, int mtu)
{
    if (x->km.state != XFRM_STATE_VALID || !type || type->proto != IPPROTO_ESP)
        return mtu - x->props.header_len;
    blksize = ALIGN(crypto_aead_blocksize(aead), 4);
    switch (x->props.mode) {
    case XFRM_MODE_TRANSPORT:
    case XFRM_MODE_BEET:
        if (x->props.family == AF_INET)
            net_adj = sizeof(struct iphdr);
        else if (x->props.family == AF_INET6)
            net_adj = sizeof(struct ipv6hdr);
        break;
    case XFRM_MODE_TUNNEL:
    case XFRM_MODE_IPTFS:
        break;
    default:
        WARN_ON_ONCE(1);
        break;
    }
    return ((mtu - x->props.header_len - crypto_aead_authsize(aead) - net_adj) & ~(blksize - 1))
           + net_adj - 2;
    /* TUNNEL is this right what is this value? */
    /* ((1500 - size(IP+ESP) - size(CRYPTAUTH) - 0) & 0xFFFFFFFc) + 0 - 2;

    /* TRANSPORT */
    /* ((1500 - size(ESP) - size(CRYPTAUTH) - size(IP)) & 0xFFFFFFFc) + size(IP) - 2; */
}

xfrm_device

  • pcode: prepping skb’s for???

This code is going to `skb_pull` `skb_data` down over the mac header and header_len of data. For IPTFS we want it to leave the IPTFS header in the data area for encryption.

static void __xfrm_transport_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb_transport_offset(skb) + x->props.header_len);
}
static void __xfrm_mode_tunnel_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb->mac_len + x->props.header_len);
}
static void __xfrm_mode_iptfs_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb->mac_len + x->props.header_len - sizeof(struct iptfs_header));
}
static void __xfrm_mode_beet_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    /* this is reversing something done earlier */
    if (x->sel.family != AF_INET6) {
        phlen = IPV4_BEET_PHMAXLEN;
        if (x->outer_mode.family == AF_INET6)
            phlen += sizeof(struct ipv6hdr) - sizeof(struct iphdr);
    }
    pskb_pull(skb, skb->mac_len + hsize + (x->props.header_len - phlen));
}

Call path for xfrm_outer_mode_prep:

static const struct proto_ops packet_ops = { .sendmsg = packet_sendmsg, packet_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) packet_snd(struct socket *sock, struct msghdr *msg, size_t len) || tpacket_snd(struct packet_sock *po, struct msghdr *msg) packet_xmit(const struct packet_sock *po, struct sk_buff *skb) dev_direct_xmit(struct sk_buff *skb, u16 queue_id) __dev_direct_xmit(struct sk_buff *skb, u16 queue_id) || sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q, validate_xmit_skb_list(struct sk_buff *skb, struct net_device *dev, bool *again) validate_xmit_skb(struct sk_buff *skb, struct net_device *dev, bool *again) validate_xmit_xfrm xfrm_outer_mode_prep

xfrm_policy

Offload vs No offload backtrace

No ESP offload

#0 esp_output (x=0xffff888107a8a000, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4.c:655 #1 0xffffffff827a5f8c in xfrm_output_one (err=0, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:600

#2 xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40, err=<optimized out>, err@entry=1) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:635 #3 0xffffffff827a6483 in xfrm_output2 (skb=0xffff888109584b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849f7e40 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662 #4 xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826 #5 0xffffffff827b9052 in iptfs_xfrm_output (skb=skb@entry=0xffff888109584b40, remaining=remaining@entry=1358) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648 #6 0xffffffff827bc427 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080 #7 iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129

#8 0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a8f468, base=0xffff88823762b3c0, cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685 #9 __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=39634970245, flags=flags@entry=642, active_mask=active_mask@entry=240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749 #10 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766 #11 0xffffffff82b50b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #12 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #13 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #14 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #15 0xffffffff82b3b5ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #16 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #17 0x0000000000000000 in ?? () (gdb)

ESP offload (INET_ESP_OFFLOAD)

#0 esp_xmit (x=0xffff888107a8e000, skb=0xffff888109589b40, features=0) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:257 #1 0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff888109589b40, features=features@entry=1, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187

#2 0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff888109589b40, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687 #3 0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff888109589b40, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709 #4 0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff888109589b40, q=q@entry=0xffff8881084e6000, dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>, validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327 #5 0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000, skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805 #6 __dev_queue_xmit (skb=skb@entry=0xffff888109589b40, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210 #7 0xffffffff82590b8d in dev_queue_xmit (skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085

#8 neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552 #9 0xffffffff826a844d in neigh_output (skip_cache=false, skb=0xffff888109589b40, n=0xffff8881092ac000) at /home/chopps/w/iptfs-dev/linux/include/net/neighbour.h:544 #10 ip_finish_output2 (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=<optimized out>, skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:229 #11 0xffffffff826aa7d6 in __ip_finish_output (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:307 #12 0xffffffff826aaa72 in ip_finish_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:317 #13 NF_HOOK_COND (pf=2 ‘\002’, hook=4, okfn=0xffffffff826aa910 <ip_finish_output>, cond=<optimized out>, out=0xffff8881053be000, in=0xffff8881053aa000, skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:292 #14 ip_output (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:431 #15 0xffffffff827b7bd9 in dst_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/include/net/dst.h:458

#16 xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40, err=err@entry=1) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:643 #17 0xffffffff827b97dd in xfrm_output2 (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662 #18 xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826 #19 0xffffffff827ce922 in iptfs_xfrm_output (skb=skb@entry=0xffff888109589b40, remaining=remaining@entry=1358) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648 #20 0xffffffff827d1cf7 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080 #21 iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129

#22 0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a91c68, base=0xffff88823762b3c0, cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685 #23 __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=123664696935, flags=flags@entry=642, active_mask=active_mask@entry=240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749 #24 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766 #25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #31 0x0000000000000000 in ?? () (gdb) c

ARP

#0 esp_xmit (x=0xffff888107a8e000, skb=0xffff8881095893c0, features=0) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:270 #1 0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff8881095893c0, features=features@entry=1, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187 #2 0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687 #3 0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709 #4 0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff8881095893c0, q=q@entry=0xffff8881084e6000, dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>, validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327 #5 0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000, skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805 #6 __dev_queue_xmit (skb=skb@entry=0xffff8881095893c0, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210 #7 0xffffffff82590b8d in dev_queue_xmit (skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085 #8 neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552 #9 0xffffffff82590fa0 in __neigh_update (neigh=neigh@entry=0xffff8881092ac000, lladdr=<optimized out>, lladdr@entry=0xffff8881058ff056 “\002\252\252\252\002\003\n”, new=<optimized out>, flags=0, nlmsg_pid=nlmsg_pid@entry=0, extack=extack@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1450 #10 0xffffffff82591c93 in neigh_update (neigh=neigh@entry=0xffff8881092ac000, lladdr=lladdr@entry=0xffff8881058ff056 “\002\252\252\252\002\003\n”, new=<optimized out>, flags=<optimized out>, nlmsg_pid=nlmsg_pid@entry=0) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1477 #11 0xffffffff8271fbc5 in arp_process (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589640) at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:932 #12 0xffffffff827206d4 in NF_HOOK (pf=3 ‘\003’, sk=0x0 <fixed_percpu_data>, okfn=0xffffffff8271f700 <arp_process>, out=0x0 <fixed_percpu_data>, in=0xffff8881053be000, skb=0xffff888109589640, net=0xffffffff849fcec0 <init_net>, hook=0) at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:303 #13 arp_rcv (skb=0xffff888109589640, dev=0xffff8881053be000, pt=<optimized out>, orig_dev=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:988 #14 0xffffffff8257b1bc in __netif_receive_skb_list_ptype (orig_dev=0xffff8881053be000, pt_prev=0xffffffff842766c0 <arp_packet_type>, head=0xffff888237609a48) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5539 #15 __netif_receive_skb_list_core (head=head@entry=0xffff8881053c2110, pfmemalloc=pfmemalloc@entry=false) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5582 #16 0xffffffff8257bd97 in __netif_receive_skb_list (head=0xffff8881053c2110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5634 #17 netif_receive_skb_list_internal (head=head@entry=0xffff8881053c2110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5725 #18 0xffffffff8257c983 in gro_normal_list (napi=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/include/net/gro.h:433 #19 napi_complete_done (n=n@entry=0xffff8881053c2008, work_done=work_done@entry=1) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6065 #20 0xffffffff81f70300 in virtqueue_napi_complete (processed=<optimized out>, vq=0xffff888105357800, napi=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:406 #21 virtnet_poll (napi=<optimized out>, budget=<optimized out>) at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:1965 #22 0xffffffff8257cc91 in __napi_poll (n=n@entry=0xffff8881053c2008, repoll=repoll@entry=0xffff888237609ed0) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6496 #23 0xffffffff8257d964 in napi_poll (repoll=0xffff888237609f00, n=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6563 #24 net_rx_action (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6696 #25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #31 0x0000000000000000 in ?? ()

nr_frags large packet

(gdb) p *skb $25 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, {dev = 0xffff888109008000, dev_scratch = 18446612686516027392}}, rbnode = { __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>, rb_left = 0xffff888109008000}, list = {next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, {sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0, skb_mstamp_ns = 0}, cb = “\b”, ‘\000’ <repeats 19 times>, “\001\000\000\000ȉ\373\377\000\000\000\000\316\000\000\000\024\000\000\000ʠ\377\377\342\000\377\377”, {{ _skb_refdst = 18446612686612525184, destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff88810ec0f080, prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686612525184}, _nfct = 0, len = 8040, data_len = 4982, mac_len = 0, hdr_len = 0, queue_mapping = 1, __cloned_offset = 0xffff88810e8fb43e “”, cloned = 0 ‘\000’, nohdr = 0 ‘\000’, fclone = 0 ‘\000’, peeked = 0 ‘\000’, head_frag = 0 ‘\000’, pfmemalloc = 0 ‘\000’, pp_recycle = 0 ‘\000’, active_extensions = 0 ‘\000’, {{__pkt_type_offset = 0xffff88810e8fb440 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff88810e8fb441 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 1 ‘\001’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048802, { csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69, sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}, headers = { __pkt_type_offset = 0xffff88810e8fb440 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff88810e8fb441 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 1 ‘\001’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048802, { csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69, sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}}, tail = 3264, end = 3776, head = 0xffff88810ed64000 “”, data = 0xffff88810ed640ce “E”, truesize = 9334, users = {refs = {counter = 1}}, extensions = 0x0 <fixed_percpu_data>} (gdb) p *(struct skb_shared_info *)(skb->head + skb->end) $26 = {flags = 0 ‘\000’, meta_len = 0 ‘\000’, nr_frags = 2 ‘\002’, tx_flags = 0 ‘\000’, gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>, hwtstamps = {{hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0, dataref = {counter = 1}, xdp_frags_size = 0, destructor_arg = 0x0 <fixed_percpu_data>, frags = {{bv_page = 0xffffea000432cb00, bv_len = 3072, bv_offset = 192}, {bv_page = 0xffffea000432cb80, bv_len = 1910, bv_offset = 192}, {bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

These 2 pages are 0x80 apart (sizeof *page == 0x40 so 2 pages), but the offset is 192 (0xc0) on both with lengths of 3072 and 1910 these regions overlap. That’s b/c the actual page address is not the page struct.

Here’s a SW based 2 frag version: (gdb) p *skb $5 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, { dev = 0xffff888109210000, dev_scratch = 18446612686518157312}}, rbnode = { __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>, rb_left = 0xffff888109210000}, list = {next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, { sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0, skb_mstamp_ns = 0}, cb = “\004”, ‘\000’ <repeats 19 times>, “\001”, ‘\000’ <repeats 11 times>, “N\000\000\000\024\000\000\000\306\340\377\377b\000\000”, {{_skb_refdst = 18446612686654115968, destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff8881113b9080, prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686654115968}, _nfct = 0, len = 9000, data_len = 8948, mac_len = 0, hdr_len = 0, queue_mapping = 1, __cloned_offset = 0xffff8881112f657e “”, cloned = 0 ‘\000’, nohdr = 0 ‘\000’, fclone = 0 ‘\000’, peeked = 0 ‘\000’, head_frag = 0 ‘\000’, pfmemalloc = 0 ‘\000’, pp_recycle = 0 ‘\000’, active_extensions = 0 ‘\000’, {{ __pkt_type_offset = 0xffff8881112f6580 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff8881112f6581 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 0 ‘\000’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0, skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66, sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, { inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98, network_header = 78, mac_header = 78}, headers = { __pkt_type_offset = 0xffff8881112f6580 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff8881112f6581 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 0 ‘\000’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0, skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66, sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, { inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98, network_header = 78, mac_header = 78}}, tail = 130, end = 320, head = 0xffff88810fa37c00 “”, data = 0xffff88810fa37c4e “E”, truesize = 9844, users = { refs = {counter = 1}}, extensions = 0x0 <fixed_percpu_data>} (gdb) p *((struct skb_shared_info *)(skb->head + skb->end)) $6 = {flags = 0 ‘\000’, meta_len = 0 ‘\000’, nr_frags = 2 ‘\002’, tx_flags = 0 ‘\000’, gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>, hwtstamps = {{ hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0, dataref = {counter = 1}, xdp_frags_size = 0, destructor_arg = 0x0 <fixed_percpu_data>, frags = {{bv_page = 0xffffea0004387000, bv_len = 3506, bv_offset = 29262}, { bv_page = 0xffffea0004387200, bv_len = 5442, bv_offset = 0}, { bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

$3 = { […] frags = {{ bv_page = 0xffffea0004387000, bv_len = 3506, bv_offset = 29262 }, { bv_page = 0xffffea0004387200, bv_len = 5442, bv_offset = 0 }, { bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0 } <repeats 15 times>} }

(gdb) p *$7->frags[0].bv_page $18 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>, mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0, share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122, _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = { counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>, pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, { pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, { _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 1}} (gdb) p/x *$7->frags[0].bv_page $20 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, { __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0, prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}}, mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0, pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0, pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = { counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = { counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x1}}

(gdb) p *$7->frags[1].bv_page $19 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>, mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0, share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122, _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = { counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>, pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, { pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, { _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 4}} (gdb) p/x *$7->frags[1].bv_page $21 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, { __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0, prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}}, mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0, pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0, pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = { counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = { counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x4}}

Sandbox

CONFIG options for tracing

CONFIG_DEBUG_INFO=y CONFIG_FRAME_POINTER=y CONFIG_FTRACE=y CONFIG_KALLSYMS=y CONFIG_KPROBES=y CONFIG_KPROBE_EVENTS=y CONFIG_LOCKDEP=y CONFIG_LOCK_STAT=y CONFIG_PERF_EVENTS=y CONFIG_TRACEPOINTS=y CONFIG_UPROBES=y CONFIG_UPROBE_EVENTS=y

Sample PPS and packet send times for 1500B IP packets

#include <stdio.h>
#define ENET_OHEAD (14 + 4 + 8 + 12)
#define _1GE_PPS(iptfs_ip_mtu) ((1e9 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _10GE_PPS(iptfs_ip_mtu) ((1e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _40GE_PPS(iptfs_ip_mtu) ((4e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _100GE_PPS(iptfs_ip_mtu) ((1e11 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _1GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _1GE_PPS(iptfs_ip_mtu))
#define _10GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _10GE_PPS(iptfs_ip_mtu))
#define _40GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _40GE_PPS(iptfs_ip_mtu))
#define _100GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _100GE_PPS(iptfs_ip_mtu))

int mtu = 64;

printf("+ 1GE 10GE 40GE 100GE\n");
printf("PPS %lu %lu %lu %lu\n", (uint64_t)_1GE_PPS(mtu),(uint64_t)_10GE_PPS(mtu),(uint64_t)_40GE_PPS(mtu),(uint64_t)_100GE_PPS(mtu));
printf("packet-time %luns %luns %luns %luns\n", (uint64_t)_1GE_PP_NANOS(mtu),(uint64_t)_10GE_PP_NANOS(mtu),(uint64_t)_40GE_PP_NANOS(mtu),(uint64_t)_100GE_PP_NANOS(mtu));

Checking uncommon flag setting code disassembly

No idea how to get this emacs-babel to work, but this saves the code for now. Babel is always evaluating the C block and saving those empty results in the file not the `code` as instructed.

echo "$filename"
gcc -O2 -c -o ${filename%.c}.o $filename
echo objdump -S -d "${filename%.c}.o"
#define SETF (1<<12)
#define CHECKF (1ull << 38)

unsigned long long is_setdm(unsigned long long bits, unsigned long long result)
{
    result |= (bits & CHECKF) / CHECKF * SETF;
    return result;
}

unsigned long long is_setto(unsigned long long bits, unsigned long long result)
{
    result |= bits & CHECKF ? SETF : 0;
    return result;
}

unsigned long long is_setif(unsigned long long bits, unsigned long long result)
{
    if (!!(bits & CHECKF))
        result |= SETF;
    return result;
}
echo "Hello World"

Performance Triaging

Testing done on a single server with 3 networks cards using munet and wiring the ports to each other. [h1] - [r1] - [r2] - [h2] ===========

Qemu emulated - single socket/core

  • Routed from h1 to h2 iperf bidir: ~2000 Mbps
  • IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~120 Mbps
  • IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir ~2 Mbps

Qemu -accel kvm - single socket/core

  • Routed from h1 to h2 iperf bidir: ~9400 Mbps
  • IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~920 Mbps
  • IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir ~2 Mbps

Qemu -accel kvm - 4 sockets

  • Routed from h1 to h2 iperf bidir: ~9400 Mbps
  • IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~7200 Mbps
  • IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir 700Kbps-3.87Mbps

Examples

These are single Core

tests/stress/trex_stress_phy.py::test_policy_imix

mode == tunnel (i.e., normal ipsec) (50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.83 Gbps version : STL @ v2.98 total_tx_L1 : 3.93 Gbps cpu_util. : 6.52% @ 2 cores (2 per dual port) total_rx : 3.04 Gbps rx_cpu_util. : 8.26% / 0 pps total_pps : 645.97 Kpps async_util. : 0% / 30.41 bps drop_rate : 792.36 Mbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.52% | 6.52% | – | | | Tx bps L2 | 1.92 Gbps | 1.91 Gbps | 3.83 Gbps Tx bps L1 | 1.97 Gbps | 1.96 Gbps | 3.93 Gbps Tx pps | 323.81 Kpps | 322.16 Kpps | 645.97 Kpps Line Util. | 4.93 % | 4.9 % | — | | | Rx bps | 1.52 Gbps | 1.51 Gbps | 3.04 Gbps Rx pps | 256.81 Kpps | 255.5 Kpps | 512.31 Kpps ---- | | | opackets | 7479318 | 7511222 | 14990540 ipackets | 5941015 | 5965888 | 11906903 obytes | 5542174638 | 5565815502 | 11107990140 ibytes | 4402286022 | 4421020888 | 8823306910 tx-pkts | 7.48 Mpkts | 7.51 Mpkts | 14.99 Mpkts rx-pkts | 5.94 Mpkts | 5.97 Mpkts | 11.91 Mpkts tx-bytes | 5.54 GB | 5.57 GB | 11.11 GB rx-bytes | 4.4 GB | 4.42 GB | 8.82 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix new 50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.84 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 6.39% @ 2 cores (2 per dual port) total_rx : 2.9 Gbps rx_cpu_util. : 6.74% / 0 pps total_pps : 647.42 Kpps async_util. : 0% / 34.3 bps drop_rate : 942.81 Mbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.39% | 6.39% | – | | | Tx bps L2 | 1.92 Gbps | 1.92 Gbps | 3.84 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 323.87 Kpps | 324.12 Kpps | 647.99 Kpps Line Util. | 4.93 % | 4.93 % | — | | | Rx bps | 1.45 Gbps | 1.45 Gbps | 2.9 Gbps Rx pps | 244.29 Kpps | 244.59 Kpps | 488.88 Kpps ---- | | | opackets | 6113860 | 6158922 | 12272782 ipackets | 4643520 | 4678016 | 9321536 obytes | 4530370260 | 4563761202 | 9094131462 ibytes | 3440826656 | 3466377360 | 6907204016 tx-pkts | 6.11 Mpkts | 6.16 Mpkts | 12.27 Mpkts rx-pkts | 4.64 Mpkts | 4.68 Mpkts | 9.32 Mpkts tx-bytes | 4.53 GB | 4.56 GB | 9.09 GB rx-bytes | 3.44 GB | 3.47 GB | 6.91 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.98 Gbps version : STL @ v2.98 total_tx_L1 : 4.2 Gbps cpu_util. : 12.07% @ 2 cores (2 per dual port) total_rx : 2.02 Gbps rx_cpu_util. : 9.33% / 0 pps total_pps : 1.41 Mpps async_util. : 0% / 36.53 bps drop_rate : 1.95 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 12.07% | 12.07% | – | | | Tx bps L2 | 2 Gbps | 1.98 Gbps | 3.98 Gbps Tx bps L1 | 2.11 Gbps | 2.09 Gbps | 4.2 Gbps Tx pps | 706.35 Kpps | 700.51 Kpps | 1.41 Mpps Line Util. | 5.28 % | 5.23 % | — | | | Rx bps | 1.02 Gbps | 1.01 Gbps | 2.02 Gbps Rx pps | 359.91 Kpps | 356.86 Kpps | 716.76 Kpps ---- | | | opackets | 10937602 | 10984861 | 21922463 ipackets | 5620736 | 5645504 | 11266240 obytes | 3866441526 | 3883148074 | 7749589600 ibytes | 1985575700 | 1995323444 | 3980899144 tx-pkts | 10.94 Mpkts | 10.98 Mpkts | 21.92 Mpkts rx-pkts | 5.62 Mpkts | 5.65 Mpkts | 11.27 Mpkts tx-bytes | 3.87 GB | 3.88 GB | 7.75 GB rx-bytes | 1.99 GB | 2 GB | 3.98 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.94 Gbps version : STL @ v2.98 total_tx_L1 : 4.16 Gbps cpu_util. : 11.57% @ 2 cores (2 per dual port) total_rx : 1.5 Gbps rx_cpu_util. : 8.49% / 0 pps total_pps : 1.39 Mpps async_util. : 0% / 0 bps drop_rate : 2.43 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 11.57% | 11.57% | – | | | Tx bps L2 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx bps L1 | 2.08 Gbps | 2.08 Gbps | 4.16 Gbps Tx pps | 694.97 Kpps | 697.46 Kpps | 1.39 Mpps Line Util. | 5.19 % | 5.21 % | — | | | Rx bps | 750.55 Mbps | 754.34 Mbps | 1.5 Gbps Rx pps | 265.62 Kpps | 266.53 Kpps | 532.14 Kpps ---- | | | opackets | 313742270 | 313798138 | 627540408 ipackets | 120083200 | 120102542 | 240185742 obytes | 110907888436 | 110927643186 | 221835531622 ibytes | 42428351448 | 42428427258 | 84856778706 tx-pkts | 313.74 Mpkts | 313.8 Mpkts | 627.54 Mpkts rx-pkts | 120.08 Mpkts | 120.1 Mpkts | 240.19 Mpkts tx-bytes | 110.91 GB | 110.93 GB | 221.84 GB rx-bytes | 42.43 GB | 42.43 GB | 84.86 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

These are 3 core

NOTE: for Qemu if multiple cores are used ipsec/iptfs seems to only use 1 thread (core). If multple sockets are used then multiple threads (sockets) are utilized.

tests/stress/trex_stress_phy.py::test_policy_imix

New IMIX (50/50)
mode == tunnel (i.e., normal ipsec) (50%/50%)

===== 0 DROP May go faster =====

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.87 Gbps version : STL @ v2.98 total_tx_L1 : 3.98 Gbps cpu_util. : 7.67% @ 2 cores (2 per dual port) total_rx : 3.87 Gbps rx_cpu_util. : 10.18% / 0.12 pps total_pps : 653.14 Kpps async_util. : 0% / 0 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 7.67% | 7.67% | – | | | Tx bps L2 | 1.95 Gbps | 1.92 Gbps | 3.87 Gbps Tx bps L1 | 2 Gbps | 1.97 Gbps | 3.98 Gbps Tx pps | 328.91 Kpps | 324.28 Kpps | 653.19 Kpps Line Util. | 5.01 % | 4.94 % | — | | | Rx bps | 1.95 Gbps | 1.92 Gbps | 3.87 Gbps Rx pps | 328.91 Kpps | 324.25 Kpps | 653.16 Kpps ---- | | | opackets | 37006482 | 37035660 | 74042142 ipackets | 37006464 | 37035618 | 74042082 obytes | 27421803162 | 27443424060 | 54865227222 ibytes | 27421789824 | 27443391520 | 54865181344 tx-pkts | 37.01 Mpkts | 37.04 Mpkts | 74.04 Mpkts rx-pkts | 37.01 Mpkts | 37.04 Mpkts | 74.04 Mpkts tx-bytes | 27.42 GB | 27.44 GB | 54.87 GB rx-bytes | 27.42 GB | 27.44 GB | 54.87 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix new 50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.85 Gbps version : STL @ v2.98 total_tx_L1 : 3.95 Gbps cpu_util. : 6.35% @ 2 cores (2 per dual port) total_rx : 2.7 Gbps rx_cpu_util. : 6.64% / 0 pps total_pps : 649 Kpps async_util. : 0% / 33.17 bps drop_rate : 1.14 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.35% | 6.35% | – | | | Tx bps L2 | 1.93 Gbps | 1.93 Gbps | 3.86 Gbps Tx bps L1 | 1.98 Gbps | 1.99 Gbps | 3.97 Gbps Tx pps | 325.41 Kpps | 326.34 Kpps | 651.75 Kpps Line Util. | 4.95 % | 4.97 % | — | | | Rx bps | 1.35 Gbps | 1.36 Gbps | 2.71 Gbps Rx pps | 228.17 Kpps | 228.88 Kpps | 457.05 Kpps ---- | | | opackets | 7075748 | 7094883 | 14170631 ipackets | 5048896 | 5062638 | 10111534 obytes | 5243129268 | 5257307626 | 10500436894 ibytes | 3741435036 | 3751414630 | 7492849666 tx-pkts | 7.08 Mpkts | 7.09 Mpkts | 14.17 Mpkts rx-pkts | 5.05 Mpkts | 5.06 Mpkts | 10.11 Mpkts tx-bytes | 5.24 GB | 5.26 GB | 10.5 GB rx-bytes | 3.74 GB | 3.75 GB | 7.49 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

Old Imix 7,4,1
mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.93 Gbps version : STL @ v2.98 total_tx_L1 : 4.16 Gbps cpu_util. : 14.76% @ 2 cores (2 per dual port) total_rx : 3.56 Gbps rx_cpu_util. : 18.21% / 0 pps total_pps : 1.39 Mpps async_util. : 0% / 30.14 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 14.76% | 14.76% | – | | | Tx bps L2 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx bps L1 | 2.08 Gbps | 2.09 Gbps | 4.17 Gbps Tx pps | 696.39 Kpps | 697.98 Kpps | 1.39 Mpps Line Util. | 5.2 % | 5.21 % | — | | | Rx bps | 1.86 Gbps | 1.71 Gbps | 3.58 Gbps Rx pps | 658.15 Kpps | 606.26 Kpps | 1.26 Mpps ---- | | | opackets | 14995714 | 15099834 | 30095548 ipackets | 14231296 | 13124265 | 27355561 obytes | 5300982114 | 5337789222 | 10638771336 ibytes | 5030619874 | 4638994972 | 9669614846 tx-pkts | 15 Mpkts | 15.1 Mpkts | 30.1 Mpkts rx-pkts | 14.23 Mpkts | 13.12 Mpkts | 27.36 Mpkts tx-bytes | 5.3 GB | 5.34 GB | 10.64 GB rx-bytes | 5.03 GB | 4.64 GB | 9.67 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.95 Gbps version : STL @ v2.98 total_tx_L1 : 4.17 Gbps cpu_util. : 10.12% @ 2 cores (2 per dual port) total_rx : 1.41 Gbps rx_cpu_util. : 5.47% / 0 pps total_pps : 1.4 Mpps async_util. : 0% / 34.58 bps drop_rate : 2.54 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 10.12% | 10.12% | – | | | Tx bps L2 | 1.97 Gbps | 1.98 Gbps | 3.95 Gbps Tx bps L1 | 2.08 Gbps | 2.09 Gbps | 4.17 Gbps Tx pps | 697.28 Kpps | 698.79 Kpps | 1.4 Mpps Line Util. | 5.21 % | 5.22 % | — | | | Rx bps | 704.24 Mbps | 703.04 Mbps | 1.41 Gbps Rx pps | 247.81 Kpps | 248.36 Kpps | 496.17 Kpps ---- | | | opackets | 11321860 | 11419036 | 22740896 ipackets | 4065024 | 4100397 | 8165421 obytes | 4002278236 | 4036629952 | 8038908188 ibytes | 1453675184 | 1455668512 | 2909343696 tx-pkts | 11.32 Mpkts | 11.42 Mpkts | 22.74 Mpkts rx-pkts | 4.07 Mpkts | 4.1 Mpkts | 8.17 Mpkts tx-bytes | 4 GB | 4.04 GB | 8.04 GB rx-bytes | 1.45 GB | 1.46 GB | 2.91 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (i.e., normal ipsec) 1G 40 octet packet
mode == tunnel (i.e., normal ipsec) 1G 40 octet packet
  • tunnel 1G 40b pkt
  • Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 36.57% @ 2 cores (2 per dual port) total_rx : 494 Mbps rx_cpu_util. : 13.65% / 0 pps total_pps : 5.86 Mpps async_util. : 0% / 35.1 bps drop_rate : 2.51 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 36.57% | 36.57% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.93 % | 4.92 % | — | | | Rx bps | 250.51 Mbps | 243.49 Mbps | 494 Mbps Rx pps | 489.27 Kpps | 475.57 Kpps | 964.84 Kpps ---- | | | opackets | 89007203 | 89443956 | 178451159 ipackets | 14835660 | 14522048 | 29357708 obytes | 5696460992 | 5724413184 | 11420874176 ibytes | 949482112 | 929411072 | 1878893184 tx-pkts | 89.01 Mpkts | 89.44 Mpkts | 178.45 Mpkts rx-pkts | 14.84 Mpkts | 14.52 Mpkts | 29.36 Mpkts tx-bytes | 5.7 GB | 5.72 GB | 11.42 GB rx-bytes | 949.48 MB | 929.41 MB | 1.88 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

More performance

routed, no tunnel

UPKT 1400, rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.17 Gbps version : STL @ v2.98 total_tx_L1 : 15.39 Gbps cpu_util. : 4.14% @ 4 cores (4 per dual port) total_rx : 14.14 Gbps rx_cpu_util. : 19.92% / 0 pps total_pps : 1.34 Mpps async_util. : 0% / 0 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.14% | 4.14% | – | | | Tx bps L2 | 7.57 Gbps | 7.61 Gbps | 15.18 Gbps Tx bps L1 | 7.68 Gbps | 7.71 Gbps | 15.39 Gbps Tx pps | 667.53 Kpps | 670.59 Kpps | 1.34 Mpps Line Util. | 19.2 % | 19.29 % | — | | | Rx bps | 6.55 Gbps | 7.6 Gbps | 14.15 Gbps Rx pps | 577.24 Kpps | 670.25 Kpps | 1.25 Mpps ---- | | | opackets | 16176393 | 16229842 | 32406235 ipackets | 14131488 | 16224312 | 30355800 obytes | 22938123860 | 23013915956 | 45952039816 ibytes | 20038449984 | 23006072998 | 43044522982 tx-pkts | 16.18 Mpkts | 16.23 Mpkts | 32.41 Mpkts rx-pkts | 14.13 Mpkts | 16.22 Mpkts | 30.36 Mpkts tx-bytes | 22.94 GB | 23.01 GB | 45.95 GB rx-bytes | 20.04 GB | 23.01 GB | 43.04 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 1K, rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.35 Gbps version : STL @ v2.98 total_tx_L1 : 15.66 Gbps cpu_util. : 4.76% @ 4 cores (4 per dual port) total_rx : 11.28 Gbps rx_cpu_util. : 20.4% / 0 pps total_pps : 1.89 Mpps async_util. : 0% / 0 bps drop_rate : 4.08 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.76% | 4.76% | – | | | Tx bps L2 | 7.7 Gbps | 7.65 Gbps | 15.35 Gbps Tx bps L1 | 7.85 Gbps | 7.8 Gbps | 15.66 Gbps Tx pps | 945.93 Kpps | 939.49 Kpps | 1.89 Mpps Line Util. | 19.64 % | 19.5 % | — | | | Rx bps | 6.09 Gbps | 5.19 Gbps | 11.28 Gbps Rx pps | 747.53 Kpps | 637.13 Kpps | 1.38 Mpps ---- | | | opackets | 39428467 | 39494064 | 78922531 ipackets | 31190218 | 26685961 | 57876179 obytes | 40138179406 | 40204954110 | 80343133516 ibytes | 31751641924 | 27166307280 | 58917949204 tx-pkts | 39.43 Mpkts | 39.49 Mpkts | 78.92 Mpkts rx-pkts | 31.19 Mpkts | 26.69 Mpkts | 57.88 Mpkts tx-bytes | 40.14 GB | 40.2 GB | 80.34 GB rx-bytes | 31.75 GB | 27.17 GB | 58.92 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 512 rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.51 Gbps version : STL @ v2.98 total_tx_L1 : 16.1 Gbps cpu_util. : 4.93% @ 4 cores (4 per dual port) total_rx : 3.21 Gbps rx_cpu_util. : 10.02% / 0 pps total_pps : 3.66 Mpps async_util. : 0% / 0 bps drop_rate : 12.3 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.93% | 4.93% | – | | | Tx bps L2 | 7.79 Gbps | 7.72 Gbps | 15.51 Gbps Tx bps L1 | 8.09 Gbps | 8.01 Gbps | 16.1 Gbps Tx pps | 1.84 Mpps | 1.82 Mpps | 3.66 Mpps Line Util. | 20.22 % | 20.03 % | — | | | Rx bps | 1.61 Gbps | 1.6 Gbps | 3.21 Gbps Rx pps | 380.82 Kpps | 377.25 Kpps | 758.07 Kpps ---- | | | opackets | 63793853 | 63793814 | 127587667 ipackets | 13199104 | 13199116 | 26398220 obytes | 33810742616 | 33810724050 | 67621466666 ibytes | 6995525120 | 6995532010 | 13991057130 tx-pkts | 63.79 Mpkts | 63.79 Mpkts | 127.59 Mpkts rx-pkts | 13.2 Mpkts | 13.2 Mpkts | 26.4 Mpkts tx-bytes | 33.81 GB | 33.81 GB | 67.62 GB rx-bytes | 7 GB | 7 GB | 13.99 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 256 rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 16.08 Gbps version : STL @ v2.98 total_tx_L1 : 17.26 Gbps cpu_util. : 7.38% @ 4 cores (4 per dual port) total_rx : 1.66 Gbps rx_cpu_util. : 10.18% / 0 pps total_pps : 7.34 Mpps async_util. : 0% / 0 bps drop_rate : 14.42 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 7.38% | 7.38% | – | | | Tx bps L2 | 8.04 Gbps | 8.04 Gbps | 16.08 Gbps Tx bps L1 | 8.63 Gbps | 8.63 Gbps | 17.26 Gbps Tx pps | 3.67 Mpps | 3.67 Mpps | 7.34 Mpps Line Util. | 21.57 % | 21.57 % | — | | | Rx bps | 830.78 Mbps | 830.68 Mbps | 1.66 Gbps Rx pps | 379.01 Kpps | 378.96 Kpps | 757.97 Kpps ---- | | | opackets | 105193737 | 105523395 | 210717132 ipackets | 10867507 | 10901568 | 21769075 obytes | 28823082318 | 28913409150 | 57736491468 ibytes | 2977696918 | 2987029632 | 5964726550 tx-pkts | 105.19 Mpkts | 105.52 Mpkts | 210.72 Mpkts rx-pkts | 10.87 Mpkts | 10.9 Mpkts | 21.77 Mpkts tx-bytes | 28.82 GB | 28.91 GB | 57.74 GB rx-bytes | 2.98 GB | 2.99 GB | 5.96 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

iptfs

1400UPKT 5G rate 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 9.47 Gbps version : STL @ v2.98 total_tx_L1 : 9.61 Gbps cpu_util. : 1.25% @ 4 cores (4 per dual port) total_rx : 2.86 Gbps rx_cpu_util. : 3.06% / 0 pps total_pps : 835.12 Kpps async_util. : 0% / 7.62 bps drop_rate : 6.62 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 1.25% | 1.25% | – | | | Tx bps L2 | 4.72 Gbps | 4.75 Gbps | 9.47 Gbps Tx bps L1 | 4.79 Gbps | 4.82 Gbps | 9.6 Gbps Tx pps | 416.15 Kpps | 418.74 Kpps | 834.89 Kpps Line Util. | 11.97 % | 12.04 % | — | | | Rx bps | 1.4 Gbps | 1.46 Gbps | 2.85 Gbps Rx pps | 123.1 Kpps | 128.47 Kpps | 251.58 Kpps ---- | | | opackets | 17519987 | 17561715 | 35081702 ipackets | 5292758 | 5474423 | 10767181 obytes | 24843341566 | 24902511870 | 49745853436 ibytes | 7505130844 | 7762730396 | 15267861240 tx-pkts | 17.52 Mpkts | 17.56 Mpkts | 35.08 Mpkts rx-pkts | 5.29 Mpkts | 5.47 Mpkts | 10.77 Mpkts tx-bytes | 24.84 GB | 24.9 GB | 49.75 GB rx-bytes | 7.51 GB | 7.76 GB | 15.27 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18908723 (75.29220704883792%) p1missed: 18699872 (74.46058807941534%)

ipsec

1400UPKT 5G rate 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 9.48 Gbps version : STL @ v2.98 total_tx_L1 : 9.61 Gbps cpu_util. : 1.17% @ 4 cores (4 per dual port) total_rx : 2.47 Gbps rx_cpu_util. : 2.57% / 0 pps total_pps : 835.27 Kpps async_util. : 0% / 0 bps drop_rate : 7.01 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 1.17% | 1.17% | – | | | Tx bps L2 | 4.75 Gbps | 4.74 Gbps | 9.49 Gbps Tx bps L1 | 4.82 Gbps | 4.8 Gbps | 9.62 Gbps Tx pps | 418.75 Kpps | 417.45 Kpps | 836.21 Kpps Line Util. | 12.04 % | 12.01 % | — | | | Rx bps | 1.24 Gbps | 1.23 Gbps | 2.47 Gbps Rx pps | 109.15 Kpps | 108.8 Kpps | 217.94 Kpps ---- | | | opackets | 10810806 | 10844294 | 21655100 ipackets | 2844712 | 2853184 | 5697896 obytes | 15329724322 | 15377208892 | 30706933214 ibytes | 4033801616 | 4045814912 | 8079616528 tx-pkts | 10.81 Mpkts | 10.84 Mpkts | 21.66 Mpkts rx-pkts | 2.84 Mpkts | 2.85 Mpkts | 5.7 Mpkts tx-bytes | 15.33 GB | 15.38 GB | 30.71 GB rx-bytes | 4.03 GB | 4.05 GB | 8.08 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18559218 (73.90052116795617%) p1missed: 18557335 (73.89302329378069%)

but then another run got:

and was showing more like 2G

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 14448450 (57.53194908692576%) p1missed: 14270021 (56.82146677611519%)

ipsec tunnel

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 4.32% @ 4 cores (4 per dual port) total_rx : 147.22 Mbps rx_cpu_util. : 3.51% / 0 pps total_pps : 5.86 Mpps async_util. : 0% / 0 bps drop_rate : 2.85 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.32% | 4.32% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.92 % | 4.92 % | — | | | Rx bps | 73.59 Mbps | 73.63 Mbps | 147.22 Mbps Rx pps | 143.74 Kpps | 143.8 Kpps | 287.54 Kpps ---- | | | opackets | 595000923 | 595235414 | 1190236337 ipackets | 29586114 | 29596946 | 59183060 obytes | 38080059072 | 38095066496 | 76175125568 ibytes | 1893511296 | 1894204544 | 3787715840 tx-pkts | 595 Mpkts | 595.24 Mpkts | 1.19 Gpkts rx-pkts | 29.59 Mpkts | 29.6 Mpkts | 59.18 Mpkts tx-bytes | 38.08 GB | 38.1 GB | 76.18 GB rx-bytes | 1.89 GB | 1.89 GB | 3.79 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

R1 HTOP

0[||| 2.7%] Tasks: 17, 0 thr, 67 kthr; 2 running 1[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Load average: 0.99 0.59 0.25 2[ 0.0%] Uptime: 00:04:31 Mem[||||||| 55.0M/3.83G] Swp[ 0K/0K]

[Main] [I/O] PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command 256 root 20 0 3164 2272 2116 S 0.7 0.1 0:00.02 /usr/sbin/dropbear -R -2 260 root 20 0 4356 3132 2572 R 0.7 0.1 0:00.34 htop 1 root 20 0 3432 648 576 S 0.0 0.0 0:00.15 init 142 root 20 0 3432 520 460 S 0.0 0.0 0:00.00 /sbin/syslogd -n 146 root 20 0 3432 492 428 S 0.0 0.0 0:00.00 /sbin/klogd -n 171 root 20 0 3164 1768 1636 S 0.0 0.0 0:00.00 /usr/sbin/dropbear -R 172 root 20 0 3432 512 444 S 0.0 0.0 0:00.00 /sbin/getty -L console 0 vt100 173 root 20 0 4528 3620 3268 S 0.0 0.1 0:00.00 /bin/sh – 175 root 20 0 3432 524 460 S 0.0 0.0 0:00.00 /sbin/getty -L ttyS0 115200 vt100 176 root 20 0 4660 4020 3472 S 0.0 0.1 0:00.04 -bash 177 root 20 0 4528 3604 3256 S 0.0 0.1 0:00.00 /bin/sh – 178 root 20 0 4528 3656 3308 S 0.0 0.1 0:00.00 /bin/sh – 254 root 20 0 3164 2276 2120 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 255 root 20 0 20904 14368 9300 S 0.0 0.4 0:00.21 perf record -F 997 -a -g -o /tmp/perf.data – sleep 1200 257 root 20 0 4660 3740 3396 S 0.0 0.1 0:00.00 bash 258 root 20 0 3300 508 448 S 0.0 0.0 0:00.00 sleep 1200 283 root 20 0 3432 524 456 S 0.0 0.0 0:00.00 /sbin/getty -n -l /bin/sh -L ttyS3 115200 vt100

F

IPTFS

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 2.98 Gbps version : STL @ v2.98 total_tx_L1 : 3.91 Gbps cpu_util. : 4.91% @ 4 cores (4 per dual port) total_rx : 275.2 Mbps rx_cpu_util. : 7.96% / 0 pps total_pps : 5.82 Mpps async_util. : 0% / 0 bps drop_rate : 2.7 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.91% | 4.91% | – | | | Tx bps L2 | 1.48 Gbps | 1.5 Gbps | 2.98 Gbps Tx bps L1 | 1.94 Gbps | 1.97 Gbps | 3.91 Gbps Tx pps | 2.89 Mpps | 2.93 Mpps | 5.82 Mpps Line Util. | 4.86 % | 4.91 % | — | | | Rx bps | 136.81 Mbps | 138.39 Mbps | 275.2 Mbps Rx pps | 267.2 Kpps | 270.29 Kpps | 537.49 Kpps ---- | | | opackets | 47519368 | 47490145 | 95009513 ipackets | 4431611 | 4424832 | 8856443 obytes | 3041239252 | 3039369040 | 6080608292 ibytes | 283623104 | 283189248 | 566812352 tx-pkts | 47.52 Mpkts | 47.49 Mpkts | 95.01 Mpkts rx-pkts | 4.43 Mpkts | 4.42 Mpkts | 8.86 Mpkts tx-bytes | 3.04 GB | 3.04 GB | 6.08 GB rx-bytes | 283.62 MB | 283.19 MB | 566.81 MB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

0[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 17, 0 thr, 67 kthr; 2 running 1[||||||||||||||||||||||||||||||||||||||||||||||||||||||| 66.2%] Load average: 0.64 0.18 0.06 2[||||| 5.4%] Uptime: 00:01:05 Mem[||||| 54.2M/3.83G] Swp[ 0K/0K]

[Main] [I/O] PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command 1 root 20 0 3432 656 580 S 0.0 0.0 0:00.11 init 142 root 20 0 3432 520 460 S 0.0 0.0 0:00.00 /sbin/syslogd -n 146 root 20 0 3432 520 452 S 0.0 0.0 0:00.00 /sbin/klogd -n 171 root 20 0 3164 1788 1656 S 0.0 0.0 0:00.00 /usr/sbin/dropbear -R 172 root 20 0 3432 516 456 S 0.0 0.0 0:00.00 /sbin/getty -L console 0 vt100 173 root 20 0 4528 3588 3236 S 0.0 0.1 0:00.00 /bin/sh – 175 root 20 0 3432 516 452 S 0.0 0.0 0:00.00 /sbin/getty -L ttyS0 115200 vt100 176 root 20 0 4660 3844 3292 S 0.0 0.1 0:00.03 -bash 177 root 20 0 4528 3756 3408 S 0.0 0.1 0:00.00 /bin/sh – 178 root 20 0 4528 3576 3216 S 0.0 0.1 0:00.00 /bin/sh – 254 root 20 0 3164 2268 2112 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 255 root 20 0 20904 14340 9276 S 0.0 0.4 0:00.12 perf record -F 997 -a -g -o /tmp/perf.data – sleep 1200 256 root 20 0 3300 520 460 S 0.0 0.0 0:00.00 sleep 1200 257 root 20 0 3164 2236 2080 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 258 root 20 0 4660 3676 3184 S 0.0 0.1 0:00.00 bash

Poor performance 150Kpps @ 40 w 4 cores:

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 13.12% @ 2 cores (2 per dual port) total_rx : 144.04 Mbps rx_cpu_util. : 9.97% / 0 pps total_pps : 5.87 Mpps async_util. : 0% / 23.93 bps drop_rate : 2.86 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 13.12% | 13.12% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.92 % | 4.93 % | — | | | Rx bps | 71.98 Mbps | 72.02 Mbps | 144 Mbps Rx pps | 140.58 Kpps | 140.66 Kpps | 281.24 Kpps ---- | | | opackets | 1380900371 | 1381337221 | 2762237592 ipackets | 69166674 | 69183583 | 138350257 obytes | 88377623744 | 88405582144 | 176783205888 ibytes | 4426667136 | 4427749312 | 8854416448 tx-pkts | 1.38 Gpkts | 1.38 Gpkts | 2.76 Gpkts rx-pkts | 69.17 Mpkts | 69.18 Mpkts | 138.35 Mpkts tx-bytes | 88.38 GB | 88.41 GB | 176.78 GB rx-bytes | 4.43 GB | 4.43 GB | 8.85 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 1,052 | 1,116 | 2,168

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

bash-5.1# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 14 0 0 0 IO-APIC 2-edge timer 1: 0 9 0 0 IO-APIC 1-edge i8042 3: 452 0 0 0 IO-APIC 3-edge ttyS1 4: 0 0 39 0 IO-APIC 4-edge ttyS0 8: 0 0 1 0 IO-APIC 8-edge rtc0 9: 0 0 0 0 IO-APIC 9-fasteoi acpi 12: 125 0 0 0 IO-APIC 12-edge i8042 16: 0 0 0 0 IO-APIC 16-fasteoi i801_smbus 24: 0 0 0 0 PCI-MSI 32768-edge virtio0-config 25: 0 0 0 26 PCI-MSI 32769-edge virtio0-virtqueues 26: 20 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2] 27: 0 0 0 0 PCI-MSI 49152-edge virtio1-config 28: 0 0 116 0 PCI-MSI 49153-edge virtio1-input.0 29: 0 0 0 126 PCI-MSI 49154-edge virtio1-output.0 30: 0 41 0 0 PCI-MSI 65536-edge eth1-TxRx-0 31: 0 0 147347 0 PCI-MSI 65537-edge eth1-TxRx-1 32: 0 0 0 147444 PCI-MSI 65538-edge eth1-TxRx-2 33: 39 0 0 0 PCI-MSI 65539-edge eth1-TxRx-3 34: 0 2 0 0 PCI-MSI 65540-edge eth1 35: 0 0 0 0 PCI-MSI 98304-edge virtio2-config 36: 0 3 0 0 PCI-MSI 98305-edge virtio2-requests 37: 0 0 0 42 PCI-MSI 81920-edge iavf-0000:00:05.0:mbx 38: 0 0 2 0 PCI-MSI 81921-edge iavf-eth2-TxRx-0 39: 0 0 0 333179 PCI-MSI 81922-edge iavf-eth2-TxRx-1 40: 0 0 0 0 PCI-MSI 81923-edge iavf-eth2-TxRx-2 41: 0 0 0 0 PCI-MSI 81924-edge iavf-eth2-TxRx-3 NMI: 0 0 0 0 Non-maskable interrupts LOC: 70266 70258 143816 70294 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 13 20 21 18 Rescheduling interrupts CAL: 72 103 73 88 Function call interrupts TLB: 1 2 1 1 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 Machine check polls HYP: 1 1 1 1 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event bash-5.1#

bash-5.1# htop bash-5.1# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 14 0 0 0 IO-APIC 2-edge timer 1: 0 9 0 0 IO-APIC 1-edge i8042 3: 606 0 0 0 IO-APIC 3-edge ttyS1 4: 0 0 49 0 IO-APIC 4-edge ttyS0 8: 0 0 1 0 IO-APIC 8-edge rtc0 9: 0 0 0 0 IO-APIC 9-fasteoi acpi 12: 125 0 0 0 IO-APIC 12-edge i8042 16: 0 0 0 0 IO-APIC 16-fasteoi i801_smbus 24: 0 0 0 0 PCI-MSI 32768-edge virtio0-config 25: 0 0 0 26 PCI-MSI 32769-edge virtio0-virtqueues 26: 29 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2] 27: 0 0 0 0 PCI-MSI 49152-edge virtio1-config 28: 0 0 167 0 PCI-MSI 49153-edge virtio1-input.0 29: 0 0 0 123 PCI-MSI 49154-edge virtio1-output.0 30: 0 46 0 0 PCI-MSI 65536-edge eth1-TxRx-0 31: 0 0 46 0 PCI-MSI 65537-edge eth1-TxRx-1 32: 0 0 0 161945 PCI-MSI 65538-edge eth1-TxRx-2 33: 193538 0 0 0 PCI-MSI 65539-edge eth1-TxRx-3 34: 0 3 0 0 PCI-MSI 65540-edge eth1 35: 0 0 0 0 PCI-MSI 98304-edge virtio2-config 36: 0 3 0 0 PCI-MSI 98305-edge virtio2-requests 37: 0 0 0 46 PCI-MSI 81920-edge iavf-0000:00:05.0:mbx 38: 0 0 2 0 PCI-MSI 81921-edge iavf-eth2-TxRx-0 39: 0 0 0 1 PCI-MSI 81922-edge iavf-eth2-TxRx-1 40: 364801 0 0 0 PCI-MSI 81923-edge iavf-eth2-TxRx-2 41: 0 0 0 0 PCI-MSI 81924-edge iavf-eth2-TxRx-3 NMI: 0 0 0 0 Non-maskable interrupts LOC: 76411 76355 76435 172972 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 14 9 34 23 Rescheduling interrupts CAL: 75 107 97 67 Function call interrupts TLB: 2 2 1 2 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 Machine check polls HYP: 1 1 1 1 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event b but even moving the IRQ didn’t change anything maybe b/c rx-2 is being shared?

New Perfs

NO GRO

no “generic-receive-offload”, no “rx-gro-hw”,

legacy IPsec - no iptfs


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 52900 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0196 sec 6.17 GBytes 5.29 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 49186 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0113 sec 6.32 GBytes 5.42 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 51934 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0103 sec 5.31 GBytes 4.55 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 45320 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0115 sec 5.32 GBytes 4.57 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

GRO

legacy IPsec - no iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 33940 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0075 sec 9.59 GBytes 8.23 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 45524 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0136 sec 9.04 GBytes 7.76 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 36432 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0125 sec 8.28 GBytes 7.10 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 40362 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0113 sec 8.79 GBytes 7.54 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 58158 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0092 sec 3.52 GBytes 3.03 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 33260 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0095 sec 6.31 GBytes 5.42 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 57584 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0105 sec 5.34 GBytes 4.58 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4


Client connecting to 10.0.2.4, TCP port 5001 TCP window size: 85.0 KByte (default)


[ 1] local 10.0.0.1 port 44142 connected with 10.0.2.4 port 5001 [ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0099 sec 4.43 GBytes 3.80 Gbits/sec root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

Debugging SLAB

5s IPv6

kmalloc-8k 282 294 16384 2 8 : tunables 0 0 0 : slabdata 147 147 0 kmalloc-4k 40861 40980 8192 4 8 : tunables 0 0 0 : slabdata 10245 10245 0 kmalloc-2k 377 528 4096 8 8 : tunables 0 0 0 : slabdata 66 66 0 kmalloc-1k 714 912 2048 16 8 : tunables 0 0 0 : slabdata 57 57 0 kmalloc-512 76535 76688 1024 16 4 : tunables 0 0 0 : slabdata 4793 4793 0 kmalloc-256 880 880 512 16 2 : tunables 0 0 0 : slabdata 55 55 0 kmalloc-192 492 544 256 16 1 : tunables 0 0 0 : slabdata 34 34 0 kmalloc-128 521 592 256 16 1 : tunables 0 0 0 : slabdata 37 37 0 kmalloc-96 6241 6656 128 32 1 : tunables 0 0 0 : slabdata 208 208 0 kmalloc-64 2984 3968 128 32 1 : tunables 0 0 0 : slabdata 124 124 0 kmalloc-32 1873 3328 64 64 1 : tunables 0 0 0 : slabdata 52 52 0 kmalloc-16 1285 2048 32 128 1 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-8 2026 4386 40 102 1 : tunables 0 0 0 : slabdata 43 43 0 kmem_cache_node 288 288 128 32 1 : tunables 0 0 0 : slabdata 9 9 0 kmem_cache 240 240 320 12 1 : tunables 0 0 0 : slabdata 20 20 0 bash-5.2# grep skb /proc/slabinfo skbuff_ext_cache 118994 166304 128 32 1 : tunables 0 0 0 : slabdata 5197 5197 0 skbuff_small_head 315 336 768 21 4 : tunables 0 0 0 : slabdata 16 16 0 skbuff_fclone_cache 108 108 640 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_head_cache 187671 284244 320 12 1 : tunables 0 0 0 : slabdata 23687 23687 0

15s IPv4

kmalloc-8k 263 274 16384 2 8 : tunables 0 0 0 : slabdata 137 137 0 kmalloc-4k 426 508 8192 4 8 : tunables 0 0 0 : slabdata 127 127 0 kmalloc-2k 8544 8592 4096 8 8 : tunables 0 0 0 : slabdata 1074 1074 0 kmalloc-1k 666 880 2048 16 8 : tunables 0 0 0 : slabdata 55 55 0 kmalloc-512 275412 275504 1024 16 4 : tunables 0 0 0 : slabdata 17219 17219 0 kmalloc-256 782 832 512 16 2 : tunables 0 0 0 : slabdata 52 52 0 kmalloc-192 463 528 256 16 1 : tunables 0 0 0 : slabdata 33 33 0 kmalloc-128 514 576 256 16 1 : tunables 0 0 0 : slabdata 36 36 0 kmalloc-96 13940 14080 128 32 1 : tunables 0 0 0 : slabdata 440 440 0 kmalloc-64 3077 4000 128 32 1 : tunables 0 0 0 : slabdata 125 125 0 kmalloc-32 1672 2944 64 64 1 : tunables 0 0 0 : slabdata 46 46 0 kmalloc-16 1198 2176 32 128 1 : tunables 0 0 0 : slabdata 17 17 0 kmalloc-8 1892 4182 40 102 1 : tunables 0 0 0 : slabdata 41 41 0 kmem_cache_node 256 256 128 32 1 : tunables 0 0 0 : slabdata 8 8 0 kmem_cache 240 240 320 12 1 : tunables 0 0 0 : slabdata 20 20 0 bash-5.2# grep skb /proc/slabinfo skbuff_ext_cache 273929 275136 128 32 1 : tunables 0 0 0 : slabdata 8598 8598 0 skbuff_small_head 50568 50589 768 21 4 : tunables 0 0 0 : slabdata 2409 2409 0 skbuff_fclone_cache 96 96 640 12 2 : tunables 0 0 0 : slabdata 8 8 0 skbuff_head_cache 175134 175716 320 12 1 : tunables 0 0 0 : slabdata 14643 14643 0 b

Why policy route failing

  • Unknown packet hist ip_route_input_noref
  • ip_route_input_rcu(struct sk_buff *skb, __be32 daddr, __be32 saddr,
  • ip_route_input_slow(struct sk_buff *skb, __be32 daddr, __be32 saddr,

#0 ip_route_input_slow (skb=0xffff888104718b00, daddr=<optimized out>, saddr=<optimized out>, tos=tos@entry=0 ‘\000’, dev=0xffff888102a74000, res=0xffffc90000003b90) at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2303 #1 0xffffffff81e8cfe3 in ip_route_input_rcu (skb=skb@entry=0xffff888104718b00, daddr=daddr@entry=84082698, saddr=saddr@entry=16777226, tos=tos@entry=0 ‘\000’, dev=dev@entry=0xffff888102a74000, res=res@entry=0xffffc90000003b90) at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2481 #2 0xffffffff81e8d086 in ip_route_input_noref (skb=skb@entry=0xffff888104718b00, daddr=84082698, saddr=16777226, tos=<optimized out>, dev=dev@entry=0xffff888102a74000) at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2492 #3 0xffffffff81e8fa1a in ip_rcv_finish_core (net=net@entry=0xffffffff833e60c0 <init_net>, skb=skb@entry=0xffff888104718b00, dev=0xffff888102a74000, hint=hint@entry=0x0 <fixed_percpu_data>, sk=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:365 #

working from strongswan

#0 __xfrm_route_forward (skb=skb@entry=0xffff888104c3c600, family=family@entry=2) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_policy.c:3794 #1 0xffffffff81e9241a in xfrm_route_forward (family=2, skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/include/net/xfrm.h:1273 #2 xfrm4_route_forward (skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/include/net/xfrm.h:1278 #3 ip_forward (skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_forward.c:124 #4 0xffffffff81e9080f in dst_input (skb=0xffff888104e70501) at /home/chopps/w/iptfs-dev/linux/include/net/dst.h:468 #5 ip_sublist_rcv_finish (head=head@entry=0xffffc90000003c48) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:580 #6 0xffffffff81e90946 in ip_list_rcv_finish (sk=0x0 <fixed_percpu_data>, head=0xffffc90000003cd0, net=0xffffffff833e60c0 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:630 #7 ip_sublist_rcv (head=head@entry=0xffffc90000003cd0, dev=dev@entry=0xffff888102f24000, net=net@entry=0xffffffff833e60c0 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:638 #8 0xffffffff81e90cbd in ip_list_rcv (head=0xffffc90000003d50, pt=<optimized out>, orig_dev=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:673 #9 0xffffffff81dd8dc8 in __netif_receive_skb_list_ptype (orig_dev=0xffff888102f24000, pt_prev=0xffffffff82ec4e60 <ip_packet_type>, head=0xffffc90000003d50) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5534 #10 __netif_receive_skb_list_core (head=head@entry=0xffff888102940110, pfmemalloc=pfmemalloc@entry=false) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5582 #11 0xffffffff81dd93cf in __netif_receive_skb_list (head=0xffff888102940110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5634 #12 netif_receive_skb_list_internal (head=head@entry=0xffff888102940110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5725 #13 0xffffffff81dd9b7e in gro_normal_list (napi=0xffff888102940008) at /home/chopps/w/iptfs-dev/linux/include/net/gro.h:433

sh-5.2# ip -d route list table 220 unicast 10.0.0.0/24 dev eth2 proto static scope link src 10.0.3.4

unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2

sh-5.2# ip -d route list table 220 unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4

BAD: ------ Host: r1 ------ unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global ------- End: r2 ------

munet> sh ip -d route list table main ------ Host: h1 ------ unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1 unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1 ------- End: h1 ------ ------ Host: h2 ------ unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.5 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.5 ------- End: h2 ------ ------ Host: r1 ------ unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.2 unicast 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2 unicast 10.0.2.4 via 10.0.1.3 dev eth2 proto boot scope global unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2 ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.1.2 via 10.0.2.3 dev eth2 proto boot scope global unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.4 unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.4 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.4 ------- End: r2 ------ ------ Host: rm ------ unicast 10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.3 unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.3 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3 ------- End: rm ------

GOOD: ------ Host: r1 ------ unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2 unicast 12.12.12.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2 ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4 unicast 11.11.11.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4 ------- End: r2 ------

munet> sh ip -d route list table main ------ Host: h1 ------ unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1 unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 proto boot scope global unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1 ------- End: h1 ------ ------ Host: h2 ------ unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 proto boot scope global unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.5 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.5 ------- End: h2 ------ ------ Host: r1 ------ unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.2 unicast 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2 unicast 10.0.2.0/24 via 10.0.1.3 dev eth2 proto boot scope global unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2 ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.1.0/24 via 10.0.2.3 dev eth2 proto boot scope global unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.4 unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.4 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.4 ------- End: r2 ------ ------ Host: rm ------ unicast 10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.3 unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.3 unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3 ------- End: rm ------

BAD FULLuTABLE

munet> sh ip -d route list table all ------ Host: h1 ------ unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.1 unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.1 local 10.0.0.1 dev eth1 table local proto kernel scope host src 10.0.0.1 broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.1 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.1 dev eth0 table local proto kernel scope host src 192.168.0.1 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.1 ------- End: h1 ------ ------ Host: h2 ------ unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.5 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.5 local 10.0.3.5 dev eth1 table local proto kernel scope host src 10.0.3.5 broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.5 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.5 dev eth0 table local proto kernel scope host src 192.168.0.5 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.5 ------- End: h2 ------ ------ Host: r1 ------ unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2

unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.2 unicast 10.0.1.0/24 dev eth2 table main proto kernel scope link src 10.0.1.2 unicast 10.0.2.4 via 10.0.1.3 dev eth2 table main proto boot scope global unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.2

local 10.0.0.2 dev eth1 table local proto kernel scope host src 10.0.0.2 broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.2 local 10.0.1.2 dev eth2 table local proto kernel scope host src 10.0.1.2 broadcast 10.0.1.255 dev eth2 table local proto kernel scope link src 10.0.1.2 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.2 dev eth0 table local proto kernel scope host src 192.168.0.2 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.2 ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4 unicast 10.0.1.2 via 10.0.2.3 dev eth2 table main proto boot scope global unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.4 unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.4 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.4 local 10.0.2.4 dev eth2 table local proto kernel scope host src 10.0.2.4 broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.4 local 10.0.3.4 dev eth1 table local proto kernel scope host src 10.0.3.4 broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.4 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.4 dev eth0 table local proto kernel scope host src 192.168.0.4 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.4 ------- End: r2 ------ ------ Host: rm ------ unicast 10.0.1.0/24 dev eth1 table main proto kernel scope link src 10.0.1.3 unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.3 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.3 local 10.0.1.3 dev eth1 table local proto kernel scope host src 10.0.1.3 broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src 10.0.1.3 local 10.0.2.3 dev eth2 table local proto kernel scope host src 10.0.2.3 broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.3 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.3 dev eth0 table local proto kernel scope host src 192.168.0.3 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.3 ------- End: rm ------ munet>

GOOD

munet> sh ip -d route list table all ------ Host: h1 ------ unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.1 unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.1 local 10.0.0.1 dev eth1 table local proto kernel scope host src 10.0.0.1 broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.1 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.1 dev eth0 table local proto kernel scope host src 192.168.0.1 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.1 ------- End: h1 ------ ------ Host: h2 ------ unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.5 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.5 local 10.0.3.5 dev eth1 table local proto kernel scope host src 10.0.3.5 broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.5 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.5 dev eth0 table local proto kernel scope host src 192.168.0.5 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.5 ------- End: h2 ------ ------ Host: r1 ------ unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2 unicast 12.12.12.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2

unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.2 unicast 10.0.1.0/24 dev eth2 table main proto kernel scope link src 10.0.1.2 unicast 10.0.2.0/24 via 10.0.1.3 dev eth2 table main proto boot scope global unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.2

local 10.0.0.2 dev eth1 table local proto kernel scope host src 10.0.0.2 broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.2 local 10.0.1.2 dev eth2 table local proto kernel scope host src 10.0.1.2 broadcast 10.0.1.255 dev eth2 table local proto kernel scope link src 10.0.1.2 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.2 dev eth0 table local proto kernel scope host src 192.168.0.2 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.2 ------- End: r1 ------ ------ Host: r2 ------ unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4 unicast 11.11.11.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4 unicast 10.0.1.0/24 via 10.0.2.3 dev eth2 table main proto boot scope global unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.4 unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.4 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.4 local 10.0.2.4 dev eth2 table local proto kernel scope host src 10.0.2.4 broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.4 local 10.0.3.4 dev eth1 table local proto kernel scope host src 10.0.3.4 broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.4 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.4 dev eth0 table local proto kernel scope host src 192.168.0.4 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.4 ------- End: r2 ------ ------ Host: rm ------ unicast 10.0.1.0/24 dev eth1 table main proto kernel scope link src 10.0.1.3 unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.3 unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.3 local 10.0.1.3 dev eth1 table local proto kernel scope host src 10.0.1.3 broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src 10.0.1.3 local 10.0.2.3 dev eth2 table local proto kernel scope host src 10.0.2.3 broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.3 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 local 192.168.0.3 dev eth0 table local proto kernel scope host src 192.168.0.3 broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.3 ------- End: rm ------

features/sharing gso/gro_cells

(py39) [0s][04:14:18 labnh:~/w/iptfs-dev]$ python scripts/feat2str.py 0x201901fd54bb3 [‘SG_BIT’, ‘IP_CSUM_BIT’, ‘IPV6_CSUM_BIT’, ‘HIGHDMA_BIT’, ‘HW_VLAN_CTAG_TX_BIT’, ‘HW_VLAN_CTAG_RX_BIT’, ‘HW_VLAN_CTAG_FILTER_BIT’, ‘GSO_BIT’, ‘GRO_BIT’,

‘TSO_BIT’, ‘TSO_ECN_BIT’, ‘TSO6_BIT’, ‘GSO_GRE_BIT’, ‘GSO_GRE_CSUM_BIT’, ‘GSO_IPXIP4_BIT’, ‘GSO_IPXIP6_BIT’, ‘GSO_UDP_TUNNEL_BIT’, ‘GSO_UDP_TUNNEL_CSUM_BIT’, ‘GSO_PARTIAL_BIT’, ‘SCTP_CRC_BIT’, ‘RXHASH_BIT’, ‘RXCSUM_BIT’, ‘HW_TC_BIT’]

(py39) [0s][04:56:02 labnh:~/w/iptfs-dev]$ python scripts/feat2str.py 0x2019017554bb3 [‘SG_BIT’, ‘IP_CSUM_BIT’, ‘IPV6_CSUM_BIT’, ‘HIGHDMA_BIT’, ‘HW_VLAN_CTAG_TX_BIT’, ‘HW_VLAN_CTAG_RX_BIT’, ‘HW_VLAN_CTAG_FILTER_BIT’, ‘GSO_BIT’, ‘GRO_BIT’,

‘TSO_BIT’, ‘TSO_ECN_BIT’, ‘TSO6_BIT’, ‘GSO_GRE_BIT’, ‘GSO_IPXIP4_BIT’, ‘GSO_IPXIP6_BIT’, ‘GSO_UDP_TUNNEL_BIT’, ‘GSO_PARTIAL_BIT’, ‘SCTP_CRC_BIT’, ‘RXHASH_BIT’, ‘RXCSUM_BIT’, ‘HW_TC_BIT’]

packets at egress:

GRO/GSO enabled

PHY

head_frag == 1, nr_frags == 0, end == 1728, headroom == 242 (data)

R2

##### CPU 1 buffer started #### ksoftirqd/1-20 [001] ..s.. 9.540379: iptfs_egress_recv: EGRESS: skb=000000006d1fa058 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=40 tail=1688 end=1728 head=ffff88810ea5a800 hdpgaddr=ffff88810ea5a000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

ksoftirqd/1-20 [001] ..s.. 9.540402: iptfs_egress_recv: EGRESS: skb=00000000156d5d79 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=7558 tailroom=40 tail=1688 end=1728 head=ffff88810ea5b800 hdpgaddr=ffff88810ea5b000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540419: iptfs_egress_recv: EGRESS: skb=00000000c70110b1 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=6116 tailroom=40 tail=1688 end=1728 head=ffff88810ea5c800 hdpgaddr=ffff88810ea5c000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540436: iptfs_egress_recv: EGRESS: skb=000000001ed1d59d len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=4674 tailroom=40 tail=1688 end=1728 head=ffff88810ea5d800 hdpgaddr=ffff88810ea5d000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540453: iptfs_egress_recv: EGRESS: skb=00000000eb724e7f len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232 tailroom=40 tail=1688 end=1728 head=ffff88810ea5e800 hdpgaddr=ffff88810ea5e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540470: iptfs_egress_recv: EGRESS: skb=0000000014591435 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790 tailroom=40 tail=1688 end=1728 head=ffff88810ea5f800 hdpgaddr=ffff88810ea5f000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540487: iptfs_egress_recv: EGRESS: skb=000000000ad8eb58 len=352 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=348 tailroom=1134 tail=594 end=1728 head=ffff88810ea60800 hdpgaddr=ffff88810ea60000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 9.540505: iptfs_egress_recv: EGRESS: skb=000000002453d179 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=40 tail=1688 end=1728 head=ffff88810ea61800 hdpgaddr=ffff88810ea61000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

NON

R1

ksoftirqd/0-14 [000] ..s1. 8.144958: iptfs_egress_recv: EGRESS: skb=000000001d284ad6 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0 tailroom=0 tail=114 end=704 head=ffff88801b667800 hdpgaddr=ffff88801b667000 pg0->addr=ffff888011520000 pg0->data=ffff88801152603e pg0->off=24638 ksoftirqd/0-14 [000] ..s1. 8.144981: iptfs_egress_recv: EGRESS: skb=000000007791ad14 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=7558 tailroom=0 tail=114 end=704 head=ffff88801b667400 hdpgaddr=ffff88801b667000 pg0->addr=ffff888011520000 pg0->data=ffff88801152663e pg0->off=26174 ksoftirqd/0-14 [000] ..s1. 8.145119: iptfs_egress_recv: EGRESS: skb=000000004969634f len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=6116 tailroom=0 tail=114 end=704 head=ffff88801b66e400 hdpgaddr=ffff88801b66e000 pg0->addr=ffff888011520000 pg0->data=ffff888011526c3e pg0->off=27710 ksoftirqd/0-14 [000] ..s1. 8.145140: iptfs_egress_recv: EGRESS: skb=00000000315dd6bd len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=4674 tailroom=0 tail=114 end=704 head=ffff88801b66e000 hdpgaddr=ffff88801b66e000 pg0->addr=ffff888011520000 pg0->data=ffff88801152723e pg0->off=29246 ksoftirqd/0-14 [000] ..s.. 8.145163: iptfs_egress_recv: EGRESS: skb=00000000695e660c len=1446 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232 tailroom=220 tail=1508 end=1728 head=ffff888011527800 hdpgaddr=ffff888011520000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s1. 8.145184: iptfs_egress_recv: EGRESS: skb=00000000702a4cee len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=1790 tailroom=0 tail=114 end=704 head=ffff88801b673c00 hdpgaddr=ffff88801b673000 pg0->addr=ffff8880119e8000 pg0->data=ffff8880119e803e pg0->off=62 ksoftirqd/0-14 [000] ..s.. 8.145206: iptfs_egress_recv: EGRESS: skb=00000000f3979ed2 len=352 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=348 tailroom=802 tail=414 end=1216 head=ffff8880119e8600 hdpgaddr=ffff8880119e8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s1. 8.145225: iptfs_egress_recv: EGRESS: skb=000000005abe0cf1 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0 tailroom=0 tail=114 end=704 head=ffff88801b673800 hdpgaddr=ffff88801b673000 pg0->addr=ffff8880119e8000 pg0->data=ffff8880119e8c3e pg0->off=3134

GRO/GSO disabled

R1

NON: small packets with a small amounts of aggregation it seems

head_frag == 1, nr_frags == 0, end == 1728, headroom == 242 (data)

ksoftirqd/0-14 [000] ..s.. 8.006165: iptfs_egress_recv: EGRESS: skb=00000000345e5161 len=368 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=786 tail=430 end=1216 head=ffff8880212db000 hdpgaddr=ffff8880212d8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.006240: iptfs_egress_recv: EGRESS: skb=00000000fec9351c len=56 data_len=0 headroom=114 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=534 tail=170 end=704 head=ffff888009cd4000 hdpgaddr=ffff888009cd4000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.006254: iptfs_egress_recv: EGRESS: skb=000000009a3b8aab len=56 data_len=0 headroom=114 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=534 tail=170 end=704 head=ffff888009926c00 hdpgaddr=ffff888009926000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.006268: iptfs_egress_recv: EGRESS: skb=0000000083f12223 len=108 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1046 tail=170 end=1216 head=ffff8880212dc200 hdpgaddr=ffff8880212d8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

NON: [000] ..s1. 15.334429: iptfs_egress_recv: EGRESS: skb=00000000029a7314 len=1412 data_len=1412 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0 tailroom=0 tail=114 end=704 head=ffff888019c8c400 hdpgaddr=ffff888019c8c000 pg0->addr=ffff888020280000 pg0->data=ffff88802028243e pg0->off=9278

PHY

PHY: [001] ..s1. 7.412970: iptfs_egress_recv: EGRESS: skb=0000000043420e6b len=212 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1274 tail=454 end=1728 head=ffff88810eb0e000 hdpgaddr=ffff88810eb0e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

ksoftirqd/0-14 [000] ..s.. 8.411730: iptfs_egress_recv: EGRESS: skb=00000000de2755bc len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1430 tail=298 end=1728 head=ffff88810eb6e000 hdpgaddr=ffff88810eb6e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

ksoftirqd/0-14 [000] ..s.. 8.411745: iptfs_egress_recv: EGRESS: skb=00000000dcac57c7 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1430 tail=298 end=1728 head=ffff88810eb6f000 hdpgaddr=ffff88810eb6f000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.411759: iptfs_egress_recv: EGRESS: skb=00000000e5e49dd8 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1430 tail=298 end=1728 head=ffff88810eb70000 hdpgaddr=ffff88810eb70000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.411774: iptfs_egress_recv: EGRESS: skb=000000002f5710b4 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1430 tail=298 end=1728 head=ffff88810eb71000 hdpgaddr=ffff88810eb71000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.411788: iptfs_egress_recv: EGRESS: skb=00000000ca57c05a len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1430 tail=298 end=1728 head=ffff88810eb72000 hdpgaddr=ffff88810eb72000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/0-14 [000] ..s.. 8.411802: iptfs_egress_recv: EGRESS: skb=00000000d813b429 len=108 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1378 tail=350 end=1728 head=ffff88810eb73000 hdpgaddr=ffff88810eb73000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

R2

NON:

ksoftirqd/0-14 [000] ..s1. 8.180352: iptfs_egress_recv: EGRESS: skb=00000000acf45ccb len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0 tailroom=0 tail=114 end=704 head=ffff88801a90b400 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214da3e pg0->off=23102

ksoftirqd/0-14 [000] ..s1. 8.180379: iptfs_egress_recv: EGRESS: skb=0000000097b1ffcb len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=7558 tailroom=0 tail=114 end=704 head=ffff88801a90b000 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214e03e pg0->off=24638

ksoftirqd/0-14 [000] ..s1. 8.180481: iptfs_egress_recv: EGRESS: skb=0000000090088909 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=6116 tailroom=0 tail=114 end=704 head=ffff88801a90b800 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214e63e pg0->off=26174

ksoftirqd/0-14 [000] ..s1. 8.180503: iptfs_egress_recv: EGRESS: skb=00000000cc18d723 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=4674 tailroom=0 tail=114 end=704 head=ffff88801a90b400 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214ec3e pg0->off=27710

ksoftirqd/0-14 [000] ..s1. 8.180525: iptfs_egress_recv: EGRESS: skb=000000008a12301d len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=3232 tailroom=0 tail=114 end=704 head=ffff88801a90b000 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214f23e pg0->off=29246

ksoftirqd/0-14 [000] ..s.. 8.180546: iptfs_egress_recv: EGRESS: skb=00000000e59cfe58 len=1446 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790 tailroom=220 tail=1508 end=1728 head=ffff88801214f800 hdpgaddr=ffff888012148000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

ksoftirqd/0-14 [000] ..s.. 8.180574: iptfs_egress_recv: EGRESS: skb=0000000079b524b0 len=352 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=348 tailroom=802 tail=414 end=1216 head=ffff888011998000 hdpgaddr=ffff888011998000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

PHY

ksoftirqd/1-20 [001] ..s.. 7.647452: iptfs_egress_recv: EGRESS: skb=0000000006bf1593 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=40 tail=1688 end=1728 head=ffff88810ea02000 hdpgaddr=ffff88810ea02000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647479: iptfs_egress_recv: EGRESS: skb=00000000b9ac19cc len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=7558 tailroom=40 tail=1688 end=1728 head=ffff88810ea03000 hdpgaddr=ffff88810ea03000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647500: iptfs_egress_recv: EGRESS: skb=00000000bac669ce len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=6116 tailroom=40 tail=1688 end=1728 head=ffff88810ea04000 hdpgaddr=ffff88810ea04000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647578: iptfs_egress_recv: EGRESS: skb=0000000052a43342 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=4674 tailroom=40 tail=1688 end=1728 head=ffff88810ea05000 hdpgaddr=ffff88810ea05000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647599: iptfs_egress_recv: EGRESS: skb=000000009e3c3cb2 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232 tailroom=40 tail=1688 end=1728 head=ffff88810ea06000 hdpgaddr=ffff88810ea06000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647620: iptfs_egress_recv: EGRESS: skb=00000000afd492e1 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790 tailroom=40 tail=1688 end=1728 head=ffff88810ea07000 hdpgaddr=ffff88810ea07000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0 ksoftirqd/1-20 [001] ..s.. 7.647643: iptfs_egress_recv: EGRESS: skb=000000009e9f8574 len=352 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=348 tailroom=1134 tail=594 end=1728 head=ffff88810ea08000 hdpgaddr=ffff88810ea08000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0