Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wg Tunnelblockade #100

Open
1977er opened this issue Jun 3, 2023 · 11 comments
Open

wg Tunnelblockade #100

1977er opened this issue Jun 3, 2023 · 11 comments

Comments

@1977er
Copy link
Member

1977er commented Jun 3, 2023 •
edited by lemoer
Loading

Symptoms:

  • Packets going into the wg interface on the supernode stop arriving on the node at the mesh_wg interface. (tcpdump -ni mesh_wg inbound).
  • Packets are still visible on br-wan.
  • wg handshakes are renewed.
  • CPU load drops significantly (to <20% on ER-X).
  • Memory load increases by a small but stable base amount.

Resulting symptoms:

  • Batman VPN Neighbors goes down to 0.
  • TX Link Quality to Supernodes (aka. TQ) no longer gets telemetry.
  • RX Link Quality from Supernodes (aka. reversed TQ) goes down to <5%.
  • Forward TX Traffic drops to 0.

See from:

Actions taken so far:

  • Monitoring checks if vpn_neighbors=0 (only works for sites with multiple uplinks).

(Edit by @lemoer: translated using deepl to english)
@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

Debug output from today:

The Situation

  • The two peers are UFU-FWH-E106-Woermannstr-Technik1 and sn10.
  • They have an established connection since approx 17 days.
  • Since 5 days (2023-06-17 15:44), the connection is broken.
  • Broken means:
    • Packets entering the wg interface on UFU-FWH-E106-Woermannstr-Technik1 are dropped somewhere before they come out of the wg interface on sn10.
    • The other way around seems to be ok.

UFU-FWH-E106-Woermannstr-Technik1

wg show on UFU-FWH-E106-Woermannstr-Technik1

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# wg show
interface: wg_mesh
  public key: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  private key: (hidden)
  listening port: 52231
  fwmark: 0x1

peer: SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
  endpoint: 81.3.6.91:51816
  allowed ips: fe80::1/128
  latest handshake: 2 seconds ago
  transfer: 1.75 TiB received, 192.12 GiB sent

Outbound on UFU-FWH-E106-Woermannstr-Technik1 wireguard interface

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i wg_mesh -c 5 outbound 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:02:22.415229 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 162: 
	0x0000:  000f 2c00 11d1 0c62 d65e 5062 8583 be18  ..,....b.^Pb....
	0x0010:  0305 c0b0 0093 0038 0401 0028 013e 0003  .......8...(.>..
	0x0020:  a145 edb7 8001 0000 e879 9090 8000 0000  .E.......y......
	0x0030:  618f fa0b 0000 0000 1300 0000 3007 4de1  a...........0.M.
	0x0040:  3f73 0000 0602 0004 0100 0000 0201 0000  ?s..............
	0x0050:  000f 2a00 fe08 c294 a6ef 7a86 2ed3 be18  ..*.......z.....
	0x0060:  0305 c0b0 0068 002c 0401 001c 0197 0003  .....h.,........
	0x0070:  041e 320d 8001 0000 4d22 4f2a 8000 0000  ..2.....M"O*....
	0x0080:  d6e4 041b 0000 0000 0602 0004 0100 0000  ................
	0x0090:  0201 0000                                ....
21:02:22.425599 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 150: 
	0x0000:  000f 3100 ba50 2c2a be18 0305 c0b3 be18  ..1..P,*........
	0x0010:  0305 c0b0 00f0 002c 0401 001c 0141 0003  .......,.....A..
	0x0020:  b244 398b 8001 0000 fb78 44ac 8000 0000  .D9......xD.....
	0x0030:  7e27 3918 0000 0000 0602 0004 0100 0000  ~'9.............
	0x0040:  0201 0000 000f 2b00 e9ad 07e2 be2f 2c4b  ......+....../,K
	0x0050:  bbc3 be18 0305 c0b0 0081 002c 0401 001c  ...........,....
	0x0060:  016b 0003 65ad 73c8 8001 0000 2c91 0eef  .k..e.s.....,...
	0x0070:  8000 0000 0944 f095 0000 0000 0602 0004  .....D..........
	0x0080:  0100 0000 0201 0000                      ........
21:02:22.535510 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 218: 
	0x0000:  000f 2c00 f565 8fdf 5e87 400e b7d3 be18  ..,..e..^.@.....
	0x0010:  0305 c0b0 0089 002c 0401 001c 01ba 0003  .......,........
	0x0020:  d01b 8003 8001 0000 9927 fd24 8000 0000  .........'.$....
	0x0030:  568f 2ea2 0000 0000 0602 0004 0100 0000  V...............
	0x0040:  0201 0000 000f 2c00 5386 3a55 4e27 fe8a  ......,.S.:UN'..
	0x0050:  718b be18 0305 c0b0 0089 002c 0401 001c  q..........,....
	0x0060:  0128 0003 37ae 48e3 8001 0000 7e92 35c4  .(..7.H.....~.5.
	0x0070:  8000 0000 b682 98ae 0000 0000 0602 0004  ................
	0x0080:  0100 0000 0201 0000 000f 2b00 8021 66f5  ..........+..!f.
	0x0090:  8253 adf4 2fab be18 0305 c0b0 0070 002c  .S../........p.,
	0x00a0:  0401 001c 01a6 0003 2dd5 ff67 8001 0000  ........-..g....
	0x00b0:  64e9 8240 8000 0000 ccd7 5534 0000 0000  d..@......U4....
	0x00c0:  0602 0004 0100 0000 0201 0000            ............
21:02:22.635524 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 218: 
	0x0000:  000f 2b00 4dff 2907 3e39 c135 b7fb be18  ..+.M.).>9.5....
	0x0010:  0305 c0b0 0070 002c 0401 001c 013f 0003  .....p.,.....?..
	0x0020:  7dba 5544 8001 0000 3486 2863 8000 0000  }.UD....4.(c....
	0x0030:  1d10 2714 0000 0000 0602 0004 0100 0000  ..'.............
	0x0040:  0201 0000 000f 2c00 7854 616e 323d f939  ......,.xTan2=.9
	0x0050:  9dfb be18 0305 c0b0 0089 002c 0401 001c  ...........,....
	0x0060:  01ad 0003 d10e 6764 8001 0000 9832 1a43  ......gd.....2.C
	0x0070:  8000 0000 be01 52f0 0000 0000 0602 0004  ......R.........
	0x0080:  0100 0000 0201 0000 000f 2c00 cce9 3c46  ..........,...<F
	0x0090:  ce87 9658 284b be18 0305 c0b0 0089 002c  ...X(K.........,
	0x00a0:  0401 001c 0114 0003 7512 562b 8001 0000  ........u.V+....
	0x00b0:  3c2e 2b0c 8000 0000 6ecd 2eec 0000 0000  <.+.....n.......
	0x00c0:  0602 0004 0100 0000 0201 0000            ............
21:02:22.775439 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 366: 
	0x0000:  000f 2c00 a645 b3e6 9272 c34b f293 be18  ..,..E...r.K....
	0x0010:  0305 c0b0 0093 0038 0401 0028 01f5 0003  .......8...(....
	0x0020:  2602 c729 8001 0000 6f3e ba0e 8000 0000  &..)....o>......
	0x0030:  5d6d 6c89 0000 0000 1300 0000 8a70 8924  ]ml..........p.$
	0x0040:  48bf 0000 0602 0004 0100 0000 0201 0000  H...............
	0x0050:  000f 2c00 8151 cf97 96ea 7c9a 6603 be18  ..,..Q....|.f...
	0x0060:  0305 c0b0 0089 002c 0401 001c 013e 0003  .......,.....>..
	0x0070:  20db 4c0d 8001 0000 69e7 312a 8000 0000  ..L.....i.1*....
	0x0080:  f32a 46b0 0000 0000 0602 0004 0100 0000  .*F.............
	0x0090:  0201 0000 000f 2d00 5823 c893 5e0e 5730  ......-.X#..^.W0
	0x00a0:  b823 be18 0305 c0b0 009d 002c 0401 001c  .#.........,....
	0x00b0:  01d3 0003 1a00 6fd0 8001 0000 533c 12f7  ......o.....S<..
	0x00c0:  8000 0000 2c79 3702 0000 0000 0602 0004  ....,y7.........
	0x00d0:  0100 0000 0201 0000 000f 2b00 7dcd 93d0  ..........+.}...
	0x00e0:  2e69 250d 193b be18 0305 c0b0 007f 002c  .i%..;.........,
	0x00f0:  0401 001c 010a 0003 bd43 6cc4 8001 0000  .........Cl.....
	0x0100:  f47f 11e3 8000 0000 d20b 9440 0000 0000  ...........@....
	0x0110:  0602 0004 0100 0000 0201 0000 000f 2c00  ..............,.
	0x0120:  6390 8721 5270 2e88 d863 be18 0305 c0b0  c..!Rp...c......
	0x0130:  0089 002c 0401 001c 0102 0003 03df 976f  ...,...........o
	0x0140:  8001 0000 4ae3 ea48 8000 0000 f230 5043  ....J..H.....0PC
	0x0150:  0000 0000 0602 0004 0100 0000 0201 0000  ................
5 packets captured
6 packets received by filter
0 packets dropped by kernel

Inbound on UFU-FWH-E106-Woermannstr-Technik1 wireguard interface

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i wg_mesh -c 5 inbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:03:23.618209 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 114: 
	0x0000:  010f 2d00 0000 2ac5 a201 1cf5 7333 3333  ..-...*.....s333
	0x0010:  ff00 0001 6637 be3b 72af 86dd 6000 0000  ....f7.;r...`...
	0x0020:  0020 3aff fdca ffee 0008 0016 8512 352b  ..:...........5+
	0x0030:  9488 28c7 ff02 0000 0000 0000 0000 0001  ..(.............
	0x0040:  ff00 0001 8700 6f3d 0000 0000 fdca ffee  ......o=........
	0x0050:  0008 0016 0000 0000 0000 0001 0101 6637  ..............f7
	0x0060:  be3b 72af                                .;r.
21:03:23.653481 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 490: 
	0x0000:  000f 3000 f837 fd3c a6d4 fd2d 9e03 0653  ..0..7.<...-...S
	0x0010:  09f0 eb74 00c6 002c 0401 001c 0160 0003  ...t...,.....`..
	0x0020:  f3d5 1bea 8001 0000 bae9 66cd 8000 0000  ..........f.....
	0x0030:  c5fc 7380 0000 0000 0602 0004 0100 0000  ..s.............
	0x0040:  0201 0000 000f 2f00 e2a6 5d06 6ef6 27c9  ....../...].n.'.
	0x0050:  27db b27c aa6e fb33 0086 002c 0401 001c  '..|.n.3...,....
	0x0060:  0152 0003 820d d32d 8001 0000 cb31 ae0a  .R.....-.....1..
	0x0070:  8000 0000 ef6d d0cc 0000 0000 0602 0004  .....m..........
	0x0080:  0100 0000 0201 0000 000f 3000 81ba 667e  ..........0...f~
	0x0090:  bec0 cdff e61b 7a83 2589 f298 00c6 002c  ......z.%......,
	0x00a0:  0401 001c 0103 0003 a69f 614b 8001 0000  ..........aK....
	0x00b0:  efa3 1c6c 8000 0000 8508 f04f 0000 0000  ...l.......O....
	0x00c0:  0602 0004 0100 0000 0201 0000 000f 2f00  ............../.
	0x00d0:  a5ab 2750 2a98 271f 0e9b 7a83 2589 f298  ..'P*.'...z.%...
	0x00e0:  00ba 002c 0401 001c 0140 0003 27f0 3775  ...,.....@..'.7u
	0x00f0:  8001 0000 6ecc 4a52 8000 0000 b72d c5a4  ....n.JR.....-..
	0x0100:  0000 0000 0602 0004 0100 0000 0201 0000  ................
	0x0110:  000f 2f00 75f4 7fce 9612 a6c8 a613 7a83  ../.u.........z.
	0x0120:  2589 f298 00ba 002c 0401 001c 0176 0003  %......,.....v..
	0x0130:  7574 0d83 8001 0000 3c48 70a4 8000 0000  ut......<Hp.....
	0x0140:  95d4 cbc1 0000 0000 0602 0004 0100 0000  ................
	0x0150:  0201 0000 000f 2d00 acca d432 bef1 c258  ......-....2...X
	0x0160:  b1f3 6667 fc1d ed22 0074 002c 0401 001c  ..fg...".t.,....
	0x0170:  017a 0003 c764 8e65 8001 0000 8e58 f342  .z...d.e.....X.B
	0x0180:  8000 0000 e359 687f 0000 0000 0602 0004  .....Yh.........
	0x0190:  0100 0000 0201 0000 000f 3000 8646 5e23  ..........0..F^#
	0x01a0:  7aac 9fb8 bc53 4685 c781 1018 00d0 002c  z....SF........,
	0x01b0:  0401 001c 0186 0003 313e 1d5a 8001 0000  ........1>.Z....
	0x01c0:  7802 607d 8000 0000 0bdc e5e7 0000 0000  x.`}............
	0x01d0:  0602 0004 0100 0000 0201 0000            ............
21:03:23.776182 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 310: 
	0x0000:  000f 3000 64ef 5a0d da9e 7e27 9ef3 b27c  ..0.d.Z...~'...|
	0x0010:  aa6e fb33 00d3 002c 0401 001c 01d7 0003  .n.3...,........
	0x0020:  1e76 cfb6 8001 0000 574a b291 8000 0000  .v......WJ......
	0x0030:  539d 50d7 0000 0000 0602 0004 0100 0000  S.P.............
	0x0040:  0201 0000 000f 2f00 7710 04c8 5603 1486  ....../.w...V...
	0x0050:  5353 6667 fc1d ed22 00ba 0044 0401 0034  SSfg..."...D...4
	0x0060:  0142 0003 9d4e 74a0 8001 0000 d472 0987  .B...Nt......r..
	0x0070:  8000 0000 91e6 90b1 0000 0000 1300 0000  ................
	0x0080:  3007 4de1 3f73 0000 1100 0000 46a1 ad8f  0.M.?s......F...
	0x0090:  fe22 0000 0602 0004 0100 0000 0201 0000  ."..............
	0x00a0:  000f 2f00 0163 7fe2 6254 3fb7 0133 6667  ../..c..bT?..3fg
	0x00b0:  fc1d ed22 00ba 002c 0401 001c 01ac 0003  ..."...,........
	0x00c0:  f0b0 35ae 8001 0000 b98c 4889 8000 0000  ..5.......H.....
	0x00d0:  dbb4 fe5b 0000 0000 0602 0004 0100 0000  ...[............
	0x00e0:  0201 0000 000f 3000 f6a6 0519 127f 1db0  ......0.........
	0x00f0:  ba2b 6667 fc1d ed22 00c6 002c 0401 001c  .+fg..."...,....
	0x0100:  0121 0003 2ef5 89ec 8001 0000 67c9 f4cb  .!..........g...
	0x0110:  8000 0000 d20b 9440 0000 0000 0602 0004  .......@........
	0x0120:  0100 0000 0201 0000                      ........
21:03:23.788811 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
	0x0000:  010f 3100 0015 92b2 72e3 f0ca 68fc ffff  ..1.....r...h...
	0x0010:  ffff ffff 88e6 40ba a016 0806 0001 0800  ......@.........
	0x0020:  0604 0001 88e6 40ba a016 0a10 6401 0000  ......@.....d...
	0x0030:  0000 0000 0a10 6726                      ......g&
21:03:23.874988 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 390: 
	0x0000:  000f 3000 e243 f342 5232 e0d2 fcab 4afd  ..0..C.BR2....J.
	0x0010:  27db 26dc 00c6 0038 0401 0028 011f 0003  '.&....8...(....
	0x0020:  1a39 5905 8001 0000 5305 2422 8000 0000  .9Y.....S.$"....
	0x0030:  dac6 e871 0000 0000 1100 0000 56f2 6d23  ...q........V.m#
	0x0040:  e2a2 0000 0602 0004 0100 0000 0201 0000  ................
	0x0050:  000f 2f00 cd58 c365 02a6 da6c 641b 6667  ../..X.e...ld.fg
	0x0060:  fc1d ed22 00ba 002c 0401 001c 016b 0003  ..."...,.....k..
	0x0070:  adcd 2363 8001 0000 e4f1 5e44 8000 0000  ..#c......^D....
	0x0080:  ac42 52b0 0000 0000 0602 0004 0100 0000  .BR.............
	0x0090:  0201 0000 000f 3100 ab2e 0529 ce9e 4182  ......1....)..A.
	0x00a0:  b5b3 42ae 25e5 5274 00cf 0044 0401 0034  ..B.%.Rt...D...4
	0x00b0:  0110 0003 9a71 8cb2 8001 0000 d34d f195  .....q.......M..
	0x00c0:  8000 0000 7a4e b02a 0000 0000 0100 0000  ....zN.*........
	0x00d0:  3333 ff78 57d3 0000 0100 0000 3333 ff6c  33.xW.......33.l
	0x00e0:  2ab9 0000 0602 0004 0100 0000 0201 0000  *...............
	0x00f0:  000f 3000 cf81 3f35 2637 87d9 76fb 4afd  ..0...?5&7..v.J.
	0x0100:  27db 26dc 00c6 002c 0401 001c 01bb 0003  '.&....,........
	0x0110:  a669 7925 8001 0000 ef55 0402 8000 0000  .iy%.....U......
	0x0120:  f34d db32 0000 0000 0602 0004 0100 0000  .M.2............
	0x0130:  0201 0000 000f 3000 8230 55ee 5e03 c2cf  ......0..0U.^...
	0x0140:  9923 2aab 8a15 cdcd 00c3 002c 0401 001c  .#*........,....
	0x0150:  0125 0003 3192 e5ae 8001 0000 78ae 9889  .%..1.......x...
	0x0160:  8000 0000 3f6b 668e 0000 0000 0602 0004  ....?kf.........
	0x0170:  0100 0000 0201 0000                      ........
5 packets captured
6 packets received by filter
0 packets dropped by kernel

sn10

wg show on sn10

[root@sn10]:~ # wg show  wg-16 | grep lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE= -A 5
peer: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  endpoint: 89.183.250.69:52231
  allowed ips: fe80::277:2bff:fe2a:2f28/128
  latest handshake: 10 seconds ago
  transfer: 188.98 GiB received, 1.77 TiB sent

Inbound on wireguard interface of sn10

(No packets appear here.)

[root@sn10]:~ # timeout 10 tcpdump -n -i wg-16 host fe80::277:2bff:fe2a:2f28 and inbound
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg-16, link-type RAW (Raw IP), snapshot length 262144 bytes

0 packets captured
44 packets received by filter
0 packets dropped by kernel

Outbound on wireguard interface on sn10

[root@sn10]:~ # tcpdump -n -i wg-16 host fe80::277:2bff:fe2a:2f28 and outbound -c 5
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg-16, link-type RAW (Raw IP), snapshot length 262144 bytes
21:11:49.536700 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 354: 
	0x0000:  000f 2f00 656b 5815 8239 d50c 8c33 6667  ../.ekX..9...3fg
	0x0010:  fc1d ed22 00b3 002c 0401 001c 01fb 0003  ..."...,........
	0x0020:  74ee b3f6 8001 0000 3dd2 ced1 8000 0000  t.......=.......
	0x0030:  80dd eb54 0000 0000 0602 0004 0100 0000  ...T............
	0x0040:  0201 0000 000f 3000 32dc 47a1 a2b5 e36f  ......0.2.G....o
	0x0050:  714b 2aab 8a15 cdcd 00c3 002c 0401 001c  qK*........,....
	0x0060:  012b 0003 4587 3e1d 8001 0000 0cbb 433a  .+..E.>.......C:
	0x0070:  8000 0000 b5f1 b26a 0000 0000 0602 0004  .......j........
	0x0080:  0100 0000 0201 0000 000f 2f00 0bef 4b93  ........../...K.
	0x0090:  8a35 5004 e96b 7a83 2589 f298 00ba 002c  .5P..kz.%......,
	0x00a0:  0401 001c 011b 0003 507a e980 8001 0000  ........Pz......
	0x00b0:  1946 94a7 8000 0000 71c3 b36c 0000 0000  .F......q..l....
	0x00c0:  0602 0004 0100 0000 0201 0000 000f 3000  ..............0.
	0x00d0:  933e 99b8 56aa 2418 4243 4685 c781 1018  .>..V.$.BCF.....
	0x00e0:  00d3 002c 0401 001c 0130 0003 aed4 43ea  ...,.....0....C.
	0x00f0:  8001 0000 e7e8 3ecd 8000 0000 150d 5812  ......>.......X.
	0x0100:  0000 0000 0602 0004 0100 0000 0201 0000  ................
	0x0110:  000f 2d00 6ef9 1b1c 9623 d51e 6b5b 6667  ..-.n....#..k[fg
	0x0120:  fc1d ed22 0087 002c 0401 001c 01dc 0003  ..."...,........
	0x0130:  76c5 732a 8001 0000 3ff9 0e0d 8000 0000  v.s*....?.......
	0x0140:  cda5 9b58 0000 0000 0602 0004 0100 0000  ...X............
	0x0150:  0201 0000                                ....
21:11:49.635886 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 230: 
	0x0000:  000f 3000 1342 8ae9 7a0e 21c0 f8f3 2aab  ..0..B..z.!...*.
	0x0010:  8a15 cdcd 00c6 002c 0401 001c 0184 0003  .......,........
	0x0020:  0701 df09 8001 0000 4e3d a22e 8000 0000  ........N=......
	0x0030:  241b 49e2 0000 0000 0602 0004 0100 0000  $.I.............
	0x0040:  0201 0000 000f 3000 83de 3a5e e6be ec5f  ......0...:^..._
	0x0050:  508b 42ae 25e5 5274 00c5 0038 0401 0028  P.B.%.Rt...8...(
	0x0060:  0140 0003 17aa e8d4 8001 0000 5e96 95f3  .@..........^...
	0x0070:  8000 0000 a671 9ed1 0000 0000 1000 0000  .....q..........
	0x0080:  e25c 0513 1a4b 0000 0602 0004 0100 0000  .\...K..........
	0x0090:  0201 0000 000f 3000 c33b 8c8e b2ec 4f8b  ......0..;....O.
	0x00a0:  0f6b 0653 09f0 eb74 00c6 002c 0401 001c  .k.S...t...,....
	0x00b0:  0145 0003 32c4 9e53 8001 0000 7bf8 e374  .E..2..S....{..t
	0x00c0:  8000 0000 11bb e639 0000 0000 0602 0004  .......9........
	0x00d0:  0100 0000 0201 0000                      ........
21:11:49.672427 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 114: 
	0x0000:  010f 2e00 0000 0a4f 5a51 32cd 36ab 3333  .......OZQ2.6.33
	0x0010:  ff00 0001 1ccc d695 a8e6 86dd 6000 0000  ............`...
	0x0020:  0020 3aff fdca ffee 0008 0016 f042 8924  ..:..........B.$
	0x0030:  9bd7 e6d0 ff02 0000 0000 0000 0000 0001  ................
	0x0040:  ff00 0001 8700 e594 0000 0000 fdca ffee  ................
	0x0050:  0008 0016 0000 0000 0000 0001 0101 1ccc  ................
	0x0060:  d695 a8e6                                ....
21:11:49.682309 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
	0x0000:  010f 2c00 0000 6787 6254 3fb7 0133 ffff  ..,...g.bT?..3..
	0x0010:  ffff ffff e076 d040 3f17 0806 0001 0800  .....v.@?.......
	0x0020:  0604 0001 e076 d040 3f17 0a10 5a8f 0000  .....v.@?...Z...
	0x0030:  0000 0000 0a10 5ad8                      ......Z.
21:11:49.683157 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
	0x0000:  010f 2c00 0000 6786 6254 3fb7 0133 ffff  ..,...g.bT?..3..
	0x0010:  ffff ffff e076 d040 3f17 0806 0001 0800  .....v.@?.......
	0x0020:  0604 0001 e076 d040 3f17 0a10 5a8f 0000  .....v.@?...Z...
	0x0030:  0000 0000 0a10 5a57                      ......ZW
5 packets captured
63 packets received by filter
0 packets dropped by kernel

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023

Currently, it seems that only handshakes are sent out at UFU-FWH-E106-Woermannstr-Technik1:

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i br-wan outbound and host 81.3.6.91
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-wan, link-type EN10MB (Ethernet), capture size 262144 bytes
21:27:47.011731 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:02.369570 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:17.792572 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:33.352853 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:48.973022 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92

And all of them seem to appear on sn10:

[root@sn10]:~ # tcpdump -n -i any host 89.183.250.69 and inbound
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
21:27:47.020876 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:02.381791 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:17.833270 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:33.362380 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:48.982652 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

It also doesn't seem to be related to the vxlan traffic that we are sending into the wg interface. If we do a normal ping, we can also just see the handshakes on br-wan:

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# ping fe80::1%wg_mesh
PING fe80::1%wg_mesh (fe80::1%16): 56 data bytes


root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i wg_mesh outbound and icmp6 -v
tcpdump: listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:41:55.055297 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 161
21:41:56.065291 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 162
21:41:57.075316 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 163
21:41:58.085348 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 164
21:41:59.095278 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 165
21:42:00.105287 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 166
21:42:01.115318 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 167
21:42:02.125331 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 168
21:42:03.135295 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 169
21:42:04.145293 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 170
21:42:05.155292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 171
21:42:06.165324 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 172
21:42:07.175321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 173
21:42:08.185316 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 174
21:42:09.195293 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 175
21:42:10.205302 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 176
21:42:11.215292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 177
21:42:12.225289 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 178
21:42:13.235292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 179
21:42:14.245295 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 180
21:42:15.255438 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 181
21:42:16.265321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 182
21:42:17.275321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 183
21:42:18.285300 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 184

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i br-wan outbound and host 81.3.6.91
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-wan, link-type EN10MB (Ethernet), capture size 262144 bytes
21:41:55.388648 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:42:10.527058 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92

wg show still shows a valid handshake:

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg show
interface: wg_mesh
  public key: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  private key: (hidden)
  listening port: 52231
  fwmark: 0x1

peer: SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
  endpoint: 81.3.6.91:51816
  allowed ips: fe80::1/128
  latest handshake: 14 seconds ago
  transfer: 1.75 TiB received, 192.12 GiB sent

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

Here is the config dump from UFU-FWH-E106-Woermannstr-Technik1:

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg showconf wg_mesh
[Interface]
ListenPort = 52231
FwMark = 0x1
PrivateKey = (redacted)

[Peer]
PublicKey = SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
AllowedIPs = fe80::1/128
Endpoint = 81.3.6.91:51816

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

Also, the TX counter of wireguard shows that packets are not sent out (second last column):

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg show all dump; sleep 15; wg show all dump
wg_mesh	(redacted)	lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=	52231	0x1
wg_mesh	SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=	(none)	81.3.6.91:51816	fe80::1/128	1687464100	1920192174532	206288920592	off
wg_mesh	(redacted)	lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=	52231	0x1
wg_mesh	SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=	(none)	81.3.6.91:51816	fe80::1/128	1687464115	1920192233560	206288920684	off

Only 92 bytes are seen in 15 seconds only. This is excactly the size of one handshake packet.

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023

Also an inbound trace on UFU-WFH-E106-Technik1 on br-wan:

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i any host 81.3.6.91 and inbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:12:53.556666 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.556714 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.592908 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.592959 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.651267 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.651317 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.723720 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.723768 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.723923 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.723942 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.753076 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.753132 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.856151 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 384
22:12:53.856198 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 384
22:12:53.856321 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.856346 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.951600 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:53.951648 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.042231 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 192
22:12:54.042275 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 192
22:12:54.042389 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042404 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042455 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042478 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.059258 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.059304 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.204977 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.205059 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.335885 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.335933 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.340871 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.340925 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.519126 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.519190 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.519374 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 256
22:12:54.519394 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 256
22:12:54.519446 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 224
22:12:54.519461 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 224
22:12:54.538861 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.538910 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.580684 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.580730 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.670918 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.670967 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
^C
44 packets captured
44 packets received by filter
0 packets dropped by kernel

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

I just found this in dmesg:

[373474.182938] ------------[ cut here ]------------
[373474.192353] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:624 rcu_eqs_enter.constprop.0+0xd8/0xe0
[373474.209825] Modules linked in: iptable_nat batman_adv xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_CT wireguard nf_nat nf_conntrack libchacha20poly1305 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_quota xt_pkttype xt_owner xt_multiport xt_mark xt_mac xt_limit xt_comment xt_addrtype xt_TCPMSS xt_LOG ts_kmp ts_fsm ts_bm poly1305_mips nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic iptable_mangle iptable_filter ip_tables ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_ip6 ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 compat chacha_mips sch_teql sch_sfq sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_pedit act_csum libcrc32c sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow act_skbedit act_mirred act_gact ip6table_mangle ip6table_filter ip6_tables
[373474.210230]  ip6t_REJECT x_tables nf_reject_ipv6 dummy ip_tunnel veth vxlan udp_tunnel ip6_udp_tunnel kpp leds_gpio cls_basic sch_tbf sch_ingress gpio_button_hotplug crc32c_generic
[373474.415982] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.161 #0
[373474.428266] Stack : 809a0000 00000000 00000000 8007c664 80830000 8072d760 00000000 00000000
[373474.445079]         807b3d24 80980000 806fdc74 807c7298 807c6dc7 00000001 807b3cc8 6f3afccb
[373474.461890]         00000000 00000000 806fdc74 807b3b68 ffffefff 00000000 ffffffea 00000000
[373474.478702]         807b3b74 0000017a 807cb9e0 ffffffff 00000000 00000000 00000000 80700000
[373474.495513]         00000009 3c016ed8 000153ac 00000000 00000000 803a42ec 00000000 80980000
[373474.512324]         ...
[373474.517357] Call Trace:
[373474.522404] [<80007fc0>] show_stack+0x30/0x100
[373474.531422] [<8031f7c8>] dump_stack+0x9c/0xcc
[373474.540272] [<8002fe64>] __warn+0xc0/0x12c
[373474.548587] [<8002ff2c>] warn_slowpath_fmt+0x5c/0xac
[373474.558629] [<80656d6c>] rcu_eqs_enter.constprop.0+0xd8/0xe0
[373474.570058] [<8045dde4>] cpuidle_enter_state+0x410/0x540
[373474.580790] [<8045dfac>] cpuidle_enter+0x84/0xac
[373474.590163] [<8005e860>] do_idle+0x26c/0x31c
[373474.598824] [<8005eb88>] cpu_startup_entry+0x2c/0x34
[373474.608887] [<80848d00>] start_kernel+0x570/0x598
[373474.618408] 
[373474.621532] ---[ end trace 74d97155b98095dd ]---

Not sure if this is related, since this happened 15 days ago.

@lemoer
Copy link
Contributor

lemoer commented Jun 22, 2023 •
edited
Loading

Listening on any interface, also just shows the handshakes:

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -ni any host 81.3.6.91 and outbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:53:01.899449 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:01.899506 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:01.899528 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252236 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252308 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252336 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634665 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634747 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634779 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233387 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233449 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233470 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588697 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588774 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588801 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947856 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947911 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947933 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
^C
18 packets captured
18 packets received by filter
0 packets dropped by kernel

I suppose every handshake is shown three times, since I have a linux bridge (or so):

root@UFU-FWH-E106-Woermannstr-Technik1:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br-mesh_other		7fff.eaf340edfbcc	no		eth3
							eth1
							eth4
							eth2
br-client		7fff.74acb9a77737	no		bat0
							local-port
br-wan		7fff.74acb9a77737	no		eth0

Unfortunately, the tcpdump doesn't show the interfaces where it saw the packets.

@1977er
Copy link
Member Author

1977er commented Jan 19, 2025

Todays debugging session showed, that the direction of "its not working vs. its working" may be interchanged, too. Today it was not possible to ping fe80::1%wg_mesh. tcpdump showed that the reply packet is withdrawn somewhere on the way back from the supernode.

@lemoer
Copy link
Contributor

lemoer commented Jan 29, 2025 •
edited
Loading

From today's debug session:

https://pad.leinelab.org/mKN6BNbHRIi5J7GPChw2FQ
https://pad.leinelab.org/aIZ6Q10kSNO0rGbwYugTrQ

Especially interesting thereby:


root@UFU-FWH-E133-Vinnhorst-Technik-1:~# while true; do ip -s link show dev wg_mesh | grep 
RX -A 1; sleep 1; done
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168584 179452429      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168676 179452430      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168768 179452431      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0 
    RX:    bytes   packets errors dropped  missed   mcast           
    214080168860 179452432      0       0       0       0

^ RX Packet increases every 30 seconds by 1 packet.

root@UFU-FWH-E133-Vinnhorst-Technik-1:~# conntrack -L | grep 2a02:790:1:ff::9001; sleep 10;
conntrack -L | grep 2a02:790:1:ff::9001
conntrack v1.4.8 (conntrack-tools): 35 flow entries have been shown.
udp 17 180 src=2003:e4:4f3a:7000:76ac:b9ff:fea7:8e77 dst=2a02:790:1:ff::9001 sport=40961 dport=51818 packets=4315 bytes=1935908 src=2a02:790:1:ff::9001 dst=2003:e4:4f3a:7000:76ac:b9ff:fea7:8e77 sport=51818 dport=40961 packets=9994 bytes=3242956 [ASSURED] use=1
conntrack v1.4.8 (conntrack-tools): 35 flow entries have been shown.
udp 17 179 src=2003:e4:4f3a:7000:76ac:b9ff:fea7:8e77 dst=2a02:790:1:ff::9001 sport=40961 dport=51818 packets=4417 bytes=1979320 src=2a02:790:1:ff::9001 dst=2003:e4:4f3a:7000:76ac:b9ff:fea7:8e77 sport=51818 dport=40961 packets=10289 bytes=3329912 [ASSURED] use=1

=> Firewall sees approx 29.5 packets/s

@lemoer
Copy link
Contributor

lemoer commented Jan 29, 2025 •
edited
Loading

If this is seen the next time on a supernode (where the supernode get's deaf), please execute:

echo 'module wireguard +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

And capture logs using journalctl...

I just checked on sn10, that this should work. On the gluon nodes, this doen't work, because we do not have debug within wireguard there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lemoer @1977er