Truenas middleware no longer responds at all with a large qbittorrent instance

[SeiyaGame]

I carried out several tests over 2 weeks and realized that when I launch my docker qbittorrent instance via my jail, after a while my Truenas middleware no longer responds.

Here's my docker-compose.yaml file

services:
  gluetun:
    image: qmcgaw/gluetun
    container_name: gluetun
    cap_add:
      - NET_ADMIN
    ports:
      - 10095:10095 # qBittorrent Web UI
    environment:
      - PUID=568
      - PGID=568
      - VPN_SERVICE_PROVIDER=custom
      - VPN_TYPE=wireguard
      - VPN_ENDPOINT_IP=IP_VPN_PROVIDER_KEY_PUT_HERE
      - VPN_ENDPOINT_PORT=1443
      - FIREWALL_VPN_INPUT_PORTS=50025
      - WIREGUARD_PUBLIC_KEY=PUBLIC_KEY_PUT_HERE
      - WIREGUARD_PRIVATE_KEY=PRIVATE_KEY_PUT_HERE
      - WIREGUARD_ADDRESSES=10.13.128.113/24
      - DNS_KEEP_NAMESERVER=on
      - TZ=Europe/Paris
      - HTTPPROXY=off
      - SHADOWSOCKS=off
      - FIREWALL_OUTBOUND_SUBNETS=192.168.5.0/24 # Your Subnet
      - WIREGUARD_MTU=1390 # Default is 1500 but some provider change this for better performance
    volumes:
      - ./gluetun:/gluetun
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=1
    restart: unless-stopped
  qbittorrent:
    image: lscr.io/linuxserver/qbittorrent:latest
    container_name: qbittorrent
    deploy:
      resources:
        limits:
          cpus: "16.0"
          memory: 16G
    environment:
      - PUID=568
      - PGID=568
      - TZ=Europe/Paris
      - WEBUI_PORT=10095
      - TORRENTING_PORT=50025
    volumes:
      - ./config:/config
      - /mnt/shared/downloads/torrents:/shared/downloads/torrents
    network_mode: service:gluetun
    depends_on:
      gluetun:
        condition: service_healthy
    healthcheck:
      test: curl --fail -s https://www.google.fr || exit 1
      interval: 1m
      timeout: 10s
      retries: 1
    restart: unless-stopped

For my jail, I used the template here, and only passthrough the nvidia gpu.

Today, my qbittorrent instance has over 8500 torrents, it works fine with kubernetes but as I said above, when I start my qbit instance in the jail, my truenas become unresponsive over time, the whole system lags and all this doesn't happen instantly!

I turn off my qbit instance and the web interface (middleware) and the system of my truenas work perfectly (sometimes I need to restart the middleware anyway).

Being a bit desperate, I'm turning to you for an idea of why this is happening...

I can do tests if you wish and don't hesitate if you want more information!

[Jip-Hop]

That's a lot of torrents. Maybe the qbittorrent App or the Apps framework itself implements some kind of resource limits? Should be possible to do with systemd-nspawn too, see their docs. I never needed it myself.

[Lockszmith-GH]

With linux, journalctl is a fountain of (unfiltered) information.

To review the log from the previous boot, run:

journalctl --boot=-1 --all --reverse

This will list all of the messages from the previous boot in reverse order in a pager (less), you migth glean some insight on why things went bad.

[SeiyaGame]

Nothing special in the logs, I've already checked many times...
I found that my dockers in my jail use a lot of Swap memory, so I forced all my containers not to use it.
I see a small improvement in fluidity but it's not quite that.

When I want to take my jail in ssh, the connection can take 1 to 2 seconds to be established the first time, as if it had to be woken up.
I'll keep testing to find out what's wrong but this behavior is very strange :/

[Jip-Hop]

Good luck! Sounds like it's running out of some resources. But which one? Idk. Maybe a TCP connection limit?

[SeiyaGame]

I think that's exactly it, when I start my qbit instance with kubernetes I get this message:

Apr 30 15:08:52 SRV-NAS kernel: TCP: request_sock_TCP: Possible SYN flooding on port 10.13.128.113:50025. Sending cookies

I tried to solve the problem by tunning the kernel settings, but without success:

# I know is temporary setup like this
sysctl -w net.core.somaxconn=16384
sysctl -w net.ipv4.tcp_max_syn_backlog=16384
sysctl -w net.core.netdev_max_backlog=16384

Also Truechart said to add this kernel value:

# I know is temporary setup like this
sysctl -w fs.inotify.max_user_instances=8192	
sysctl -w fs.inotify.max_user_watches=1048576

But with my jail, I don't get the Possible SYN flooding message.
Do I need to apply the kernel settings inside the jail ? From what I understand, this isn't necessary

Two question is comming because I read the entire manpage systemd-nspawn

Duo to the fact for my qbit instance I use CAP_NET_ADMIN for my wireguard VPN, so do I need to add this settings to my jail ?

--capability=CAP_NET_ADMIN

Also do I need to modify RLIMIT ?

--rlimit=RLIMIT_NOFILE=32768
--rlimit=RLIMIT_NPROC=4096

root@SRV-DOCKER:~# ulimit -Ha
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 513382
max locked memory           (kbytes, -l) 16471004
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 513382
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@SRV-DOCKER:~# ulimit -Sa
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 513382
max locked memory           (kbytes, -l) 16471004
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 513382
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

but these settings don't solve the network latency problem I'm having :/

Do I need to switch this issue as discussion ?

[Jip-Hop]

I have converted the issue into a discussion. I don't think there's anything wrong with jailmaker which is causing this situation.

I don't use qbittorrent or wireguard in a jail so I don't know how to fix this.

[SeiyaGame]

I've found my problem, it's a Truenas Scale bug
For those interested in solving this problem :

• https://forums.truenas.com/t/dragonfish-swap-usage-high-memory-memory-leak-issues/3680/35
• https://www.reddit.com/r/truenas/comments/1crfzsx/truenas_scale_dragonfish_slow_ui_responsiveness/

The problem will be solved in the next version 24.04.1 and higher

Temporary fix :

echo n >/sys/kernel/mm/lru_gen/enabled
sysctl -w vm.swappiness=1

[SeiyaGame]

No more problems on my side, the middleware works well and without any latency !