Clarification regarding swap: Possible to define swap volume?

[Kramerican]

LXD 3.16
Ubuntu Bionic

I've been reading up on swap limiting as I've encountered poor performance (unsurprisingly) in containers that need to swap to disk, as on my new systems the main OS is on spinning rust.

As the containers are zfs backed on an SSD, I've been looking around for information on whether it is possible to tell my containers/lxd to swap out to their zfs volume instead of the host OS disk?

I also tried for kicks and giggles to test out the limits.memory.swap = false option in my profile, but that seems to do nothing whatsoever. The container still swaps happily away to the host OS drive. Which I found interesting - possibly a kernel param I need to enable here?

lxc info shows no information regarding swap on my system - the container has 1GB of swap space available as seen in top/htop - where is this defined?

Thank you for your time as always :)

[Kramerican]

Ok as I've never had to fiddle with swap allocation before, I didn't know it was so easy to just create a volume on the zpool as a swap volume and then tell the host to use that instead of the disk. I basically just followed this guide:

https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device

Problem solved I guess :) I'll leave this open for now in case the experts have any further relevant information to add here.

[Kramerican]

OK maybe not solved after all. I found this issue:

openzfs/zfs#7734

And after doing my own stress test I encountered the deadlock as described in this issue. I am now considering buying a dedicated (small) SSD drive to hold my swap.

[stgraber]

Swap devices are global so there is currently no way to use a separate swap file or device for specific containers and it's unclear that it's something which would ever be practical to implement due to how the kernel decides what to move to swap (and whether it can always track this down to a particular container).

The main thing you can do through LXD is control the swapiness through limits.memory.swap.priority making it less likely for a container to hit the swap. Or even disable swap usage for the container (which effectively sets the swapiness to such a level that the kernel shouldn't swap anything for the container, unless the system is about to completely crash).

Swap files/devices are pretty picky. They can't be stored on most copy on write filesystems (btrfs, zfs, ...) and as you've noticed, even when using a block device, they can be a bit picky as is the case on top of zvol. A dedicated partition tends to work best.

Another option is using zram as swap, which effectively takes a bit of ram, compresses it and turns it into swap. That's sometimes a reasonable option for systems that aren't expected to hit the swap very often and yield reasonable performance so long as you don't completely run the system out of memory.

[Kramerican]

Just to add some details I've found out over the last few days:

• zram and zswap works, but not ideally so
• Best is to have a dedicated swap partition as normal and have it on an SSD drive. This is by far the fastest and most "standard" option.
• If you plan on having lots of memory-limited containers which may end up swapping, then you should probably consider giving your system lots of swap space (I've gone for a (s)whopping 80Gigs on my system)
• You may want to look into vm.swappiness and vm.vfs_cache_pressure - look here:

https://haydenjames.io/linux-performance-almost-always-add-swap-space/

• Turning off swap entirely is not recommended unless you enjoy having processes being killed in containers that run out of memory. In my case that's pretty bad customer service, so yeah, lots of swap space for me it is! :)