[bug] new cgroups-related error on start

[mirabilos]

Describe the bug with a clear and concise description of what the bug is.

2024/06/09 20:38:43 ERROR failed to set GOMEMLIMIT package=github.com/KimMachineGun/automemlimit/memlimit error="failed to set GOMEMLIMIT: cgroups: cgroup mountpoint does not exist"

What's your GoToSocial Version?

GoToSocial 0.16.0-rc1+git-048339a 🦥

GoToSocial Arch

amd64 binary

What happened?

I upgraded, and this message on start is new.

What you expected to happen?

No response

How to reproduce it?

No response

Anything else we need to know?

This is Debian with sysvinit, so no cgroups in use or prepared for use.

[mirabilos]

From peeking at https://github.com/KimMachineGun/automemlimit?tab=readme-ov-file and https://tip.golang.org/doc/gc-guide#Memory_limit it sounds to me like everyone not running GtS inside cgroup’d Docker (with already-configured limits?) needs to set the memory limits themselves, so other programs running on the same machine (at least Apache httpd 2 reverse proxy and PostgreSQL, but often also other services) aren’t moved out?

If so, should there be a guide on how (export GOMEMLIMIT?) and sizing?

[mirabilos]

That being said, as-is it uses about 150 MiB RSS, which is fine.

[tsmethurst]

The only thing that's changed from previous releases is that this is now logged, but it really shouldn't be since it's not actually an error, so we just need to figure out how to suppress it to avoid confusing people.

should there be a guide on how (export GOMEMLIMIT?) and sizing?

Not sure how necessary a guide is. We have a little note here with some links: https://docs.gotosocial.org/en/latest/getting_started/#memory-and-cpu-limits. Likely the sort of users who'd want to create those sorts of constraints already know what they're doing.

[daenney]

From peeking at https://github.com/KimMachineGun/automemlimit?tab=readme-ov-file and https://tip.golang.org/doc/gc-guide#Memory_limit it sounds to me like everyone not running GtS inside cgroup’d Docker (with already-configured limits?) needs to set the memory limits themselves, so other programs running on the same machine (at least Apache httpd 2 reverse proxy and PostgreSQL, but often also other services) aren’t moved out?

The GOMEMLIMIT sets a maximum amount of memory the process may use, and by default that's unconstrained. But that's no different from Apache2 or PostgreSQL, which aren't constrained in their memory allocations or CPU usage either. Just because the Go runtime isn't constrained doesn't mean it's going to allocate or consume everything so you can leave the limit unset just fine. When all processes run without resource constraints you can end up in an OOM situation triggered by any of those processes, and the kernel will decide on its victim based on its own heuristics at that point.

When running in cgroups, whether manually managed or as a consequence of running a container, you may set resource constraints on the cgroup either directly or by specifying something like 256MB of memory, amount of vCPUs etc. when starting a container. However, running within a cgroup the Go runtime still sees the hosts' total available memory, total CPU etc. The Go runtime isn't cgroup-aware. This is unfortunate as it can cause the process to try to use resources beyond what it can use and get killed as the runtime ends up making incorrect decisions and isn't able to adjust the GC based on the actual resources available. By passing on any cgroup limits to the Go runtime, it becomes aware of the resource constraints its under and can tweak its behaviour to match.

[mirabilos]

Daenney dixit:

Just because the Go runtime isn't constrained doesn't mean it's going to allocate or consume everything so you can leave the limit unset just fine.

Right. I was concerned that some part of sizing wouldn’t work or something, and this was logged as an error. It behaves with this unset as well (ok, I upgraded to rc2 yesterday so it’s got under a day uptime atm, but 220 MiB is totally fine).

However, running within a cgroup the Go runtime still sees the hosts' total available memory, total CPU etc.

Ah, okay. So it’s for only that? Thanks, that explains it. So, perhaps adjust the “error” message, but otherwise, nothing to do (as the automemlimit code catches the cgroups case and the nōn-cgroups case doesn’t need handling)?

[daenney]

Yup, the library introduced a change that causes that error to be logged by them. We just need to silence their logger.

[daenney]

I've raised #3002 which should fix this. Feel free to try it out if you like!

[mirabilos]

That, or raise the issue with them, so they can exclude such errors.

But the idea of præfixing all messages from their logger when displaying them in the GtS log (and stripping this one) is also nice.

[mirabilos]

Yup, is fixed in rc3, thanks!