Issue starting on Ubuntu 18.04 for RPI3B+

[mihaigalos]

I'm having problems with log2ram on Ubuntu 18.04. Any help would be much appreciated!

➜  ~ systemctl status log2ram
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2019-08-10 18:36:11 UTC; 1min 29s ago
  Process: 1669 ExecStart=/usr/local/bin/log2ram start (code=exited, status=1/FAILURE)
 Main PID: 1669 (code=exited, status=1/FAILURE)
➜  ~ sudo service log2ram restart
Job for log2ram.service failed because the control process exited with error code.
See "systemctl status log2ram.service" and "journalctl -xe" for details.
➜  ~ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            440M     0  440M   0% /dev
tmpfs            92M  4.4M   87M   5% /run
/dev/mmcblk0p2   28G  1.7G   26G   7% /
tmpfs           457M     0  457M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           457M     0  457M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   54M  199M  22% /boot/firmware
tmpfs            92M     0   92M   0% /run/user/1001
➜  ~ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[mihaigalos]

no mention of log2ram in output of journalctl -xe

[mihaigalos]

OK. Solved it.

➜  ~ sudo cat /var/log/syslog | grep log2ram
Aug 10 18:13:56 bambam log2ram[1068]: ERROR: RAM disk too small. Can't sync.
➜  ~ sudo du -sch /var/log
211M	/var/log
211M	total
➜  ~ cat /etc/log2ram.conf | grep SIZE
SIZE=40M

I increased the SIZE to 1024M and it worked:

➜  ~ cat /etc/log2ram.conf | grep SIZE
SIZE=1024M
➜  ~ sudo service log2ram restart     
➜  ~ sudo service log2ram status 
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2019-08-10 19:11:56 UTC; 3s ago
  Process: 2408 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
 Main PID: 2408 (code=exited, status=0/SUCCESS)

[mihaigalos]

@azlux , I would love to help in adding an automatic check for the OS and creating a bigger by default size in case of i.e. Ubuntu Server.

[azlux]

Feel free to open a PR.
1GB reserved on ram is quite big just for the logs.

[pb66]

The option to set it by percentage would be nice!

eg SIZE=10%

would give you 50MB on Pi Zero and early Pi models but 100MB on a Pi 2 or 3.

Likewise it would give you 100MB, 200MB or 400MB on a Pi 4 depending on model.

I do not know how much memory your Ubuntu server has, @mihaigalos but I doubt it needs more than 10% of that for Log2ram.

This could alternatively just be done during the install stage? By checking the RAM and overwriting the size setting with sed, but it might be nicer to have it done dynamically at start up instead.

[edit] Woooaaa! I just noticed the "RPI3B+" in the title, that only has 1024MB of ram, it wouldn't be wise to set log2ram to use all 1024MB of RAM for logfiles, it could easily choke the OS, that is the reason we have a size limit set!

[azlux]

@pb66 I prefer have a value change at the install. If you make it dynamic during startup, you may have issues and surprises. Not sure peoples like to have that.

[pb66]

@mihaigalos you should look at using the olddir directive of logrotate to manage the log2ram size in ram. See #65.

We've been using olddir for quite a while now and this arrangement works really well to keep the amount of ram used by log2ram down.

@azlux - yeah, I'm probably of the same mind truth be told. But I still think it could be done dynamically by the installer and fixed at whatever 10% might be, then the user can override it should they choose. 50MB is just a bit too ridged for a wider audience, it makes manually setting the size on each install almost mandatory.

[mihaigalos]

I personally don't care of old logs Ubuntu produced on my Pi. I very much care of the microSD card getting corrupt. 😊

[pb66]

@mihaigalos - I'm not convinced you fully understand the problem. we are all using log2ram to extend sdcard life!

If you don't care for the old logs, why are you allowing them to reside in your valuable RAM? You need to get rid of them. Logrotate is a great way to remove old logs from the RAM.

Your linked PR will not resolve this issue! Why do you think you had 211M of logs? Because they has built up over time. What do you think will happen over time after you have deleted those logs when installing log2ram? They will build up again unless you take steps to manage the size of the logs.

You can either stop logs going into RAM by changing various log levels etc OR you can get the logs out of RAM on a regular basis. Leaving them in RAM to grow to a huge size is not an option!

By increasing the size of the log partition to equal the size of the installed RAM you are allowing the logs to grow to the point of stopping your Pi dead in it's tracks. But before it crashes out, it will use the swap space as aggressively as it can to avoid a crash, that can be far more disk-writes than a regular and controlled hourly or daily "move old files to disk" by logrotate.

Besides if you really do not want to keep those log files, try using /dev/null as the olddir path so the old logs just get discarded rather than saved to disk in a calm and orderly way.

The part that shortens the sdcard life is the "lots of small writes". Due to the block size on the sdcard, each time a small message is written by any SW that whole block needs re-writing, 1x 100byte message every second, means the same 4k blocks get re-written 41 times in as many seconds. That's why log2ram works so well because it prevents those lots of small rapid writes going to disk and then writes them all (or at least the changes) in one hit every hour (each 4k block written just once). The same for logrotate, moving a few files once an hour/day will not burn out your sdcard.

Forcing your OS to use the swapspace so you can keep lots of old logs in RAM will reduce your sdcard life. So you must make that log partition smaller, much smaller.

Then, even when you do delete all your logs during the install, when they grow again, and they will grow unless you change something (less logs in or more logs out). At some point you will reboot and the logs saved to disk will again be too big to load. So deleting them during install is not a permanent solution.

You can use the logrotate olddir directive to keep the log size under control, even if you choose to discard them rather than persist them (move them to /dev/null?) and I also recommend you set swappiness to 0 to avoid the os using the sdcard as swap space. It will not stop the inevitable crash if there is not enough RAM available, but it will stop your sdcard life being so heavily reduced in the lead up to a crash.

If there is a real threat of a crash (due to full log partition or running out of RAM) that you cannot manage using logrotate, then you could install monit and set that to delete old logs if the partition fills up. Obviously prevention is better than cure!

[mihaigalos]

@pb66, I could not install log2ram on a fresh Ubuntu box because the allocated size for the log2ram log destination in ram was smaller than the size of the microSD folder. So no, the logs didn't accumulate over time. Please, create a PR with a better implementation if you're not satisfied with the current.