oldlog

[StuartIanNaylor]

log2ram keeps current logs in memory but also keeps old logs from logrotate and vastly increase mem usage via what are essentially old non current logs not in use and already copied to /var/hdd.log

logrotate has an oldlog directive that unfortunately can not be used with different devices.

There is a workaround though as postrotate can call scripts to mv the contents of /oldlog elsewhere
So use the oldlog directive with /var/log/oldlog
postrotate scripts could be used but log2ram has an hourly routine that could equally move old logs form /var/log/oldlog to /var/hdd.log

I am sat here awaiting my next hourly hoping a global oldlog directive in /etc/logrotate.conf will move all old logs to /var/log/oldlogs
via a simple olddir /var/log/oldlog inclusion to /etc/logrotate.conf

My optimism that it maybe that simple is likely just that and expecting it will not capture logs in /etc/logrotate.d
But its an example that really something should and that the oldlog directive can be used as a staging directory for log pruning and moving to /log/hdd.log and not just copying.

[azlux]

I understand your idea.
But you need additional changes. Log2Ram is a KISS project. I don't want to tell people they need to modify logrotate configuration.

[StuartIanNaylor]

The problem is currently the whole rationale of log2ram to minimize flash writes is flawed because the hourly copy of logs that have changed in the majority of installs is likely no more beneficial than the actual log writes themselves.

The stop function should be the one that writes current to hdd.log
The hourly should just move oldlog files to hdd.log files if /var/log usage > x% of /var/log device size.
Or better would be usage since last check < x% of remaining. Because log rotation will restart on each move.

Also no-one should be telling anyone they need to modify logrotate configuration as the install.sh should add the olddir /var/log/oldlog directive and that is all that is need as /etc/logrotate.conf is global.

It maybe a KISS project and the reality is that currently the whole project is an unnecessary addition as the hourly overwrite of log changes is still intensive.
As its quite likely complete files are going to be written for what is maybe a couple of log line inclusions and yes write frequency is reduced but amount of data is massively increased hourly.

I never really thought until now, but we could be causing more flash wear than we are reducing and stealing memory for log drives for little benefit.

Also oldlog should be transferred as by looking at https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-ramlog and the RecreateLogs routine it looks like L2R will cause problems with some programs if the original log structure is not there on start.
Using the oldlog mechanism means it is easy to copy on start existing log structure excluding oldlog or like armbian be forced to start having individual creation routines for specific applications.

I wish they had maybe fed back a little but hey ho.
Then again I thought I private bind had the directory sub-tree but just didn't propagate but haven't tried.

[StuartIanNaylor]

The olddir directive is a single line at the top of /etc/logrotate.conf and because its there its global unless specific noolddir directives are given.

The limitations of olddir is that it must be on the same device as logrotate only makes swift and efficient file allocation table changes and will no bulk file copies due to the I/O and load that will make.
Shame really as having olddir on hdd.log for L2R would of been sweet.

What it does is create a really simple container and separator of live logs and oldlogs that makes pruning a ramlog extremely easy as logrotate will do this for you via the noolddir directive.

The hourly syncToDisk routine is a complete curveball to anyone employing L2R as L2R doesn't state anywhere that L2R is only beneficial for high frequency logging applications and the lesser your logging requirements the nearer you get to low qty log line inclusions being swapped for hourly large file bulk copies.
For many the syncToDisk command practically negates the whole rationale for employing L2R and there is absolutely zero benefit to be keeping redundant copies of live logs onto the hard media that L2R is in operation to stop.

syncToDisk should only be called by the stop command of the service so that L2R provides the minimum writes to hard media.
But currently its just a bulk copy every hour and it undoes much of the benefit and rationale to employ L2R whilst just create problems of minimal logging space in a restricted size memory drive.

    if [ "$USE_RSYNC" = true ]; then
        rsync -aXWv --delete --links $RAM_LOG/ $HDD_LOG/ 2>&1 | $LOG_OUTPUT
    else
        cp -rfup $RAM_LOG/ -T $HDD_LOG/ 2>&1 | $LOG_OUTPUT
    fi

I apologise to be so critical as the easy answer is why is it there? It doesn't need to be and it shouldn't be and please remove.

The hourly routine should be doing something far more important than creating redundant copies of live logs and that is pruning the critical small space of a memory disk log by moving oldlogs to hdd.log because this action of intense bulk copies across devices only needs to be done once for those logs.
Log2Ram should have a configurable option that if the remaining log space is less than $PRUNE_LEVEL it will move all old logs from the olddir /var/log/oldlog directive in one extremely infrequent copy and move.
The now empty olddir /var/log/oldlog will then start to recreate the logrotation and continue separating live logs from old via the olddir /var/log/oldlog directive until the next condition when remaining space is less than the $PRUNE_LEVEL set in /etc/log2ram.conf.
If all runs well the remaining space will never drop below and the only writes to hard media are on a stop.

The hourly routine of syncToDisk should be changed to pruneOldLog and its a one line comparison.

if [ "$LOG_REMAINING" -lt "$PRUNE_BYTES" ];then
	    if [ "$USE_RSYNC" = true ]; then
			rsync -aXWv --delete --links $RAM_LOG/oldlog/ $HDD_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
			rm -rf $RAM_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
		else
			cp -rfup $RAM_LOG/oldlog/ -T $HDD_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
			rm -rf $RAM_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
		fi
	fi

The only additional parts I have added at my is the conversion of remaining size with suffix to bytes so that comparison is an easy one liner.

getUsage () {
	if [ "$ZL2R" = true ]; then
		USAGE=$(zramctl -b | grep /var/log | awk '{ print $5 }')
	else
		USAGE=$(df "/var/log" | tail -1 | awk '{ print $3 }')
	fi
}

toBytes() {
 echo $1 | echo $((`sed 's/.*/\L\0/;s/t/Xg/;s/g/Xm/;s/m/Xk/;s/k/X/;s/b//;s/X/ *1024/g'`))
}

pruneOldLog() {
	isSafe
	getUsage
	PRUNE_BYTES=$(toBytes ${PRUNE_LEVEL})
	MEM_SIZE=$(toBytes ${SIZE})
	LOG_REMAINING=$(( ${MEM_SIZE} - ${USAGE} ))
	if [ "$LOG_REMAINING" -lt "$PRUNE_BYTES" ];then
	    if [ "$USE_RSYNC" = true ]; then
			rsync -aXWv --delete --links $RAM_LOG/oldlog/ $HDD_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
			rm -rf $RAM_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
		else
			cp -rfup $RAM_LOG/oldlog/ -T $HDD_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
			rm -rf $RAM_LOG/oldlog/ 2>&1 | $LOG_OUTPUT
		fi
	fi
	
}

As the retention of old logs in live log memory is as wasteful and pointless as the writes of live logs to the redundant log hard media.
The above one line comparison is actually beneficial for a huge reduction in hard media writes and also provides pruning of precious memory space.
The inclusion of the olddir directive means logrotate will decide what are old and new logs and not us.

I made one dumb boob as thought for some reason rm -rf $RAM_LOG/oldlog/ the default action would be to keep the root directory and forgot to check prob a rm -rf $RAM_LOG/oldlog/* should of been used doh!

Alternatives

Either that or create some form of cat to diff or cmp routine to just append to hhd.log but the problem with hourly schedules irrespective of what you do on a system crash the most critical log information close to crash is always lost and having the hours previous is of little gain for just negative effects on the main purpose of L2R.
If mission critical logging is required where the unlikely event of total system crashes is unacceptable then L2R can not be used.
For most purposes where media wear is a concern L2R is an excellent choice and enabling a hardware watchdog routine to trigger a stop services orderly shutdown would add system resilience if needed.
For most system crashes are a rare exception and the logs are there in the L2R ram disk and shipped out via the oldlog method anyway.

[pb66]

I agree with the idea of not holding the rotated files in RAM and have made changes to use olddir.

Actually you can use the olddir to move the logs to a different partition if you use the renamecopy setting. Here's the changes I've made.

master...pb66:pb66

Please ignore the changes to the log folder naming. I just preferred to find all my log files using something like ls -la /var/log* and since, in my case it's predominantly used on a sdcard rather than a hdd, the naming seemed more in tune with what I was using it for.

As you can see I opted for a simple additional file to be dropped into logrotate.d so there is no editing required. The 00 prefix ensures that the olddir directive is loaded before any of the other logrotate conf files and since the include /etc/logrotate.d is included ahead of the default global settings in the main logrotate.conf file, it gets the olddir setting in early enough for it to be global.

I also added a simple symlink so that all the rotated logs can be found in /var/log/rotated_logs too, hopefully even the users unfamiliar with log2ram should still be able to find the rotated logs without knowing about it's inner workings.

I've also updated the installer and uninstaller to suit.

Whilst I have configured Pi's to run (Raspbian) RO for years now, I'm fairly new to using log2ram so I cannot tell you how this works over time or with heavier traffic, but I have had it running for a while on a test RPi and it seems to work well.

[StuartIanNaylor]

renamecopy setting ? olddir is just a logrotate directive that is set globally in /etc/logrotate.conf via sed.

I can not find a renamecopy setting is that just a post-rotate script? Logrotate will do all the oldlog movement but as far as I know its limited to the same device so its making quick fat (file allocation table) edits and not bulk moves.

The olddir directive just makes an easy container for oldlogs and should be only moved out on on low available log space.

If you check https://github.com/StuartIanNaylor/zramdrive its a minimal and tidied log2ram for any directory as the whole action of writing out logs to another partition every hour is totally pointless as at any point you have up to an hours missing logs.
It contains zramdrive purge and zramdrive write methods to allow it to be done at any time if needed but completely lacks what is essentially a pointless cron job.
If you where to move to another partition then again you are just making writes for no reason as the most important logs are likely to be the live ones.

Also using the reload function of systemd seems to periodically get triggered on boot and at times not by the cron job. Not sure why but check syslog and you will see it happening.
Reload is a specific systemd event to reload conf of a running service and even though its hardly ever implemented it shouldn't be used for completely different functionality.

https://github.com/StuartIanNaylor/log2zram
Can take over the logrotate schedule and works on minutes and also doesn't require cron and only purges on low space requirements.
I named it log2zram because I can see little point of the default tmpfs usage or even its inclusion if the massive ram savings are avail by zram.

[pb66]

This issue started out about not holding the rotated logs in RAM and using olddir to rotate them out, that is the discussion I was aiming my comments at.

olddir is just a logrotate directive that is set globally in /etc/logrotate.conf via sed.

olddir is indeed a logrotate directive and it CAN be set in /etc/logrotate.conf or as I have demonstrated, it can be set in a logrotate.d file too, this is how logrotate works!

The use of sed is irrelevant to the discussion, it is one of many ways that a text file can be edited and has nothing directly to do with logrotate or log2ram.

I can not find a renamecopy setting is that just a post-rotate script?

https://manpages.debian.org/testing/logrotate/logrotate.8.en.html

renamecopy
Log file is renamed to temporary filename in the same directory by adding ".tmp" extension to it. After that, postrotate script is run and log file is copied from temporary filename to final filename. This allows storing rotated log files on the different devices using olddir directive. In the end, temporary filename is removed.

Logrotate will do all the oldlog movement but as far as I know its limited to the same device

Not when you use renamecopy, that's why I said

Actually you can use the olddir to move the logs to a different partition if you use the renamecopy setting. Here's the changes I've made.

and gave you the code.

The olddir directive just makes an easy container for oldlogs and should be only moved out on on low available log space.

You are right it is easy, but I disagree with "only on low space", it's just as easy to read logfiles from disc as tmpfs IF you need to read the rotated logs at all. disc space is cheap, ram is less so, especially on older/smaller dev boards.

I hope that clears up any confusion for you and gets the discussion back to potentially using olddir and rotating the logs out of ram in log2ram. My comments were more aimed at hopefully convincing @azlux to reconsider this option as it is not as complex as you first suggested, so it keeps in line with the KISS objectives whilst reducing the RAM occupied by the logs managed by log2ram.

[StuartIanNaylor]

renamecopy must be a debian thing as doesn't seem to exist in https://linux.die.net/man/8/logrotate

Not sure how it works though as some such as apache have postrotate scripts that would override? The global one?
Dunno as renamecopy is new to me, if it does then great as yeah but the main point in log2ram is to reduce writes and reduce blockwear that is why it only moves if space is getting low on the amount of ram you have allocated for your zram drive.

[pb66]

I think that is a documentation issue perhaps!

It's been in the main source since Feb 2015 https://github.com/logrotate/logrotate/blob/master/ChangeLog.md#389---2015-02-13

Is it included in the output of man logrotate on whatever distro you are using?

I do not think https://www.die.net/ is the defining place to look for such things. Many online man pages are out of date or for specific distro's, you should always use the man pages installed to your system (sometimes as a separate package).

the renamecopy entry in the man page source code https://github.com/logrotate/logrotate/blob/3.15.0/logrotate.8.in#L528-L534

The rotated logs will end up on disc sooner or later and by grouping them altogether and doing the lot in one move could actually write to fewer blocks than you think. We do not need to try and eradicate disk wear, just lower it enough so that the sdcards do not wear prematurely, no one expects an sdcard to last forever, but even with moderate writes an sdcard can last a significant time these days. Log2ram just needs to buffer and bulk write to make a serious impact on disk wear, to reduce it further gets more complex and has less gain per effort than the initial simple buffering.

I see no reason why apache2 will override the olddir and renamecopy directives, but I haven't specifically tested it nor do I claim to be an expert. But I did research this for my own needs to get the rotated files out of ram.

[StuartIanNaylor]

The question marks where questions on the info you provided that I wasn't aware of, not statements

Log file is renamed to temporary filename in the same directory by adding
".tmp" extension to it. After that, \fBpostrotate\fR script is run
and log file is copied from temporary filename to final filename. This allows
storing rotated log files on the different devices using \fBolddir\fR
directive.

The info is slightly confusing as to why the postrotate script is mentioned and where the directive should be placed as have never seen an example.
If it does work then great that is initially how I wanted to implement logrotate and scrap the hourly write out altogether as its a bit pointless, it was just all the examples I could find all seemed to state it must be same with the olddir directive.
Like I said 'if it does then great', create a branch, let us have a look as I am interested as it would more than halve the /var/log footprint.
Do it from either log2ram or https://github.com/StuartIanNaylor/log2zram as you can drop the whole scheduler bit, which ever you find easier.

Also reading through fBcreateolddir should be added also as its that would seem simpler and tidier than mkdir in scripts

[StuartIanNaylor]

https://github.com/StuartIanNaylor/zram-config implemented renamecopy

[azlux]

@pb66

olddir directory
              Logs are moved into directory for rotation. The directory must be on the same physical device as the log file being rotated, unless copy, copytruncate or renamecopy option is used. The directory is assumed to be relative to the directory holding  the
              log file unless an absolute path name is specified. When this option is used all old versions of the log end up in directory.  This option may be overridden by the noolddir option.

Same physical device. I wonder why ?

[azlux]

I see a reason It will not work. Some software don't use logrotate but have it own rotation implemented.
2nd question, how to you manage software installed after the log2ram install ?

[pb66]

Same physical device. I wonder why ?

Possibly because mv is used ordinarily and mv cannot be used across devices? That is bypassed using renamecopy so not really a issue (although the mv vs renamecopy does confuse things a little, name aside that feature gets around the "same device" issue)

I see a reason It will not work. Some software don't use logrotate but have it own rotation implemented.

Indeed it may not fully work with some edge cases, but even those that currently self-rotate also get rotated by logrotates global settings unless an exclusion is added to logrotates conf. There will always be exceptions and edge cases but this method catches almost all as it's the more common practice for logrotations to be left to logrotate, and when this isn't the case, global rotations tend to not break anything despite some softwares self-rotating. The worse case scenario is that those specific self-rotated logs are not relocated on disk, but that still results on a significantly reduced size in RAM and also significantly reduced load time/effort at boot as well as potentially reduced save time/effort hourly and at shutdown.

how to you manage software installed after the log2ram install ?

Just install as per the software's guide or installer without concerning yourself with log2ram, it is totally transparent. The current position of retaining newly created logs providing the machine isn't stopped ungracefully before the next hourly log2ram runs is retained, users can manually run a log2ram as part of a install regime if they want to be sure the logfiles are definitely going to be there on the next reboot. (but that is standard with log2ran, not related to this olddir directive)

All this change does is keeps the rotated logs (created by logrotate) out of RAM, the operation of the original logs are unaffected and any additional logrotate config files added by a software's package installer should be honoured, only the location of the rotated files is altered.

[StuartIanNaylor]

Same physical device is default speed as files are not copied across devices as all that is needed is FAT changes on the same device.
I actually can not think of many apps that do self log rotate that are common, in fact struggling to think of one.
Because it would not rotate a few but rotate the majority does not mean it would not work.
I took @pb66 info and applied it and so far haven't found an app it doesn't work with and it seems to work great.

Also its only a 3 line insert in /etc/logrotate that is choice on start or install.

		sed -i "/^weekly.*/i olddir ${OLDLOG_DIR}" /etc/logrotate.conf 
		sed -i "/^weekly.*/i createolddir 755 root root" /etc/logrotate.conf 
		sed -i "/^weekly.*/i renamecopy" /etc/logrotate.conf 

It would mean at least 50% space and likely much more.
Oldlog= 1 full copy of last and then archived logs might even account for 70%+ of log space just to store archived oldlogs in ram for no reason and to be copied back on each start.

[pb66]

Also its only a 3 line insert in /etc/logrotate that is choice on start or install.

IMO that is a rather messy way of doing it.

A file added to /etc/logrotate.d using install is the better way to do it, it avoids editing the default conf file and is inkeeping with the existing install procedure. The 00_ prefix ensures it is included in the conf ahead of any installed packages additional conf files.

[edit] Plus if you wanted to make the install even neater you might consider having the 00_olddir and log2ram files combined into just one 00_log2ram file and reduce the (automated) install steps and file count.

eg

# These settings will save the rotated logfiles to a seperate location.
# This file needs to be located in /etc/logrotate.d.
# The "00" prefix in the file name is required to ensure this file is
# loaded before any program specific logrotate settings are loaded as
# contents of /etc/logrotate.d are read in alphabetical order.

# The path & name for rotated logfiles folder.
olddir /var/log.old

# Create the path if it doesn't exist
createolddir 755 root root

# To allow the files to be "moved" (ie copied and original deleted)
# to another device/partition (eg out of RAM)
renamecopy

/var/log/log2ram.log
{
  rotate 7
  daily
  missingok
  notifempty
  delaycompress
  compress
}

[StuartIanNaylor]

Guess so as that would prob override any existing and saves you having to molest /etc/logrotate.conf
I think I will do that also but for me its a nobrainer as the majority of var/log are actually oldlogs that just take precious mem space for no reason as they are not live.

[edit]
I do it from conf as its optional depending on user setting and its not set on install but on run, depending on conf.
Never been sure about the validity of a /var/log/log2ram log when its likely lost due to mount / umount with /var/log and other activities of log2ram.
I just stick it elsewhere off /var/log

[azlux]

Because if another discussion about log2ram, I think I will reconsider this issue.
Maybe a prompt during the install, but olddir cannot be set by default.
@pb66 Your simple config file is quite easy to add.

[ejony]

Just a quick comment that may help someone in an edge use case like mine.

Background:
Not using logrotate to prune an arbitrary directory with daily network device logs, '/var/log/central/'.
Using logs with file name convention 'YYYY_MM_DD_IPADDRESS.log'.

Because of the naming convention and other reasons, I didn't want to use logrotate, instead moving the logs to a different server every week using a script.

Observation:
Moved logs from /var/log/central/ to a different server.
Upon reboot of Pi, the files that were moved from the directory would be restored back, probably from '/var/hdd.log/central/'.

My Work Around:
service log2ram stop
move logs from /var/log/central/ to different server
service log2ram start

After reboot, the logs are no longer restored back into /var/log/central/.