Back to index
Last updated: 13 Oct 2019 (complete changelog since 2017
here.)
I've not at all been shy in sharing my hate for systemd and its horrible design with regard to logging (for example); I utterly despise systemd but there isn't a bloody thing I can do to make it go away. But what I can do over time is workaround (and that's unfortunately what it is) the problems it causes; in the case of systemd-journald it is log clutter (making it harder to analyse), log messages which are also logged where they belong (where context is kept unlike in the journal), takes more disk space, etc. To make matters worse, parsing text files is much easier than binary files - especially with the basic Linux command line utilities. Even if all these standard utilities were implemented for systemd-journald what if there are customised scripts or utilities written by an individual? And why do we need to duplicate everything in the first place? More importantly, what can be done about this?
Now on the subject of systemd's binary logging system (a completely stupid idea even without the cesspool of useless messages it dumps) I by chance had a limited amount of time before I was to be out for some hours - and had a sudden thought that turned out to be wonderful with the subject matter. I previously stated that when using rsyslogd it isn't at all odd that when comparing the syslog message you can't include the name of the program because in fact it's separate from the message. It never occurred to me until roughly nine hours ago that maybe, just maybe rsyslog is clever enough to also filter by other things e.g. the program name itself! And so indeed it can. But what I discovered is it goes beyond this.
As it happens rsyslog has for many versions had a script engine called RainerScript; and this script engine makes it easy to not only check the program name (the program with the log message) but also check for the message at the same time. This means that I can make sure stupid log messages are never written to disk. To be sure most of the log messages (at least that show up in excess) are useless crap; but I don't know that all messages are completely useless (I'd be surprised if they are 100% useful though) so I've started out slow: this is a work in progress. But indeed this means that if you were so inclined you could remove all log entries from systemd-journald. How do you do this? It depends on what you're after; do you want the journal to be written to disk? And what do you want to filter in what way?
To prevent the journal from being written to disk you want to modify the setting 'Storage' in /etc/systemd/journald.conf to be 'volatile'; that is to say, uncomment (remove the '#') the line:
#Storage=autoand change it to:
Storage=volatileNote: If what follows the '=' isn't auto the idea is the same and that is you're changing it from the default to 'volatile'.
You will then need to run the following command (keeping in mind that in this context the hash sign is the root prompt):
# systemctl restart systemd-journald
Before you can properly filter you have to make sure a specific rsyslogd module is loaded. Open /etc/rsyslog.conf in your favourite editor (e.g. VIM or VIM) and make sure that there is a line like this (without a '#' anywhere on the line before it):
$ModLoad imuxsockIf there isn't such a line or it is commented out you will need to add it (where the other modules are added i.e. towards the top of the file) or uncomment it (where the other modules are being loaded i.e. near the top of the file). Then you need to make sure you have a line like this (after the modules are loaded):
$SystemLogSocketName /run/systemd/journal/syslog
Now, as for the files under /etc/rsyslog.d/ I have an example systemd.conf file (note that it needn't be called 'systemd.conf' - this was arbitrary on my part but it makes most sense because it's actually a filter for 'systemd' messages). I've added comments below but do know that I only recently found out about RainerScript and what I came up with took no time to speak of; there may very well be better ways and that arguably starts with reading the documentation of RainerScript. This is a work in progress for me (since I originally wrote this document I've updated the file). But do not forget that logging is important; the point of this is to remove much of the redundant and other useless crap from the logs. This is why I don't filter 'systemd' outright: this has to be done in steps because there may very well be some systemd log entries that are important. I have friends that aren't all that bothered by systemd or its journal (and all the more power to them) but it bothers me and so I do filter it; once again remember that logging is a good thing: if you blindly filter out log messages you could be blindly filtering out something important. I may very well have not foreseen any number of problems (and amusingly and ironically I would be doing exactly as I caution against); but this is why I have done it in steps: I do not want to filter out important log messages (if I didn't care I would simply filter out all systemd log messages).
So the file in question does the following:
The following logic applies:
Don't forget to restart rsyslogd when you update or add a new .conf file under /etc/rsyslog.d/! You do that like so:
# systemctl restart rsyslog
Essentially it comes down to keeping the journal in memory only and removing the redundant and useless crap from /var/log/messages (stopping the logging isn't the only thing, of course; you can for example make it write to another file and then stop it from going to /var/log/messages also). More to the point the trick is that you can indeed filter by not only the message but also the name of the program.