Back to index
Last updated: 13 Oct 2019 (complete changelog since 2017 here.)

Introduction

I've not at all been shy in sharing my hate for systemd and its horrible design with regard to logging (for example); I utterly despise systemd but there isn't a bloody thing I can do to make it go away. But what I can do over time is workaround (and that's unfortunately what it is) the problems it causes; in the case of systemd-journald it is log clutter (making it harder to analyse), log messages which are also logged where they belong (where context is kept unlike in the journal), takes more disk space, etc. To make matters worse, parsing text files is much easier than binary files - especially with the basic Linux command line utilities. Even if all these standard utilities were implemented for systemd-journald what if there are customised scripts or utilities written by an individual? And why do we need to duplicate everything in the first place? More importantly, what can be done about this?

The solution: rsyslogd

Now on the subject of systemd's binary logging system (a completely stupid idea even without the cesspool of useless messages it dumps) I by chance had a limited amount of time before I was to be out for some hours - and had a sudden thought that turned out to be wonderful with the subject matter. I previously stated that when using rsyslogd it isn't at all odd that when comparing the syslog message you can't include the name of the program because in fact it's separate from the message. It never occurred to me until roughly nine hours ago that maybe, just maybe rsyslog is clever enough to also filter by other things e.g. the program name itself! And so indeed it can. But what I discovered is it goes beyond this.

As it happens rsyslog has for many versions had a script engine called RainerScript; and this script engine makes it easy to not only check the program name (the program with the log message) but also check for the message at the same time. This means that I can make sure stupid log messages are never written to disk. To be sure most of the log messages (at least that show up in excess) are useless crap; but I don't know that all messages are completely useless (I'd be surprised if they are 100% useful though) so I've started out slow: this is a work in progress. But indeed this means that if you were so inclined you could remove all log entries from systemd-journald. How do you do this? It depends on what you're after; do you want the journal to be written to disk? And what do you want to filter in what way?

Part 1: Confine journal to RAM

To prevent the journal from being written to disk you want to modify the setting 'Storage' in /etc/systemd/journald.conf to be 'volatile'; that is to say, uncomment (remove the '#') the line:

#Storage=auto
and change it to:
Storage=volatile
Note: If what follows the '=' isn't auto the idea is the same and that is you're changing it from the default to 'volatile'.

You will then need to run the following command (keeping in mind that in this context the hash sign is the root prompt):

# systemctl restart systemd-journald

Part 2: Configure rsyslogd

Before you can properly filter you have to make sure a specific rsyslogd module is loaded. Open /etc/rsyslog.conf in your favourite editor (e.g. VIM or VIM) and make sure that there is a line like this (without a '#' anywhere on the line before it):

$ModLoad imuxsock
If there isn't such a line or it is commented out you will need to add it (where the other modules are added i.e. towards the top of the file) or uncomment it (where the other modules are being loaded i.e. near the top of the file). Then you need to make sure you have a line like this (after the modules are loaded):
$SystemLogSocketName /run/systemd/journal/syslog

Part 3: Installing rsyslogd filters

Now, as for the files under /etc/rsyslog.d/ I have an example systemd.conf file (note that it needn't be called 'systemd.conf' - this was arbitrary on my part but it makes most sense because it's actually a filter for 'systemd' messages). I've added comments below but do know that I only recently found out about RainerScript and what I came up with took no time to speak of; there may very well be better ways and that arguably starts with reading the documentation of RainerScript. This is a work in progress for me (since I originally wrote this document I've updated the file). But do not forget that logging is important; the point of this is to remove much of the redundant and other useless crap from the logs. This is why I don't filter 'systemd' outright: this has to be done in steps because there may very well be some systemd log entries that are important. I have friends that aren't all that bothered by systemd or its journal (and all the more power to them) but it bothers me and so I do filter it; once again remember that logging is a good thing: if you blindly filter out log messages you could be blindly filtering out something important. I may very well have not foreseen any number of problems (and amusingly and ironically I would be doing exactly as I caution against); but this is why I have done it in steps: I do not want to filter out important log messages (if I didn't care I would simply filter out all systemd log messages).

So the file in question does the following:

  1. Remove all log messages from 'systemd-logind' which are logged elsewhere (and with useful information and context).
  2. If the program is 'systemd' then we filter out log messages that match different regular expressions. The following message patterns (very loosely described!!) aren't logged to disk (where place holders are surrounded by square brackets):
    1. Starting [service]
    2. Stopping [service]
    3. Started Session [...]
    4. Configuration file [file] is marked world-inaccessible. [...]
    5. Started System Logging Service [...]
    6. Created slice user[...]
    7. Removed slice user[...]
    8. Mail
    9. mail

The following logic applies:

  1. I don't care to see if a service is being started; any decently programmed service will have its own start up message (and possibly in a more relevant log file).
  2. I don't care when a service is being stopped; any decently programmed service will have its own stopped message (and possibly in a more relevant log file).
  3. I honestly can't recall what this one was about; I believe it was something to do with a user logging in which once again is logged elsewhere with proper context and more relevant information.
  4. Configuration file being marked world-inaccessible is irrelevant because they override it anyway; the fact the files are the defaults (in system installed locations e.g. under /usr/lib/systemd/) which I have not touched it and the fact they override it makes it as good as spam.
  5. The brilliance of showing me that the logging service was started; it just doesn't get any better than that when you consider that the logging service itself has (once again) its own start/stop messages.
  6. More user login rubbish.
  7. More user logout rubbish.
  8. Mail related messages are in in /var/log/maillog so why the hell do I need to see it under /var/log/messages or anywhere else? This is just like the other things I've filtered.
  9. Mail related messages are in in /var/log/maillog so why the hell do I need to see it under /var/log/messages or anywhere else? This is just like the other things I've filtered.

Don't forget to restart rsyslogd when you update or add a new .conf file under /etc/rsyslog.d/! You do that like so:

# systemctl restart rsyslog

Essentially it comes down to keeping the journal in memory only and removing the redundant and useless crap from /var/log/messages (stopping the logging isn't the only thing, of course; you can for example make it write to another file and then stop it from going to /var/log/messages also). More to the point the trick is that you can indeed filter by not only the message but also the name of the program.