Enable Rsyslog Logging on Debian

This step by step tutorial will take you through how to enable rsyslog logging on Debian 12. Debian 12 bookworm uses systemd-journald logging by default. As such, it has deprecated the use of Rsyslog for logging so as to prevent the log messages being written twice on disk. So, how can you enable rsyslog logging on Debian 12 and higher?

Rsyslog vs Journald

Rsyslog and Systemd-journald are both logging systems commonly used in Linux distributions for collecting and storing log data. However, they have different features, functionalities, and configurations.

Rsyslog

rsyslog is a traditional and widely-used logging system in Linux distributions.

It uses a client-server architecture where log messages can be received from various sources and forwarded to remote syslog servers or stored locally.

rsyslog supports a wide range of log inputs and outputs, including files, network protocols (like syslog, TCP, and UDP), and database backends.

It provides flexible configuration options, allowing customization of log routing, filtering, and processing rules.

rsyslog supports advanced features such as log rotation, compression, log rate limiting, and filtering based on severity levels or message content.

Configuration is typically done through the /etc/rsyslog.conf file and additional configuration files in the /etc/rsyslog.d/ directory.

rsyslog.service is responsible for starting, stopping, and managing the rsyslog daemon.

By default, on many Linux distributions, rsyslog stores logs in /var/log/ directory, with different log files for various system components and services.

Systemd-Journald

systemd-journald is a new logging system that is part of the systemd initialization system, which is becoming the standard in many modern Linux distributions.

It uses a binary log format and stores log data in a structured manner, making it efficient for log retrieval and analysis.

systemd-journald captures log messages directly from services and processes that use the systemd journal API.

It provides advanced features such as log compression, rate limiting, and the ability to store metadata along with log entries.

systemd-journald integrates well with other systemd components and can capture additional system information like boot logs and kernel messages.

Logs stored by systemd-journald are accessed using the journalctl command-line tool.

Configuration options for systemd-journald are specified in the /etc/systemd/journald.conf file.

Starting, stopping, and managing the systemd-journald daemon is managed by the systemd-journald.service.

By default, the log data is stored in /var/log/journal/ directory. The logs are organized by system and user, with separate directories for each.

Install Rsyslog on Debian system

Since Rsyslog has been deprecated and made optional on Debian 12, it doesn’t come installed by default now on Debian 12. It is still however possible to install the rsyslog package and it will work as usual. Thus, execute the commands below to install:

apt install rsyslog

When installed, it is started and enabled to run on system boot by default;

~] systemctl status rsyslog

● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
     Active: active (running) since Tue 2023-09-12 12:27:52 UTC; 25min ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             man:rsyslog.conf(5)
             https://www.rsyslog.com/doc/
   Main PID: 589 (rsyslogd)
      Tasks: 4 (limit: 4644)
     Memory: 3.4M
        CPU: 27ms
     CGroup: /system.slice/rsyslog.service
             └─589 /usr/sbin/rsyslogd -n -iNONE

Sep 12 12:27:52 syslog-smartcon systemd[1]: Starting rsyslog.service - System Logging Service...
Sep 12 12:27:52 syslog-smartcon systemd[1]: Started rsyslog.service - System Logging Service.
Sep 12 12:27:52 syslog-smartcon rsyslogd[589]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2302.0]
Sep 12 12:27:52 syslog-smartcon rsyslogd[589]: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="589" x-info="https://www.rsyslog.com"] start

Once started, you should now be able to see more log files written to /var/log.

Disable Systemd-Journald Logging

Note that, when rsyslog is installed and enabled, systemd-journald is still logging as well. As such, you will end up with logs being written twice on disk. See sample SSH logs:

tail -f /var/log/auth.log

2023-09-12T12:27:52.472393+00:00 syslog-smartcon sshd[659]: Server listening on 0.0.0.0 port 22.
2023-09-12T12:28:15.202602+00:00 syslog-smartcon sshd[937]: Accepted password for rrastik from 192.168.48.208 port 56661 ssh2
2023-09-12T12:28:15.209173+00:00 syslog-smartcon sshd[937]: pam_unix(sshd:session): session opened for user rrastik(uid=0) by (uid=0)
2023-09-12T12:28:15.225313+00:00 syslog-smartcon systemd-logind[596]: New session 1 of user root.

journalctl --since=today | grep sshd

Sep 12 12:27:52 syslog-smartcon sshd[659]: Server listening on 0.0.0.0 port 22.
Sep 12 12:28:15 syslog-smartcon sshd[937]: Accepted password for rrastik from 192.168.48.208 port 56661 ssh2
Sep 12 12:28:15 syslog-smartcon sshd[937]: pam_unix(sshd:session): session opened for user rrastik(uid=0) by (uid=0)
Sep 12 12:28:15 syslog-smartcon sshd[937]: pam_env(sshd:session): deprecated reading of user environment enabled

If you want to save some disk space, you can disable systemd-journald logging. This can be done by removing the logging directory, /var/log/journal , thus preventing journald from using its own message persistence logic.

rm -rf /var/log/journal

And that is it. You are now back to traditional logging on Debian 12 and higher.

rsyslog configuration (optional)

My recomandation for rsyslog config:

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat



###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth/auth.log
*.*;auth,authpriv.none;cron.none;lpr.none;mail.none;user.none;local5.none               -/var/log/syslog
cron.*                          /var/log/cron/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr/lpr.log
mail.*                          -/var/log/mail/mail.log
user.*                          -/var/log/user/user.log
# local5.*                        -/var/log/esets/nod32.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail/mail.info
mail.warn                       -/var/log/mail/mail.warn
mail.err                        /var/log/mail/mail.err

rsyslog - remote logging

Having a separate remote Linux server for storing logs has its benefits. Here's how you can set up a remote log aggregation server using rsyslog.

Configuring the Log Host Server

There are two protocols you can use for sending/receiving log files with rsyslog: TCP and UDP. This guide shows you how to configure TCP.

Our rsyslog server has hostname rsyslog-server. The rsyslog configuration resides in the /etc/rsyslog.conf file. Open this configuration file for editing using a text editor and find directives:

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

These lines load the imudp and imtcp modules for listening at the specified UDP or TCP ports (both at the port 514). You can listen to both ports if you wish or change the port number as you see fit.

For this tutorial, we will use a TCP connection alone. Enable it by uncommenting the following two lines of code (remove the # prefix):

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Save changes in file /etc/rsyslog.conf and exit the editor.

Setting up the remote log storage location

By default, all logs received from TCP port 514 will be merged in the /var/log directory with the system's log file. This is typically unwanted behavior because it mixes all remote logs with the host server's local logs, making it more difficult to search and filter later on. To prevent this behavior, you need to edit the Rsyslog configuration.

Once again, open up the /etc/rsyslog.conf file with your text editor and add the following lines at the beginning of the file:

# Our previous config
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# define log storage
$template remote-incoming-logs,"/var/log/remote_logs/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?remote-incoming-logs
& ~

The $template remote-incoming-logs directive instructs Rsyslog to store all incoming log entries in the location that is defined by the third parameter. In our case, the remote logs will continue to be stored in /var/log/remote_logs/ directory, but each client will have its own subdirectory with a name equivalent to client hostname. This subdirectory will store each log entry in a file that matches the client program that generated it.

On the following line, the *.* ?remote-incoming-logs directive applies the remote-incoming-logs configuration rule at all facilities with all priority levels (in other words, to all logs).

optional: Finally, the & ~ directive defines that rsyslog stops processing log input after it is stored to a file defined in previous lines. The default configuration will overwrite the previous rule without this line.

Save the file, then restart the rsyslog service with systemctl:

root@rsyslog-server:/var/log ~] systemctl restart rsyslog

Forwarding logs from an Rsyslog client

Log to the rsyslog client (with hostname web) and edit the default rsyslog configuration at /etc/rsyslog.conf with a text editor.

At the *.* @@<your_rsyslog_server_ip_address>:514 directive on a new line at the beginning of the file. Make sure to replace the placeholder with the actual IPv4 address of the server that you retrieved in step 1.

#  Default rules for rsyslog.
#
#                       For more information see rsyslog.conf(5) and /etc/rsyslog.conf

*.* @@1.1.1.1:514

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
#daemon.*                       -/var/log/daemon.log
kern.*                          -/var/log/kern.log
#lpr.*                          -/var/log/lpr.log
mail.*                          -/var/log/mail.log
#user.*                         -/var/log/user.log

The directive you just added above defines that the Rsyslog service should send all facilities with all priority levels (in other words, all logs) to the IP address (1.1.1.1 in the above example) of the centralized server at TCP port 514. If you set up this directive using @ instead of @@, then it will forward the logs to the UDP port.

Note that the *.* syntax determines that all log entries on the server should be forwarded. If you want to forward only specific logs, you can specify the service name instead of * such as cron.* @@1.1.1.1:514 or apache2.* @@1.1.1.1:514. You can also forward logs to more than one server:

#  Default rules for rsyslog.
#
#                       For more information see rsyslog.conf(5) and /etc/rsyslog.conf

*.*       @@1.1.1.1:514
*.*       @@192.168.122.235
cron.*    @@192.168.122.237:514

After saving your changes to the file, restart the rsyslog service to apply the new configuration:

root@web:~] systemctl restart rsyslog.service

At this point, your Rsyslog client is now fully configured to send its logs to the centralized Rsyslog server.

Testing your configuration

Now that you've configured both the rsyslog client (with hostname web) and centralized server (with hostname rsyslog-server), let's go ahead and verify that our changes have had the desired effect. Go ahead and manually create a log entry on the client through the logger command:

root@web:~] logger 'test from client'

Afterward, confirm that the entry is present in the /var/log/syslog file through the tail command. You should see the following message near the end of the output:

root@web:~] tail /var/log/syslog
...
2024-06-12T11:19:05.440014+00:00 web systemd[1]: Started rsyslog.service - System Logging Service.
2024-06-12T11:19:40.291611+00:00 web root: test from client

Now, switch to the centralized rsyslog server and view the contents of the /var/log/remote_logs directory:

root@rsyslog-server:/var/log ~] tree remote_logs/
remote_logs/
└── web
    ├── CRON.log
    ├── root.log
    ├── rsyslogd.log
    ├── sshd.log
    └── systemd.log

You should see a directory that corresponds to the hostname of the client system. In this example, the hostname of the client server is web and you can observe a web directory in the output above.

Examine the root.log file in this subdirectory:

root@rsyslog-server:/var/log ~] tail /var/log/remote_logs/web/root.log 
2024-06-12T11:19:40+00:00 web root: test from client

You should see the identical log entry observed on the client and server.