How to Setup Rsyslog Remote Logging on Linux (Central Log Server)
Every Linux distribution have some kind of logging mechanism that
records all the system activities. A while back we provided a list of
20 log files
that are stored under /var/log that you might be helpful during
troubleshooting. These logs are very critical for sysadmin
for troubleshooting purpose.
The following are the three common methods to log a message:
- Logging on the same server: Messages get written into the local hard drive/local database
- Logging on a remote server: Many systems forward their logs over the
network to a central log server. On the central log server, the
messages from various systems are written to the local hard
drive/database.
- Relay logging: Branch ‘A’ and Branch ‘B’ logs the messages on 2
different servers. These server in-turn logs the message to the ‘Head
Office’.
Rsyslog is the default logging program on several Linux distributions
including Debian and Red Hat based systems. Apart from implementing the
syslog protocol, rsyslog adds additional
features such as content-based filtering. This also uses TCP for transporting, and provides lot of configuration options.
This article explains how to implement the method 2 mentioned above.
i.e This explains how to setup a central logging server, and send logs
from individual servers to the central logging server.
This setup will help you to analyze the log files of all the servers in your infrastructure from a central log server.
Installation
Rsyslog comes as the default logging program in Debian Distribution
and Red Hat based systems. If you system doesn’t have rsyslog, install
it as shown below depending on your distro.
apt-get install rsyslog rsyslog-doc
(or)
yum install rsyslog rsyslog-doc
Rsyslog configurations are stored in /etc/ryslog.conf file and the files under /etc/rsyslog.d/ directory.
Configuration Structure
Before understanding how to setup the central logging sever, it is good to understand the configuration structure of rsyslog.
Rsyslog configuration files are structed in the following manner
- Modules
- Configuration Directives
- Rule line
Modules
Rsyslog has a modular architecture. It enables functionalities to be
added dynamically through these modules. The modules are categorized as:
- Input Modules – Used to gather messages from various sources
- Output Modules – Used to write the messages to various places ( file, socket etc.. )
- Parser Modules – Used to parse the message content
Please note that there are also other categories of modules available. This is to give an overview of what modules can do.
Configuration Directives
All configuration directives must be specified one per line and must start with dollar sign ($). It affects the rules.
Rule line
Every rule line consists of two fields, a ‘selector field’ and an
‘action field’. The selector field is divided into two, ‘facilities
& priorities’. Action specifies what action must be taken for the
matched rule.
A Sample Configuration
######################
MODULES
######################
$ModLoad imuxsock
$ModLoad imklog
######################
Directives
######################
# Set the default permissions for all log files.
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
######################
RULES
######################
mail.info /var/log/mail.info
mail.warn /var/log/mail.warn
mail.err /var/log/mail.err
daemon.* /var/log/daemon.log
Note:
10 Examples for Viewing Huge Log Files in Linux might be helpful when you are manipulating log files.
Templates
Templates are a very important features provided by rsyslog. It
allows the user to log the messages in their desirable format. It can
also be used to create dynamic file names to log the messages. In case
of database logging, the templates are used to convert the message into a
proper SQL statement.
A sample template will look like:
$template mytemplate “Text-Before %msg% Text-After\n”
The above template will log the message “This is hello from rsyslog” as:
Text-Before This is hello from rsyslog Text-After
We will see how to use the template for generate the log files dynamically.
Central Logging Server
The above sections should have given an overview about rsyslog and
how to configure it. Now we will move on to setup a central logging
system.
For our discussion we will have server IP as “192.168.1.1” for the
central log server, where all the log messages from client should be
forwarded.
Add the following lines to the rsyslog.conf of the central log server
servers (In this example, the following line was added on the log
server with ip-address 192.168.1.1):
# provides support for local system logging
$ModLoad imuxsock
# provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# provides UDP syslog reception. For TCP, load imtcp.
$ModLoad imudp
# For TCP, InputServerRun 514
$UDPServerRun 514
# This one is the template to generate the log filename dynamically, depending on the client's IP address.
$template FILENAME,"/var/log/%fromhost-ip%/syslog.log"
# Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by the template FILENAME.
*.* ?FILENAME
After adding the above lines to the rsyslog.conf, restart the rsyslog
process. Now the rsyslog server will be ready to accept messages.
# service rsyslog restart
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Add the following lines to the rsyslog.conf on the individual client
machines that should send their log messages to the central server.
$ModLoad imuxsock
$ModLoad imklog
# Provides UDP forwarding. The IP is the server's IP address
*.* @192.168.1.1:514
# Provides TCP forwarding. But the current server runs on UDP
# *.* @@192.168.1.1:514
Restart the rsyslog process on the clients. Now the rsyslog central
server (In this example, 192.168.1.1) will receive all the log messages
from the configured clients and each client’s log will be placed under a
separate directory.
