Documentation for a newer release is available. View Latest

Viewing and Managing Log Files

indexterm:[log files,System Log]indexterm:[log files,description] Last review: and a log file for cron tasks.

Log files can be very useful when trying to troubleshoot a problem with the system such as trying to load a kernel driver or when looking for unauthorized login attempts to the system. This chapter discusses where to find log files, how to view log files, and what to look for in log files. Some log files are controlled by a daemon called rsyslogd. The rsyslogd daemon is an enhanced replacement for sysklogd, and provides extended filtering, encryption protected relaying of messages, various configuration options, input and output modules, support for transportation via the TCP or UDP protocols. Note that rsyslog is compatible with sysklogd.

Log files can also be managed by the journald daemon – a component of systemd. The journald daemon captures Syslog messages, kernel log messages, initial RAM disk and early boot messages as well as messages written to standard output and standard error output of all services, indexes them and makes this available to the user. The native journal file format, which is a structured and indexed binary file, improves searching and provides faster operation, and it also stores meta data information like time stamps or user IDs. Log files produced by journald are by default not persistent, log files are stored only in memory or a small ring-buffer in the /run/log/journal/ directory. The amount of logged data depends on free memory, when you reach the capacity limit, the oldest entries are deleted. However, this setting can be altered – see Enabling Persistent Storage. For more information on Journal see Using the Journal.

By default, only journald is installed on your system. You have to install rsyslog youself. Also do not forget to enable and start it after install before continuing with rest of this guide. The journald daemon is the primary tool for troubleshooting. It also provides additional data necessary for creating structured log messages. Data acquired by journald is forwarded into the /run/systemd/journal/syslog socket that may be used by rsyslogd to process the data further. However, rsyslog does the actual integration by default via the imjournal input module, thus avoiding the aforementioned socket. You can also transfer data in the opposite direction, from rsyslogd to journald with use of omjournal module. See Interaction of Rsyslog and Journal for further information. The integration enables maintaining text-based logs in a consistent format to ensure compatibility with possible applications or configurations dependent on rsyslogd. Also, you can maintain rsyslog messages in a structured format (see Structured Logging with Rsyslog).

Locating Log Files

A list of log files maintained by rsyslogd can be found in the /etc/rsyslog.conf configuration file. Most log files are located in the /var/log/ directory. Some applications such as httpd and samba have a directory within /var/log/ for their log files. You may notice multiple files in the /var/log/ directory with numbers after them (for example, cron-20100906). These numbers represent a time stamp that has been added to a rotated log file. Log files are rotated so their file sizes do not become too large. The logrotate package contains a cron task that automatically rotates log files according to the /etc/logrotate.conf configuration file and the configuration files in the /etc/logrotate.d/ directory.

Basic Configuration of Rsyslog

The main configuration file for rsyslog is /etc/rsyslog.conf. Here, you can specify global directives, modules, and rules that consist of filter and action parts. Also, you can add comments in the form of text following a hash sign (#).

Filters

A rule is specified by a filter part, which selects a subset of syslog messages, and an action part, which specifies what to do with the selected messages. To define a rule in your /etc/rsyslog.conf configuration file, define both, a filter and an action, on one line and separate them with one or more spaces or tabs.

rsyslog offers various ways to filter syslog messages according to selected properties. The available filtering methods can be divided into Facility/Priority-based, Property-based, and Expression-based filters.

Facility/Priority-based filters: The most used and well-known way to filter syslog messages is to use the facility/priority-based filters which filter syslog messages based on two conditions: facility and priority separated by a dot. To create a selector, use the following syntax:

FACILITY.PRIORITY

where:

FACILITY specifies the subsystem that produces a specific syslog message. For example, the mail subsystem handles all mail-related syslog messages. FACILITY can be represented by one of the following keywords (or by a numerical code): kern (0), user (1), mail (2), daemon (3), auth (4), syslog (5), lpr (6), news (7), uucp (8), cron (9), authpriv (10), ftp (11), ntp (12), logaudit (13), logalert (14), clock (15), and local0 through local7 (16 - 23).
PRIORITY specifies a priority of a syslog message. PRIORITY can be represented by one of the following keywords (or by a number): debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0).

The aforementioned syntax selects syslog messages with the defined or higher priority. By preceding any priority keyword with an equal sign (=), you specify that only syslog messages with the specified priority will be selected. All other priorities will be ignored. Conversely, preceding a priority keyword with an exclamation mark (!) selects all syslog messages except those with the defined priority.

In addition to the keywords specified above, you may also use an asterisk (*) to define all facilities or priorities (depending on where you place the asterisk, before or after the comma). Specifying the priority keyword none serves for facilities with no given priorities. Both facility and priority conditions are case-insensitive.

To define multiple facilities and priorities, separate them with a comma (,). To define multiple selectors on one line, separate them with a semi-colon (;). Note that each selector in the selector field is capable of overwriting the preceding ones, which can exclude some priorities from the pattern.

Contoh 1. Facility/Priority-based Filters

The following are a few examples of simple facility/priority-based filters that can be specified in /etc/rsyslog.conf. To select all kernel syslog messages with any priority, add the following text into the configuration file:

kern.*

To select all mail syslog messages with priority crit and higher, use this form:

mail.crit

To select all cron syslog messages except those with the info or debug priority, set the configuration in the following form:

cron.!info,!debug

Property-based filters: Property-based filters let you filter syslog messages by any property, such as timegenerated or syslogtag. For more information on properties, see Properties. You can compare each of the specified properties to a particular value using one of the compare-operations listed in Property-based compare-operations. Both property names and compare operations are case-sensitive.

Property-based filter must start with a colon (:). To define the filter, use the following syntax:

:PROPERTY, [!]COMPARE_OPERATION, "STRING"

where:

The PROPERTY attribute specifies the desired property.
The optional exclamation point (!) negates the output of the compare-operation. Other Boolean operators are currently not supported in property-based filters.
The COMPARE_OPERATION attribute specifies one of the compare-operations listed in Property-based compare-operations.
The STRING attribute specifies the value that the text provided by the property is compared to. This value must be enclosed in quotation marks. To escape certain character inside the string (for example a quotation mark (")), use the backslash character (\).

Tabel 1. Property-based compare-operations
Compare-operation	Description
`contains`	Checks whether the provided string matches any part of the text provided by the property. To perform case-insensitive comparisons, use `contains_i`.
`isequal`	Compares the provided string against all of the text provided by the property. These two values must be exactly equal to match.
`startswith`	Checks whether the provided string is found exactly at the beginning of the text provided by the property. To perform case-insensitive comparisons, use `startswith_i`.
`regex`	Compares the provided POSIX BRE (Basic Regular Expression) against the text provided by the property.
`ereregex`	Compares the provided POSIX ERE (Extended Regular Expression) regular expression against the text provided by the property.
`isempty`	Checks if the property is empty. The value is discarded. This is especially useful when working with normalized data, where some fields may be populated based on normalization result.

Contoh 2. Property-based Filters

The following are a few examples of property-based filters that can be specified in /etc/rsyslog.conf. To select syslog messages which contain the string error in their message text, use:

:msg, contains, "error"

The following filter selects syslog messages received from the host name host1:

:hostname, isequal, "host1"

To select syslog messages which do not contain any mention of the words fatal and error with any or no text between them (for example, fatal lib error), type:

:msg, !regex, "fatal .* error"

Expression-based filters: Expression-based filters select syslog messages according to defined arithmetic, Boolean or string operations. Expression-based filters use rsyslog's own scripting language called RainerScript to build complex filters.

The basic syntax of expression-based filter looks as follows:

if EXPRESSION then ACTION else ACTION

where:

The EXPRESSION attribute represents an expression to be evaluated, for example: $msg startswith 'DEVNAME' or $syslogfacility-text == 'local0'. You can specify more than one expression in a single filter by using and and or operators.
The ACTION attribute represents an action to be performed if the expression returns the value true. This can be a single action, or an arbitrary complex script enclosed in curly braces.
Expression-based filters are indicated by the keyword if at the start of a new line. The then keyword separates the EXPRESSION from the ACTION. Optionally, you can employ the else keyword to specify what action is to be performed in case the condition is not met.

With expression-based filters, you can nest the conditions by using a script enclosed in curly braces as in Expression-based Filters. The script allows you to use facility/priority-based filters inside the expression. On the other hand, property-based filters are not recommended here. RainerScript supports regular expressions with specialized functions re_match() and re_extract().

Contoh 3. Expression-based Filters

The following expression contains two nested conditions. The log files created by a program called prog1 are split into two files based on the presence of the "test" string in the message.

if $programname == 'prog1' then {
   action(type="omfile" file="/var/log/prog1.log")
   if $msg contains 'test' then
     action(type="omfile" file="/var/log/prog1test.log")
   else
     action(type="omfile" file="/var/log/prog1notest.log")
}

See Online Documentation for more examples of various expression-based filters. RainerScript is the basis for rsyslog's new configuration format, see Using the New Configuration Format

Actions

Actions specify what is to be done with the messages filtered out by an already-defined selector. The following are some of the actions you can define in your rule:

Saving syslog messages to log files: The majority of actions specify to which log file a syslog message is saved. This is done by specifying a file path after your already-defined selector:

FILTER PATH

where FILTER stands for user-specified selector and PATH is a path of a target file.

For instance, the following rule is comprised of a selector that selects all cron syslog messages and an action that saves them into the /var/log/cron.log log file:

cron.* /var/log/cron.log

By default, the log file is synchronized every time a syslog message is generated. Use a dash mark (-) as a prefix of the file path you specified to omit syncing:

FILTER -PATH

Note that you might lose information if the system terminates right after a write attempt. However, this setting can improve performance, especially if you run programs that produce very verbose log messages.

Your specified file path can be either static or dynamic. Static files are represented by a fixed file path as shown in the example above. Dynamic file paths can differ according to the received message. Dynamic file paths are represented by a template and a question mark (?) prefix:

FILTER ?DynamicFile

where DynamicFile is a name of a predefined template that modifies output paths. You can use the dash prefix (-) to disable syncing, also you can use multiple templates separated by a colon (;). For more information on templates, see Generating Dynamic File Names.

If the file you specified is an existing terminal or /dev/console device, syslog messages are sent to standard output (using special terminal-handling) or your console (using special /dev/console-handling) when using the X Window System, respectively.

Sending syslog messages over the network: rsyslog allows you to send and receive syslog messages over the network. This feature allows you to administer syslog messages of multiple hosts on one machine. To forward syslog messages to a remote machine, use the following syntax:

@(zNUMBER)HOST:PORT

where:

The at sign (@) indicates that the syslog messages are forwarded to a host using the UDP protocol. To use the TCP protocol, use two at signs with no space between them (@@).
The optional zNUMBER setting enables zlib compression for syslog messages. The NUMBER attribute specifies the level of compression (from 1 – lowest to 9 – maximum). Compression gain is automatically checked by rsyslogd, messages are compressed only if there is any compression gain and messages below 60 bytes are never compressed.
The HOST attribute specifies the host which receives the selected syslog messages.
The PORT attribute specifies the host machine’s port.

When specifying an IPv6 address as the host, enclose the address in square brackets ([, ]).

Contoh 4. Sending syslog Messages over the Network

The following are some examples of actions that forward syslog messages over the network (note that all actions are preceded with a selector that selects all messages with any priority). To forward messages to 192.168.0.1 via the UDP protocol, type:

*.* @192.168.0.1

To forward messages to "example.com" using port 18 and the TCP protocol, use:

*.* @@example.com:18

The following compresses messages with zlib (level 9 compression) and forwards them to 2001:db8::1 using the UDP protocol

*.* @(z9)[2001:db8::1]

Output channels: Output channels are primarily used to specify the maximum size a log file can grow to. This is very useful for log file rotation (for more information see Log Rotation). An output channel is basically a collection of information about the output action. Output channels are defined by the $outchannel directive. To define an output channel in /etc/rsyslog.conf, use the following syntax:

$outchannel NAME, FILE_NAME, MAX_SIZE, ACTION

where:

The NAME attribute specifies the name of the output channel.
The FILE_NAME attribute specifies the name of the output file. Output channels can write only into files, not pipes, terminal, or other kind of output.
The MAX_SIZE attribute represents the maximum size the specified file (in FILE_NAME) can grow to. This value is specified in bytes.
The ACTION attribute specifies the action that is taken when the maximum size, defined in MAX_SIZE, is hit.

To use the defined output channel as an action inside a rule, type:

FILTER :omfile:$NAME

Contoh 5. Output channel log rotation

The following output shows a simple log rotation through the use of an output channel. First, the output channel is defined via the $outchannel directive:

 $outchannel log_rotation, /var/log/test_log.log, 104857600, /home/joe/log_rotation_script

and then it is used in a rule that selects every syslog message with any priority and executes the previously-defined output channel on the acquired syslog messages:

*.* :omfile:$log_rotation

Once the limit (in the example 100 MB) is hit, the /home/joe/log_rotation_script is executed. This script can contain anything from moving the file into a different folder, editing specific content out of it, or simply removing it.

Sending syslog messages to specific users: rsyslog can send syslog messages to specific users by specifying a user name of the user you want to send the messages to (as in Specifying Multiple Actions). To specify more than one user, separate each user name with a comma (,). To send messages to every user that is currently logged on, use an asterisk (*).
Executing a program: rsyslog lets you execute a program for selected syslog messages and uses the system() call to execute the program in shell. To specify a program to be executed, prefix it with a caret character (^). Consequently, specify a template that formats the received message and passes it to the specified executable as a one line parameter (for more information on templates, see Templates).

FILTER ^EXECUTABLE; TEMPLATE

Here an output of the FILTER condition is processed by a program represented by EXECUTABLE. This program can be any valid executable. Replace TEMPLATE with the name of the formatting template.

Contoh 6. Executing a Program

In the following example, any syslog message with any priority is selected, formatted with the template template and passed as a parameter to the test-program program, which is then executed with the provided parameter:

*.* ^test-program;template

Be careful when using the shell execute action

When accepting messages from any host, and using the shell execute action, you may be vulnerable to command injection. An attacker may try to inject and execute commands in the program you specified to be executed in your action. To avoid any possible security threats, thoroughly consider the use of the shell execute action.

Storing syslog messages in a database: Selected syslog messages can be directly written into a database table using the database writer action. The database writer uses the following syntax:

:PLUGIN:DB_HOST,DB_NAME,DB_USER,DB_PASSWORD;TEMPLATE

where:

The PLUGIN calls the specified plug-in that handles the database writing (for example, the ommysql plug-in).
The DB_HOST attribute specifies the database host name.
The DB_NAME attribute specifies the name of the database.
The DB_USER attribute specifies the database user.
The DB_PASSWORD attribute specifies the password used with the aforementioned database user.
The TEMPLATE attribute specifies an optional use of a template that modifies the syslog message. For more information on templates, see Templates.

Using MySQL and PostgreSQL

Currently, rsyslog provides support for MySQL and PostgreSQL databases only. In order to use the MySQL and PostgreSQL database writer functionality, install the rsyslog-mysql and rsyslog-pgsql packages, respectively. Also, make sure you load the appropriate modules in your /etc/rsyslog.conf configuration file:

$ModLoad ommysql    # Output module for MySQL support
$ModLoad ompgsql    # Output module for PostgreSQL support

For more information on rsyslog modules, see Using Rsyslog Modules.

Alternatively, you may use a generic database interface provided by the omlibdb module (supports: Firebird/Interbase, MS SQL, Sybase, SQLLite, Ingres, Oracle, mSQL).

Discarding syslog messages: To discard your selected messages, use the tilde character (~).

FILTER ~

The discard action is mostly used to filter out messages before carrying on any further processing. It can be effective if you want to omit some repeating messages that would otherwise fill the log files. The results of discard action depend on where in the configuration file it is specified, for the best results place these actions on top of the actions list. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines.

For instance, the following rule discards any cron syslog messages:

cron.* ~

Specifying Multiple Actions

For each selector, you are allowed to specify multiple actions. To specify multiple actions for one selector, write each action on a separate line and precede it with an ampersand (&) character:

FILTER ACTION
& ACTION
& ACTION

Specifying multiple actions improves the overall performance of the desired outcome since the specified selector has to be evaluated only once.

Contoh 7. Specifying Multiple Actions

In the following example, all kernel syslog messages with the critical priority (crit) are sent to user user1, processed by the template temp and passed on to the test-program executable, and forwarded to 192.168.0.1 via the UDP protocol.

kern.=crit user1
& ^test-program;temp
& @192.168.0.1

Any action can be followed by a template that formats the message. To specify a template, suffix an action with a semicolon (;) and specify the name of the template. For more information on templates, see Templates.

Using templates

A template must be defined before it is used in an action, otherwise it is ignored. In other words, template definitions should always precede rule definitions in /etc/rsyslog.conf.

Templates

Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates. To create a template use the following syntax in /etc/rsyslog.conf:

$template TEMPLATE_NAME,"text %PROPERTY% more text", OPTION

where:

$template is the template d