As we have seen, the names and locations of log files are configurable as are the types of messages they contain, simply by the customization of rsyslog.conf and rsyslog.d/* In this section we will go over the types of events that are normally logged, as well as the files that are used on a sample Linux system as their destination.

Startup Messages

As you know, Unix systems are very noisy during startup. Most of these messages are hardware status and are logged by the kernel, via rsyslogd. The kernel boot messages are usually saved in a log file in the master log directory. On Linux, this is /var/log/dmesg. You can examine this text file to see a copy of most of the kernel startup messages.

Once the system is up and running, kernel messages are normally processed by rsyslogd and are filtered as usual by the rules in /etc/rsyslog.conf.

The General Log file

Most systems have one master log file that is the default destination for general log messages. Messages that do not have their own specific log file are loggged in this file. We can examine the rsyslog.conf file for this default log file on our Linux system:

As you can see, some types of messages are not logged to the general log file. These are either deemed unimportant or are logged separately. Some types of messages are logged in the general log file and also in a specialized log file.

Specialized log files

Besides the general log file, which may be world-readable, certain types of messages are logged separately. We describe some of these separate logs below:

Kernel messages are usually hardware issues. Besides being logged in the general log file, they are also logged either on the system console, if there is one, or to a special log file. Emergency messages, which normally come from the kernel, are usually broadcast or written to root's terminal. Here is the configuration for emerg messages on linux

Standard packages

Often messages from certain programs or daemons are logged separately. Examples here include mail messages, news messages and messages from cron. (Note the dash prefix on mail's log file.)

Authentication messages are often logged in a file that is readable only by root:

Although not under the purview of rsyslog, one other log file deserves mention: wtmp. The wtmp file contains login/logout records. The wtmp file is filtered by the command last.

Logging your own information

One benefit and drawback of logfiles is that anyone can write to them. You do not have to be running as a privileged user. Most log entries are written by a C program calling the syslog(2) system call. From a shell script, however, you can use logger(1):

logger simply takes a facilty and priority pair, a tag, and a message. It then logs the indicated message to sys. For example, the logger command

logger -p authpriv.info -t 'gboyd-fooled u" "this is not really an authpriv message"

generates the following message in /var/log/secure

The remainder of this section is optional

Managing Log Files

Log files can get unwieldy, especially if the system has a lot of users or noisy hardware. Many systems provide some type of command to allow you to control the log files by some combination of archiving, rotating, or deleting old entries. On RedHat Linux, this command is logrotate.

Before discussing logrotate, we should define log file rotation. Log file rotation is one solution to unbounded log files. This solution involves periodic examination of log files and the "rotation" of a file that has reached some threshold value of time or size. To rotate, the log file is simply renamed with an extension. This extension used to be a simple number, it has now been replaced with a date string. Several such files are kept - the number is configurable. For example, if we are keeping 4 old versions of /var/log/messages, when it is time to rotate, the current messages.x where x is the oldest date is deleted, and the current messages becomes messages.y (y=today). If the rotation is done on a time basis, which is common, this results in the archival of 4 periods of old messages. The archived messages may also be compressed.

logrotate allows automatic rotation, compression and removal of log files. Each log file can be configured separately and managed by periodically running logrotate, usually automatically from a cron job. It uses a configuration file /etc/logrotate.conf that indicates the default configuration parameters. Then, for each log file under its control, it needs a section for that logfile like this

The specific directives can override the default paramenters in /etc/logrotate.conf. There is even the ability to run shell commands prior to or after rotation. 

As an example, consider the following simple logrotate.conf file

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

This results in the following log files for the messages file, for example:

$ ls /var/log/messages*
/var/log/messages        /var/log/messages-20111113
/var/log/messages-20111030  /var/log/messages-20111120
/var/log/messages-20111106

As you can see, logrotate's configuration file even has the ability to include logfile-specific sections from other files or from every file in a directory. On our systems, a directory of these files exists so that newly-installed programs that need logging can simply drop their parameters in the directory. The include directive in logrotate.conf would include the logfile-specific sections. For example, the syslog file in /etc/logrotate.d has a section that describes the rotation of all of the files that rsyslog manages:

For more information, see logrotate(8)

Analyzing log files

Sifting through voluminous logfiles can be tedious. Even using grep to analyze them is difficult. Since system processes write log messages in a standard way, however, tools can be used to analyze the files. On RedHat Linux the tool logwatch is useful.

To explain logwatch, we should examine a few standard log entries (from /var/log/secure) that might be interesting:

Nov 21 12:25:36 localhost pam: gdm-password[2273]: pam_unix(gdm-password:session): session opened for user gboyd by (uid=0)
Nov 21 16:55:27 localhost unix_chkpwd[7894]: password check failed for user (root)
Nov 21 16:55:27 localhost su: pam_unix(su-l:auth): authentication failure; logname=gboyd uid=501 euid=0 tty=pts/7 ruser=gboyd rhost=  user=root
Nov 21 16:55:35 localhost su: pam_unix(su-l:session): session opened for user root by gboyd(uid=501)

In the above entries note the following about the format:

• the lines begin with the date-stamp and the host. This is standard. (my machine at home does not have a host name)

• the next piece consists of information about the service that produced the entry. This is normally the tag of the rsyslog entry. The service information is followed by a colon. (Note that some lines have two service tags.) 

• the remainder of the entry is the message itself.

logwatch analyzes these messages using a set of Perl scripts that are custom-built for particular log entries organized by the service name. You can see these scripts beneath /usr/share/logwatch

We will point logwatch at a file that has only the entries above and see what it outputs. To accomplish the task, logwatch needs several pieces of information, given as options on the commandline:

--detail d  (d is high, med, or low)

--logfile logfilename  (actually, logfilename is a logfile-group. logwatch will analyze logfilename, logfilename.1, etc)

--logdir logfiledir  (defaults to /var/log)

--service s    (you can use all for all services)

--range  r  (here r is a date-range, or one of the known ranges: all, today, yesterday)

--print  (required to output the result. go figure)

Since this is not a real logfile, we have placed it in /tmp, with the name secure. (You must use the name of the logfile that it comes from: logwatch has scripts that understand the various logfile formats.)

Copyright 2013 Greg Boyd - All Rights Reserved.