Contents

Overview

System logging is done in the folder "/var/log/" . There are different log files in this folder. We can have system log files and we can also have applications log in this folder. As an example for the hills server we have the system log file "messages" and for application "Anaconda" the log files are in the folder:

/var/log/anaconda

General system logging

Debian-based systems like Ubuntu store this in /var/log/syslog, while Red Hat-based systems like RHEL or CentOS use /var/log/messages. A sample of the log file may look like:

Dec 18 10:08:55 localhost kernel: sdb: Current [descriptor]: sense key: Recovered Error Dec 18 10:08:55 localhost kernel: Add. Sense: ATA pass through information available Dec 18 10:08:55 localhost kernel: Dec 18 10:08:55 localhost kernel: Descriptor sense data with sense descriptors (in hex): Dec 18 10:08:55 localhost kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Dec 18 10:08:55 localhost kernel: 00 4f 00 c2 00 50 The format of the file is:

The format of the log file fields is as follows:

 the date and time of the message.
 the name of the hosst from which the message came from.
 the program that generated the message.
 the process number (enclosed in square brackets) of the program that generated the message.
 the actual log message.

/var/log/auth.log and /var/log/secure store all security-related
events such as logins, root user actions, and output from
pluggable authentication modules (PAM). Ubuntu and Debian use
/var/log/auth.log, while Red Hat and CentOS use /var/log/secure.

/var/log/kern.log stores kernel events, errors, and warning
logs, which are particularly helpful for troubleshooting custom kernels.


/var/log/cron stores information about scheduled tasks (cron jobs). Use
this data to verify your cron jobs are running successfully.

Syslog and journald are both logging systems used on Linux systems, but they differ in their approach to storing and managing log data. Syslog traditionally uses plain text files for storing logs, while journald, part of systemd, utilizes a binary format for storing logs in a journal. Journald offers features like structured logging, indexing for fast search, and integration with systemd, making it a popular choice for modern Linux distributions.

journald

We shall only look at "journald" and not "syslog". The journald uses a configuration file located at:

/etc/systemd/journald.conf

This file also exists on the hills server. The contents of this file on the hills server are:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

We see that the settings are commented out which means that the default settings are being used. We can override those setting by modifying the "journald.conf" file. We can have sectioons in this file:

Sections in journald.conf
[Journal]: This section contains settings related to the journal
itself, such as storage, compression, and rotation.

[ForwardToSyslog]: This section controls forwarding
of messages to syslog.

[System] and [User]: These sections configure system-wide
and user-specific settings, respectively.

Rotation refers to the process of managing files. What should
happen when a file becomes too large. One option is to archive it and
start with an empty log file. The other option is to delete it. Rotation
can also depend on multiple criteria such as the time elapsed since the file
was created.

SystemMaxFileSize= and RuntimeMaxFileSize=: Set the maximum size of individual
journal files for system and runtime journals.

MaxFileSec=: Controls the maximum time to store journal entries.

MaxRetentionSec=: Specifies the maximum time to keep

We do not have a utility to set the configuration for jounald but we do have something that can be used to view the entries. Unfortunately we do not have permissions on the Linux system to run this utility.

[amittal@hills ~]$ journalctl
Hint: You are currently not seeing messages from other users and the system.
      Users in the 'systemd-journal' group can see all messages. Pass -q to
      turn off this notice.
No journal files were opened due to insufficient permissions.
[amittal@hills ~]$

Display the current boot messages.
$ journalctl -b

Using the time ranges "since" and "until" .

$ journalctl --since "1 hour ago"

$ journalctl --since "2 days ago"

$ journalctl --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00"

We can choose to display jounal entries by a particular unit.
This shows all messages logged by the Nginx web server.
$ journalctl -u nginx.service


$ journalctl -f

Prints messages to the console.

$ journalctl -u mysql.service -f

Prints only messages produced by the mysql.service .

We have the -o option to print the logs in a different format.
For example, the following command prints logs from the Apache web server in json-pretty format
$ journalctl -u apache2.service -r -o json-pretty


We can also display messages by user. First obtain the id of the user.

[amittal@hills ~]$ id amittal
uid=2548(amittal) gid=1258(csdept) groups=1258(

and then use ( if we had permissions which we don't on the hills server):
# journalctl _UID=2548