Home Linux Admin Introduction Directory Tree Disks/File Systems Memory Mgmt Monitoring Startup/ShutDown Logging in/out User Accounts Backups Processes Cron Packages Books

Startup and Shutdown

Contents

Overview

After a Linux installation the system has a configuration. This configuration consists of things like what filesystems to mount, what daemons to start, what mode ( single user or multi user mode ) to start and so on. How can we change this configuration ? What steps should we take to debug an issue at startup ? How should a system be shutdown ? What changes can be made to perform custom tasks during shutdown ? This chapter covers these topics.

Any Linux system will follow the below process for starting up.

The init version have some differences and some commonality. The hills server is using "systemd" We can see this on the hills server:


[amittal@hills ~]$ ps -aef | grep systemd | head
root           1       0  0 Jan12 ?        00:30:46 /usr/lib/systemd/systemd --switched-root --system --deserialize 18
root

Also the "init" program points to the systemd program

[amittal@hills sbin]$ pwd
/usr/sbin
[amittal@hills sbin]$ ls -l init*
lrwxrwxrwx. 1 root root 22 Jul 28  2021 init -> ../lib/systemd/systemd
[amittal@hills sbin]$


The init process will have the process id of 1. The process id's are assigned sequentially and "1" is the first process id as zero is not used. The init process fathers all the rest of the processes directly or indirectly. A process is created from another process called the parent process. The init process keeps running until the system is shutdown. The init process can create other processes needed for startup and execute shell scripts also. The init process will also become parent to orphaned processes. We shall study processes in detail in a later chapter. We change the bootup configurations by either changing the arguments to the kernel or changing the configuration files. What configuration files do we change. This is dependent on the init implementation. An init process might use a concept called runlevel or something very similar.

Runlevel

A runlevel is an operating system state. It defines what services are currently running. The run level is a number from 0 to 6. 

Runlevel 0
shuts down the system

Runlevel 1
single-user mode. No networking.

Runlevel 2
multi-user mode without networking

Runlevel 3
multi-user mode with networking and no graphics.

Runlevel 4
user-definable

Runlevel 5
multi-user mode with networking with graphics.

Runlevel 6
reboots the system to restart it


When the system boots up only 1 runlevel is run. Runlevel 1 is single user mode. It is also called the maintenance mode. It allows us to fix issues like reset root password. We need physical access to the computer. We also have the "runlevel" command 

[amittal@hills ~]$ runlevel
N 3
[amittal@hills ~]$


As expected the hills server is running on level 3. The concept of runlevel is not applicable to systemd and the hills server run systemd. The runlevels are mapped to targets ( a target is similar to runlevel but is a systemd concept ) .

SysVinit

This is the old style implementation of init. The executable is at "/bin/init". It uses a configuration file called "/etc/inittab". We can find this file on the hills server but if we look at the contents we shall see that it is not being used because the hills server is using "SystemD". SysVinit uses the concept of runlevels. An entry in "inittab" is of the following format: 

id:rstate:action:process

id     -- A unique indentifier for the entry.
rstate -- Lists the runlevel to which the entry applies to
action -- How is the next field going to be run. Possible values
          include: initdefault, sysinit, boot, bootwait, wait, and respawn.
          If the value is "initdefault" then it identifies the default run level.
process -- Defines the process or script to execute.

Sample inittab file:

1 ap::sysinit:/sbin/autopush -f /etc/iu.ap
2 ap::sysinit:/sbin/soconfig -f /etc/sock2path
3 fs::sysinit:/sbin/rcS sysinit   >/dev/msglog 2<>/dev/msglog </dev/console
4 is:3:initdefault:
5 p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/...
6 sS:s:wait:/sbin/rcS              >/dev/msglog 2
7 s0:0:wait:/sbin/rc0              >/dev/msglog 2
8 s1:1:respawn:/sbin/rc1           >/dev/msglog 2
9 s2:23:wait:/sbin/rc2             >/dev/msglog 2
10 s3:3:wait:/sbin/rc3             >/dev/msglog 2
11 s5:5:wait:/sbin/rc5             >/dev/msglog 2
12 s6:6:wait:/sbin/rc6             >/dev/msglog 2
13 fw:0:wait:/sbin/uadmin 2 0      >/dev/msglog 2
14 of:5:wait:/sbin/uadmin 2 6      >/dev/msglog 2
15 rb:6:wait:/sbin/uadmin 2 1      >/dev/msglog 2
16 sc:234:respawn:/usr/lib/saf/sac -t 300
17 co:234:respawn:/usr/lib/saf/ttymon -g -h -p "`uname -n` console login: "
   -T terminal-type -d /dev/console -l console
-m ldterm,ttcompat


From the above we notice the entry at line 4. This states that the default runlevel is 3. The man page for "inittab" contains a more detailed descripton of each of these fields. The "wait" entry in the theird field starts the process and waits till the process is up and running before moving to the next entry. Init starts the different scripts and processes. We can change the runlevel once we are logged in with the "telinit" command. 

root@ajkumar08-PC:/home/ajay#
Prints the current runlevel of 5. "N" means there is no previous runlevel.
root@ajkumar08-PC:/home/ajay# runlevel
N 5
Change the runlevel to 3.
root@ajkumar08-PC:/home/ajay# telinit 3

Prints the previous run
root@ajkumar08-PC:/home/ajay# runlevel
5 3
root@ajkumar08-PC:/home/ajay# telinit 5
root@ajkumar08-PC:/home/ajay# runlevel
3 5

We can also use the "who -r" command to display the runlevel.
ajay@ajkumar08-PC:~$ who -r
         run-level 5  2023-03-17 23:12                   last=3
ajay@ajkumar08-PC:~$


The process now has to trigger scripts to start subsystems. It does this by running shell scripts depending on the runlevel we have. There is a folder for each runlevel. The below is a snapshot for the Debin 11 Linux folders.

See full image


Each folders contains scripts that are links to scripts in the "init.d" folder. 

ajay@ajkumar08-PC:~$ cd /etc/rc0.d
ajay@ajkumar08-PC:/etc/rc0.d$ ls -l
total 0
lrwxrwxrwx 1 root root 20 Feb 12 19:45 K01alsa-utils -> ../init.d/alsa-utils
lrwxrwxrwx 1 root root 22 Feb 12 19:48 K01avahi-daemon -> ../init.d/avahi-daemon
lrwxrwxrwx 1 root root 19 Feb 12 19:45 K01bluetooth -> ../init.d/bluetooth
lrwxrwxrwx 1 root root 22 Feb 12 19:49 K01cups-browsed -> ../init.d/cups-browsed
lrwxrwxrwx 1 root root 14 Feb 12 19:51 K01gdm3 -> ../init.d/gdm3
lrwxrwxrwx 1 root root 20 Feb 12 19:01 K01hwclock.sh -> ../init.d/hwclock.sh
lrwxrwxrwx 1 root root 23 Feb 19 17:12 K01lvm2-lvmpolld -> ../init.d/lvm2-lvmpolld
lrwxrwxrwx 1 root root 20 Feb 12 19:02 K01networking -> ../init.d/networking
lrwxrwxrwx 1 root root 18 Feb 12 19:46 K01plymouth -> ../init.d/plymouth
lrwxrwxrwx 1 root root 37 Feb 12 19:49 K01pulseaudio-enable-autospawn -> ../init.d/pulseaudio-enable-autospawn
lrwxrwxrwx 1 root root 17 Feb 12 19:02 K01rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root 15 Feb 12 19:49 K01saned -> ../init.d/saned
lrwxrwxrwx 1 root root 27 Feb 12 19:48 K01speech-dispatcher -> ../init.d/speech-dispatcher
lrwxrwxrwx 1 root root 14 Feb 12 19:02 K01udev -> ../init.d/udev
lrwxrwxrwx 1 root root 29 Feb 12 19:48 K01unattended-upgrades -> ../init.d/unattended-upgrades


The scripts within each directory are named with either a capital S, or a capital K, followed by a two-digit number, followed by the name of the service being referenced. The files beginning with capital S represent scripts which are started upon entering that runlevel, while files beginning with capital K represent scripts which are stopped. The numbers specify the order in which the scripts should be executed. The main script "rc script" may reside on a folder "/etc/rc.d/rc" and is responsible for starting the scripts. Note that even though the Hills server does not use "SysVinit" it still has some folders related to the utility "SysVinit". These are in the "/etc/rc.d" folder.
It is possible to get the status of a subsystem and even start/stop these subsystems manually. Hills service redirects the "service" command because it is running "SystemD". 

To start the crond subsystem
service crond start


[amittal@hills /]$ service crond status
Redirecting to /bin/systemctl status crond.service
? crond.service - Command Scheduler
   Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor prese>
   Active: active (running) since Thu 2023-01-12 09:57:48 PST; 2 months 4 days >
 Main PID: 1629 (crond)
    Tasks: 1 (limit: 75135)
   Memory: 51.0M
   CGroup: /system.slice/crond.service
           +-1629 /usr/sbin/crond -n

To stop the crond subsystem
service crond stop


SystemD

SystemD came out in 2011 and was developed by engineers working for Red Hat. Some definitions related to the SystemD utility.

Unit
A unit is a configuration file and can refer to a service, or a target or a socket, mount point or some other resource. It is the smallest object used to configure the system. A unit can also contain dependencies upon other units and information about the order in which the services should run.



Target
A target is a group of services, sockets, mount points and can be thought of as a state such as . It is similar to a runlevel but is more generic. Just like a default runlevel we have a default target.

Service
A service is a daemon that can be started and stopped. A daemon is a process that runs in the background. We can think of a service as a susbsytem also. So for example cron ( scheduler ) is a service. 

The configuration files can be found in the following folders. SystemD will look in the
order that the folders are listed in.We can find these folders on the hills server.

/etc/systemd/system: Local configuration
We can make copies of unit files and place them in this folder. This can be
used for customization.

/run/systemd/system: Runtime configuration
This directory holds transient unit files, typically generated at runtime.

/usr/lib/systemd/system:
Distribution-wide configuration. This will contain the unit files that came
with the distribution. Copies of these files can be made and placed
in the "/etc/systemd/system" folder.


SystemD will search the first folder for a unit configuration file and then the next one stopping till it finds one. This allows for customization. Suppose there is a distribution-wide configuration then we can place a custom unit configuration file of the same name in the lcoal configuration. A unit file can contain dependencies; these can be other units such as targets and services. A unit file begins with a section called unit. 



Target

On hills, the file "graphical.target" exists on the
"/usr/lib/systemd/system" folder .


File: graphical.target


[Unit]
Description=Graphical Interface
Documentation=man:systemd.special(7)
Requires=multi-user.target
Wants=display-manager.service
Conflicts=rescue.service rescue.target
After=multi-user.target rescue.service rescue.target display-manager.service
AllowIsolate=yes

This file does not specify what sercvices to run. The services will be
decided upon the "Requires" and "Wants" section. SystemD will examine those units
and ultimately the services will be run.
Unit dependencies are expressed though Requires, Wants, and Conflicts:

Requires: A list of units that this unit depends on, which is
started when this unit is started.

Wants: A weaker form of Requires: the units listed are started
but the current unit is not stopped if any of them fail.
This is not the only way we specify the wants units. We also have
folders such as:
/usr/lib/systemd/system/graphical.target.wants
This folder will contain the services that need to be run. It can
only contain symbolic links. We can manually create the symbolic
links or have the system create them for us by inserting something
like :

[Install]
WantedBy=multi-user.target

in a service file.

Conflicts: A negative dependency: the units listed are stopped
when this one is started and, conversely, if one of them is
started, this one is stopped

Before: This unit should be started before the units listed
After: This unit should be started after the units listed
AllowIsolate: Allows a unit to be used in a manner similar to a runlevel


Service

A unit can also be a service and will have a service section. 



File: sshd.service


[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/crypto-policies/back-ends/opensshserver.config
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target


From the above:
Type=notify
The "Type" parameter can have the values "simple" or "notify" . If the type is simple then systemd assumes the service completes it's startup as soon as it is spawned. However some services may not be ready till they do something like establish a connection to the database. The "notify" can be used in such cases. The "notify" ability allows the service which can be a script or a program to send messages to the systemd system. These messages can indicate to the systemd when the service is ready and also allow the user to query the systemd to view the messages. We shall look at a sample script and the commands involved in using it. 
We are going to use the site at:
https://www.redhat.com/en/interactive-labs/enterprise-linux
for this exercise.

First we create the script.

File: aj1.sh



function handle_signal()
{   systemd-notify --status "Exiting..."
    sleep 10
    exit 0
}

trap handle_signal SIGINT

systemd-notify --ready --status "Starting..."
sleep 10

while true
do
    systemd-notify --status "Sleeping..."
    sleep 5

    systemd-notify --status "Pinging..."
    timeout 5 ping 240.0.0.0 >& /dev/null
done

This script has a signal handler when the script is to be stopped.
This sends a message to the systemd system.
systemd-notify --status "Exiting..."
The systemd-notify is a shell command. We also have a "C" api call if
we want to send a message from a program.

We send a "ready" message with
systemd-notify --ready --status "Starting..."

and then run an infinite loop that does some work and sends status
messages to the systemd.

We also create a service for the script:

File: aj1.service

[Unit]
Description=Example script using systemd-notify

[Service]
Type=notify
ExecStart=/root/ajay/aj1.sh


[Install]
WantedBy=multi-user.target




We place this script and the service in the folder "ajay":

$mkdir ajay
$cd ajay
$vi aj1.sh
$vi aj1.service
$ls
$aj1.sh aj1.service
$ln -s /root/ajay/aj1.service /etc/systemd/system/aj1.service
$setenforce 0
$systemctl enable aj1
Created symlink /etc/systemd/system/multi-user.wants/aj1.service ->
/root/ajay/aj1.service
$setenforce 0
$systemctl daemon-reload
$systemctl status aj1.service
aj1.service - Example script using systemd-notify
Loaded
Active: inactive(dead)

$systemctl start aj1


We create a symbolic link with the command:
$ln -s /root/ajay/aj1.service /etc/systemd/system/aj1.service

We could also have created and placed the ".service" file in the
"/etc/systemd/system/" folder directly.
We enable it with the command:
systemctl enable aj1

This creates a symbolic link in the
multi-user.wants folder to the service .

Then we reload the systemd configuration.
systemctl daemon-reload
This command by itself will not run the service.
We check the status of the service using the command:

systemctl status aj1.service

and we can run the service with the command:

systemctl start aj1

The virtual system we have, does not have the socket setup for notify so
we get an error but this example shows how to use notify in the
service configuration file.




Let's go back to the sshd.service.

ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY

ExecReload=/bin/kill -HUP $MAINPID




The "ExecStart" as we can guess is for starting the service while "ExecReload" is
for restarting it. The command "man systemd.service" has more information
on these options. Targets have names ending in ".target". It is similar to a
runlevel in that it is a state although the scheme is different and more flexible
for selecton of services. We can create our own targets and use existing targets
as dependencies. The "ExecReload" is executed when the system restarts and here
we are basically killing the process. We have different parameters for different
actions.

If we wanted to run a custom script upon shutdown of a service we can
use the following "ExecStop" paramter.
ExecStop=/path/to/cleanup_script.sh




Mount Pt

We can also have a unit file that does mounting.

File: data.mount

[Unit]
Description=Data mount

[Mount]
What=/dev/disk/by-uuid/filesystem_UUID
Where=/mnt/data
Type=xfs
Options=defaults

[Install]
WantedBy=multi-user.target

Usually a simple mount command will look like:
mount /dev/sda1 /mnt/media
In the mount file the device path is denoted by "What" and the
mount point is the denoted by "Where" .

Socket

A unit file can also be a socket file. This makes socket communication a bit easier in certain cases. We can configure the unit files so that the systemd listens at a socket and we can manipulate it by using stdin and stdout. Let's look at a simple example.
File: echo.socket
# echo.socket
[Unit]
Description = Echo server

[Socket]
ListenStream = 4444
Accept = yes

[Install]
WantedBy = sockets.target

File: echo@.service
# echo@.service
[Unit]
Description=Echo server service

[Service]
ExecStart=/path/to/echo.py
StandardInput=socket

File: echo.py
#!/usr/bin/python
import sys
sys.stdout.write(sys.stdin.readline().strip().upper() + 'rn')

We first define a file "echo.socket" .

ListenStream = 4444
It can listen for datagrams, sequential packages. We are stating here
that listen for a stream at port "4444" .
Accept = yes
If this is AF_UNIX socket we can set it to false else for AF_INET it is
true.

We then define our service file with the same name
but with "@" added in: "echo@.service" .

ExecStart=/path/to/echo.py
StandardInput=socket

We specify the path of the program that is a Python file.
The "StandardInput=socket" setting states that we are using the
socket file for the standard input and output and the python file
is going to use that. The python file has the statement:

sys.stdout.write(sys.stdin.readline().strip().upper() + 'rn')

This reads from the sockets and converts the input line to upper case and
then writes the line back to the socket.
A sample run may look like:

$ socat - TCP:server_IP_address:4444
hello computer
HELLO COMPUTER

We use the tool "socat" to send the string "hello computer" to the listening
socket and the python program reads the string, converts it to uppercase and
writes it back to the socket. The systemd then prints the output to the console.

The kernel will start the SystemD process by starting init. On the hills server: 

[amittal@hills sbin]$ pwd
/sbin
[amittal@hills sbin]$ ls -l init
lrwxrwxrwx. 1 root root 22 Jul 28  2021 init -> ../lib/systemd/systemd
[amittal@hills sbin]$


Now "systemd" will run the default target by searching the 4 folders mentioned previously. 

[amittal@hills sbin]$ cd /etc/systemd/system
[amittal@hills system]$ ls -l default.target
lrwxrwxrwx. 1 root root 37 May  4  2021 default.target -> /lib/systemd/system/multi-user.target


[amittal@hills system]$
[amittal@hills system]$ cat /lib/systemd/system/multi-user.target
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
[amittal@hills system]$




If we study the folder "/usr/lib/systemd/system/multi-user.target.wants"
we notice that it does not contain the "crond.service". However the
"crond.service" should be somewhere in systemd because we have it on hills.


[amittal@hills ~]$ ps -ael | grep cron
4 S     0    1659       1  0  80   0 -  6483 -      ?        00:00:02 crond

Remember the order of the folders mentioned in the beginning.
The first folder is:
/etc/systemd/system
and if we look inside this folder, we can find the "crond.service" .
/etc/systemd/system/multi-user.target.wants
On the hills server the default.target is linked to the multi-user.target. This in turn has dependencies that will be evaluated and the process will repeat. We can also change the default target to boot to by passing in parameters to the kernel at boot time. Recall that the boot loader will load up the kernel . This can be done by editing the boot entry in the boot loader's selection menu. We can also set the parameters permanently by modifying the boot loader's configuration file. Suppose we have to add a "quiet" parameter to the kernel.The below steps apply to Grub loader. Press "e" when the menu shows up and add the parameter to the line at:

linux /boot/vmlinuz-linux root=UUID=0a3407de-014b-458b-b5c1-848e92a327a3 rw quiet

We can also edit the file:
/boot/grub/grub.cfg

Another way is to edit the file "/etc/default/grub" and change the line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

And then regenerate the "grub.cfg" file with the command:
grub-mkconfig -o /boot/grub/grub.cfg




We use the systemctl command to manually manage the services. It is part of systemd. 

[amittal@hills ~]$ systemctl status crond
? crond.service - Command Scheduler
   Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor prese>
   Active: active (running) since Tue 2025-04-29 14:41:27 PDT; 3 weeks 5 days a>
 Main PID: 1659 (crond)
    Tasks: 1 (limit: 74636)
   Memory: 37.8M
   CGroup: /system.slice/crond.service
           +-1659 /usr/sbin/crond -


We can also do "systemctl start", "systemctl stop" and "systemctl restart" to start, stop and restart the service. As expected we do not have permissions on the hills server for these functions. 

This enables the unit. By default the units are disabled.
ajay@ajkumar08-PC:/etc/default$ systemctl enable cron.service
The "systemctl" command by itself will list all the active unit files. The command "systemctl list-units --all" will list all the units that have been loaded or an attempt has been made to load them.

View the dependency tree for a unit. 

We list the dependencies for the default.target which in turn depends on units like
the multi-user.target.

ajay@ajkumar08-PC:/etc/default$ systemctl list-dependencies default.target | more
default.target
? +-accounts-daemon.service
? +-e2scrub_reap.service
? +-gdm.service
? +-switcheroo-control.service
? +-systemd-update-utmp-runlevel.service
? +-udisks2.service
? +-multi-user.target
?   +-anacron.service
?   +-avahi-daemon.service
...


Shutdown

We should always shutdown a system gracefully using either a GUI option or from the command line. This ensures that the file system is in a consistent states and data from the RAM is flushed out to the hard drive. Just powering off the system can and will create delays when bringing the system up.The shutdown command is: 

shutdown option time message


We can use just the command "shutdown" or specify a time at which a message will be sent to all the users. The "shutdown" without any options will power off the system. The time can be one of the following: Example:
$ shutdown -r +5 "Emergency maintenance required. Please log-off"

This states that after 5 minutes the system is going to restart. The "-H' means halt and is legacy with little use. It stops the CPU but the power remains on. We can diagnose it at a hardware level by checking voltage levels as an example. Once we are done we can power off manually. The options are: 

       --help
           Print a short help text and exit.

       -H, --halt
           Halt the machine.

       -P, --poweroff
           Power-off the machine (the default).

       -r, --reboot
           Reboot the machine.

       -h
           Equivalent to --poweroff, unless --halt is specified.

       -k
           Do not halt, power-off, reboot, just write wall message.

      --no-wall
           Do not send wall message before halt, power-off, reboot.

       -c
           Cancel a pending shutdown. This may be used cancel the effect of an invocation of
           shutdown with a time argument that is not "+0" or "now".


As expected, we cannot run the "shutdown" command on the hills server and need to run it on a personal or virtual machine in order to try it out. We can try it out on browser red hat virtual machine. 
https://www.redhat.com/en/interactive-labs/enterprise-linux

shutdown -r +5 "Emergency maintenance required. Please log-off"
$ shutdown -r +5 "Emergency maintenance required. Please log-off"
Reboot scheduled for Mon 2025-05-26 13:55:17 UTC, use 'shutdown -c'
to cancel

The message shold appear on the other users screen who are logged
on but it will not appear for the user who ran the command.
After about 5 minutes we will see on the screen.
$ Connection lost, trying to reconnect