File Systems: Part 1
Contents
Introduction
A filesystem has been explained in the overview section. overview.html
It is a specification for how data is stored on a storage media such as a hard drive and also the software that manipulates this data. We can see the filesystem on our Windows machines by right clicking on the drive and choosing properties.
We can see that in the above that the filesystem is "NTFS" . Similarly viewing the properties on a flash drive yields the following.
The filesystem is "FAT32". The filesystem is different on the flash drive because the flash drive could be used on a Unix or Apple machine and NTFS can only be used on a Windows machine.
Some filesystems that may be present on a Unix machine( can be):ext2,ext3,ext4, vfat .
We can also have partitions on a hard drive. We can divide a hard drive up into
partitions such that a file that is growing by size cannot enter another partition.
If a folder is growing by accumulating more files then it also won't enter another
partition.
Open up the "fdsk" utility on your windows machine to see the partitions.
In the above image we have 2 partitions on the hard disk drive. The "Disk1"
refers to the flash drive and has only 1 partition. Of the 2 partitions one is the
regular "C:" drive and the other is "System Reserved" . The "System Reserved" partition
can be used to recover or correct a corrupted system on the "C:" drive. The "System Reserved"
partition does not have a letter associated with it. So we cannot see it from the
file explorer or make changes to it. So we first have a partition on a device and
then we place a file system on it. In the example above we have 2 partitions that
have the file system "NTFS" installed on them.
Later on we will
see how partitions are used on a Linux system.
In Windows we have the lettering concept for different filesystems. Usually the lettering
scheme starts with "C:" and then "D:" . On Linux we have the root path indicated by
"/" and then we have folders/subfolders. If we insert a flash drive in Linux then how
do we access it ? We have to use a path. What path ? That depends. On Cygwin I have:
/cygdrive/d .
The mount command lists all the currently mounted systems. We can also use the "mount" command to actually mount a device file .The first argument is the device file and the second argument is the mount point. Ex: mount device1 /usr $ mount C:/cygwin64/bin on /usr/bin type ntfs (binary,auto) C:/cygwin64/lib on /usr/lib type ntfs (binary,auto) C:/cygwin64 on / type ntfs (binary,auto) C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto) D: on /cygdrive/d type vfat (binary,posix=0,user,noumount,au On Debain Linux /dev/sdc1 /media/ajay/65D0-3A85The process of associating the folder to use for the device is called mounting. If we plug in a device ( say a flash drive ) then a file is created in the "/dev" folder that is a device file. Linux will automatically mount this file for us. We can then use the "mount" command to associate a folder with this device file. Example:
mount /dev/sdc1/ /ajay/flashdriveWe can also unmount the folder from the device. We studied the folder structure of Linux from the root. We can have subfolders such as "tmp" and "usr" . Using the "mount" command we can map "tmp" to one hard drive and "usr" another" . This way if "tmp" gets too big it won't bring down rest of the system. To the user who logs in to the terminal the whole structure is seen as one entity. We can think of this folder structure as "logical filesystem" while the actual mount point ( folder ) can be thought of as a "physical filesystem" . Most of the file systems are mounted during boot/startup time. We can however plug in things like the flash drives. Linux has some empty folders that we can use to mount these devices. They may be named: "/mnt" , "/media" .
This introduction gives us some idea about filesystems, partitions, mounting and hard drives.
Building the directory tree
We saw in the chapter on directory tree what the common folders are underneath the root folder "/" . Many of these folders are actually on different filesystems ( different partitions ) and possibly on different hard drives also. Let us take a hypothetical example . Say we have 3 different filesystems on 3 different partitions.device 0 Has the current folder structure / dev tmp users etc boot The folders tmp and users are empty. device 1 has: / amittal lkao jwu device 2 has: / We first mount device 0 at "/" with the command. mount device0 / Now when we do "cd /" we can see the folders in device 0. Then we mount device 1 . mount device1 /users Now when the user does "cd /users" the user can see the folders "amittal" etc. Usually the folder mount path is empty. The device contents are placed underneath the folder path. Finally we mount device2 mount device2 /tmp The user does "cd /tmp" and does not see any folders because device2 is empty but the user is on a different filesystem.The concept of mounting is sometimes confusing becuase we are so used to windows filesystems.
Multiple Filesystems
We touched upon this topic in the previous section. Some advantages of multiple filesystems are:1) Size. A single file system can limit size. By mounting different folders to different devices we can oversome this limitation. We could have different partitions on a single hard drive and mount these partitions to the folders also. Some common folders that are mapped to their own filesystems are:
/tmp
This is the temporary folder and can grow easily. It is recommended that users clean up the files they have placed in this folder but often forget to do so.
/home
This folder can be "/users" ( or "/students" on hills ) also depending on your Linux version. It contains the user folders and these can grow pretty fast also as users use and write in their folders. Also a new user has to have a home folder.
/var
This folder can contain log files that can grow.
2) Data Protection
It's possible that a part of the hard drive can go bad but the problem will be localized to a particular filesystem and other parts of the operating system will not be affected. Of course if the kernel or an important part is corrupted then the system won't work. However the kernel contains static files that are not changed while user, tmp and var folders involve writing of files and that is where the disk problems can occur. The kernel is stored under "/boot" . Run the following command on the hills server and confirm that "/boot" is on it's own filesystem.
[amittal@hills home]$ mount | grep "sd" /dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
3) Efficiency
Any time we do a lot of rewrites to a filesystem we introduce fragmentation. Suppose we wrote a file at a certain position that was 4 kb. Then we wrote a file after that of 1 kb. After that another file was written on this track of 4kb. Now we delete this file of 1Kb. That leaves a gap in track. If most of our files are greater than 1kb then this gap stays. Over time this causes fragmentation. By separating our folders into different filesystems we can localize the fragmentation to one filesystem and other filesystems can operate in an efficient manner. There are tools that can "defragment" a drive. The process is a bit time consuming.
4) Backup
Backup tools can back up certain filesystems ( such as the ones whose data changes frequently ) .
5) Flexibility
Having multiple filesystems allows us to change a device more easily and mount another filesystem. A filesystem can also be mounted for 2 different systems allowing the systems to share the data.
Disadvantages
1) Having multiple physical filesystems adds complexity. We need to keep track of the different
partitions and possible different devices.
2) Space utilization. We might allocate too much space in which case space does not get
utilized or not enough space for a filesystem. In such cases the data will have to be
transferred to another filesystem. Logical volume managers rectify some of these problems.
The folders "/tmp" , "/var", "/users" can be on a different partition. The folder "/boot" is usually on a separate partition. It contains files that are used only at start up. After startup the Linux system needs certain folders to be on the same partition such as "/bin", "/etc" , "/dev" . We can find out the filesystems on our system with the "df" command. As an example on the Hills server:
[amittal@hills /]$ df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 6010900 0 6010900 0% /dev tmpfs 6028272 0 6028272 0% /dev/shm tmpfs 6028272 590904 5437368 10% /run tmpfs 6028272 0 6028272 0% /sys/fs/cgroup /dev/mapper/vg00-root 52416000 27564716 24851284 53% / /dev/mapper/vg00-tmp 5232640 69668 5162972 2% /tmp /dev/mapper/vg00-var 26201600 1019420 25182180 4% /var /dev/mapper/vg02-students 524027908 61478512 462549396 12% /students /dev/sda1 2086912 318876 1768036 16% /boot /dev/mapper/vg01-users 52403200 14806644 37596556 29% /users /dev/mapper/vg00-logs 5232640 69544 5163096 2% /logs tmpfs 1205652 0 1205652 0% /run/user/13479 tmpfs 1205652 0 1205652 0% /run/user/2548The device names containing "lv", "vg", "vol" are logical volumes managed by a LVM (Logical Volumen Management ).
Formatting
We have 2 kinds of formatting: low-level and high-level formatting.
1) Low-level formatting.
This is done at the factory and the disk is burned. The tracks are sectors are marked
on the blank disk. Bad sectors are identified.
The formatting is done on the whole disk.
2) High-level formatting.
Once the hard drive has been formatted at a low-level we can perform
a high-level formatting.
This is done at the user level. We can choose a file
system( FAT32, ext2, NTFS) and an empty file system is written to
a partition. The original data is not completely erased from the hard drive.
We cannot access the original data as a user but the drive
may contain the data and it may be retrievable by special means.
In Windows we can format from the file explorer.
For Linux we are going to explore how to format from the terminal. Please
save all the data on the flash drive as this will erase the existing
contents.
The below shows how a flash drive can be formatted in Debian Linus. If you have direct access to a Linux system then you may try the below. We cannot test this on the hills server. We first need to find out the mount point for our flash drive. We can do that with the "df" command. This shows the file systems and the "-h" option shows the size in KilBytes, MegaBytes etc. -rwxr-xr-x 1 root root 5 Jan 16 15:43 1.txt root@ajkumar08-PC:/home/ajay/flash# df -h Filesystem Size Used Avail Use% Mounted on udev 925M 0 925M 0% /dev tmpfs 189M 1.6M 187M 1% /run /dev/sda1 145G 4.9G 133G 4% / tmpfs 942M 0 942M 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 189M 932K 188M 1% /run/user/1000 /dev/sdc1 15G 16K 15G 1% /media/ajay/NEW VOLUME As we can guess the "/dev/sda1" is the hard drive device file and "/dev/sdc1" is the flash drive device file. We can also use the "mount" command. root@ajkumar08-PC:/home/ajay/flash# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=946364k,nr_inodes=207866,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=192844k,mode=755) /dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12684) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=192840k,nr_inodes=48210,mode=700,uid=1000,gid=1000) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) portal on /run/user /1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) /dev/sdc1 on /media/ajay/NEW VOLUME type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) root@ajkumar08-PC:/media/ajay/NEW VOLUME# Now we "unmount" the flash drive. umount /dev/sdc1 At this point we can switch to the root user. su root And now we format the flash drive. root@ajkumar08-PC:/home/ajay# /sbin/mkfs.vfat /dev/sdc1 mkfs.fat 4.2 (2021-01-31) We know that "sbin" is not for normal users so it is not on our path and we can give the full path for the "mkfs" command. We choose the "vfat" file system. We create an empty folder and use that to mount the flash drive. mount /dev/sdc1 /home/ajay/flash Once that is created we can confirm it using the "mount" command. /dev/sdc1 on /home/ajay/flash type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022, codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush, errors=remount-ro,uhelper=udisks2)
Block Size
When we read/write some data from the hard drive we cannot just read a few bytes, there is a minimum amount set that we have to read. This is called the block size. This is in the context of hard drives. Usually the size is 512 bytes.
Finding out the block size of the flash drive root@ajkumar08-PC:/media/ajay/0A3C-7AD3# /sbin/blockdev --getbsz /dev/sdc1 512
We also have a filesystem block size that we can set when formatting and this is usually a multiple of the hard drive block size. We can find the filesystem block size for our flash drive.
root@ajkumar08-PC:/media/ajay/0A3C-7AD3# stat -f /home/ajay/flash
File: "/home/ajay/flash"
ID: 83100000000 Namelen: 1530 Type: msdos
Block size: 8192 Fundamental block size: 8192
Blocks: Total: 1951244 Free: 1951239 Available: 1951239
Inodes: Total: 0 Free: 0
root@ajkumar08-PC:/media/ajay/0A3C-7AD3#
From the above, the block size is 8K.
Device Files
We covered little bit about the "/dev" folder in the section on Linux directory tree.There are different file types in Linux:Regular file Directory file Denotes a folder. Link file Symbolic link file These 2 files are device files Character special file Reads/writes a character at a time. Block special file Reads/writes a block of characters at a time. Socket file Used to network communication Named pipe file Similar to pipes in shell but the pipe has a name and processes can write and read to it.A device file contains information that allows the kernel to associate a device driver with it.We can also use it ( in certain cases ) for mounting. There are 3 properties associated with a device file:
1) Character or block device. 2) Major number. This signifies the type of device. Example: hard drive, printer, etc. 3) Minor number. This signifies the device number for the same type of device. If there are say 2 hard drives then this number could be 1 or 2 ( assuming each drive has a single partition) .
$ cd /dev $ ls -l sd* brw-rw---- 1 root disk 8, 0 Jan 12 09:57 sda brw-rw---- 1 root disk 8, 1 Jan 12 09:57 sda1 brw-rw---- 1 root disk 8, 2 Jan 12 09:57 sda2 brw-rw---- 1 root disk 8, 16 Jan 12 09:57 sdb brw-rw---- 1 root disk 8, 32 Jan 12 09:57 sdc brw-rw---- 1 root disk 8, 48 Jan 12 09:57 sdd brw-rw---- 1 root disk 8, 49 Jan 12 09:57 sdd1 $The above is an output on the hills server. The device files are special files. They do not contain information like the size . Instead information is kept in the inode. The first letter tells us whether the device file is character or block) c or b ). Instead of the size we have the major number ( 8 ) and minor number. The letter "a" in "sda" denotes that it is the first SCSI disk and the first partition. The "sda1" denotes the second partition.If the minor number differs by 1 then it's the next partition. The "b" , "c" and "d" are the other 3 disks with "d" having 2 partitions.
$ mount | grep "sd" /dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota) $
Partitioning
A partition allows us to use a single hard drive as multiple isoloated sections. Why would we want to do that ? A partition can have it's own filesystem. Suppose we wanted to have 2 operating systems. An operating system will have it's own file system. We can achieve this by creating 2 partitions and each operating system can be installed in it's own partition. The operating system files cannot interfere with each other. It's as if we have created 2 hard drives from 1 hard drive. With Linux we have the cocept of mounting. We can mount say "/boot" on 1 partition and "/tmp" on another partition. That way if the space on "/tmp" gets filled up it does not affect rest of the Linux system.It is important to note that the partition is a hardware concept. We can divide a hard disk into partitions without any file system on them. On Windows we can use "fdisk" utility. We can experiment with partitions using a flash drive. It is important to select the right drive before performing any operating else you will lose all the data on your computer. Also note that we need at least 1 partition if we want to install a file system on it.
In the above we have an unallocated flash drive of 14 Gb. There is no file system on it and now we are going to create a partition of 2 Gb on it.We choose a size and a volume and then are presented with an option of a file system.
So now we have a partition with a letter of "D:". We can't use it yet because it doesn't have a file system on it !
C:\WebSite\Learn\2\linuxa\fs>d: The volume does not contain a recognized file system. Please make sure that all required file system drivers are loaded and that the volume is not corrupted. C:\WebSite\Learn\2\linuxa\fs>We can select it and format it from "fdisk" . The partition information is kept on the hard drive in the MBR ( Master boot record ) . This is located on the first sector of the hard drive.
The MBR is very small ( 512 Bytes ) . It has space to store information about 4 partitions. When we create or manipulate partitions the information is stored in the MBR. It also has a small program called "Bootstrap code" that will look for the bootable partition ( only 1 partition can be bootable ) and start the boot up program on the bootable partition. This is usually a bootloader program ( GRUB in case of Linux ) . The 4th partition can be used to contain the location of entries that contain information about other partitions thus providing the ability to create more than 4 partitions. These partitions are called logical or extended partitions.
There are 2 utilities in Linux to manipulate partitions: "fdisk" and "parted". We shall create a partition on our floppy drive just as we did with Windows. As expected, we do not have permissions to run these utilities on the hills server. I am using the stand alone Debian 11 Linux environment that I created from the section on setup earlier.
root@ajkumar08-PC:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 946364 0 946364 0% /dev
tmpfs 192844 1544 191300 1% /run
/dev/sda1 151741000 5522132 138438056 4% /
tmpfs 964212 0 964212 0% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 192840 936 191904 1% /run/user/1000
/dev/sdc 482304 482304 0 100% /media/ajay/Debian 11.4.0 i386 n
root@ajkumar08-PC:~# /sbin/fdisk /dev/sdc
Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The device contains 'iso9660' signature and it will be removed by a write command. See fdisk(8) man page and --wipe option for more details.
Command (m for help): p
Disk /dev/sdc: 14.91 GiB, 16005464064 bytes, 31260672 sectors
Disk model: Infinitive
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x69ec8367
----
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-31260671, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31260671, default 31260671): +2G
Created a new partition 1 of type 'Linux' and of size 2 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
We first find out the device file for the flash drive. It will usually be on the
"mnt" or "media" folder. In this case we run the "df" command and see that the device
is "/dev/sdc" . Notice I have switched to the root user using the "su root" command and
that's why the prompt is "root@ajkumar08-PC:~#" . We run the fdisk command
"/sbin/fdisk /dev/sdc" . This runs the fdisk utility. The "m" for menu, print the
available commands. Also note that fdisk does not make any permanent changes till we
use the "w" command.
Command (m for help): m Help: DOS (MBR) a toggle a bootable flag b edit nested BSD disklabel c toggle the dos compatibility flag Generic d delete a partition F list free unpartitioned space l list known partition types n add a new partition p print the partition table t change a partition type v verify the partition table i print information about a partition Misc m print this menu u change display/entry units x extra functionality (experts only) Script I load disk layout from sfdisk script file O dump disk layout to sfdisk script file Save & Exit w write table to disk and exit q quit without saving changes Create a new label g create a new empty GPT partition table G create a new empty SGI (IRIX) partition table o create a new empty DOS partition table s create a new empty Sun partition tableWe use the "p" command to print the partition table which right now doesn't contain any entries. We use the "n" command to create a new partition. We choose a primary partition. A primary partition is one of the 4 partitions in the MBR vs the extended partition we can have by the 4th partition pointing to some place else. Now an "active" partition is the one that is bootable. There can be only 1 active partition and it must be a primary partition. We can make an active partition using the "a" command. The very first sector is sector 2048 and we use that as default. We specify the size as 2Gb with the option "+2G". Finally we use "w" to finalize the changes. Let's create another 2Gb partition using the "parted" utility. The "parted" utility can be run from the command line or in interactiv mode. From the command line we can give command such as.
root@ajkumar08-PC:/home/ajay# /sbin/parted /dev/sdc1 printWe supply the device file in a single command .
root@ajkumar08-PC:/home/ajay# /sbin/parted /dev/sdc1
(parted) help
align-check TYPE N check partition N for TYPE(min|opt)
alignment
help [COMMAND] print general help, or help on
COMMAND
mklabel,mktable LABEL-TYPE create a new disklabel (partition
table)
mkpart PART-TYPE [FS-TYPE] START END make a partition
name NUMBER NAME name partition NUMBER as NAME
print [devices|free|list,all|NUMBER] display the partition table,
available devices, free space, all found partitions, or a particular
partition
quit exit program
rescue START END rescue a lost partition near START
and END
resizepart NUMBER END resize partition NUMBER
rm NUMBER delete partition NUMBER
select DEVICE choose the device to edit
disk_set FLAG STATE change the FLAG on selected device
disk_toggle [FLAG] toggle the state of FLAG on selected
device
set NUMBER FLAG STATE change the FLAG on partition NUMBER
toggle [NUMBER [FLAG]] toggle the state of FLAG on partition
NUMBER
unit UNIT set the default unit to UNIT
version display the version number and
copyright information of GNU Parted
(parted)
File system type? [ext2]?
Start? 2GB
End? 4GB