Flag g

The flag "g" will make the replacements globally.

[amittal@hills Flags]$ echo "Testing the Tesla car." | sed 's/Tes/TES/'

TESting the Tesla car.

Normally the substitution is done on the first pattern match that sed found. Using the "g" flag causes the replacements to occur throughout the line.

[amittal@hills Flags]$  echo "Testing the Tesla car." | sed 's/Tes/TES/g'
TESting the TESla car.

[amittal@hills Flags]$ echo "Testing the Tesla car." | sed -r 's/[^ ]+/(&)/g'
(Testing) (the) (Tesla) (car.)

The above line uses the not operator to mean a
combination does not contain a space.

Exercise

1)
echo "suden  unflatering noncommital subcommitee" | sed -r 'TODO'

Complete the sed command above so that the d is replaced by
2 dd's and one t is replaced by 2 t's .

2)
Remove both the duplicated words.
This This contains a mistake mistake


Solutions

1)
$ echo "suden  unflatering noncommital subcommitee" | sed -r 's/d|t/&&/g'
sudden  unflattering noncommittal subcommittee
2)
$ echo "This This contains a mistake mistake" | sed -r 's/([A-Za-z]+) \1/\1/g'
This contains a mistake

Specifying the occurrence

We can specify which  matching pattern should be applied.

$echo "Testing the Tesla car." | sed -r 's/[^ ]+/(&)/4'

Testing the Tesla (car.)

In the above line we are stating that the pattern
match should apply to the 4th word only. We can
use the number with the "g" flag to specify
apply the match pattern to the nth
occurrence and beyond.

$echo "Testing the Tesla car." | sed -r 's/[^ ]+/(&)/2g'
Testing (the) (Tesla) (car.)

In the above line we are stating that apply the pattern
matching to 2nd word and beyond.

Exercises:

1) Write a sed command to work on the file "data.txt"
to keep just the first 3 words in each line.

Solution

1)

$ cat data.txt | sed -r 's/[^ ]+//4g'

Writing the output to a file

File: even.txt


22 Even number
23 Odd Number
24 Even Number
25 Odd Number

sed -n 's/^[0-9]*[02468] /&/w even' even.txt

The above command creates a new file named "even".
The command looks for an even number in each line.
File: "even" :

22 Even number
24 Even Number

This can also be done using redirection:

sed -n 's/^[0-9]*[02468] /&/' > even.txt

We can also combine flags such as:

 sed -n -r 's/^[0-9]*[02468] /&/w even' even.txt

 sed -nr 's/^[0-9]*[02468] /&/w even' even.txt

Suppresses the output. The file "even" still gets
created.
$ sed -n -r 's/^[0-9]*[02468] /&/w even' even.txt

$ cat even
22 Even number
24 Even Number

If we take out the "-n" then we get the whole file
printed out.

$ sed -r 's/^[0-9]*[02468] /&/w even' even.txt
22 Even number
23 Odd Number
24 Even Number
25 Odd Number

Writing to variable

$ var1=`sed -n  's/^[0-9]*[02468] /&/p' even.txt`
$ echo "$var1"
22 Even number
24 Even Number

We need to place the double quotes otherwise we run into
problems with the end of line character.

$ var1=`sed -n  's/^[0-9]*[02468] /&/p' even.txt`
$ echo $var1
 24 Even Number

Exercises

1) Using the file "even.txt"  place the
values  22,23,24,25 in a variable "var1".
There should only be spaces between the numbers.
Write your commands in a script and execute the script.
Do not hard code the numbers.

Solutions

1)


File: even.sh

var1=`sed -r 's/(^[0-9]+ ).*/\1/' even.txt | tr -d '\n'`
echo "$var1"

Ignoring case

$ echo "cAt ate the fish" | sed -r  's/cat/Cat/i'

Cat ate the fish

echo "Abc" | sed -n  '/abc/I p'

We can use the capitol letter "I" character to
ignore the case. The above sed does not substitute
but merely searches for a pattern in a manner
similar to grep.

Exercises
1) What does the below do

$ echo "a dog jumps A fence" | sed -n 's/a/A/2ipw data'

Multiple Commands

Instead of pipes we can use "-e" option to give multiple commands.

$ echo "cab is coming" | sed -e 's/a/A/g' -e 's/c/C/g'
CAb is Coming

Exercises:

1) What does the below print ?
$ echo "cab is coming" | sed -e 's/a/A/g' -e 's/A/C/ig'

Using Multiple FileNames

Sed can work with multiple files at the same time. Assume we have the following files.

File "f1.txt"
#12 This is the first line in file f1.
#Abc This is the second line in file f1

File: "f2.txt"

#132 This is a line in f2.
#Abcdef This is another line in f2

$ sed -r 's/^#[^ ]+ //' f1.txt f2.txt

Output:
$ sed -r 's/^#[^ ]+ //' f1.txt f2.txt
This is the firstline in file f1.
This is the second line in file f1
This is a line in f2.
This is another line in f2

Exercises:

1) Use the above sed command to save the
output in a variable "var1" and output the
contents of the variable.

work with a script in a file

We can use Sed to work with a script in a file.

If we have many commands we can place the
commands in a file and use the "-f" option
to run the sed command.

contents of the "myscript" file.

File: myscript

# sed comment - This script changes lower case vowels to upper case
s/a/A/g
s/e/E/g
s/i/I/g
s/o/O/g
s/u/U/g

$ echo "cat is sitting on the roof" | sed -f myscript

cAt Is sIttIng On thE rOOf

Exercises

1) Place your sed command in a file to
increment a 2 digit number so that each digit
gets converted to the one higher with 9
getting converted to 0.

45 ->  56
91 ->  02
99  -> 00
00 -> 11

Solution
1)
s/8/9/g
s/7/8/g
s/6/7/g
s/5/6/g
s/4/5/g
s/3/4/g
s/2/3/g
s/1/2/g
s/0/1/g
s/9/0/g

$ echo "45" | sed -f myscript1
56

Printing a specific line using Sed

Let's create a file called "data1.txt" containing
the following 10 lines.

File: data1.txt

Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

To print out the 5th line we can use the command:

$ cat data1.txt | sed -n 5p
Line 5

To print out lines 3 to 5 we can use:

$ cat data1.txt | sed -n 3,5p
Line 3
Line 4
Line 5

We can also specify that a pattern should apply to a specific line.

$ cat data1.txt | sed -n '3 s/3/31/p'
$ Line 31

$ cat data1.txt | sed -n '4 s/3/3/p'
$

There is no output for the above command because
at line 4 we don't have text with the digit "3" .

Applying a range of line numbers

$ cat data.txt | sed -n '1,4 s/3/31/p'
Line 31

The above states that look in the lines 1 to 4 and
apply the substitute operation if a match for the
string "3" is found.

$ cat data1.txt | sed  '1,4 s/Line/LINE/'
LINE 1
LINE 2
LINE 3
LINE 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

The above states that look in the lines from 1 to 4 and change the small "Line" to "LINE" .

The "$" sign means till the end of the file.

$ cat data.txt | sed '3,$ s/Line/LINE/'
Line 1
Line 2
LINE 3
LINE 4
LINE 5
LINE 6
LINE 7
LINE 8
LINE 9
LINE 10

Searching for a range by pattern

$ sed '/3/,/5/ s/Line//' data.txt
Line 1
Line 2
 3
 4
 5
Line 6
Line 7
Line 8
Line 9
Line 10

We apply the substitute command upon
encountering the first pattern up to the
second pattern.


We can apply the range and pattern also.
$ cat data.txt | sed  '2,/4/ s/Line/LINE/'
Line 1
LINE 2
LINE 3
LINE 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

The above states that start at line 2 and
then go up the line that contains the pattern "4".



File: data2.txt


This is a test.
BEGIN The dog is chasing the cat.
A test is coming up.
Are we having fun in this class ?  END
Some more lines.

Write a sed command that will place a "#" in the
section ( beginning of each line) marked BEGIN to END.

Deleting a line

To delete the 3rd line.

$ sed 3d data1.txt
Line 1
Line 2
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

Note the original file is not changed.

Deleting by a range:

$ sed 1,3d data.txt

Deleting by a pattern:

$ sed '/3/ d' data.txt
Line 1
Line 2
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

In the above we delete lines with the pattern "3".


sed '5,$ d' data.txt

$ sed '5,$ d' data1.txt
Line 1
Line 2
Line 3
Line 4

In the above we delete lines from 5 to
the end of the file.

Exercise

In this exercise we combine the line range
with a pattern. Write a sed command to delete
from line "1" to the pattern "3" .

Adding a line after a pattern match

$ sed '/3/ a\ Add' data.txt
Line 1
Line 2
Line 3
 Add
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

Adding a line after a line number.

sed '3 a\ Add' data1.txt

Adding a line at the end of the file.

sed '$ a\ Add' data1.txt

Changing a line using the "c" flag

$ sed '/3/ c\ Change a line' data1.txt
Line 1
Line 2
 Change a line
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

Exercises

1) Write a shell script using sed and line ranges
to create a file "data3.txt" with the following contents.

Line 6
Line 7
Line 8
Line 9
Line 10
Line 1
Line 2
Line 3
Line 4
Line 5

Solution:

Adding a line number

The "=" command can be used to insert line numbers
before each line.

$ sed = data1.txt
1
Line 1
2
Line 2
3
Line 3
4
Line 4
5
Line 5
...



File: data4.txt


Line a
Line b
Line c
Line d
Line e
Line f
Line g
Line h
Line i
Line j

$ sed -n '/c/ =' data4.txt
$ 3

The above states that match the line with "c"
in it and print it's line number.

Transforming Characters

$ sed 'y/ie/IE/' data1.txt
LInE 1
LInE 2
LInE 3
...

We use the "y" option to state that "i"
should be changed to "I" and "e" should
be changed to "E" .

1)

Place your sed command in a file to increment a 2 digit number so that each digit gets converted to the one higher with 9 getting converted to 0 .

45 ->  56
91 ->  02
99  -> 00
00 -> 11

Do the above using the "y" option with sed.



The "\u" option


$ echo "cat" | sed -r 's/.$/\u&/'
caT

$ echo "cat" | sed -r 's/.*/\u&/'
Cat

$ echo "cat" | sed -r 's/a/\u&/'
cAt

The small "/u" option turns the next character into upper case.

$ echo "cat" | sed -r 's/.*/\U&/'
CAT

The capital "U" converts the rest of the string to uppercase.

Grouping

We can use curly braces to group multiple sed
commands together.



File: data5.txt

BEGIN The dog is chasing the cat.
A test is coming up.
#Comment1

Are we having fun in this class ?  END
Some more lines.
#Comment2



File: 1.sh


sed -n '
/BEGIN/,/END/ {
     s/#.*//
     /^$/ d
     p
}
'

Grouping allows us to combine multiple sed commands together.