You are all acquainted with redirection of standard input and standard output, and with the append standard output operator. You probably also know about the other standard stream for output called standard error. Standard error is the pathway that system commands use to output error data. This allows control of normal output (through standard output) and error output (through standard error) separately. Consider the following simple command

cat a b > c

where file a exists, but file b doesn't. When you execute this command, the file a is copied to c (using standard output redirection) and you get this message

$ cat a b > c
cat: b: No such file or directory
$

If both error messages and output were sent through the same data pathway, the file c would contain both the copy of a and the error message!

Control of standard error can be accomplished using output redirection. In fact, output redirection can be used to redirect output from any data stream, whether it is provided for you by default (as is standard output and standard error) or not if you know how to specify it. You specify the output stream by its number. The predefined numbers are

The number must be specified immediately preceding the redirection operator. It is part of the operator, so there may not be any spaces. As we know, > redirects standard output, so it is implicitly 1>  Similarly, < redirects standard input so it is implicitly 0<  Of course, it does not make any sense to redirect output to an input stream, so 0> and 1< would be silly. However, if you want to redirect standard error (an output stream) you can use 2>  

Let's try a few things with our command above. First, let's check out what is in our input file:

$ cat a
this is file a
$

If we redirect standard error instead of standard output to the file c, this is what happens:

$ cat a b 2> c
this is file a
$

Now if we examine c

$ cat c
cat: b: No such file or directory
$ 

We can redirect each individually on the same command line:

$ cat a b > c 2> errs
$ cat c
this is file a
$ cat errs
cat: b: No such file or directory
$

Sometimes you need to redirect both standard error and standard output to the same file. Hopefully your basic Unix experience would make the following command look suspect:

Redirecting twice to the same file certainly looks suspicious, and, indeed, it doesnt work very well

As you see, standard output was lost! Our instincts were correct: redirecting twice to the same file is problematic. Instead, a special syntax can be used to merge two output streams. It involves redirecting one stream to where the other is going. Here's how it works:

1. redirect the first output stream to the file  (e.g.,  > c)
2. then redirect the second output stream to where the first is going (e.g, 2>&1) In this syntax, the &1 means 'to where stream #1 is going'

$ cat a b > c 2>&1
$ cat c
this is file a
cat: b: No such file or directory
$

Here we redirected standard output to the file c, then redirected standard error to where standard output is going. You can reverse the order as well: below we redirect standard error to the file c, then redirect standard output to where standard error is going

Just make sure you redirect to the file first, then redirect the second stream to where the first is going. In other words, the order is important! In the example below

$ cat a b >&2 2>c
this is file a
$ 

standard output was first redirected to where standard error was going. But when the >&2 was encountered on the command line, standard error was going to the screen! So the first redirection operator was ineffective.

Outputting error messages

These redirection operators may seem arcane, but they are important to control your output. Just like regular Unix commands, we want to write our shell scripts so that our error messages go to standard error. However, the only mechanism we have for outputting a message, echo, only outputs to standard output! This is easily solved by our redirection operators:

echo "Error: cannot open file '$ipfile'"

outputs to standard output. If we want it to output to standard error instead, we simply send standard output to where standard error is going:

echo "Error: cannot open file '$ipfile'" >&2

Now a user of our shell script can control the output and error messages separately.

Now can you decipher the preview question from the last section?

make install > make.out 2>&1 &

This runs the command make install, placing all the output it produces in the file make.out. The only thing that may be surprising is the trailing &, which means run in the background.

Note that most of the spaces in the above command are redundant, since redirection operators and the backgrounding & are delimiters as well as metacharacters. You only need a space between make install and before the 2 of the last redirection operator.

make install>make.out 2>&1&

throwing away output

Sometimes you want to ignore the output of a Unix command. Instead of redirecting it to a file that must later be deleted, just redirect it to the null device /dev/null. Any output that is redirected to /dev/null just disappears. (/dev/null is lovingly called 'the bit bucket'). As an example, the command 

gives an error because one file that matches the pattern file*, named file2, is not readable. If you don't care about this, you could just throw away the error message that cat outputs. Since it outputs its error message to standard error, you can throw it away without disturbing the regular output (standard output):

$ cat file* > allfiles 2>/dev/null

tee

Have you ever wanted standard output to go both to a file and be piped to another command? The command tee can help. tee can be placed in any pipeline to save a copy of the data currently coming down the pipe in a file while still sending it on. For example, the pipeline

sort -t: -k1,1 /etc/passwd | head -n 10

displays the first ten lines of the passwd file after sorting on the first field. If you wanted to create a sorted copy of the passwd file in addition, you would have to use two commands. This is where tee can be useful. Simply add another stage in your pipe and instruct tee to save the pipe contents to the file passwd.sorted:

sort -t: -k1,1 /etc/passwd | tee passwd.sorted | head -n 10

The first 10 lines of the sorted passwd file are still displayed on the screen, and the sorted file is also saved in passwd.sorted.

tee [-a] file

tee gets its name from a plumbing analogy, the same as a pipe. It acts like a T-junction, where water passes along a pipe and is siphoned off at a right angle to the normal flow. tee simply copies standard input to standard output, saving a copy in the attached file. The -a option appends to the output file rather than overwriting it.

Here documents

From time to time there is a need to create a file or to generate a significant amount of input to a command from within a shell script. You can do this using a here document. The syntax is simple:

The input to the command consisted of a block of text that looked like this:

Text to use as input to the command (using standard input) goes
here. The text can have variable, arithmetic and command substitutions
The input continues until a line is encountered that has TAG on it (TAG can be any text string)
(Just checking: look at the output of LOGNAME: gboyd)

One common use of this is to generate a mail message in response to an error condition. An example of this is shown in the Examples section of this module.

Copyright 2016 Greg Boyd - All Rights Reserved.