You are all acquainted with redirection of standard input and standard output, and with the append standard output operator. You probably also know about the other standard stream for output called standard error. Standard error is the pathway that system commands use to output error data. This allows control of normal output (through standard output) and error output (through standard error) separately. Consider the following simple command

cat a b > c

where file a exists, but file b doesn't. When you execute this command, the file a is copied to c (using standard output redirection) and you get this message

$ cat a b > c
cat: b: No such file or directory
$

If both error messages and output were sent through the same data pathway, the file c would contain both the copy of a and the error message!

Control of standard error can be accomplished using output redirection. In fact, output redirection can be used to redirect output from any data stream, whether it is provided for you by default (as is standard output and standard error) or not if you know how to specify it. You specify the output stream by its number. The important numbers are

The number must be specified immediately preceding the redirection operator. It is part of the operator, so there may not be any spaces. As we know, > redirects standard output, so it is implicitly 1>  Similarly, < redirects standard input so it is implicitly 0<  Of course, it does not make any sense to redirect output to an input stream, so 0> and 1< would be silly. However, if you want to redirect standard error (an output stream) you can use 2>  

Let's try a few things with our command above. First, let's check out what is in our input file:

$ cat a
this is file a
$

If we redirect standard error instead of standard output to the file c, this is what happens:

$ cat a b 2> c
this is file a
$

Now if we examine c

$ cat c
cat: b: No such file or directory
$ 

We can redirect each individually on the same command line:

$ cat a b > c 2> errs
$ cat c
this is file a
$ cat errs
cat: b: No such file or directory
$

Sometimes you need to redirect both standard error and standard output to the same file. Hopefully your basic Unix experience would make the following command look suspect:

Redirecting twice to the same file certainly looks suspicious, and, indeed, it doesnt work very well

As you see, standard output was lost! Our instincts were correct: redirecting twice to the same file is problematic. Instead, a special syntax can be used to merge two output streams. It involves redirecting one stream to where the other is going. Here's how it works:

1. redirect the first output stream to the file  (e.g.,  > c)
2. then redirect the second output stream to where the first is going (e.g, 2>&1) In this syntax, the &1 means 'to where stream #1 is going'

$ cat a b > c 2>&1
$ cat c
this is file a
cat: b: No such file or directory
$

Here we redirected standard output to the file c, then redirected standard error to where standard output is going. You can reverse the order as well: below we redirect standard error to the file c, then redirect standard output to where standard error is going

Just make sure you redirect to the file first, then redirect the second stream to where the first is going. In other words, the order is important! In the example below

$ cat a b >&2 2>c
this is file a
$ 

standard output was first redirected to where standard error was going. But when the >&2 was encountered on the command line, standard error was going to the screen! So the first redirection operator was ineffective.

Outputting error messages

These redirection operators may seem arcane, but they are important to control your output. Just like regular Unix commands, we want to write our shell scripts so that our error messages go to standard error. However, the only mechanism we have for outputting a message, echo, only outputs to standard output! This is easily solved by our redirection operators:

echo "Error: cannot open file '$ipfile'"

outputs to standard output. If we want it to output to standard error instead, we simply send standard output to where standard error is going:

echo "Error: cannot open file '$ipfile'" >&2

Now a user of our shell script can control the output and error messages separately.

Now can you decipher the preview question from the last section?

make install > make.out 2>&1 &

This runs the command make install, placing all the output it produces in the file make.out. The only thing that may be surprising is the trailing &, which means run in the background.

Note that most of the spaces in the above command are redundant, since redirection operators and the backgrounding & are delimiters as well as metacharacters. You only need a space between make install and before the 2 of the last redirection operator.

make install>make.out 2>&1&

throwing away output

Sometimes you want to ignore the output of a Unix command. Instead of redirecting it to a file that must later be deleted, just redirect it to the null device /dev/null. Any output that is redirected to /dev/null just disappears. (/dev/null is lovingly called 'the bit bucket'). As an example, the command 

gives an error because one file that matches the pattern file*, named file2, is not readable. If you don't care about this, you could just throw away the error message that cat outputs. Since it outputs its error message to standard error, you can throw it away without disturbing the regular output (standard output):

$ cat file* > allfiles 2>/dev/null

tee

Have you ever wanted standard output to go both to a file and be piped to another command? The command tee can help. tee can be placed in any pipeline to save a copy of the data currently coming down the pipe in a file while still sending it on. For example, the pipeline

sort -t: -k1,1 /etc/passwd | head -n 10

displays the first ten lines of the passwd file after sorting on the first field. If you wanted to create a sorted copy of the passwd file in addition, you would have to use two commands. This is where tee can be useful. Simply add another stage in your pipe and instruct tee to save the pipe contents to the file passwd.sorted:

sort -t: -k1,1 /etc/passwd | tee passwd.sorted | tail -n 10

The last 10 lines of the sorted passwd file are still displayed on the screen, and the sorted file is also saved in passwd.sorted.

tee [-a] file

tee gets its name from a plumbing analogy, the same as a pipe. It acts like a T-junction, where water passes along a pipe and is siphoned off at a right angle to the normal flow. tee simply copies standard input to standard output, saving a copy in the attached file. The -a option appends to the output file rather than overwriting it.

Note that standard output only flows down the pipe as long as both ends are "open". If tee sends standard output down the pipe to another command and that command exits prematurely, the pipe will be "broken" and that may result in the termination of the entire pipeline. In the example above, this may happen if tail is replaced with head.

Other redirection operators

Believe it or not, there are over a dozen redirection operators. Most are very special-purpose and not interesting to us. The force overwrite operator can be useful, however:

Here the user has the noclobber option set in his shell. This option disallows the overwriting of existing files with redirection. This user would have to remove the make.out file prior to re-running the make command. Alternatively the user could have used the force overwrite redirection operatory >|

$ make >| make.out 2>&1
$

Controlling the file attached to a stream

Normally when you write to stdout or stderr, the dataa goes wherever the stream goes. Suppose your shell script is myss, and your script is executed as 

myss > xxx

Inside myss, any output to stdout will go to xxx by default. For example, if a line in myss is

sort thedata

the [standard] output of sort will go into xxx. The destination of stdout set at the commandline is propigated to (or inherited by) standard output of every command in myss, effectively becoming the new default for stdout.

You can, of course, change this for a single command in myss by redirecting stdout for that command:

sort thedata > sort.out

but this must be done on a command by command basis. 

If you want stdout (or stdin or stderr) to be redirected to the same place for a group of commands you can use any compound command, such as a loop, an if statement, a function, or a block.

Example:

could be written as

Similarly you could redirect a stream for an anonymous block of code:

yyy will be opened at the beginning of the block, each of cmd1, cmd2, cmd3 will send stdout to yyy, and yyy will be closed at the end of the block. After the block, stdout will revert to whatever its default was.

You can also redirect a stream premanantly within your script. Suppose your script should always write its output to a particular file, say, /tmp/user.out, where user is the current user. When your script begins, you can simply redirect it permanantly using exec:

exec > "/tmp/$LOGNAME.out"

From this point on, everything written to stdout will go the file you chose, irrespective of how stdout was redirected when the script was started.

Copyright 2009 Greg Boyd - All Rights Reserved.