Redirection

January 29, 2007

A lot of the power of the *nix shell is in redirection. The model is called “streams”, and we even have “pipes” for these “streams”. It’s not the best metaphor ever, but it’s good enough, I suppose. There are 3 streams associated with a process: the standard input (stdin), standard output (stdout) and standard error output (stderr). As you might expect, you get user input on stdin, output to stderr, and send errors to stderr.

stdout : Redirecting Output

Let’s start with stdout, the output stream, since that is the most common:

$ ls > /tmp/foo.txt
$

This will run the “ls” (list files) command, but instead of showing you the results, it will write to the file you specify (/tmp/foo.txt, in this example). If there is already a /tmp/foo.txt file, then it gets replaced.

$ ls >> /tmp/foo.txt
$

By using >> instead of the single >, we can append to the file instead of overwriting it.

If we want to see the output like we normally would, but log it to a file as well, we can use the “tee” utility:

$ ls | tee /tmp/foo.txt
[ the usual ls output shown here ]
$ ls | tee -a /tmp/foo.txt
[ the usual ls output shown here ]
$

The first command will write to /tmp/foo.txt; the second (with “-a“) will append to /tmp/foo.txt.

We have used a different redirector here; “|” (pipe) instead of “>” and “>>” . This is because we’re not funnelling the output to a different place (/tmp/foo.txt), but passing it to a program (tee) which does some more funky redirection.

Another common use of the pipe (“|“) is to go to the more (or less) utility. If a command would produce loads of output, faster than you can read it, then more will pause after the screen is full, and prompt you with a “— more” prompt (hence the name). SPACE will show the next screen; RETURN will show the next line. less is just like more, but it can also go backwards (PgUp and PgDn):

$ ls | more
file1.txt
file2.txt
file3.txt
--- more  (PRESS SPACE)
file4.txt
file5.txt
file6.txt
--- more (PRESS SPACE)
file7.txt
$ 

stderr : Redirecting Errors

Well-written programs will send their errors to a different “device” than they send their normal output to. Both stdout and stderr are usually the same place (your terminal), but they can be treated separately:

$ ls
foo.txt
$ ls fo.txt
ls: fo.txt: No such file or directory
$ ls fo.txt > output.txt 2>errors.txt
$ ls
foo.txt    output.txt   errors.txt
$ cat output.txt
$ cat errors.txt
ls: fo.txt: No such file or directory
$

What happened there? Well, we asked for “ls fo.txt“, but fo.txt doesn’t exist (foo.txt does). So we see an error from ls. If we direct stdout to “output.txt” and stderr to “errors.txt”, then we can see the difference. What ls actually did, was that it wrote *nothing* to stdout and the error message was sent to stderr. (stderr has file descriptor #2, so we say “2>” to direct stderr.) So when we “cat output.txt“, we get nothing (there was no output), but when we “cat errors.txt“, we see the error.

stdin: Redirecting Input

This is most commonly done by system utilities, but many shell scripts use the functionality.
The simple way is to use the “<” director:

$ mycommand < myfile.txt

This will take input from “myfile.txt” instead of from the keyboard.

The second way is probably more common. The more example above had more redirecting its input, via the pipe (“|“). We can create entire pipelines this way:

$ find . -print | grep foo | more

This will attach the stdout of the find command to the stdin of the grep command, and attach the stdout of grep to the stdin of more. Got that?! So what the *nix kernel will do, is that it will start more first, and then start grep telling it that its stdout is more‘s stdin. It will then start find, and tell it that its stdout is grep‘s stdin.

There’s actually more to it than that, but that’s got the basics of stdin, stdout and stderr covered briefly.


Simple Maths in the Unix Shell

January 29, 2007

One thing that often confuses new users to the Unix / Linux shell, is how to do (even very simple) maths. In most languages, x = x + 1 (or even x++) does exactly what you would expect. The Unix shell is different, however. It doesn’t have any built-in mathematical operators for variables. It can do comparisons, but maths isn’t supported, not even simple addition.

Following the Unix tradition (“do one thing, and do it well”) to the extreme, because the expr and bc utilites can do maths, there is absolutely no need for sh to re-invent the wheel.

Yes, I agree. This is frustrating. If I’ve got one gripe against shell programming, then this is it.

Addition and Subtraction

So how do we cope? There are basically two ways, depending on whether we choose expr or bc:

#!/bin/sh

echo "Give me a number: "
read x
echo "Give me another number: "
read y

######  Here's where we have the two options:
# The expr method:
exprans=`expr $x + $y`

# The bc method:
bcans=`echo $x + $y | bc`
###### Did you see the difference?


echo "According to expr, $x + $y = $exprans"
echo "According to bc,   $x + $y = $bcans"

As you can see, the language is slightly different for the two commands; expr parses an expression passed to it as arguments: expr something function something whereas bc takes the expression as its input (stdin): echo something function something | bc. Also, for expr, you must put spaces around the arguments: “expr 1+2” doesn’t work. “expr 1 + 2” works.

Multiplication

Multiplication is a little awkward, too; the * asterisk, which traditionally denotes multiplication, is a special character to the shell; it means “every file in the current directory”, so we have to delimit it with a backslash. “*” becomes “\*“:

#!/bin/sh

echo "Give me a number: "
read x
echo "Give me another number: "
read y

######  Here's where we have the two options:
# The expr method:
exprans=`expr $x \\* $y`

# The bc method:
bcans=`echo $x \\* $y | bc`
###### Did you see the difference?


echo "According to expr, $x * $y = $exprans"
echo "According to bc,   $x * $y = $bcans"

The other thing to note here, is the backtick (`). This grabs the output of the command it surrounds, and passes it back to the caller. So a command like

x=`expr 1 + 2`

means that, while if you type expr 1 + 2 at the command line, you’d get “3” back:

steve@nixshell$ expr 1 + 2
3
steve@nixshell$ 

If you enclose it with backticks, then the variable $x becomes set to the output of the command. Therefore,

x=`expr 1 + 2`

Is equivalent to (but of course more flexible than):

x=3

One last thing about assigning values to variables: Whitespace MATTERS. Don’t put spaces around the = sign. “x = 3” won’t work. “x=3” works.

Update: 17 Feb 2007 : Division, and Base Conversion

As noted by Constantin, the “scale=x” function can be useful for defining precision (bc sometimes seems to downgrade your precision: “echo 5121 / 1024 | bc” claims that the answer is “5”, which isn’t quite true; 5120/1024=5. echo "scale = 5 ; 5121 / 1024" | bc produces an answer to 5 decimal places (5.00097)).

Another important note I would like to add, is that bc is great at converting between bases:

Convert Decimal to Hexadecimal

steve@nixshell$ bc
obase=16
12345
3039

This tells us that 12345 is represented, in Hex (Base 16) as “0x3039″.
Similarly, we can convert back to decimal (well, we can use bc to convert any base to any other base)…

steve@nixshell$ bc
ibase=16
3039
12345

Or we can convert from Binary:

steve@nixshell$ bc
ibase=2
01010110
86
steve@nixshell$

… which tells us that 01010110 (Binary) is 86 in decimal. We can get that in hex, like this:

steve@nixshell$ bc
obase=16
ibase=2
01010110
56
steve@nixshell$

Which tells bc that the input base is 2 (Binary), and the output base is 16 (Hex). So, 01010110 (base 2) = 56 (hex) = 86 (decimal).

Note that the order does matter a lot; if we’d have said “ibase=2; obase=16″, that would be interpreted differently from “obase=16; ibase=2″.

I hope that this article will help some people out with some of the more frustrating aspects of shell programming. Please, let me know what you think.


Follow

Get every new post delivered to your Inbox.