A serious publisher has contacted me about writing a serious book about Linux shell programming.

It is all really very serious. I’m not used to being serious, as you can probably tell from the fact that I have now used the word “serious” four times in this three-sentence post.

I am rather keen to write a book on the subject, not because I’m vain, or desperate for money, but because the stuff I have seen out there in dead-tree format has been of rather low quality. Also because of all the emails I’ve received over the years, they have all been positive, and none has said anything along the lines of “I didn’t need any of that because I bought Book[X]“, or indeed any book. People have emailed me, asking for advice as to what book to buy, and I have been unable to recommend any book that I have seen.

So:

What would you like to see in your ideal book about UNIX / Linux shell scripting, be it Bourne, Bash, ksh, tcsh, zsh, whatever?

Please don’t be timid; if you want to know how to work out how many nose-flutes can be fitted into the area of a Boeing 757, you won’t be anything like as strange as some of the correspondants I’ve had over the years, so please, tell me what is bugging you, what has bugged you, or even what you think might be likely to bug you in days / months / years to come.

I’m likely to answer any specific questions here and now, whether or not they end up in the book, but anything you’d like to see in a book, too… post that here, and I’ll have a stab at it.

Also, I would of course be interested to know if you have found any useful books on or around the subject, and what they did particularly well.

Steve

This blog has now been running for a year; the first post was Hello World on 17th Jan 2007.

I hadn’t realised it had been going for so long; in that time, I’ve made 41 posts, so I haven’t quite managed to make one post per week I have been a bit slack lately, for which I do apologise. New Years Resolution: I must make more posts here!

In the meantime, my main site, steve-parker.org, has celebrated its seventh birthday, having been born in June 2000 - looking forward to making the 8th birthday celebrations this June!

There are lots of small little quirks to the *nix shells; this is just one of them.

If you want to list the files in a directory, then ls will list them all for you, in alphabetical order.

If you want to list them by size, you can use ls -S; by timestamp: ls -t, and so on.

But ls is a particular utility. What happens when we do this:


for myfile in *
do
  echo “My file is called $myfile”
done

We get an alphabetically sorted list (see man ascii for the actual detail; they’re sorted by ASCII value, so numbers first, then uppercase letters, then lowercase letters).

This can be a pain, but it can also be quite useful. If you’ve got a bunch of files:

1.install.txt
2.setup.txt
3.use.txt
4.uninstall.txt

Then you can play with them in order, just by using the asterisk:

for i in *
do
  echo “File $i” >> all.txt
  cat $i >> all.txt
done

And it will sort them into order for you (”1″ comes before “2″ in ASCII, and so on…)

Or you could just do this:

more * > all.txt

Because more will prefix each file with its name in a header, if there is more than one file to process.

http://snap.nlc.dcccd.edu/reference/bash1/features_29.html has a bash-specific technique for processing calculations:

$[ expression ]
$(( expression ))

See the link for further details.

It seems like an esoteric concept, but it’s actually very useful.

If your input file is “1 apple steve@example.com”, then your script could say:

while read qty product customer
do
  echo "${customer} wants ${qty} ${product}(s)"
done

The read command will read in the three variables, because they’re spaced out from each other.

However, critical data is often presented in spreadsheet format. If you save these as CSV files, it will come out like this:

1,apple,steve@example.com

This contains no spaces, and the above code will not be able to understand it. It will take the whole thing as one item - the first thing, quanity, $qty, and set the other two fields as blank.

The way around this, is to tell the entire shell, that “,” (the comma itself) separates fields; it’s the “internal field separator”, or IFS.

The IFS variable is set to space/tab/newline, which isn’t easy to set in the shell, so it’s best to save the original IFS to another variable, so you can put it back again after you’ve messed around with it. I tend to use “oIFS=$IFS” to save the current value into “oIFS”.

Also, when the IFS variable is set to something other than the default, it can really mess with other code.

Here’s a script I wrote today to parse a CSV file:

#!/bin/sh
oIFS=$IFS     # Always keep the original IFS!
IFS=","          # Now set it to what we want the "read" loop to use
while read qty product customer
do
  IFS=$oIFS
  # process the information
  IFS=","       # Put it back to the comma, for the loop to go around again
done < myfile.txt

It really is that easy, and it’s very versatile. You do have to be careful to keep a copy of the original (I always use the name oIFS, but whatever suits you), and to put it back as soon as possible, because so many things invisibly use the IFS - grep, cut, you name it. It’s surprising how many things within the “while read” loop actually did depend on the IFS being the default value.

Whilst not directly related to shell programming, understanding of basic logic operations - AND, OR, NOR, XOR, NAND, etc, are as important to shell programmers as to C, Java, .Net and other coders.

My recent interactive logic gate page seems to have become quite popular; it’s just a simple implementation of each of the major logic circuits in use. If you want to see more, say so - I’ll add anything you ask for

The (often a bit geeky for this blog) FreeBSD-World website has a good “Top-25″ list of 25 useful commands in Linux/UNIX for Beginners

I’m not sure that #24 (dig) and #25 (host) are absolutely necessary, #18 (startx) is possibly outdated these days, and the compression tools (6-9) are much of a muchness, but apart from that, #1 - #23 should be familiar to anyone who claims to be experienced with UNIX/Linux. If somebody was missing one, it would have to be #18 (startx), as (a) it’s not needed on servers, and (b) modern *nix distros will boot into a GUI automatically when possible.

So what’s the list?
25. host
24. dig
23. mkdir
22. rm
21. cp
20. grep
19. ls
18. startx
17. nano / vi
16. pwd
15. cat
14. man
13. kill
12. locate
11. ifconfig
10. ssh
9. gzip
8. bzip2
7. zip
6. tar (I would put 6-9 in one category, personally. rar should probably be in there too)
5. mount
4. passwd
3. ping
2. tail
1. top

I use vi every day; to me, there is no better text editor. It is apparently a little intimidating for the newcomer, though…. here’s a beginner’s guide to the VI editor.

I would write one of these, but - much as I love vi, and I really do, everyone who uses vi, seems to have their own experience of it, and their own shortcuts. We’ve all got our own quirks. One thing’s for sure; vi is not notepad!

Greg Wooledge has an excellent list of Bash Pitfalls, with good explanations as to why they are wrong, and what the correct syntax should be.

UNIX and Linux systems use “init scripts” - scripts typically placed in /etc/init.d/ which are run when the system starts up and shuts down (or changes runlevels, but we won’t go into that level of detail here, being more of a sysadmin topic than a shell scripting topic). In a typical setup, /etc/init.d/myservice is linked to /etc/rc2.d/S70myservice. That is to say, /etc/init.d/myservice is the real file, but the rc2.d file is a symbolic link to it, called "S70myservice". The “S” means “Start”, and “70″ says when it should be run - lower-numbered scripts are run first. The range is usually 1-99, but there are no rules. /etc/rc0.d/K30myservice (for shutdown), or /etc/rc6.d/K30myservice (for reboot; possibly a different scenario for some services), will be the corresponding “Kill” scripts. Again, you can control the order in which your services are shut down; K01* first, to K99* last.

All of these rc scripts are just symbolic links to /etc/init.d/myservice, so there is just one actual shell script, which takes care of starting or stopping the service. The Samba init script from Solaris is a nice and simple script to use as an example:

case "$1" in
start)
	[ -f /etc/sfw/smb.conf ] || exit 0

	/usr/sfw/sbin/smbd -D
	/usr/sfw/sbin/nmbd -D
	;;
stop)
	pkill smbd
	pkill nmdb
	;;
*)
	echo “Usage: $0 { start | stop }”
	exit 1
	;;
esac
exit 0

The init daemon, which controls init scripts, calls a startup script as "/etc/rc2.d/S70myservice start", and a shutdown script as "/etc/rc0.d/K30myservice stop". So we have to check the variable $1 to see what action we need to take. (See http://steve-parker.org/sh/variables2.shtml to read about what $1 means - in this case, it’s either “start” or “stop”).

So we use case (follow link for more detail) to see what we are required to do.

In this example, if it’s “start”, then it will run the three commands:

	[ -f /etc/sfw/smb.conf ] || exit 0
	/usr/sfw/sbin/smbd -D
	/usr/sfw/sbin/nmbd -D

Where line 1 checks that smb.conf exists; there is no point continuing if it doesn’t exist, just “exit 0″ (success) so the system continues booting as normal. Lines 2 and 3 start the two daemons required for Samba.

If it’s “stop”, then it will run these two commands:

	pkill smbd
	pkill nmdb

pkill means “Process Kill”, and it simply kills off the two processes started by the “start” option.

The "*)" construct catches any other uses, and simply replies that the correct syntax is to call it with either “start” or “stop” - nothing else will do. Some services allow for status reports, restarting, and so on. The one thing we do need to provide is “start”. Most services also have a “stop” function. All others are optional.

The simplest possible init script would be this, to control an Apache webserver:

#!/bin/sh
/usr/sbin/apachectl $1

Apache comes with a program called “apachectl” (or “apache2ctl”), which will take “stop” and “start” as arguments, and act accordingly. It will also take “restart”, “status”, “configtest”, and a few more options, but that one-line script would be enough to act as /etc/init.d/apache, with /etc/rc2.d/S90apache and /etc/rc0.d/K10apache linking to it. To be frank, even that is not necessary; you could just link /usr/sbin/apachectl into /etc/init.d/apache. In reality, it’s normally good to provide a few sanity-checks in addition to the basic stop/start functionality.

The vast majority of init scripts use the case command; around that, you can wrap all sorts of other things - most GNU/Linux distributions include a generic reporting script (typically /lib/lsb/init-functions - to report “OK” or “FAILED”), read in a config file (like the Samba example above), define functions for the more involved aspects of starting, stopping, or reporting on the status of the service, and so on.

Some (eg, SuSE) have an “INIT INFO” block, which may allow the init daemon a bit more control over the order in which services are started. Ubuntu’s Upstart is another; Solaris 10 uses pmf (Process Monitor Facility), which starts and stops processes, but also monitors them to check that they are running as expected.

After a good decade of stability, in 2007 the world of init scripts appears to be changing, potentially quite significantly. However, I’m not here to speculate on future developments, this post is just to document the stable interface which is init scripts. Even if other things change, the basic “start|stop” syntax is going to be with us for a long time to come. It is easy, but often important, to understand what is going on.

In closing, I will list the run-levels, and what each run-level provides:

0: Shut down the OS (without powering off the machine)
1, s, S: Single-User mode. Networking is not enabled.
2: Networking enabled (not NFS, Printers)
3: Normal operating mode (including NFS, Printers)
4: Not normally used
5: Shut down the OS and power off the machine
6: Reboot the OS.

Some GNU/Linux distributions change these definitions - in particular, Debian provides all network services at runlevel 2, not 3. Run-level 5 is also sometimes used to start the graphical (X) interface.

Pipes, piping, pipelines… whatever you call them, are very powerful - in fact, they are one of the core tenets of the philosophy behind UNIX (and therefore Linux). They are also, really, very simple, once you understand them. The way to understand them, is by playing with them, but if you don’t know what they do, you don’t know where to start… Catch-22!

So, here are some simple examples of how the pipe works.

Let’s see the code

$ grep steve /etc/passwd | cut -d: -f 6
/home/steve
$
What did this do? There are two UNIX commands there: grep and cut. The command “grep steve /etc/passwd” finds all lines in the file /etc/passwd which contain the text “steve” anywhere in the line. In my case, this has one result:
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash
The second command, “cut -d: -f6” cuts the line by the delimiter (-d) of a colon (”:“), and gets field (-f) number 6. This is, in the /etc/passwd file, the home directory of the user.

So what? Show me some more

This is the main point of this article; once you’ve seen a few examples, it normally all becomes clear.

EG2

$ find . -type f -ls | cut -c14- | sort -n -k 5
rw-r--r--   1 steve    steve       28 Jul 22 01:41 ./hello.txt
rwxr-xr-x   1 steve    steve     6500 Jul 22 01:41 ./a/filefrag
rwxr-xr-x   1 steve    steve     8828 Jul 22 01:42 ./c/hostname
rwxr-xr-x   1 steve    steve    30848 Jul 22 01:42 ./c/ping
rwxr-xr-x   1 steve    steve    77652 Jul 22 01:42 ./b/find
rwxr-xr-x   1 steve    steve    77844 Jul 22 01:41 ./large
rwxr-xr-x   1 steve    steve    93944 Jul 22 01:41 ./a/cpio
rwxr-xr-x   1 steve    steve    96228 Jul 22 01:42 ./b/grep
$

What I did here, was three commands: “find . -type f -ls” finds regular files, and lists them in an “ls”-style format: permissions, owner, size, etc.
“cut -c14-” cuts out the first 14 characters, which mess up the formatting on this website (!), and aren’t very interesting.
“sort -n -k 5” does a numeric (-n) sort, on field 5 (-k5), which is the size of the file.
So this gives me a list of the files in this directory (and subdirectories), ordered by file size. That’s much more useful than “ls -lS“, which restricts itself to the current directory, but not subdirectories.

(As an aside, I have to admit that I only concocted this by trying to think of an example; it actually seems really useful, and worth making into an alias… I must do a post about “alias” some time!)

So how does it work?

This seems pretty straightforward: get lines containing “steve” from the input file (”grep steve /etc/passwd“), and get the sixth field (where fields are marked by colons) (”cut -d: -f6“). You can read the full command from left to right, and see what happens, in that order.

How does it really work?

EG1 Explained

There are some gotchas when you start to look at the plumbing. Because we’re using the analogy of a pipe (think of water flowing through a pipe), the OS actually sets up the commands in the reverse order. It calls cutfirst, then it calls grep. If you have (for example) a syntax error in your cut command, then grep will never be called.
What actually happens is this:

1. A “pipe” is set up - a special entity which can take input, which it passes, line by line, to its output.
2. cut is called, and its input is set to be the “pipe”.
3. grep is called, and its output is set to be the “pipe”.
4. As grep generates output, it is passed through the pipe, to the waiting cut command, which does its own simple task, of splitting the fields by colons, and selecting the 6th field as output.

EG2 Explained

For EG2, “sort” is called first, which ties to the second (rightmost) pipe for its input. Then “cut” is called, which ties to the second pipe for its output, and the first (leftmost) pipe for its input. Then, “find” is called, which ties to the first pipe for its output.
So, the output of “find” is piped into “cut“, which strips off the first 14 characters of the “find” output. This is then passed to “sort“, which sorts on field 5 (of what it receives as input), so the output of the entire pipeline, is a numerically sorted list of files, ordered by size.

There doesn’t seem to be any decent shell CheatSheet out there, so I have undertaken to write one.

This is my first attempt; once I actually printed it out, I realised that the the font was actually rather large, so I had room to include much more than I had originally sketched out.

Cheatsheet You may notice that there are a lot of blanks; Shell Cheatsheet has the same content, with some ideas for new content. Please tell me what you want to see in there - There is still lots of room, it’s only about 60% full.

What would you fill the other 40% with?

Does this seem to be a useful cribsheet? What would you like to see in it?

Also, what format would you prefer? PDF? PNG? GIF? DOC?!

I could talk for hours about TERMCAP, but I won’t; instead, here’s a link to one usage of termcap:

‘less’ colours for man pages

The post is interesting, the comments are probably at least as useful as the post.

A tutorial reader, Maarten Zeinstra, emailed me the other day about Variables - Part II, where I "kinda implicitly state that there are only 9 available params to be given in any program $1..$9 so if you wanted do have more then those you might want to do something like your example. But ${n} also works to get above 9 params."

It turns out that this is entirely true - for bash, dash, csh, ksh, and probably other shells too. The ${9} limitation seems only to apply to the traditional Bourne shell - that is, #!/bin/sh on commercial Unices.

If you can be sure that your script will be interpreted by bash, or similar, you can quite happily refer to your tenth argument as ${10}.

Sorry, another link to an external site. I will get some good hints and tips back on here soon, I promise.

In the meantime, here’s a summary of bash keyboard shortcuts, for those of us who use Bash, be it under Linux or any other OS.

It seems interesting to me, that a few years ago, only Linux users used Bash (because it was the default, being a GNU shell), whilst now, most UNIX users I know, use Bash (because it’s so flexible, particularly for interactive use), whilst Ubuntu, the most popular Linux distro, uses dash as /bin/sh. Dash is Posix compliant, but omits a few features that Bash enjoys. Debian notes that “‘bash’ is a better shell for most users, since it has some nice features absent from ‘dash’, and is a required part of the system.”

Nobody deals with the really low-level stuff any more; I learned it from UNIX Gurus in the 90s. I was really lucky to have met some real experts, and was stupid not to have better understood the opportunity to pick their brains.

Write to a file

$ echo foo > file

Append to a file

$ echo foo >> file

Read from a file (1)

$ cat < file
Read from a file (2)
$ cat file
Read lines from a file
$ while read f
> do
>   echo LINE: $f
> done < file
$
       ]]>
			http://nixshell.wordpress.com/2007/05/30/redirection-simple-stuff/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/05/18/korn-shell-website/
		http://nixshell.wordpress.com/2007/05/18/korn-shell-website/#comments
		Fri, 18 May 2007 22:56:16 +0000
		unixshell
		
		

		http://nixshell.wordpress.com/2007/05/18/korn-shell-website/
		
			
David Korn’s Korn shell (ksh) lives at http://www.kornshell.com/
I have to admit to a preference for the Bourne shell; partly because of the if/then/else syntax (what is “[[” all about?!), partly because /bin/sh is bourne (or compatible) on all traditional UNIX and Linux (where it’s bash) systems, but ksh is another good, solid shell. The source is also available from AT&T. It’s also a nice interactive shell, in the absence of bash.
There’s also a nice anecdotal story about David Korn - at a launch of Microsoft’s “Services for UNIX” (which actually provides UNIX-like services for Windows), I believe.
You can also see David Korn hanging with KoRn
       ]]>
			http://nixshell.wordpress.com/2007/05/18/korn-shell-website/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/05/08/pipes-primer/
		http://nixshell.wordpress.com/2007/05/08/pipes-primer/#comments
		Tue, 08 May 2007 00:42:03 +0000
		unixshell
		
		

		

		

		

		

		

		

		

		

		http://nixshell.wordpress.com/2007/05/08/pipes-primer/
		
			
The previous post dealt with pipes, though the example may not have been the best for those who are not accustomed to the concept.
There are a few concepts to be understood - mainly, that of two (or more) processes operating together, how they put their data out, and how the get their data in. UNIX deals with multiple processes, all running (conceptually, at least) at the same time, on different CPUs, each with a standard input (stdin), and standard output (stdout). Pipes connect one process’s stdout to another’s stdin.
What do we want to pipe? Let’s say we’ve got a small 80×25 terminal screen, and lots of files. The ls command will spew out tons of data, faster than we can read it. There’s a handy utility called “more“, which will show a screen-worth of text, then prompt “more”. When you hit the space bar, it will scroll down a screen. You can hit ENTER to scroll one line.
I’m sure that you’ve worked this out already, but here is how we combine these two commands:


$ ls | more

<the first screenful of files is  shown>

–More–


What happens here, is that the “more” command is started up first, then the “ls” command. The output of “ls” is piped to the input of “more”, so it can read the data.
Most such tools can also work another way, too:



$ more myfile.txt

<the first screenful of “myfile.txt” is shown>

–More–


That is to say, “myfile.txt” is taken as standard input (stdin).
       ]]>
			http://nixshell.wordpress.com/2007/05/08/pipes-primer/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/05/03/pipelines-in-the-shell/
		http://nixshell.wordpress.com/2007/05/03/pipelines-in-the-shell/#comments
		Thu, 03 May 2007 00:13:01 +0000
		unixshell
		
		

		

		

		

		

		

		http://nixshell.wordpress.com/2007/05/03/pipelines-in-the-shell/
		
			
One of the most powerful things of the *nix shell, and one which is currently not even covered in the tutorial, is the pipeline. I try to keep this blog and the tutorial from overlapping, but I really must rectify this gap in the main site some time.
In the meantime, this is what it is all about.
UNIX (and therefore GNU/Linux) is full of small text-based utilities : wc to count words (and lines, and characters) in a text file; sort to sort a text file; uniq to get only the unique lines from a text file; grep to get certain lines (but not others) from a text file, and so on.
Did you see the common trait there? Yes, it’s not just that “Everything is a file”, nearly everything is also text. It’s largely from this tradition that HTML, XML, RSS, Email (SMTP, POP, IMAP) and the like are all text-based. Contrast with MS Office, for example, where all data is in binary files which can only (really) be manipulated by the application which created them.
So what? It’s crude, simple, works on an old-fashioned green-and-black screen. How could that be relevant in the 21st Century?
It’s relevant, not because of what each tool itself provides, but what they can do when combined.
Ugly Apache Access Log
Here’s a line from my access.log (Apache2) - Yes, honest, it’s one line. It just looks very ugly:

12.106.111.10 - - [02/May/2007:12:09:58 -0700] “GET /sh/sh.shtml HTTP/1.1″ 200 33080 “http://www.google.ca/search?hl=en&q=bourne+shell&btnG=Google+Search&meta=” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

That’s interesting (well, kind of). But it doesn’t tell us much about this visitor. They came from Google Canada (google.ca), searched for “bourne shell”, and clicked on the link to give them http://steve-parker.org/sh/sh.shtml. They’re using FireFox 1.5.0.3 on Windows.
Great. What happened to them? Did they like the site? Did they stay? Did they actually read anything?
Filtering it
We can use grep to filter out just this visitor. However, this gives us lots of stuff we’re not interested in - all the CSS, PNG, GIF files which support the pages themselves:

$ grep "^12.106.111.10 " access_log

12.106.111.10 - - [02/May/2007:12:09:58 -0700] “GET /sh/sh.shtml HTTP/1.1″ 200 33080 “http://www.google.ca/search?hl=en&q=bourne+shell&btnG=Google+Search&meta=” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:09:58 -0700] “GET /steve-parker.css HTTP/1.1″ 200 8757 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:09:59 -0700] “GET /images/1.png HTTP/1.1″ 200 471 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:09:59 -0700] “GET /images/prevd.png HTTP/1.1″ 200 1397 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:10:00 -0700] “GET /images/2.png HTTP/1.1″ 200 648 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

… etc …
This is looking ugly already, and not just because of the small width of the webpage - even at 80 characters wide, this is hard to understand.
Filtering Some More
At first glance, I should be able to pipe this through a grep for html files:
$ grep "^12.106.111.10 " access_log | grep shtml
However, Apache also logs the referrer, so even the “GET /images/2.png” request above, includes a .shtml request. So I can use “grep -v“. I’ll add the “-w” option to grep to say that the search term must match a whole word. So - “grep -v gif” would let “gifford.html” through, whereas “grep -vw gif” would not. I’ll add “\” to the code, so that you can cut and paste it… the “\” means that although the line breaks, it’s still really part of the same line of code:

$ grep "^12.106.111.10 " access_log | grep -vw css \

   | grep -vw gif | grep -vw jpg \

   | grep -vw png | grep -vw ico

12.106.111.10 - - [02/May/2007:12:09:58 -0700] “GET /sh/sh.shtml HTTP/1.1″ 200 33080 “http://www.google.ca/search?hl=en&q=bourne+shell&btnG=Google+Search&meta=” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:10:32 -0700] “GET /sh/variables1.shtml HTTP/1.1″ 200 48431 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:13:23 -0700] “GET /sh/external.shtml HTTP/1.1″ 200 41322 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:13:45 -0700] “GET /sh/quickref.shtml HTTP/1.1″ 200 42454 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

12.106.111.10 - - [02/May/2007:12:14:27 -0700] “GET /sh/test.shtml HTTP/1.1″ 200 48844 “http://steve-parker.org/sh/sh.shtml” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3″

This pumps the output through a grep which gets rid of CSS files (”| grep -vw css“), then gif, then jpg, then png, then ico. That should just leave us with the HTML files, which is what we’re really interested in.
It’s really hard to see what’s going on here. The narrow web page doesn’t help, but we just want to get the key information out, which should be nice and simple. It should look good however we view it.
If you look carefully, you can see that this visitor accessed /sh/sh.shtml, then /sh/variables1.shtml (after a minute), /sh/external.shtml (after 3 minutes), then /sh/quickref.shtml (about 20 seconds later, presumably - given the referrer is the same, in a new tab). A second later, they opened /sh/test.shtml (which also suggests that they’ve loaded the two pages in tabs, to read at their leisure).
Getting the Data we need
However, none of this is really very easy to read. If we just want to know what pages they visited, and when, we need to do some more filtering. awk is a very powerful tool, of whose abilities we will only scratch the surface here. We will get “fields” 4 and 7 - the timestamp and the URL accessed.

$ grep "^12.106.111.10 " access_log \

| grep -vw css | grep -vw gif \

| grep -vw jpg | grep -vw png \

| grep -vw ico | awk ‘{ print $4,$7 }’

[02/May/2007:12:09:58 /sh/sh.shtml

[02/May/2007:12:10:32 /sh/variables1.shtml

[02/May/2007:12:13:23 /sh/external.shtml

[02/May/2007:12:13:45 /sh/quickref.shtml

[02/May/2007:12:14:27 /sh/test.shtml

Okay, it’s the info we wanted, but it’s still not great. That “[” looks out of place now. We can use cut to tidy things up. In this case, we’ll use its positional spacing, because we want to get rid of the first character. Cut’s “-c” paramater tells it what character to cut from. We want the 2nd character onwards, so we just add it to the end of the pipe line:

$ grep "^12.106.111.10 " access_log | grep -vw css | grep -vw gif | grep -vw jpg | grep -vw png | grep -vw ico | awk '{ print $4,$7 }'|cut -c2-

02/May/2007:12:09:58 /sh/sh.shtml

02/May/2007:12:10:32 /sh/variables1.shtml

02/May/2007:12:13:23 /sh/external.shtml

02/May/2007:12:13:45 /sh/quickref.shtml

02/May/2007:12:14:27 /sh/test.shtml
And that’s the kind of thing that we can do with a pipe. We can get exactly what we want from a file.
Moving on
At the start of this post, we mentioned sort and uniq. Actually, “sort -u” will do the same as “sort  | uniq“. So if we want to get the unique visitors, we can just get the first field (”cut -d" " -f1“) and sort it uniquely:

$ cut -d" " -f1 access_log | sort -u
       ]]>
			http://nixshell.wordpress.com/2007/05/03/pipelines-in-the-shell/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/04/25/new-navigation-buttons/
		http://nixshell.wordpress.com/2007/04/25/new-navigation-buttons/#comments
		Wed, 25 Apr 2007 01:11:20 +0000
		unixshell
		
		

		http://nixshell.wordpress.com/2007/04/25/new-navigation-buttons/
		
			
Long-lasting readers may have noticed a few minor tweaks over the past week or so to the main site - the background to the main text has been getting lighter, so that it has now actually become white. Also, I have added new navigation buttons:Click to View
What do you think? Are these useful navigation aids, Do they make the pages too slow to load (there should only be one new 4Kb image per page), or is it much as before?
Any feedback on the new look of the main shell tutorial is, as ever, greatly appreciated.
       ]]>
			http://nixshell.wordpress.com/2007/04/25/new-navigation-buttons/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/04/21/suid-shell-scripts-setting-the-sticky-bit/
		http://nixshell.wordpress.com/2007/04/21/suid-shell-scripts-setting-the-sticky-bit/#comments
		Sat, 21 Apr 2007 23:58:24 +0000
		unixshell
		
		

		http://nixshell.wordpress.com/2007/04/21/suid-shell-scripts-setting-the-sticky-bit/
		
			
I just found an excellent explanation of why suid shell scripts are a very bad idea.
So just be glad that most modern OSes don’t allow it. You really don’t want to do it, in the first place.
Yes, yes, yes. I know. You do want to do it, and you’re really really sure (really really really sure) that it will all be nice and secure; you’ve got a firewall, completely trusted users, regular password changes, and all the rest of it.
You still don’t want to do it.
Do it in C - you could even get away with a C wrapper, if you must.
But - and let me be quite clear on this:
Don’t do it. It really isn’t worth it, however much it seems, right now, to be a good idea.
What is the “SUID bit”?
The suid bit, is what you get from a “chmod u+s” command, which will set the permissions to “-rwsr-xr-x”, or similar (note the “s” replacing the usual “x”). That means that you can run certain commands with the permissions of a different user (typically root). This is very useful -for example, a normal user under Linux can’t open an ICMP socket (see footnote), so they can’t run the ping command. As a fix, it’s suid root:
$ ls -l /bin/ping
-rwsr-xr-x 1 root root 30848 2007-03-05 04:25 /bin/ping

okay, it’s owned by “root”, and the SUID bit is set. It will work fine, because ping itself has root permissions:
$ /bin/ping -c 1 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.060 ms
--- localhost ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
Good stuff. Now, what if we make a copy? That copy will be owned by a normal user (”steve”, in this example), so it won’t have root permissions:
$ cp /bin/ping /tmp/
$ ls -l /tmp/ping
-rwsr-xr-x 1 steve steve 30848 2007-04-22 00:46 /tmp/ping
$ /tmp/ping localhost
ping: icmp open socket: Operation not permitted
It still had the SUID bit set, it’s just the owner that changed. Let’s just confirm that it’s the “suid + root” that mattered:
$ chmod 755 /tmp/ping
$ ls -l /tmp/ping
-rwxr-xr-x 1 steve steve 30848 2007-04-22 00:46 /tmp/ping
$ /tmp/ping localhost
ping: icmp open socket: Operation not permitted
And what if it’s owned by root?:
$ sudo chown root:root /tmp/ping
$ ls -l /tmp/ping
-rwxr-xr-x 1 root root 30848 2007-04-22 00:46 /tmp/ping
$ /tmp/ping localhost
ping: icmp open socket: Operation not permitted

[footnote]: Why can’t I “ping”?
The reason that Linux ping has to be run as root, isn’t because ping itself is inherently dangerous, but the things that it does (open network connections, sends data over the network to other machines, and receives data back from them) are potentially dangerous. Therefore, those functions are restricted to the root user. So ping can be seen as a special subset of commands which - whilst they do these potentially dangerous things, do them in a safe and secure manner. This means that the ping command can be elevated to superuser status by the “SUID bit”: It’s a trusted user of certain potentially dangerous commands.
A taxi cab is a useful thing, but it could be used for a ram-raid. The taxi cab, in this case, is like the network protocol. The taxi driver is ping. We trust ping to do the Good Stuff, and not the Bad Stuff. So, we don’t let anyone get in and start driving the taxi, but we do let anyone get in and ask the taxi driver to take them somewhere. The command “ping DoS attack www.google.com” will not work - although another network utility, with the same permissions, may well perform a DoS attach. Similarly, getting into a taxi and telling the driver to “drive into that electronics store front window, wait for me to pile the car up with goods, and take me to my hiding place” is not likely to work. If you had a car (had root access) yourself, you could do it, but you can’t (well, should not be able to) convince these trusted services to do nasty stuff on your behalf.
For that reason, we need to be very picky about what code gets the “SUID bit” promotion - the restrictions are there for a reason, so any program with a SUID bit must be able to prove that it will “only use its powers for good, and not for evil”, in the good old melodramatic cliche.
Black-hat hackers will spend a lot more time looking at a suid program like ping than at they will at a normal program like cat, even though cat is far more flexible. If mkdir had the SUID bit set, all kinds of mayhem would ensue.
For shell scripts, where security-through-obscurity is not even possible (chmod permissions of r-x are required, at a minimum, so everyone who can run the script, can read exactly how it was written), suid is always to be avoided.
       ]]>
			http://nixshell.wordpress.com/2007/04/21/suid-shell-scripts-setting-the-sticky-bit/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/04/18/regular-expressions/
		http://nixshell.wordpress.com/2007/04/18/regular-expressions/#comments
		Wed, 18 Apr 2007 09:20:13 +0000
		unixshell
		
		

		

		

		

		

		

		

		

		http://nixshell.wordpress.com/2007/04/18/regular-expressions/
		
			
http://etext.lib.virginia.edu/services/helpsheets/unix/regex.html has a good introduction to Regular Expressions - grep, sed, and friends. 
It includes a brief discussion on Backreferences (aka “the stuff that * matched”)
       ]]>
			http://nixshell.wordpress.com/2007/04/18/regular-expressions/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/04/14/tool-tip-read-it-does-what-it-says/
		http://nixshell.wordpress.com/2007/04/14/tool-tip-read-it-does-what-it-says/#comments
		Sat, 14 Apr 2007 00:48:24 +0000
		unixshell
		
		

		

		

		

		

		

		http://nixshell.wordpress.com/2007/04/14/tool-tip-read-it-does-what-it-says/
		
			
read is a very useful tool; it might seem too simple to bother mentioning, but there are at least three different ways to use it. (Okay, two, and the third isn’t really anything special about read, just a nifty thing that the shell itself provides)…
1. Read the whole line
Let’s start with an interactive script:
$ cat readme.sh
#!/bin/sh
echo “I’m a parrot!”
while read a
do
    echo “A is $a”
done
$ ./readme.sh
I’m a parrot!
hello
A is hello
one two three
A is one two three
piglet eeyore pooh owl
A is piglet eeyore pooh owl
^D
$
Yes, you’ll need to hit CTRL-D to exit this loop, it’s just a simple example.
So far, so stupid. But wait; what if I wanted to get that “one” “two” “three” and use them differently?
2. Read the words
$ cat readme.sh
#!/bin/sh
echo “I’m a parrot!”
while read a b c
do
        echo “A is $a”
        echo “B is $b”
        echo “C is $c”
done
$ ./readme.sh
I’m a parrot!
hello
A is hello
B is
C is
one two three
A is one
B is two
C is three
piglet eeyore pooh owl
A is piglet
B is eeyore
C is pooh owl
^D
$
So, just by naming some variables, we can pick what we get. And - did you see that last one? We don’t lose anything, either… Just because we asked for three variables (a, b, c) and we got 4 values (piglet eeyore pooh owl), we didn’t lose anything; the last one was treated like a normal read.
This is actually pretty handy stuff; you’d have to do a bit of messing about with pointers to get the same effect in C, for example.
3. Read from a file
We can do all this from a file, too. This isn’t special to read, but it’s often used in this way. See that “while - do - done” loop? It’s a sub-shell, and we can direct whatever we want to its input (everything is a file, remember, so the keyboard, a text file, a device driver, whatever you want, it’s all just a file)
We do this with the “<” operator. Just add “< filename.txt” after the “done” end of the loop:
$ cat readme.sh
#!/bin/sh
echo “I’m a parrot!”
while read a b c
do
        echo “A is $a”
        echo “B is $b”
        echo “C is $c”
done  < myfile.txt
$ cat myfile.txt
1 2 3
4
5 6
7
8 9 10 11 12 13
14
15 16 17
$  ./readme.sh
I’m a parrot!
A is 1
B is 2
C is 3
A is 4
B is
C is
A is 5
B is 6
C is
A is 7
B is
C is
A is 8
B is 9
C is 10 11 12 13
A is 14
B is
C is
A is 15
B is 16
C is 17

So we can process tons of data, wherever it comes from.
4. I only mentioned 3 uses
We could make the script a bit more useful, by allowing the user to specify the file, instead of hard-coding it to “myfile.txt“:
$ cat readme.sh
#!/bin/sh
echo “I’m a parrot!”
while read a b c
do
        echo “A is $a”
        echo “B is $b”
        echo “C is $c”
done < $1
$ cat someotherfile.txt
123
1 2 3
one two three four
$ ./readme.sh someotherfile.txt
I’m a parrot!
A is 123
B is
C is
A is 1
B is 2
C is 3
A is one
B is two
C is three four
$
Update 14 April
Updated to fix the “done < filename.txt” from the example code of the last two examples.
       ]]>
			http://nixshell.wordpress.com/2007/04/14/tool-tip-read-it-does-what-it-says/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/04/08/more-maths-stuff-bc-in-detail/
		http://nixshell.wordpress.com/2007/04/08/more-maths-stuff-bc-in-detail/#comments
		Sun, 08 Apr 2007 00:22:13 +0000
		unixshell
		
		

		

		

		

		

		

		http://nixshell.wordpress.com/2007/04/08/more-maths-stuff-bc-in-detail/
		
			
There’s a great post about bc at basicallytech.com - I think that I’ve already covered most of the same ground, but it’s got lots of great examples.
       ]]>
			http://nixshell.wordpress.com/2007/04/08/more-maths-stuff-bc-in-detail/feed/
	
		
			unixshell
		
	
		
		
		http://nixshell.wordpress.com/2007/03/26/calculating-averages/
		http://nixshell.wordpress.com/2007/03/26/calculating-averages/#comments
		Mon, 26 Mar 2007 22:30:27 +0000
		unixshell
		
		

		

		

		

		

		http://nixshell.wordpress.com/2007/03/26/calculating-averages/
		
			
The Simple Maths post seems to be the most popular article in the so-far short life of this blog. 
It’s also something that I have received a few emails about recently, so I feel like posting a bit more on the subject.
I think that the code can speak for itself… We implement a loop, which calls the builtin read function (I’m not sure the “-p” flag, to provide a prompt, is universal. It does work with the Bash builtin. If it doesn’t work on your *nix, it’s really only for show, so you can live without it.
Because read works on standard input (aka “stdin”), it will work interactively from the keyboard, or direct from a file (one number per line).
We use two methods of doing maths in the shell:

expr, because it’s a simple and easily-read way to do simple maths: n=`expr $n + 1`

bc, because it is more powerful. Do have a play with bc interactively, it can do a lot… see below.

So, we can write a fairly simple script (read down, it’s only actually 11 lines of code without the comments), which is actually quite versatile - it can do running averages, it can be interactive or run from cron, called from another script, even used as a function.
So, here’s the code. It should be fairly self-explanatory, but do have a look at the interactive bc sample session below, to see what we are doing with bc. Also, do play with bc (some Linux distros have dropped it from the default install recently, so you’ll have to yast -i bc, or equivalent)
The Script - Calculate Averages
#!/bin/sh
# Calculate mean (average) of integer data

# Initialise the variables
n=0     # n being the number of (valid) data provided
sum=0   # sum being the running total of all data

# Note that by using ^D (aka "EOF") to quit, this
# script will work just as well interactively, as
# when provided with a file containing the data.
while read -p "Enter a number (^D to quit): " x
do
        # expr is useful for simple maths
  sum=`expr $sum + $x`
  # If this fails, it was non-numeric input
  if [ "$?" -eq "0" ]; then
    # Okay, it was valid input.
    n=`expr $n + 1`
    # We can provide a “running average” here;
    # I’ll comment it out for now.
    # echo “Running Average:”
    # echo “scale=2;$sum/$n” | bc
    # echo
  fi
done

# Okay, we’ve done the loop.
# Present the data.
echo “Overall Average:”
        # bc is more useful than expr for
        # more involved maths, though its
        # syntax, particularly in a script,
        # is possibly less obvious.
        # Using bc interactively is easier
        # than using it in a shell script
echo “scale=2;$sum/$n” | bc

Interactive bc
The bold text is user input. The rest is from bc:



$ bc

bc 1.06

Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.

This is free software with ABSOLUTELY NO WARRANTY.

For details type `warranty’.


ibase=2
I’ll be entering base2 (binary)


01001001
So, I enter 1001001 (73)


73
And it replies with the answer in base 10


ibase=10
Does this set the input base back to 10?


10
Let’s input “10″, it should reply “10″


2
No, we entered “10″ in base 2, which is 2!


ibase=1010
So, 10 in binary is 1010 (8+2)


10
We say 10


10
And bc says 10. Good, we’re back to normal


11
And the same for 11


11
Good, it works. Now for some maths..


1 + 2
(tricky stuff!)


3
Yes, that’s good, 1+2=3


23 + 34 + 45 + 56
We’re not limited to x+y


158
So we can build up our sums


10/3
10/3 = 3 and a third, right?


3
Not to 0 decimal places.


scale=2
Okay, let’s have 2 decimal places


10/3
Now ask again


3.33
That’s better


scale=5
Or to 5 points?


10/3
Ask again…


3.33333
And it works!


scale=1
One point: 


10/3
And ask again


3.3
As we expected.


scale=0
So, scale=0 means 0 places


10/3
Should say 3


3
Yes, we’re back to where we started.


Back to the Script
That made a nice break. Now we’ll go back to the script… it’s only actually 11 lines long:
#!/bin/sh
n=0
sum=0
while read x
do
  sum=`expr $sum + $x`
  if [ "$?" -eq "0" ]; then
    n=`expr $n + 1`
  fi
done
echo “scale=2;$sum/$n” | bc

And as I said, we can use it interactively, or with a file of data:
$ cat data.txt
4
5
6
$ average.sh
5.00

Because, under *nix, EVERYTHING IS A FILE, even the keyboard!
       ]]>
			http://nixshell.wordpress.com/2007/03/26/calculating-averages/feed/
	
		
			unixshell