Shell Scripting Tutorial on Kindle

Unix & Linux Shell Scripting Tutorial on Kindle

The Shell Scripting tutorial at http://steve-parker.org is now available natively on the Kindle!

USA (amazon.com)

UK (amazon.co.uk)

Similarly, you can search for “B00C2EGNSA” on any Amazon site, or just go to http://www.amazon.COUNTRY/dp/B00C2EGNSA (where “COUNTRY” is .fr, .de, etc) for your local equivalent.

Track memory usage on Solaris

When a Solaris server is overloaded, this is one way to check what actual memory each process is using. Here I am restricting the checks to one user (“steve”) but by omitting the “-u steve” flag to ps, the whole system will be checked.

$ps -ea -o pid,rss -u steve | sort -n -k 2 | while read pid rss
> do
> echo -en "RSS : ${rss} Kb: "
> pmap -x $pid | tail -1
> done
RSS: 50104 Kb: total Kb 1384400 712744 494312 -
RSS: 669296 Kb: total Kb 1414648 819496 657584 -

This uses the pmap tool to inspect the actual memory usage, which as the ps(1) man page says, is more accurate than just the RSS field of ps.

I suspect that a method using pmap alone might be possible, but this is just a quick note as I go…

Shell Scripting page on Facebook

Shell Scripting

My Shell Scripting book, due out on August 12th by Wrox, now has a page on Facebook: http://www.facebook.com/pages/Shell-Scripting/175263275869249. Feel free to “Like” it, and get the latest updates on the project.

I have the final pages to proofread this week, ready to go to the printers. It’s looking like 576 pages, a little bit over the target of 504 pages, but close enough.

I will update the Table of Contents at http://sgpit.com/book/ once the page count is finalised.

Update on Shell Scripting Recipes book

Wow, it’s been nearly two months since I last made a post about the upcoming book on shell scripting. I’m really sorry, I had intended to give much more real-time updates here. The book focusses on GNU/Linux and the Bash shell in particular, but it does cover the other environments too – Solaris, Bourne Shell, as well as mentions for ksh, zsh, *BSD and the rest of the Unix family.

In terms of page count, it is currently 89% finished. There is still the proof-reading to be done, and whatever delivery details the publishers need to deal with, so the availability date of some time in August is still on schedule. I notice that http://amzn.com/1118024486 is already offering a massive discount on the cover price; I have no idea what that is about, I’m trying not to take offence – they can’t have dismissed the book already as I have not quite finished writing it yet! So hopefully you can get a bargain while it’s cheap.

The subject matter has the potential to be quite boring if presented as a list of tedious system administration tasks, so I have tried to make it light and fun whenever I can; it’s still with Legal at the moment, but I hope to have a Space Invaders clone written entirely in the shell published in the book. People don’t tend to see the Shell as being capable of doing anything interactive at all, so it is nice to write a playable interactive game in the shell. The main problem in terms of playability is in working out how much to slow it down, and at what stage! Of course, being a shell script, you can tweak the starting value, the level at which it speeds up, and anything else about the gameplay. If the game doesn’t make it in to the book, I’ll post it here anyway, and will welcome your contributions on gameplay.

Other than games, I’ve got recipes for init scripts, conditional execution, translating scripts into other (human) languages, even writing CGI scripts in the shell. There is coverage of arrays, functions, libraries, process control, wildcards and filename expansion, pipes and pipelines, exec and redirection of input and output; this book aims to cover pretty much all that you need to know about shell scripting without being a tedious list of what the bash shell can do.

There is a status page at http://sgpit.com/book which also has order information; you can pre-order your copy from there.

Ten Good Unix Habits

IBM’s DeveloperWorks has 10 Good Unix Habits, which apply to GNU/Linux at least as much as to Unix.

I would expect that most experienced admins can second-guess the content to 5-7 of these 10 points, just from the title (for example, item 1 is a reference to “mkdir -p”, plus another related syntax available to Bash users). I would be surprised if you knew all ten:

1. Make directory trees in a single swipe.
2. Change the path; do not move the archive.
3. Combine your commands with control operators.
4. Quote variables with caution.
5. Use escape sequences to manage long input.
6. Group your commands together in a list.
7. Use xargs outside of find .
8. Know when grep should do the counting — and when it should step aside.
9. Match certain fields in output, not just lines.
10. Stop piping cats.

How many did you get?

Use of pipes, and other nifty tricks

http://www.tuxradar.com/content/command-line-tricks-smart-geeks has some useful tricks. A lot of it is presented as being bash-specific, but isn’t. Also, a lot seems Linux-specific, but isn’t. Lots of useful info for all Unix/Linux admins here. These hints go on and on; hardly any of them are the generic stuff you often see on Ubuntu forums, stumbleupon, and so on.

Book

A serious publisher has contacted me about writing a serious book about Linux shell programming.

It is all really very serious. I’m not used to being serious, as you can probably tell from the fact that I have now used the word “serious” four times in this three-sentence post.

I am rather keen to write a book on the subject, not because I’m vain, or desperate for money, but because the stuff I have seen out there in dead-tree format has been of rather low quality. Also because of all the emails I’ve received over the years, they have all been positive, and none has said anything along the lines of “I didn’t need any of that because I bought Book[X]”, or indeed any book. People have emailed me, asking for advice as to what book to buy, and I have been unable to recommend any book that I have seen.

So:

What would you like to see in your ideal book about UNIX / Linux shell scripting, be it Bourne, Bash, ksh, tcsh, zsh, whatever?

Please don’t be timid; if you want to know how to work out how many nose-flutes can be fitted into the area of a Boeing 757, you won’t be anything like as strange as some of the correspondants I’ve had over the years, so please, tell me what is bugging you, what has bugged you, or even what you think might be likely to bug you in days / months / years to come.

I’m likely to answer any specific questions here and now, whether or not they end up in the book, but anything you’d like to see in a book, too… post that here, and I’ll have a stab at it.

Also, I would of course be interested to know if you have found any useful books on or around the subject, and what they did particularly well.

Steve

Happy First Birthday!

This blog has now been running for a year; the first post was Hello World on 17th Jan 2007.

I hadn’t realised it had been going for so long; in that time, I’ve made 41 posts, so I haven’t quite managed to make one post per week 😦 I have been a bit slack lately, for which I do apologise. New Years Resolution: I must make more posts here!

In the meantime, my main site, steve-parker.org, has celebrated its seventh birthday, having been born in June 2000 – looking forward to making the 8th birthday celebrations this June!

Shell Pipes by Example

Pipes, piping, pipelines… whatever you call them, are very powerful – in fact, they are one of the core tenets of the philosophy behind UNIX (and therefore Linux). They are also, really, very simple, once you understand them. The way to understand them, is by playing with them, but if you don’t know what they do, you don’t know where to start… Catch-22!

So, here are some simple examples of how the pipe works.

Let’s see the code

$ grep steve /etc/passwd | cut -d: -f 6
/home/steve
$
What did this do? There are two UNIX commands there: grep and cut. The command “grep steve /etc/passwd” finds all lines in the file /etc/passwd which contain the text “steve” anywhere in the line. In my case, this has one result:
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash
The second command, “cut -d: -f6” cuts the line by the delimiter (-d) of a colon (“:“), and gets field (-f) number 6. This is, in the /etc/passwd file, the home directory of the user.

So what? Show me some more

This is the main point of this article; once you’ve seen a few examples, it normally all becomes clear.

EG2

$ find . -type f -ls | cut -c14- | sort -n -k 5
rw-r--r--   1 steve    steve       28 Jul 22 01:41 ./hello.txt
rwxr-xr-x   1 steve    steve     6500 Jul 22 01:41 ./a/filefrag
rwxr-xr-x   1 steve    steve     8828 Jul 22 01:42 ./c/hostname
rwxr-xr-x   1 steve    steve    30848 Jul 22 01:42 ./c/ping
rwxr-xr-x   1 steve    steve    77652 Jul 22 01:42 ./b/find
rwxr-xr-x   1 steve    steve    77844 Jul 22 01:41 ./large
rwxr-xr-x   1 steve    steve    93944 Jul 22 01:41 ./a/cpio
rwxr-xr-x   1 steve    steve    96228 Jul 22 01:42 ./b/grep
$

What I did here, was three commands: “find . -type f -ls” finds regular files, and lists them in an “ls”-style format: permissions, owner, size, etc.
“cut -c14-” cuts out the first 14 characters, which mess up the formatting on this website (!), and aren’t very interesting.
“sort -n -k 5” does a numeric (-n) sort, on field 5 (-k5), which is the size of the file.
So this gives me a list of the files in this directory (and subdirectories), ordered by file size. That’s much more useful than “ls -lS“, which restricts itself to the current directory, but not subdirectories.

(As an aside, I have to admit that I only concocted this by trying to think of an example; it actually seems really useful, and worth making into an alias… I must do a post about “alias” some time!)

So how does it work?

This seems pretty straightforward: get lines containing “steve” from the input file (“grep steve /etc/passwd“), and get the sixth field (where fields are marked by colons) (“cut -d: -f6“). You can read the full command from left to right, and see what happens, in that order.

How does it really work?

EG1 Explained

There are some gotchas when you start to look at the plumbing. Because we’re using the analogy of a pipe (think of water flowing through a pipe), the OS actually sets up the commands in the reverse order. It calls cutfirst, then it calls grep. If you have (for example) a syntax error in your cut command, then grep will never be called.
What actually happens is this:

1. A “pipe” is set up – a special entity which can take input, which it passes, line by line, to its output.
2. cut is called, and its input is set to be the “pipe”.
3. grep is called, and its output is set to be the “pipe”.
4. As grep generates output, it is passed through the pipe, to the waiting cut command, which does its own simple task, of splitting the fields by colons, and selecting the 6th field as output.

EG2 Explained

For EG2, “sort” is called first, which ties to the second (rightmost) pipe for its input. Then “cut” is called, which ties to the second pipe for its output, and the first (leftmost) pipe for its input. Then, “find” is called, which ties to the first pipe for its output.
So, the output of “find” is piped into “cut“, which strips off the first 14 characters of the “find” output. This is then passed to “sort“, which sorts on field 5 (of what it receives as input), so the output of the entire pipeline, is a numerically sorted list of files, ordered by size.