Update on Shell Scripting Recipes book

April 23, 2011

Wow, it’s been nearly two months since I last made a post about the upcoming book on shell scripting. I’m really sorry, I had intended to give much more real-time updates here. The book focusses on GNU/Linux and the Bash shell in particular, but it does cover the other environments too – Solaris, Bourne Shell, as well as mentions for ksh, zsh, *BSD and the rest of the Unix family.

In terms of page count, it is currently 89% finished. There is still the proof-reading to be done, and whatever delivery details the publishers need to deal with, so the availability date of some time in August is still on schedule. I notice that http://amzn.com/1118024486 is already offering a massive discount on the cover price; I have no idea what that is about, I’m trying not to take offence – they can’t have dismissed the book already as I have not quite finished writing it yet! So hopefully you can get a bargain while it’s cheap.

The subject matter has the potential to be quite boring if presented as a list of tedious system administration tasks, so I have tried to make it light and fun whenever I can; it’s still with Legal at the moment, but I hope to have a Space Invaders clone written entirely in the shell published in the book. People don’t tend to see the Shell as being capable of doing anything interactive at all, so it is nice to write a playable interactive game in the shell. The main problem in terms of playability is in working out how much to slow it down, and at what stage! Of course, being a shell script, you can tweak the starting value, the level at which it speeds up, and anything else about the gameplay. If the game doesn’t make it in to the book, I’ll post it here anyway, and will welcome your contributions on gameplay.

Other than games, I’ve got recipes for init scripts, conditional execution, translating scripts into other (human) languages, even writing CGI scripts in the shell. There is coverage of arrays, functions, libraries, process control, wildcards and filename expansion, pipes and pipelines, exec and redirection of input and output; this book aims to cover pretty much all that you need to know about shell scripting without being a tedious list of what the bash shell can do.

There is a status page at http://sgpit.com/book which also has order information; you can pre-order your copy from there.


Ten Good Unix Habits

June 22, 2010

IBM’s DeveloperWorks has 10 Good Unix Habits, which apply to GNU/Linux at least as much as to Unix.

I would expect that most experienced admins can second-guess the content to 5-7 of these 10 points, just from the title (for example, item 1 is a reference to “mkdir -p”, plus another related syntax available to Bash users). I would be surprised if you knew all ten:

1. Make directory trees in a single swipe.
2. Change the path; do not move the archive.
3. Combine your commands with control operators.
4. Quote variables with caution.
5. Use escape sequences to manage long input.
6. Group your commands together in a list.
7. Use xargs outside of find .
8. Know when grep should do the counting — and when it should step aside.
9. Match certain fields in output, not just lines.
10. Stop piping cats.

How many did you get?


Use of pipes, and other nifty tricks

December 18, 2009

http://www.tuxradar.com/content/command-line-tricks-smart-geeks has some useful tricks. A lot of it is presented as being bash-specific, but isn’t. Also, a lot seems Linux-specific, but isn’t. Lots of useful info for all Unix/Linux admins here. These hints go on and on; hardly any of them are the generic stuff you often see on Ubuntu forums, stumbleupon, and so on.


Linux Command Directory

May 16, 2009

I just found this page on the OReilly website – a Linux Command Directory

Click on any of the 687 commands below to get a description and list of available options. All links in the command summaries point to the online version of the book on Safari Bookshelf.

It doesn’t cover everything (what could?) but it could be a useful page to bookmark.


awk one-liners

April 1, 2009

I have previously plugged the great list of sed 1-liners at http://sed.sourceforge.net/sed1line.txt.

Here is a similar (if shorter) list of handy awk 1-liners:

http://www.sap-basis-abap.com/unix/awk-one-liner-tips.htm:

Print column1, column5 and column7 of a data file or output of any columns list

awk '{print $1, $5, $7}' data_file

cat file_name |awk '{print $1 $5 $7}'

ls –al |awk '{print $1, $5, $7}' -- Prints file_permissions,size and date

List all files names whose file size greater than zero.

ls –al |awk '$5 > 0 {print $9}'

List all files whose file size equal to 512bytes.

ls –al |awk '$5 == 512 {print $9}'

print all lines

awk '{print }' file_name

awk '{print 0}' file_name

Number of lines in a file


awk ' END {print NR}' file_name

Number of columns in each row of a file

awk '{print NF}' file_name

Sort the output of file and eliminate duplicate rows

awk '{print $1, $5, $7}' |sort –u

List all file names whose file size is greater than 512bytes and owner is “oracle”

ls –al |awk '$3 == "oracle" && $5 > 512 {print $9}'

List all file names whose owner could be either “oracle” or “root”

ls –al |awk '$3 == "oracle" || $3 == "root" {print $9}'

list all the files whose owner is not “oracle

ls –al |awk '$3 != "oracle" {print $9}'

List all lines which has at least one or more characters

awk 'NF > 0 {print }' file_name

List all lines longer that 50 characters

awk 'length($0) > 50 {print }' file_name

List first two columns

awk '{print $1, $2}' file_name

Swap first two columns of a file and print

awk '{temp = $1; $1 = $2; $2 = temp; print }' file_name

Replace first column as “ORACLE” in a data file

awk '{$1 = "ORACLE"; print }' data_file

Remove first column values in a data file

awk '{$1 =""; print }' data_file

Calculate total size of a directory in Mb

ls –al |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'

Calculate total size of a directory including sub directories in Mb

ls –lR |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'


Find largest file in a directory including sub directories

ls –lR |awk '{print $5 "\t" $9}' |sort –n |tail -1


ssh port forwarding without starting a new session

December 10, 2008

You can forward ports with ssh like this:

$ ssh -L 8080:localhost:80 user@remotehost

This will log you in to remotehost as user, and port 8080 on your local machine will be tunnelled to port 80 on remotehost. If remotehost can see a machine that you can’t (for example, if it’s on an internal network), you can even do this:

$ ssh -L 8080:internalhost:80 user@borderhost

This will log you in to borderhost, and localhost:8080 will be directed to internalhost:80, even though you may not be able to see internalhost directly yourself.

What I didn’t know until I read Nico Golde’s blog today, is that you can do this interactively, with an existing session. Tilde (~) is the default escape character, and ~C (note that’s an uppercase C) gets you a shell session within ssh itself:

$ ssh user@remotehost
user@remotehost$ ~C
ssh> -L 8080:localhost:80
Forwarding port.
user@remotehost$


Emptying deleted files

November 8, 2008

Mike Hommey has a nice article on clearing down deleted (but still-open) files:
http://glandium.org/blog/?p=211. He explains the problem clearly, and goes on to provide the solution, too. Well worth bookmarking, for that 2am emergency!

someone had to free some space on a 1GB filesystem, and thought a good idea would be to delete that 860MB log file that nobody cares about. Except that it didn’t really remove it, but he didn’t really check.

Later, the “filesystem full” problem came back at someone else, who came to ask me what files from a small list he could remove. But the files were pretty small, and that wouldn’t have freed enough space. That gave me the feeling that we probably were in this typical case I introduced this post with, which du -sk confirmed: 970MB used on the filesystem according to df, but only 110MB worth of data…

Mike has solutions to this for Solaris and Linux; lsof is also useful for this kind of thing, on systems which have it.


Some useful Linux / *nix admin words of experience

August 21, 2008

Entitled “Lazy Linux: 10 essential tricks for admins“, this is a pretty good set of essential commands for *nix administrators.

If I have to suggest anything for in-depth study, it’s “Trick 6″: “Remote VNC session through an SSH tunnel”, because you can actually do just about anything through an SSH tunnel. If you’re faced with you -> firewall -> internet -> remotemachine -> internet -> restoftheworld, you can do whatever you like with “restof theworld” if you can ssh to “remotemachine”, whatever the local firewall thinks. And you can probably ssh over port 443 through the firewall. Hmm; maybe I should write this up properly at some point. I’m sure it’s well documented on the web if you look for it….


Book

April 23, 2008

A serious publisher has contacted me about writing a serious book about Linux shell programming.

It is all really very serious. I’m not used to being serious, as you can probably tell from the fact that I have now used the word “serious” four times in this three-sentence post.

I am rather keen to write a book on the subject, not because I’m vain, or desperate for money, but because the stuff I have seen out there in dead-tree format has been of rather low quality. Also because of all the emails I’ve received over the years, they have all been positive, and none has said anything along the lines of “I didn’t need any of that because I bought Book[X]“, or indeed any book. People have emailed me, asking for advice as to what book to buy, and I have been unable to recommend any book that I have seen.

So:

What would you like to see in your ideal book about UNIX / Linux shell scripting, be it Bourne, Bash, ksh, tcsh, zsh, whatever?

Please don’t be timid; if you want to know how to work out how many nose-flutes can be fitted into the area of a Boeing 757, you won’t be anything like as strange as some of the correspondants I’ve had over the years, so please, tell me what is bugging you, what has bugged you, or even what you think might be likely to bug you in days / months / years to come.

I’m likely to answer any specific questions here and now, whether or not they end up in the book, but anything you’d like to see in a book, too… post that here, and I’ll have a stab at it.

Also, I would of course be interested to know if you have found any useful books on or around the subject, and what they did particularly well.

Steve


Happy First Birthday!

January 6, 2008

This blog has now been running for a year; the first post was Hello World on 17th Jan 2007.

I hadn’t realised it had been going for so long; in that time, I’ve made 41 posts, so I haven’t quite managed to make one post per week :( I have been a bit slack lately, for which I do apologise. New Years Resolution: I must make more posts here!

In the meantime, my main site, steve-parker.org, has celebrated its seventh birthday, having been born in June 2000 – looking forward to making the 8th birthday celebrations this June!


Follow

Get every new post delivered to your Inbox.