Unix & Linux Shell Scripting Tutorial on Kindle

USA (amazon.com)

UK (amazon.co.uk)

Similarly, you can search for “B00C2EGNSA” on any Amazon site, or just go to http://www.amazon.COUNTRY/dp/B00C2EGNSA (where “COUNTRY” is .fr, .de, etc) for your local equivalent.

Any feedback on this work is welcome here, I will pass on all praise to Mahmood,

Steve

$ps -ea -o pid,rss -u steve | sort -n -k 2 | while read pid rss
> do
> echo -en "RSS : ${rss} Kb: "
> pmap -x $pid | tail -1
> done
RSS: 50104 Kb: total Kb 1384400 712744 494312 -
RSS: 669296 Kb: total Kb 1414648 819496 657584 -

This uses the pmap tool to inspect the actual memory usage, which as the ps(1) man page says, is more accurate than just the RSS field of ps.

I suspect that a method using pmap alone might be possible, but this is just a quick note as I go…

If you find yourself in a shell session, but don’t know what type of shell it is, there are a few ways to find out whether it’s Bourne shell, Bash, ksh, csh, zsh, tcsh, or whatever.

The simple answer is that you type in the command

echo $SHELL

which should tell you the path to your current shell; if it’s /bin/ksh or /usr/bin/ksh then it’s the KornShell; if it’s /bin/csh then it’s the C shell, and so on.

However, it’s not always that simple; Bash will set $SHELL only if it was not already set, so if you were running csh and use that to call bash, then Bash’s $SHELL will still say csh, not bash. However, Bash will set the $BASH_VERSION variable, but that’s not really a guarantee that you have that version; it only tells you that there exists a variable which specifies that version. You may not even be running bash at all.

Similarly, some shells (particularly /bin/sh) are symbolic links to others – whether to bash (GNU/Linux and newer Solaris), dash (some Linux distros more recently), or ksh (AIX).

So $SHELL is a useful start, but not at all definitive. You can search for your UserID in the /etc/passwd file:

$ grep steve /etc/passwd
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash
$ 

But there may be many entries containing the text “steve”, and you could be using NIS, LDAP, AD or some other authentication mechanism. $UID should be a read-only variable containing your UserID, and you can search your system-specific passwd via the getent command:

$ getent passwd $UID
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash
$ 

However, as my UID is 1000, this would match any line containing “1000″ such as an account called “HAL1000″. So this should be one way to get your shell:

$ getent passwd $UID | cut -d: -f3,7|grep "^${UID}:"
1000:/bin/bash
$ 

Still, you could have run another shell after picking up your default shell. You can always check $0 – that should tell you how your current shell was called. You don’t know what the $PATH variable was when that was called – if $0 says “/usr/bin/zsh” then that’s what was called (of course, it is possible that/usr/bin/zsh could have changed since your shell was invoked!). If it just says “sh” then it’s whatever “sh” was found first in the $PATH of the calling shell. And of course, you can’t find out for sure what state that shell was in at that time.

On a Linux system, “cat /proc/$$/cmdline” should also give a good clue; “ls -l /proc/$$/exe” is better (but not definitive; it may be marked “(deleted)” if it’s been deleted, so you should check if it’s been subsequently replaced by some other shell.

So – it depends why you need to know. If you need to know for sure, on an unknown system, exactly what shell you are in, then I’m sorry, it’s not possible to be absolutely certain. To be reasonably confident, check $SHELL or $0. If you need to be more certain than that, then check for the features you require.

If you’re writing a shell script which requires arrays, then define and access an array, check that the results are as expected, and if not, bail out with a message along the lines of “an array-capable shell is expected – we suggest bash or ksh”.

The classic rm -rf / gets the message “rm of / is not allowed” and a return code of 2. Some variations get the same, whilst others (such as cd /; rm -rf .) get no message, a return code of zero, but nothing happens. Here is a transcript:

$ ssh sunflare
Password:
Last login: Tue Jan 31 21:38:05 2012 from goldie
Oracle Corporation SunOS 5.10 Generic Patch January 2005
steve@sunflare:~$ su -
Password:
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# bash
root@sunflare:/# df -h .


Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d10        9.6G   3.6G   5.9G    38%    /
root@sunflare:/# rm -rf /
rm of / is not allowed
root@sunflare:/# echo $?
2
root@sunflare:/# cd /usr
root@sunflare:/usr# rm -rf ../
rm of / is not allowed
root@sunflare:/usr# cd /tmp
root@sunflare:/tmp# rm -rf ../
rm of / is not allowed
root@sunflare:/tmp# cd /usr/sfw/bin
root@sunflare:/usr/sfw/bin# rm -rf ../../..
root@sunflare:/usr/sfw/bin# cd /
root@sunflare:/# rm -rf .
root@sunflare:/# echo $?
0
root@sunflare:/# pwd
/
root@sunflare:/# df -h .

/dev/md/dsk/d10        9.6G   3.6G   5.9G    38%    /
root@sunflare:/# cd /dev
root@sunflare:/dev# rm -rf ../tmp/../
rm of / is not allowed
root@sunflare:/dev# rm -rf ../tmp/..
root@sunflare:/dev# exit
steve@sunflare:~$ logout
Connection to sunflare closed.
$

Paper books:
Amazon USA
Amazon UK
Also available at Barnes & Noble, WH Smiths, Waterstones, Supermarkets (online, if not in-store), and as the saying goes, every good bookshop (and some others too).

Kindle: Kindle

iPad / etc: iTunes

Nook: Nook

There’s also a Facebook page at facebook.com/shellscript.

Space Game

The shell is often seen as old hat, hard to use, green screens and no interaction. In my new book, Shell Scripting (http://facebook.com/shellscript), I attempt to throw a new perspective on this; the shell can be used in lots of creative and imaginative ways.

The book also covers creating CSV files to easily plot complex data in spreadsheets, creating HTML as well as parsing HTML documents; I plan to post more updates before its launch on 12th August 2011.

Shell Scripting by Steve Parker. ISBN 1118024486

http://amzn.com/1118024486
http://facebook.com/shellscript
http://sgpit.com/book

You can get the code for the game here. The description and explanation is in the book, of course. But at least the code itself is available online for inspection.

Shell Scripting

I have the final pages to proofread this week, ready to go to the printers. It’s looking like 576 pages, a little bit over the target of 504 pages, but close enough.

I will update the Table of Contents at http://sgpit.com/book/ once the page count is finalised.

By logging in to the console, I was somehow able to get a shell (one process). Having got that shell, even though I was root, it was difficult to be able to spawn other processes. It turned out that this was because we had restricted the CPU count on the kernel command line (maxcpus=2) so that a dual processor, 16-core machine had only one eighth of its processing power available. The dynamic change to the nproc value does not take this into account, so this unprivileged user was able to fork-bomb the entire machine.

The first thing you might want to do in this situation is to run ps -eaf. That’s another process, and even as root, you don’t get to do it. Being Linux, you can see how many processes exist on the system by listing /proc:
cd /proc
echo *

Neither of these commands spawn a new shell, they are both shell builtin commands, so they will work. In this case, with over 69,000 processes, I killed the output before I got too bored. Since there are usually around 200 processes running, that was enough to tell me that something was wrong.

After many attempts, a ps command did work, and confirmed that a certain shell script was being run a lot of times. I couldn’t cat that file, and didn’t even have its full name (ps truncates output to match the terminal’s width). I had the PID, so /proc/$PID/fd gave the filename.

It’s not possible to cat the script to see what it’s doing, so more builtin commands are required.
$ while read f
> do
> echo $f
> done < /path/to/script.sh
This uses all builtin commands, and tells you what the script is. From there, you may have some insight into what it is doing, and how to stop it.

The full chapter list is at http://sgpit.com/book/ along with links to other pages with more details about the book.

In terms of page count, it is currently 89% finished. There is still the proof-reading to be done, and whatever delivery details the publishers need to deal with, so the availability date of some time in August is still on schedule. I notice that http://amzn.com/1118024486 is already offering a massive discount on the cover price; I have no idea what that is about, I’m trying not to take offence – they can’t have dismissed the book already as I have not quite finished writing it yet! So hopefully you can get a bargain while it’s cheap.

The subject matter has the potential to be quite boring if presented as a list of tedious system administration tasks, so I have tried to make it light and fun whenever I can; it’s still with Legal at the moment, but I hope to have a Space Invaders clone written entirely in the shell published in the book. People don’t tend to see the Shell as being capable of doing anything interactive at all, so it is nice to write a playable interactive game in the shell. The main problem in terms of playability is in working out how much to slow it down, and at what stage! Of course, being a shell script, you can tweak the starting value, the level at which it speeds up, and anything else about the gameplay. If the game doesn’t make it in to the book, I’ll post it here anyway, and will welcome your contributions on gameplay.

Other than games, I’ve got recipes for init scripts, conditional execution, translating scripts into other (human) languages, even writing CGI scripts in the shell. There is coverage of arrays, functions, libraries, process control, wildcards and filename expansion, pipes and pipelines, exec and redirection of input and output; this book aims to cover pretty much all that you need to know about shell scripting without being a tedious list of what the bash shell can do.

There is a status page at http://sgpit.com/book which also has order information; you can pre-order your copy from there.

I hope to publish more details here as things progress; for now, it is well on the way, but it is not too late for readers to contact me (steve@steve-parker.org) if there is anything that you see as vital for a Shell Scripting Recipes book which was maybe missing from some other book you saw.

http://amzn.com/1118024486

Part I covers Language and Usage; all of the concepts of the Shell and how it works.
Part II is Recipes using System Tools. This covers the commands that are necessary for shell scripting, and includes quite a few surprising ways to use them.
Part III is Recipes using Shell Features. This is similar to Part II but it gives concrete uses for the theory presented in Part I.
Part IV is Recipes for Systems Administration. This provides (and explains) various recipes for real-world systems administration tasks of and beyond the ordinary.

I do intend to keep you appraised of progress; you can also follow my personal blog at http://steve-parker.org/urandom/ for more detailed updates. The RSS feed for that blog is http://steve-parker.org/urandom/rss.php.

There is a book in the pipeline, and I have lots to say about all these things, but am very interested to hear what you think is easy / hard / intuitive / arcane / stupid about these commands and the whole job control side of Unix/Linux and the different shells.

lsof is great, but almost only GNU/Linux; fuser is good, but restricted in how much it actually tells you – you have to go digging into PIDs to see what has to be KILLed or otherwise dealt with.

What, oh faithful few who may still be following this terribly intermittent blog, do you want to see on the subject of processes and job control in the *nix shell?

The Blog-Health-o-Meter™ reads Fresher than ever.

Crunchy numbers

About 3 million people visit the Taj Mahal every year. This blog was viewed about 26,000 times in 2010. If it were the Taj Mahal, it would take about 3 days for that many people to see it.

The busiest day of the year was November 9th with 142 views. The most popular post that day was Simple Maths in the Unix Shell.

Where did they come from?

The top referring sites in 2010 were steve-parker.org, google.com, google.co.in, ubuntuforums.org, and rackerhacker.com.

Some visitors came searching, mostly for bash maths, suid bit, shell script timestamp, awk one liners, and bash field separator.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Simple Maths in the Unix Shell January 2007
12 comments

Timestamps for Log Files March 2007
4 comments

suid shell scripts – setting “the SUID bit” April 2007
5 comments

IFS – Internal Field Separator September 2007
10 comments

Calculating Averages March 2007
7 comments

Unix keeps 3 timestamps for each file: mtime, ctime, and atime. Most people seem to understand atime (access time), it is when the file was last read. There does seem to be some confusion between mtime and ctime though. ctime is the inode change time while mtime is the file modification time. “Change” and “modification” are pretty much synonymous. There is no clue to be had by pondering those words. Instead you need to focus on what is being changed. mtime changes when you write to the file. It is the age of the data in the file. Whenever mtime changes, so does ctime. But ctime changes a few extra times. For example, it will change if you change the owner or the permissions on the file.

Let’s look at a concrete example. We run a package called Samba that lets PC’s access files. To change the Samba configuration, I just edit a file called smb.conf. (This changes mtime and ctime.) I don’t need to take any other action to tell Samba that I changed that file. Every now and then Samba looks at the mtime on the file. If the mtime has changed, Samba rereads the file. Later that night our backup system runs. It uses ctime, which also changed so it backs up the file. But let’s say that a couple of days later I notice that the permissions on smb.conf are 666. That’s not good..anyone can edit the file. So I do a “chmod 644 smb.conf”. This changes only ctime. Samba will not reread the file. But later that night, our backup program notices that ctime has changes, so it backs up the file. That way, if we lose the system and need to reload our backups, we get the new improved permission setting.

Here is a second example. Let’s say that you have a data file called employees.txt which is a list of employees. And you have a program to print it out. The program not only prints the data, but it obtains the mtime and prints that too. Now someone has requested an employee list from the end of the year 2000 and you found a backup tape that has that file. Many restore programs will restore the mtime as well. When you run that program it will print an mtime from the end of the year 2000. But the ctime is today. So again, our backup program will see the file as needing to be backed up.

Suppose your restore program did not restore the mtime. You don’t want your program to print today’s date. Well no problem. mtime is under your control. You can set it to what ever you want. So just do:
$ touch -t 200012311800 employees.txt
This will set mtime back to the date you want and it sets ctime to now. You have complete control over mtime, but the system stays in control of ctime. So mtime is a little bit like the date on a letter while ctime is like the postmark on the envelope.

This is a really clear, thorough explanation of ctime and mtime. Unfortunately, it is not possible to find the original creation time of a file, though that is somewhat meaningless as things are copied, moved, linked, changed; what is the creation time of a file which was created, removed, then created afresh, for example?

I believe you can write shell scripts that will run either in the Bourne shell or Bash. It may have some additional features that aren’t in the Bourne shell. I believe Bash was intended as a strictly compatible open source version of the Bourne shell. Honestly I haven’t looked at it in any detail so I could be wrong. I have used Bash myself because I run a Linux/Gnu system at home and it appears to do what I would expect.

I have nearly finished reading Coders At Work – Steve Bourne could have been an interesting interviewee for that book.

When I first posted this link at urandom, I was not aware that I myself was quoted, at the top of page 5 of the 7-page interview:

Unix Specialist Steve Parker has posted ‘Steve’s Bourne / Bash scripting tutorial’ in which he writes: “Shell script programming has a bit of a bad press amongst some Unix systems administrators. This is normally because of one of two things: a) The speed at which an interpreted program will run as compared to a C program, or even an interpreted Perl program; b) Since it is easy to write a simple batch-job type shell script, there are a lot of poor quality shell scripts around.” Do you agree?

It would be hard to disagree because he probably knows more about it than I do. The truth of the matter is you can write bad code in any language, or most languages anyway, and so the shell is no exception to that. Just as you can write obfuscated C you can write obfuscated shell. It may be that it is easier to write obfuscated shell than it is to write obfuscated C. I don’t know. But that’s the first point.

The second point is that the shell is a string processing language and the string processing is fairly simple. So there is no fundamental reason why it shouldn’t run fairly efficiently for those tasks. I am not familiar with the performance of Bash and how that is implemented. Perhaps some of the people that he is talking about are running Bash versus the shell but again I don’t have any performance comparisons for them. But that is where I would go and look. I know when I wrote the original implementation of the shell I spent a lot of time making sure that it was efficient. And in particular with respect to the string processing but also just the reading of the command file. In the original implementation that I wrote, the command file was pre-loaded and pre-digested so when you executed it you didn’t have to do any processing except the string substitutions and any of the other semantics that would change values. So that was about as efficient as you could get in an interpretive language without generating code.

I think that the points were presented to Steve Bourne in reverse order; his answer to the first point seems to relate to “b” (quality of scripts), and his longer answer to the second point seems to relate to “a” (performance).

Regarding performance, as he says, the real cost is of the Unix exec() call, which makes “cat /etc/hosts | grep localhost” half as fast as “grep localhost /etc/hosts”. There is nothing that the shell itself can do about that.

Regarding quality, deliberately obfusacated C is an institution; my point was merely that it is easy to write a bad shell script simply by not knowing how to write a better shell script. As this quote was from the introduction to a shell scripting tutorial, it should hopefully be clear from the context that the tutorial aims to enable the reader to write better shell scripts.

It creates a ZIP file with the XML Manifest file, and the startup/shutdown script, based on what you tell it.

There is much more that SMF can do – create entire new runlevels, and so on – but this does the basic single-instance startup and shutdown stuff that /etc/init.d scripts did.

Feel free to go and check it out at sgpit.com/smf/

Here are a few of my favourites (I have taken the liberty of slightly altering some of the code and/or descriptions):
From the original:
Search recursively for “expr” in all *.c and *.h files:
find -name '*.[ch]' | xargs grep -E 'expr'

Concatenate lines with training backslash:
sed ':a; /\\$/N; s/\\\n//; ta'

Delete line 42 from .known_hosts:
sed -i 42d ~/.ssh/known_hosts

From the new post:
Echo the path one item per line (assumes GNU tr):
echo $PATH | tr : '\n'

Top for Network:
iftop
Top for Input/Output (I/O):
iotop

Get SSL website Certificate:
openssl s_client -connect http://www.google.com:443 < /dev/null

List processes with Port 80 open:
lsof -i tcp:80

Edit a remote file directly in vim:
vim scp://user@remote//path/to/file

Add 20ms latency to loopback device (for testing):
tc qdisc add dev lo root handle 1:0 netem delay 20msec
Remove the latency:
tc qdisc del dev lo root

I would expect that most experienced admins can second-guess the content to 5-7 of these 10 points, just from the title (for example, item 1 is a reference to “mkdir -p”, plus another related syntax available to Bash users). I would be surprised if you knew all ten:

1. Make directory trees in a single swipe.
2. Change the path; do not move the archive.
3. Combine your commands with control operators.
4. Quote variables with caution.
5. Use escape sequences to manage long input.
6. Group your commands together in a list.
7. Use xargs outside of find .
8. Know when grep should do the counting — and when it should step aside.
9. Match certain fields in output, not just lines.
10. Stop piping cats.

How many did you get?

Now would be a very good time to get in touch (training@sgpit.com) as things are at a very early stage and very fluid – if you can bring a few people along, we can even run a bespoke course for you, and tailor everything to your need.

Depending on subject, duration, location and so on, it should be possible to run the first few courses for as little as £250 – £300 per person per day – much less than the £400 – £500 or so you’d pay for a corporate course where you all get is a trainer who has no experience of the actual situation you face at work, and who delivers powerpoint slides to you, then doles out the free mousepads and t-shirts at the end of the course.

None of us have been overly impressed by many of the available training courses – we are hoping to redefine how personal IT training can be delivered. Here’s how:

The kind of training session I would envisage us providing, would involve a fairly small class size (certainly fewer than 6 people), allowing us to focus on your current issues, and tailor the course around the needs, interests and skills of the attendees. The courses are likely to be between 2 and 5 days, most being 2-3 day courses.

Of course, there will be no corners cut – we will insist on great location and facilities, free internet access, PCs for all candidates (preinstalled with Linux, Solaris, *BSD, you name it – contact us before the course and we’ll build the PC to suit you), tons of good quality course notes, including certificates and the obligatory full VAT receipts, of course. I’m sure that we can find a few freebies to throw in, too!

If you have specific queries or concerns that you would like to be addressed in the course, let us know up-front, and we can find a way to work it in to the course.

If any of this sounds vaguely interesting, please do get in touch (training@sgpit.com) and we can mold things around your requirements.

vm.dirty_ratio – The highest % of your memory that can be used to hold dirty data. If you set this to a low value, the kernel will flush small writes to the disk more often. Higher values allow the small writes to stack up in memory. They’ll go to the disk in bigger chunks.

vm.dirty_background_ratio – The lowest % of your memory where pdflush is told to stop when it is writing dirty data. You’ll want to keep this set as low as possible.

linuxhaxor has a useful introduction to all five, with the most common uses for each of them.

Note that locate requires a regular run of updatedb – the article says that “The database is automatically created and updated daily” which is true for most distributions, but it depends on your cron setup – you can update the locate db as frequently as you wish. Another thing to note about locate is that it will not use the (normally root-generated) database to tell you (as a non-privileged user) about files which you would not otherwise know about.

diff -y -W `stty size | cut -d” ” -f2` –suppress-common-lines oldfile newfile

Note: This stty option is not available on Solaris, however, if you have it installed, the /usr/openwin/bin/resize command sets the COLUMNS variable.

update: This post originally said “width of your Linux terminal” but as noted in the comments, this feature of stty is also available in *BSD implementations, even though it is not part of the POSIX standard. So you should expect this to work on GNU and BSD systems (eg, most GNU/Linux distros, most *BSDs, including OSX) but not on all POSIX-compliant systems (eg, Solaris). I would assume that AIX, HPUX, SCO, the other “traditional” UNIX systems would also not support this, though I have not (yet) tested any of them. YMMV.