Efficient Shell Scripting

January 22, 2014

In an article on ITworld.com, Sandra Henry-Stocker gives advice about writing efficient shell scripts.

Whilst a lot of the principles provided would appear to make sense, most of them actually do not make any significant difference, and some are entirely wrongly measured.

First, Henry-Stocker suggests replacing this script:

for day in Mon Tue Wed Thu Fri
    echo $day
    touch $day.log

… with this one:

for day in Mon Tue Wed Thu Fri
    if [ $verbose ]; then echo $day; fi
    touch $day.log

I ran both scripts 5,000 times, like this:

for x in `seq 1 5000`
  for day in Mon Tue Wed Thu Fri
      echo $day
      touch $day.log

… and similarly for the second script.

The “slow” script ran in 21.425 seconds on my PC, the “fast” script, which although it does not echo anything, instead parses and executes the test, which means that it took longer – 25.178 seconds, or 17% slower than simply running “echo” every time.

I would also note that the syntax if [ $verbose ] is asking for trouble, in real scripts I’m sure she would agree that you should use something like: “if [ "$verbose" -eq "y" ]“.

If the code is running on an old Sun framebuffer console, which will update the screen at around one second per line, all this needless echoing would make a difference, but in any real-world situation in 2014, the overhead of the test is far slower than writing the output.

Over on page two (because it’s all about selling advertising space :-) ), order of comparison is taken on. Whilst in principle, it could make a significant difference, the example given involves a single if statement, no fork()ing, and some simple variable comparisons:

echo -n "enter foo> "; read foo;
echo -n "enter bar> "; read bar;
echo -n "enter tmp> "; read tmp;

if [[ $foo -eq 1 && $bar -eq 2 && $tmp -eq 3 ]]; then
    echo ok

Taking out the read from the tests, we find that it takes 0.083 seconds to do 5,000 runs of the full test, with all variables matching (so all three conditions have to be tested each time), and 0.033 seconds when the first condition does not match, so it takes just over twice as long to run three tests as it does to run one test.

This is a significant difference, but it’s not the 1.195 seconds per iteration suggested by the article, it’s 0.00001 second per iteration. Taking Sandra Henry-Stocker’s results at face value, my tests which each took well under 1 second, would have taken 4 hours 5 minutes, or 2 hours 26 minutes respectively.

If one comparison was particularly time-consuming, it would be a more effective example. Here, if the find command takes 10 seconds to run, but foo is usually 1, then this will take 11 seconds:
if find /var -name foo.txt && [ "$foo" -eq "93" ]; then ...
whilst this will take 1 second, 1000% faster:
if [ "$foo" -eq "93" ] && find /var -name foo.txt; then ...
The example provided just doesn’t match the claimed results.

Avoiding unnecessary cat, echo and similar statements is good advice; not as significant as it was 10 years ago, and much less significant on Linux, where fork()ing is much faster than on Unix.

VMWare Balloon Size

October 29, 2013

This is just a quick note as it took me a while to find. I assumed that vmware_balloon.c would write stats into /proc, like most Linux kernel modules do.

That’s not how vmware roll, however. To find out how much RAM has been claimed by VMWare’s “Balloon” driver from within the guest OS itself, use the vmware-tools command:

# vmware-toolbox-cmd stat balloon
4460 MB

Shell Scripting Tutorial on Kindle

March 29, 2013

Unix & Linux Shell Scripting Tutorial on Kindle

Unix & Linux Shell Scripting Tutorial on Kindle

The Shell Scripting tutorial at http://steve-parker.org is now available natively on the Kindle!

USA (amazon.com)

UK (amazon.co.uk)

Similarly, you can search for “B00C2EGNSA” on any Amazon site, or just go to http://www.amazon.COUNTRY/dp/B00C2EGNSA (where “COUNTRY” is .fr, .de, etc) for your local equivalent.

Persian translation of Shell Scripting Tutorial

December 15, 2012

Mahmood Pahlevani has translated my Shell Scripting tutorial at http://steve-parker.org/sh/sh.shtml into Persian, at http://bashlinux.persiangig.com/stev/sh/index.shtml

Any feedback on this work is welcome here, I will pass on all praise to Mahmood,


Track memory usage on Solaris

November 15, 2012

When a Solaris server is overloaded, this is one way to check what actual memory each process is using. Here I am restricting the checks to one user (“steve”) but by omitting the “-u steve” flag to ps, the whole system will be checked.

$ps -ea -o pid,rss -u steve | sort -n -k 2 | while read pid rss
> do
> echo -en "RSS : ${rss} Kb: "
> pmap -x $pid | tail -1
> done
RSS: 50104 Kb: total Kb 1384400 712744 494312 -
RSS: 669296 Kb: total Kb 1414648 819496 657584 -

This uses the pmap tool to inspect the actual memory usage, which as the ps(1) man page says, is more accurate than just the RSS field of ps.

I suspect that a method using pmap alone might be possible, but this is just a quick note as I go…

Updated CheatSheet

September 5, 2012

I have updated the cheatsheet at http://steve-parker.org/sh/cheatsheet.pdf – it’s still a single-page PDF, or a PNG at http://steve-parker.org/sh/cheatsheet.png, but it squeezes a bit more content in than the previous one had, and is slightly more Linux/Bash biased.

What shell am I running?

March 18, 2012

Whether you’ve got an interactive shell session, or are writing a shell script, it is very difficult to be certain. It’s better to check for the capabilities that you require.

If you find yourself in a shell session, but don’t know what type of shell it is, there are a few ways to find out whether it’s Bourne shell, Bash, ksh, csh, zsh, tcsh, or whatever.

The simple answer is that you type in the command

echo $SHELL

which should tell you the path to your current shell; if it’s /bin/ksh or /usr/bin/ksh then it’s the KornShell; if it’s /bin/csh then it’s the C shell, and so on.

However, it’s not always that simple; Bash will set $SHELL only if it was not already set, so if you were running csh and use that to call bash, then Bash’s $SHELL will still say csh, not bash. However, Bash will set the $BASH_VERSION variable, but that’s not really a guarantee that you have that version; it only tells you that there exists a variable which specifies that version. You may not even be running bash at all.

Similarly, some shells (particularly /bin/sh) are symbolic links to others – whether to bash (GNU/Linux and newer Solaris), dash (some Linux distros more recently), or ksh (AIX).

So $SHELL is a useful start, but not at all definitive. You can search for your UserID in the /etc/passwd file:

$ grep steve /etc/passwd
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash

But there may be many entries containing the text “steve”, and you could be using NIS, LDAP, AD or some other authentication mechanism. $UID should be a read-only variable containing your UserID, and you can search your system-specific passwd via the getent command:

$ getent passwd $UID
steve:x:1000:1000:Steve Parker,,,:/home/steve:/bin/bash

However, as my UID is 1000, this would match any line containing “1000″ such as an account called “HAL1000″. So this should be one way to get your shell:

$ getent passwd $UID | cut -d: -f3,7|grep "^${UID}:"

Still, you could have run another shell after picking up your default shell. You can always check $0 – that should tell you how your current shell was called. You don’t know what the $PATH variable was when that was called – if $0 says “/usr/bin/zsh” then that’s what was called (of course, it is possible that/usr/bin/zsh could have changed since your shell was invoked!). If it just says “sh” then it’s whatever “sh” was found first in the $PATH of the calling shell. And of course, you can’t find out for sure what state that shell was in at that time.

On a Linux system, “cat /proc/$$/cmdline” should also give a good clue; “ls -l /proc/$$/exe” is better (but not definitive; it may be marked “(deleted)” if it’s been deleted, so you should check if it’s been subsequently replaced by some other shell.

So – it depends why you need to know. If you need to know for sure, on an unknown system, exactly what shell you are in, then I’m sorry, it’s not possible to be absolutely certain. To be reasonably confident, check $SHELL or $0. If you need to be more certain than that, then check for the features you require.

If you’re writing a shell script which requires arrays, then define and access an array, check that the results are as expected, and if not, bail out with a message along the lines of “an array-capable shell is expected – we suggest bash or ksh”.


Get every new post delivered to your Inbox.