There are lots of well-written articles, such as this [westnet.com] and especially this [kerneltrap] on Page Cacheing and pdflush, but RackerHacker (although the title says “reads”, it really seems to address lots of small writes) summarises it very well:

vm.dirty_ratio – The highest % of your memory that can be used to hold dirty data. If you set this to a low value, the kernel will flush small writes to the disk more often. Higher values allow the small writes to stack up in memory. They’ll go to the disk in bigger chunks.

vm.dirty_background_ratio – The lowest % of your memory where pdflush is told to stop when it is writing dirty data. You’ll want to keep this set as low as possible.

I suspect that most Linux admins know 3 or 4 of these five commands, and regularly use 2 or 3 of them.

linuxhaxor has a useful introduction to all five, with the most common uses for each of them.

Note that locate requires a regular run of updatedb – the article says that “The database is automatically created and updated daily” which is true for most distributions, but it depends on your cron setup – you can update the locate db as frequently as you wish. Another thing to note about locate is that it will not use the (normally root-generated) database to tell you (as a non-privileged user) about files which you would not otherwise know about.

A quick and easy way to get the width of your terminal is the command stty size. I have used it with diff like this:

diff -y -W `stty size | cut -d” ” -f2` –suppress-common-lines oldfile newfile

Note: This stty option is not available on Solaris, however, if you have it installed, the /usr/openwin/bin/resize command sets the COLUMNS variable.

update: This post originally said “width of your Linux terminal” but as noted in the comments, this feature of stty is also available in *BSD implementations, even though it is not part of the POSIX standard. So you should expect this to work on GNU and BSD systems (eg, most GNU/Linux distros, most *BSDs, including OSX) but not on all POSIX-compliant systems (eg, Solaris). I would assume that AIX, HPUX, SCO, the other “traditional” UNIX systems would also not support this, though I have not (yet) tested any of them. YMMV.

The TTY Demystified tells you everything you need to know (and quite possibly more) about the *nix tty history. (If you do need to know more than this, you are probably already writing your own *nix kernel, or at least libc)

Hopefully that should give some insight into the terminfo, and other obscure details of the *nix kernel and its obsession with piping characters from one place to another.

I just found this page on the OReilly website – a Linux Command Directory

Click on any of the 687 commands below to get a description and list of available options. All links in the command summaries point to the online version of the book on Safari Bookshelf.

It doesn’t cover everything (what could?) but it could be a useful page to bookmark.

I have previously plugged the great list of sed 1-liners at http://sed.sourceforge.net/sed1line.txt.

Here is a similar (if shorter) list of handy awk 1-liners:

http://www.sap-basis-abap.com/unix/awk-one-liner-tips.htm:

Print column1, column5 and column7 of a data file or output of any columns list

awk '{print $1, $5, $7}' data_file

ls –al |awk '{print $1, $5, $7}' -- Prints file_permissions,size and date

List all files names whose file size greater than zero.

ls –al |awk '$5 > 0 {print $9}'

List all files whose file size equal to 512bytes.

ls –al |awk '$5 == 0 {print $9}'

print all lines

awk '{print }' file_name

awk '{print 0}' file_name

Number of lines in a file

awk ' END {print NR}' file_name

Number of columns in each row of a file

awk '{print NF}' file_name

Sort the output of file and eliminate duplicate rows

awk '{print $1, $5, $7}' |sort –u

List all file names whose file size is greater than 512bytes and owner is “oracle”

ls –al |awk '$3 == "oracle" && $5 > 512 {print $9}'

List all file names whose owner could be either “oracle” or “root”

ls –al |awk '$3 == "oracle" || $3 == "root" {print $9}'

list all the files whose owner is not “oracle

ls –al |awk '$3 != "oracle" {print $9}'

List all lines which has at least one or more characters

awk 'NF > 0 {print }' file_name

List all lines longer that 50 characters

awk 'length($0) > 50 {print }' file_name

List first two columns

awk '{print $1, $2}' file_name

Swap first two columns of a file and print

awk '{temp = $1; $1 = $2; $2 = temp; print }' file_name

Replace first column as “ORACLE” in a data file

awk '{$1 = "ORACLE"; print }' data_file

Remove first column values in a data file

awk '{$1 =""; print }' data_file

Calculate total size of a directory in Mb

ls –al |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'

Calculate total size of a directory including sub directories in Mb

ls –lR |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'

Find largest file in a directory including sub directories

ls –lR |awk '{print $5 "\t" $9}' |sort –n |tail -1

http://www.seanius.net/blog/2009/03/saving-and-restoring-positional-params/ has a really good post about how “shift” loses the original command-line arguments, discusses various seemingly-obvious ways of restoring them, explores why they fail for certain types of input, and provides what seems to be a working solution:

one way you could try to store/save them would be to save the original copy of $@ and use eval set — later on to change it back to the saved variable

… but as it turns out, that doesn’t work with spaces, or quotes. Excellent, really concise description of the problem(s) and solution(s).

Shell Script: GetCluster – a shell script I happen to have written today; uses a few features of GNU Grep, whilst being tolerant of non-GNU greps (at the cost of some functionality).

This simple shell script is an example of the kind of thing that can easily be done with a few simple commands. As it is intended for the Solaris platform, which does not necessarily include GNU’s grep, some additional complexity is required to see what the platform’s grep utility is capable of.

This is achieved by:
1) Set the PATH to pick up the GNU tools first: PATH=/usr/sfw/bin:$PATH
2) Find grep (ggrep if possible; the existence of ggrep suggests that grep itself is not GNU’s grep):
type ggrep >/dev/null 2>&1
if [ "$?" -eq "0" ]; then
GREP=`which ggrep`
else
GREP=`which grep`
fi


It then finds a file, which could be in one of many places:
CLUSTERTOC=`ls clustertoc .clustertoc /cdrom/cdrom0/Solaris*/Product/.clustertoc /cdrom/Solaris*/Product/.clustertoc /mnt/Solaris*/Product/.clustertoc 2>/dev/null|head -1`
What this does, is identify the first succesful match (the rest go to /dev/null). If none is found, the CLUSTERTOC will not be a valid file (note: this gets us the added bonus that it will complain if clustertoc is a device driver, pipe, etc).

It then looks for “SUNW_CSRMEMBER” in the clustertoc, using “grep -n” to get the line number, then passing that linenumber on to head so as to limit the output to only contain those lines. Any lines below that point would refer to other clusters, and be irrelevant. We want to get the last of the relevant part of the file, so that the tail command below will give us what we need.

GNU Grep has a “-A n” and “-B n” facility to say “include n lines After the matching line” or “include n lines Before the matching line”. We use “-A” here to get the following lines which describe the pacakge in more detail:
head -${linenum} $CLUSTERTOC |$GREP -A4 "^CLUSTER="|cut -d"=" -f2-|tail -4

See the readme file for specific output – the Solaris (non-GNU) version has notably less detail than the GNU version.

Hopefully the script (http://steve-parker.org/code/sh/getcluster/getcluster.sh) is in other ways sufficiently self-explanatory.

The input file (http://steve-parker.org/code/sh/getcluster/clustertoc is relatively simple, and should explain any other queries.

Still – this simple script may provoke a few questions; please do play with it and ask away, if anything is unclear.

I have missed the second birthday of this blog; starting with Hello World on 17th January 2007, this blog has seen a small growth, but the comments posted show that people have found it to be useful, which is good.

Traffic

Statistics

I hope that you continue to find it useful; I really will try to post a bit more often in 2009!

Steve

echo unccl puevfgznf | tr A-Za-z N-ZA-Mn-za-m

This is a very simplistic thing called “rot13″, whereby “a” becomes “n”, “b” becomes “o”, “c” becomes “p”, and so on. The tr utility is ideal for simple string manipulation like this.

Happy Christmas to all!

You can forward ports with ssh like this:

$ ssh -L 8080:localhost:80 user@remotehost

This will log you in to remotehost as user, and port 8080 on your local machine will be tunnelled to port 80 on remotehost. If remotehost can see a machine that you can’t (for example, if it’s on an internal network), you can even do this:

$ ssh -L 8080:internalhost:80 user@borderhost

This will log you in to borderhost, and localhost:8080 will be directed to internalhost:80, even though you may not be able to see internalhost directly yourself.

What I didn’t know until I read Nico Golde’s blog today, is that you can do this interactively, with an existing session. Tilde (~) is the default escape character, and ~C (note that’s an uppercase C) gets you a shell session within ssh itself:

$ ssh user@remotehost
user@remotehost$ ~C
ssh> -L 8080:localhost:80
Forwarding port.
user@remotehost$

Mike Hommey has a nice article on clearing down deleted (but still-open) files:
http://glandium.org/blog/?p=211. He explains the problem clearly, and goes on to provide the solution, too. Well worth bookmarking, for that 2am emergency!

someone had to free some space on a 1GB filesystem, and thought a good idea would be to delete that 860MB log file that nobody cares about. Except that it didn’t really remove it, but he didn’t really check.

Later, the “filesystem full” problem came back at someone else, who came to ask me what files from a small list he could remove. But the files were pretty small, and that wouldn’t have freed enough space. That gave me the feeling that we probably were in this typical case I introduced this post with, which du -sk confirmed: 970MB used on the filesystem according to df, but only 110MB worth of data…

Mike has solutions to this for Solaris and Linux; lsof is also useful for this kind of thing, on systems which have it.

When using the more tool (this works with less, also, and less is generally more useful; you can scroll backwards and forwards with less), you can press “v” to start editing the file, in vi, at the line you are currently viewing.

This works on Linux and Solaris; please confirm on other OSes as appropriate, though I would assume that what works on more on Solaris, will work on more or less on just about any OS.

This makes using more or less far more powerful than before, and a new way into vi.

Ben Hutchings has noted a quirk in the chmod syntax: If you use the “chmod o-x” syntax, but omit the “o” (Others) (or the “u” (User) or “g” (Group)), it will default to “a” (All).

So “chmod -x foo” becomes “chmod a-x foo“; similarly, “chmod +x foo” becomes “chmod a+x foo“

Interesting list of 15 examples of bash history syntax.

Most people probably know #5:
# !ps
ps aux | grep yp

I didn’t know #13, that looks really useful for arcane purposes:
# cp ~/longname.txt /really/a/very/long/path/long-filename.txt
# ls -l !cp:2
ls -l /really/a/very/long/path/long-filename.txt

Entitled “Lazy Linux: 10 essential tricks for admins“, this is a pretty good set of essential commands for *nix administrators.

If I have to suggest anything for in-depth study, it’s “Trick 6″: “Remote VNC session through an SSH tunnel”, because you can actually do just about anything through an SSH tunnel. If you’re faced with you -> firewall -> internet -> remotemachine -> internet -> restoftheworld, you can do whatever you like with “restof theworld” if you can ssh to “remotemachine”, whatever the local firewall thinks. And you can probably ssh over port 443 through the firewall. Hmm; maybe I should write this up properly at some point. I’m sure it’s well documented on the web if you look for it….

Network Theory have a Bash Quiz!

Out of the ten questions, one I’m not immediately sure of without checking; another I only happened to come across earlier today, and one (echo "\'")I got wrong

So, 7/10 really; 8/10 by good fortune of coming across the definition earlier today

A serious publisher has contacted me about writing a serious book about Linux shell programming.

It is all really very serious. I’m not used to being serious, as you can probably tell from the fact that I have now used the word “serious” four times in this three-sentence post.

I am rather keen to write a book on the subject, not because I’m vain, or desperate for money, but because the stuff I have seen out there in dead-tree format has been of rather low quality. Also because of all the emails I’ve received over the years, they have all been positive, and none has said anything along the lines of “I didn’t need any of that because I bought Book[X]“, or indeed any book. People have emailed me, asking for advice as to what book to buy, and I have been unable to recommend any book that I have seen.

So:

What would you like to see in your ideal book about UNIX / Linux shell scripting, be it Bourne, Bash, ksh, tcsh, zsh, whatever?

Please don’t be timid; if you want to know how to work out how many nose-flutes can be fitted into the area of a Boeing 757, you won’t be anything like as strange as some of the correspondants I’ve had over the years, so please, tell me what is bugging you, what has bugged you, or even what you think might be likely to bug you in days / months / years to come.

I’m likely to answer any specific questions here and now, whether or not they end up in the book, but anything you’d like to see in a book, too… post that here, and I’ll have a stab at it.

Also, I would of course be interested to know if you have found any useful books on or around the subject, and what they did particularly well.

Steve

This blog has now been running for a year; the first post was Hello World on 17th Jan 2007.

I hadn’t realised it had been going for so long; in that time, I’ve made 41 posts, so I haven’t quite managed to make one post per week I have been a bit slack lately, for which I do apologise. New Years Resolution: I must make more posts here!

In the meantime, my main site, steve-parker.org, has celebrated its seventh birthday, having been born in June 2000 – looking forward to making the 8th birthday celebrations this June!

There are lots of small little quirks to the *nix shells; this is just one of them.

If you want to list the files in a directory, then ls will list them all for you, in alphabetical order.

If you want to list them by size, you can use ls -S; by timestamp: ls -t, and so on.

But ls is a particular utility. What happens when we do this:


for myfile in *
do
  echo "My file is called $myfile"
done

We get an alphabetically sorted list (see man ascii for the actual detail; they’re sorted by ASCII value, so numbers first, then uppercase letters, then lowercase letters).

This can be a pain, but it can also be quite useful. If you’ve got a bunch of files:

1.install.txt
2.setup.txt
3.use.txt
4.uninstall.txt

Then you can play with them in order, just by using the asterisk:

for i in *
do
  echo "File $i" >> all.txt
  cat $i >> all.txt
done

And it will sort them into order for you (“1″ comes before “2″ in ASCII, and so on…)

Or you could just do this:

more * > all.txt

Because more will prefix each file with its name in a header, if there is more than one file to process.

http://snap.nlc.dcccd.edu/reference/bash1/features_29.html has a bash-specific technique for processing calculations:

$[ expression ]
$(( expression ))

See the link for further details.

It seems like an esoteric concept, but it’s actually very useful.

If your input file is “1 apple steve@example.com”, then your script could say:

while read qty product customer
do
  echo "${customer} wants ${qty} ${product}(s)"
done

The read command will read in the three variables, because they’re spaced out from each other.

However, critical data is often presented in spreadsheet format. If you save these as CSV files, it will come out like this:

1,apple,steve@example.com

This contains no spaces, and the above code will not be able to understand it. It will take the whole thing as one item – the first thing, quanity, $qty, and set the other two fields as blank.

The way around this, is to tell the entire shell, that “,” (the comma itself) separates fields; it’s the “internal field separator”, or IFS.

The IFS variable is set to space/tab/newline, which isn’t easy to set in the shell, so it’s best to save the original IFS to another variable, so you can put it back again after you’ve messed around with it. I tend to use “oIFS=$IFS” to save the current value into “oIFS”.

Also, when the IFS variable is set to something other than the default, it can really mess with other code.

Here’s a script I wrote today to parse a CSV file:

#!/bin/sh
oIFS=$IFS     # Always keep the original IFS!
IFS=","          # Now set it to what we want the "read" loop to use
while read qty product customer
do
  IFS=$oIFS
  # process the information
  IFS=","       # Put it back to the comma, for the loop to go around again
done < myfile.txt

It really is that easy, and it’s very versatile. You do have to be careful to keep a copy of the original (I always use the name oIFS, but whatever suits you), and to put it back as soon as possible, because so many things invisibly use the IFS – grep, cut, you name it. It’s surprising how many things within the “while read” loop actually did depend on the IFS being the default value.

Whilst not directly related to shell programming, understanding of basic logic operations – AND, OR, NOR, XOR, NAND, etc, are as important to shell programmers as to C, Java, .Net and other coders.

My recent interactive logic gate page seems to have become quite popular; it’s just a simple implementation of each of the major logic circuits in use. If you want to see more, say so – I’ll add anything you ask for

The (often a bit geeky for this blog) FreeBSD-World website has a good “Top-25″ list of 25 useful commands in Linux/UNIX for Beginners (note: new URL updated 31 Aug 2008)

I’m not sure that #24 (dig) and #25 (host) are absolutely necessary, #18 (startx) is possibly outdated these days, and the compression tools (6-9) are much of a muchness, but apart from that, #1 – #23 should be familiar to anyone who claims to be experienced with UNIX/Linux. If somebody was missing one, it would have to be #18 (startx), as (a) it’s not needed on servers, and (b) modern *nix distros will boot into a GUI automatically when possible.

So what’s the list?
25. host
24. dig
23. mkdir
22. rm
21. cp
20. grep
19. ls
18. startx
17. nano / vi
16. pwd
15. cat
14. man
13. kill
12. locate
11. ifconfig
10. ssh
9. gzip
8. bzip2
7. zip
6. tar (I would put 6-9 in one category, personally. rar should probably be in there too)
5. mount
4. passwd
3. ping
2. tail
1. top

I use vi every day; to me, there is no better text editor. It is apparently a little intimidating for the newcomer, though…. here’s a beginner’s guide to the VI editor.

I would write one of these, but – much as I love vi, and I really do, everyone who uses vi, seems to have their own experience of it, and their own shortcuts. We’ve all got our own quirks. One thing’s for sure; vi is not notepad!