lsof, fuser, nohup, disown, bg, fg, and jobs

February 4, 2011

Bit of a cheeky one here – what does anybody want to know about these topics?

There is a book in the pipeline, and I have lots to say about all these things, but am very interested to hear what you think is easy / hard / intuitive / arcane / stupid about these commands and the whole job control side of Unix/Linux and the different shells.

lsof is great, but almost only GNU/Linux; fuser is good, but restricted in how much it actually tells you – you have to go digging into PIDs to see what has to be KILLed or otherwise dealt with.

What, oh faithful few who may still be following this terribly intermittent blog, do you want to see on the subject of processes and job control in the *nix shell?


2010 in review

January 2, 2011

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads Fresher than ever.

Crunchy numbers

Featured image

A helper monkey made this abstract painting, inspired by your stats.

About 3 million people visit the Taj Mahal every year. This blog was viewed about 26,000 times in 2010. If it were the Taj Mahal, it would take about 3 days for that many people to see it.

The busiest day of the year was November 9th with 142 views. The most popular post that day was Simple Maths in the Unix Shell.

Where did they come from?

The top referring sites in 2010 were steve-parker.org, google.com, google.co.in, ubuntuforums.org, and rackerhacker.com.

Some visitors came searching, mostly for bash maths, suid bit, shell script timestamp, awk one liners, and bash field separator.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

Simple Maths in the Unix Shell January 2007
12 comments

2

Timestamps for Log Files March 2007
4 comments

3

suid shell scripts – setting “the SUID bit” April 2007
5 comments

4

IFS – Internal Field Separator September 2007
10 comments

5

Calculating Averages March 2007
7 comments


inodes – ctime, mtime, atime

October 7, 2010

http://www.unix.com/tips-tutorials/20526-mtime-ctime-atime.html has a really good explanation of the different timestamps in a Unix/Linux inode. GNU/Linux has a useful utility called “stat” which displays most of the inode contents:
$ stat .bashrc
File: `.bashrc'
Size: 3219 Blocks: 8 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 33 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ steve) Gid: ( 1000/ steve)
Access: 2010-10-07 01:11:21.000000000 +0100
Modify: 2010-08-19 21:22:20.000000000 +0100
Change: 2010-08-19 21:22:21.000000000 +0100
$

As Perderabo explains in the above-linked post:

Unix keeps 3 timestamps for each file: mtime, ctime, and atime. Most people seem to understand atime (access time), it is when the file was last read. There does seem to be some confusion between mtime and ctime though. ctime is the inode change time while mtime is the file modification time. “Change” and “modification” are pretty much synonymous. There is no clue to be had by pondering those words. Instead you need to focus on what is being changed. mtime changes when you write to the file. It is the age of the data in the file. Whenever mtime changes, so does ctime. But ctime changes a few extra times. For example, it will change if you change the owner or the permissions on the file.

Let’s look at a concrete example. We run a package called Samba that lets PC’s access files. To change the Samba configuration, I just edit a file called smb.conf. (This changes mtime and ctime.) I don’t need to take any other action to tell Samba that I changed that file. Every now and then Samba looks at the mtime on the file. If the mtime has changed, Samba rereads the file. Later that night our backup system runs. It uses ctime, which also changed so it backs up the file. But let’s say that a couple of days later I notice that the permissions on smb.conf are 666. That’s not good..anyone can edit the file. So I do a “chmod 644 smb.conf”. This changes only ctime. Samba will not reread the file. But later that night, our backup program notices that ctime has changes, so it backs up the file. That way, if we lose the system and need to reload our backups, we get the new improved permission setting.

Here is a second example. Let’s say that you have a data file called employees.txt which is a list of employees. And you have a program to print it out. The program not only prints the data, but it obtains the mtime and prints that too. Now someone has requested an employee list from the end of the year 2000 and you found a backup tape that has that file. Many restore programs will restore the mtime as well. When you run that program it will print an mtime from the end of the year 2000. But the ctime is today. So again, our backup program will see the file as needing to be backed up.

Suppose your restore program did not restore the mtime. You don’t want your program to print today’s date. Well no problem. mtime is under your control. You can set it to what ever you want. So just do:
$ touch -t 200012311800 employees.txt
This will set mtime back to the date you want and it sets ctime to now. You have complete control over mtime, but the system stays in control of ctime. So mtime is a little bit like the date on a letter while ctime is like the postmark on the envelope.

This is a really clear, thorough explanation of ctime and mtime. Unfortunately, it is not possible to find the original creation time of a file, though that is somewhat meaningless as things are copied, moved, linked, changed; what is the creation time of a file which was created, removed, then created afresh, for example?


Interview with Steve Bourne

August 26, 2010

ARNnet have an Interview with Steve Bourne

I believe you can write shell scripts that will run either in the Bourne shell or Bash. It may have some additional features that aren’t in the Bourne shell. I believe Bash was intended as a strictly compatible open source version of the Bourne shell. Honestly I haven’t looked at it in any detail so I could be wrong. I have used Bash myself because I run a Linux/Gnu system at home and it appears to do what I would expect.

I have nearly finished reading Coders At Work – Steve Bourne could have been an interesting interviewee for that book.

When I first posted this link at urandom, I was not aware that I myself was quoted, at the top of page 5 of the 7-page interview:

Unix Specialist Steve Parker has posted ‘Steve’s Bourne / Bash scripting tutorial’ in which he writes: “Shell script programming has a bit of a bad press amongst some Unix systems administrators. This is normally because of one of two things: a) The speed at which an interpreted program will run as compared to a C program, or even an interpreted Perl program; b) Since it is easy to write a simple batch-job type shell script, there are a lot of poor quality shell scripts around.” Do you agree?

It would be hard to disagree because he probably knows more about it than I do. The truth of the matter is you can write bad code in any language, or most languages anyway, and so the shell is no exception to that. Just as you can write obfuscated C you can write obfuscated shell. It may be that it is easier to write obfuscated shell than it is to write obfuscated C. I don’t know. But that’s the first point.

The second point is that the shell is a string processing language and the string processing is fairly simple. So there is no fundamental reason why it shouldn’t run fairly efficiently for those tasks. I am not familiar with the performance of Bash and how that is implemented. Perhaps some of the people that he is talking about are running Bash versus the shell but again I don’t have any performance comparisons for them. But that is where I would go and look. I know when I wrote the original implementation of the shell I spent a lot of time making sure that it was efficient. And in particular with respect to the string processing but also just the reading of the command file. In the original implementation that I wrote, the command file was pre-loaded and pre-digested so when you executed it you didn’t have to do any processing except the string substitutions and any of the other semantics that would change values. So that was about as efficient as you could get in an interpretive language without generating code.

I think that the points were presented to Steve Bourne in reverse order; his answer to the first point seems to relate to “b” (quality of scripts), and his longer answer to the second point seems to relate to “a” (performance).

Regarding performance, as he says, the real cost is of the Unix exec() call, which makes “cat /etc/hosts | grep localhost” half as fast as “grep localhost /etc/hosts”. There is nothing that the shell itself can do about that.

Regarding quality, deliberately obfusacated C is an institution; my point was merely that it is easy to write a bad shell script simply by not knowing how to write a better shell script. As this quote was from the introduction to a shell scripting tutorial, it should hopefully be clear from the context that the tutorial aims to enable the reader to write better shell scripts.


Solaris 10 SMF Manifests

July 17, 2010

I have recently written a web service which creates Solaris 10 SMF manifests based on the information you give it.

It creates a ZIP file with the XML Manifest file, and the startup/shutdown script, based on what you tell it.

There is much more that SMF can do – create entire new runlevels, and so on – but this does the basic single-instance startup and shutdown stuff that /etc/init.d scripts did.

Feel free to go and check it out at sgpit.com/smf/


Useful GNU/Linux Commands

June 23, 2010

Pádraig Brady has some useful, if somewhat basic hints, at http://www.pixelbeat.org/cmdline.html. He has updated them to include more powerful commands at http://www.pixelbeat.org/docs/linux_commands.html.

Here are a few of my favourites (I have taken the liberty of slightly altering some of the code and/or descriptions):
From the original:
Search recursively for “expr” in all *.c and *.h files:
find -name '*.[ch]' | xargs grep -E 'expr'

Concatenate lines with training backslash:
sed ':a; /\\$/N; s/\\\n//; ta'

Delete line 42 from .known_hosts:
sed -i 42d ~/.ssh/known_hosts

From the new post:
Echo the path one item per line (assumes GNU tr):
echo $PATH | tr : '\n'

Top for Network:
iftop
Top for Input/Output (I/O):
iotop

Get SSL website Certificate:
openssl s_client -connect http://www.google.com:443 < /dev/null

List processes with Port 80 open:
lsof -i tcp:80

Edit a remote file directly in vim:
vim scp://user@remote//path/to/file

Add 20ms latency to loopback device (for testing):
tc qdisc add dev lo root handle 1:0 netem delay 20msec
Remove the latency:
tc qdisc del dev lo root


Ten Good Unix Habits

June 22, 2010

IBM’s DeveloperWorks has 10 Good Unix Habits, which apply to GNU/Linux at least as much as to Unix.

I would expect that most experienced admins can second-guess the content to 5-7 of these 10 points, just from the title (for example, item 1 is a reference to “mkdir -p”, plus another related syntax available to Bash users). I would be surprised if you knew all ten:

1. Make directory trees in a single swipe.
2. Change the path; do not move the archive.
3. Combine your commands with control operators.
4. Quote variables with caution.
5. Use escape sequences to manage long input.
6. Group your commands together in a list.
7. Use xargs outside of find .
8. Know when grep should do the counting — and when it should step aside.
9. Match certain fields in output, not just lines.
10. Stop piping cats.

How many did you get?


Follow

Get every new post delivered to your Inbox.