Copy a filesystem

January 23, 2007

When you need to copy or move an entire filesystem, the first, most obvious, answers do not quite do the job. cp needs lots of flags to get the permissions, ownership, access times and so on to match. tar takes care of most of that, but it still doesn’t deal with special files (the most common being character or block devices, such as those you’d find in /dev). So there’s a little magic incantation I’ve memorised. It does the job, not 99% of the job, it just does the job.

If you have two volumes, let’s call them /data and /backup, and you want to copy everything from /data into /backup, then you’d do this:

# cd /data
# find . -print | cpio -pudvm /backup

And off it goes. Your inode attributes, special files, everything, just as it was in the /data filesystem. If there are other filesystems mounted under /data – eg., /data/customer1 and /data/customer2, then they’ll be backed up by this method. If that’s not what you want, then add -mount to the find command.

I used it recently when a hard disk was dying, and it contained the /home filesystem; just get a new disk, and off you go:

# mkfs.ext3 /dev/hdc1
# mount /dev/hdc1 /newhomes
# cd /home
# find . -print | cpio -pudvm /newhomes

And then just unmount the old one, edit the /etc/fstab to point to the new one, and remount it. A little bit of sed is easier to show than the editing… I’d use vi in real life:

# mount | grep home
/dev/hda1 on /home type ext3 (rw)
# umount /home
# grep home /etc/fstab
/dev/hda1    /home     ext3  defaults    0   2
# sed -i hda1 hdc1 /etc/fstab
# grep home /etc/fstab
/dev/hdc1    /home     ext3  defaults    0   2
# mount /home
# mount | grep home
/dev/hdc1 on /home type ext3 (rw)

The cpio switches are (with descriptions from a Solaris 9 man page):
-p (pass) Reads a list of file path names from the standard input and conditionally copies those files into the destination directory tree.
-d Creates directories as needed.
-m Retains previous file modification time. This option is ineffective on directories that are being copied.
-u Copies unconditionally. Normally, an older file will not replace a newer file with the same name.
-v Verbose. Prints a list of file and extended attribute names. When used with the -t option, the table of contents looks like the output of an ls -l command (see ls(1) ).

I don’t know why, but I find the order pudvm is stuck in my head, it just seems like a nice little word in its own right. And it’s been really helpful on a number of occasions.

Tool Tip: “find”

January 19, 2007

find is a very powerful command. After the last post about grep, in which I mentioned that DOS has a command called “find” which is a simplistic version of grep, I now feel obliged to tell all about the real (that is, the *nix) find command.

Find works on the basis of find (from somewhere) (something); the “from somewhere” is often “.” (here, the current directory), or “/” (root, to search the entire filesystem). it’s not terribly interesting. What is interesting, is the “something” bit. You can specify a file name, for example:

$ find / -name "foo.txt"

That wasn’t very exciting, and it will take a long time to complete, too. Systems with “slocate” installed could just say locate foo.txt and get the answer back in a fraction of a second (by looking it up in a database) ,without trawling through the whole hard disk (or, indeed, all attached disks). So that’s not what’s exciting about find. What is exciting about find, is what else it can do, instead of just “-name foo.txt”.

Don’t get me wrong; the “-name” switch is useful. More useful with wildcards: find . -name "*.txt" will find all text files.

You can restrict the search to one filesystem with the “-mount” (aka “-xdev”) flag.

If you want to find files newer than /var/log/messages, you can use find . -newer /var/log/messages

If you want to find files over 10Kb, then find . -size +10k will do the job. To get a full listing, find . -size +10k -ls.

Want to know what files I own? find . -uname steve

How about listing all files over 10Kb with their owner and permissions?

$ find . -size +10k -printf "%M %u %f\n"
-rwxr-xr-x steve foo.txt
-rw------- steve bar.doc
-rwxr-xr-x steve fubar.iso
-rwxr-xr-x steve fee.txt
-rw------- steve jg.tar

Here, the “%M” shows the permissions (-rwxr-xr-x), “%u” shows the username (“steve”), and “%f” shows the filename. The “\n” puts a “newline” character after each matching file, so that we get one file per line.

There is much more to find than this; I’ve not really covered the actions (other than printf) at all in this article, just a quick glimpse of how find can search for files based on just about any criteria you can think of. Search terms can be combined, so find . -size +10k -name "*.txt" will only find text files over 10Kb, and so on.

Tool Tip: “grep”

January 17, 2007

A powerful and useful tool in the shell scripter’s arsenal is grep. If you’ve not come across it before, it’s similar to the “find” tool that DOS had; it finds strings in files. Grep stands for “get regular expression”; a “regular expression” is a string, or something more than just a string.

$ grep foo myfile.txt
and Steve said, "foo! that's crazy"

That searches for “foo” in the file called “myfile.txt”. It gets any line (yes, the whole line) which contains the search text.

But you can do other stuff, with “switches”. For example “-i” means “insensitive to case”:
$ grep -i foo myfile.txt
"Foo" is a word, associated with "Bar".
and Steve said, "foo! that's crazy"

This time, grep finds that the word “foo” is actually mentioned twice in “myfile.txt”; once as “Foo” and once as “foo”.

The “-i” flag is a pretty common one, then, because it’s often what we really want it to find.

Here’s a good one, though: Under Linux, a special file /proc/bus/usb/devices lists your USB devices. That’s good, but yuck, it’s a mess of (too much) detailed information:

T:  Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc=  0/900 us ( 0%), #Int=  0, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.15-27-server uhci_hcd
S:  Product=UHCI Host Controller
S:  SerialNumber=0000:00:07.2
C:* #Ifs= 1 Cfg#= 1 Atr=c0 MxPwr=  0mA
I:  If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

T:  Bus=01 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#=  2 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=ff(vend.) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=06b9 ProdID=4061 Rev= 0.00
S:  Manufacturer=ALCATEL
S:  Product=Speed Touch USB
S:  SerialNumber=0090D00D0B25
C:* #Ifs= 3 Cfg#= 1 Atr=80 MxPwr=500mA
I:  If#= 0 Alt= 0 #EPs= 1 Cls=ff(vend.) Sub=00 Prot=00 Driver=usbfs

How do I just get what I need from the file? One switch to grep, which I don’t use as much as I should, is “-A”, for “After”. (Note that it’s a capital “A”).

After the Vendor ID and Product ID, /proc/bus/usb/devices includes the name of the device, so I can find out what I’ve got installed with a Vendor ID of 06b9 quite easily:

$ grep -A 2 06b9 /proc/bus/usb/devices
P: Vendor=06b9 ProdID=4061 Rev= 0.00
S: Manufacturer=ALCATEL
S: Product=Speed Touch USB

Or what have I got from Alcatel?
$ grep -i -A1 Alcatel /proc/bus/usb/devices
S: Manufacturer=ALCATEL
S: Product=Speed Touch USB

I can also ask: Who made my Speed Touch modem, or what’s its ID? “-B” displays lines before the line that matches:

$ grep -B 2 Speed /proc/bus/usb/devices
P: Vendor=06b9 ProdID=4061 Rev= 0.00
S: Manufacturer=ALCATEL
S: Product=Speed Touch USB

There’s a lot you can do with grep; I’ve only really covered the first line from “man grep”

Hello world!

January 17, 2007

Okay, here we go.

As a side-project to my *nix shell programming tutorial, I thought it might be an idea to start a blog, with regular postings of general hints and tips.

Not the overview stuff that the tutorial provides, but more “cheat codes” type stuff; crib-sheets to get you through the day.

Here’s one for starters: Sed 1-Liners There’s an awful lot you can do with sed if you have the secret sauce. Without this file, I’d be stuck with sed s/foo/bar/g, and that’d be it. You can do a lot with sed. The file linked above contains many very useful tricks.


Get every new post delivered to your Inbox.