[Tfug] data snapshot/backup system

[John Gruenenfelder]

Hello all,

I offer this with the hope that some of you might find it useful and/or
interesting.

  http://bach.as.arizona.edu/bachwiki/index.php/Snapshot_facility

Some time ago I needed a short-term backup system for my work machine.  After
digging around on the Net for a while I found one such system that used rsync
and heavy use of hard links.  In effect, for each new snapshot of the target
data, a full copy of the directory tree is made on the snapshot drive, but the
new files are hard linked to the old files if they have not changed.  This
allows a large number of snapshots to be made in a very small amount of space.

I heavily modified the the scripts I found to make the system I am now using.
It creates a power series out of the many snapshots.  A new snapshot is
generated every two hours and the series is shifted down each time, eventually
terminating at 1024.  This gives a span of roughly 85 days from the oldest to
the most recent snapshot which is plenty for my needs.

The original script made use of a C++ helper program to perform the series
shifting.  I rewrote that as a shell script, partly so that there would be no
dependency on anything other than the shell and partly as an exercise for
myself.  (I was very happy when I got it to work)

If anybody else finds this useful or has any comments, I'd be pleased to hear
them.


I also have a question to ask... what filesystem would you recommend be used
on this snapshot drive?  On this machine, the snapshot drive is an external
160GB drive in a USB2 enclosure.  Firewire would have been preferable, but
this is what we had.  At first I was using XFS on this drive for several
reasons.  I figured having the log would be a good thing in case the drive
lost power and I have been using XFS on several machines for a number of years
without an issues.

However, when used on this external USB2 drive I have had frequent problems.
Rather than protecting the data, XFS seems prone to oblitering the filesystem
when something bad happens (like a machine crash, losing power, or losing the
cable connection).  The amount of damage to the filesystem varies, but it's
usually pretty bad.

Because of this, I have recently switched over to using ext3 on the drive.  So
far there have been no problems, but I suppose only time will tell.  Do any of
you have any suggestions for which filesystem should be used?  The main
requirements are that it supports normal UNIX properties (permissions,
ownership, special files, etc.) and, of course, hard links.


-- 
--John Gruenenfelder    Research Assistant, UMass Amherst student
                        Systems Manager, MKS Imaging Technology, LLC.
Try Weasel Reader for PalmOS  --  http://gutenpalm.sf.net
"This is the most fun I've had without being drenched in the blood
of my enemies!"
        --Sam of Sam & Max