[Tfug] HDD size and RAID queries

[John Gruenenfelder]

On Wed, May 08, 2013 at 01:50:56PM -0700, Nick Lopez wrote:
>
>On May 8, 2013, at 4:49 AM, John Gruenenfelder <jetpackjohn at gmail.com> wrote:
>> But, the time has come to upgrade.  Mostly because of MythTV, I find I am
>> constantly on the verge of running out of space.  I've already had to resort
>> to some symlink juggling to put things in different places where more space is
>> fre.
>  MythTV added the ability to have multiple recording directories and put new
>recordings where there is space a version or two ago.

Actually, I wasn't referring to the MythTV data.  When I originally set this
iteration of the machine up some years back I didn't put enough forethought
into how big to make some of the volumes.  Worse still, I also didn't leave
any free space in the volume group as a whole so that I might later grow a
volume if needed.  Oops.  :)

>> The other related question deals with using these new giant drives to begin
>> with.  I came across a few vague sentences that *seemed* to indicate that the
>> current BIOS methods for accessing large drives (LBA) won't do the trick for
>> these 1 TB+ drives and that UEFI is required.  Is this really true?  The
>> motherboard they will be used with is not new, nor is it very old either.
>> I've used 1 TB+ drives on a much older machine at work, but they were accessed
>> via USB2 and FireWire and not via a SATA or eSATA port.  Since I saw this
>> mentioned in only one place I tend to think that this isn't true, but I
>> thought I should check first.
>  That mess only matters when booting, once Linux is up it can see
>  everything, and since you already had boot partitions set up at the
>  beginning of the drives you're safe.

I suspected as much, but I wanted to make sure.

>  As others have mentioned, fancier file systems can be fun too. ZFS on
>  Solaris or FreeBSD is supposed to be really solid but on Linux it was in
>  the "it might not freak out and destroy all your data stage". BTRFS is in
>  roughly the same level of development; maybe more confident in integrity
>  but the tools are rougher. I recently used BTRFS's mirroring magic to move
>  my laptop from a HDD to an SSD hot, but I had to compile my own kernel to
>  remove the HDD because the version in Debian doesn't have the code to
>  remove a mirror implemented. It did work completely online though.

Using something other than XFS had occurred to me, but I don't know that I
trust BTRFS fully just yet.  ZFS has longer track record, but the last time I
gave it a look (admittedly, some time ago), the only way to use it with Linux
is via FUSE and that resulted it very poor performance.

Is anybody on the list using either BTRFS or ZFS *on Linux* for data they
truly feel is important?


To answer the other questions...

With four 500 GB drives creating a single encrypted array to hold the entire
system was the route of least resistance.  I've been using MythTV going on ten
years now and in that time I've built a decent collection of recordings,
videos (MythTV-speak for any video that isn't recorded by MythTV), and other
miscellaneous bits.  Oh, and a fair amount of cruft, too.

How important/valuable this data is varies greatly.  Recordings can certainly
be recorded again in the future.  The vast majority of my videos are DVD rips
so, if necessary, I could re-rip them all, time consuming as that might be.

My backup solution is... poor, I'm afraid.  Certainly nothing systematic.  If
something is truly valuable to me I try to keep copies of that data on more
than one machine and occasionally on optical media.  In general, though, I
don't trust CDs/DVDs for anything long term.

Part of the problem is that my expectations are cheerfully rose colored by
experience.  Throughout the years I've never had a HDD in a system fail or
even come close to failing.  Once, a few years back, an smallish (160 GB)
external HDD that was used for periodic snapshots failed, but it was never
determined if it was the drive or the enclosure that failed.  Chalk that
uncertainty up to my nonexistent parts budget for the computer at work.
Consequently, even though I *know* how often drives fail, I have no personal
memories of the pain of losing important data.  (knock on wood)

As I loathe having to rebuild a functioning system, I simply found it easiest
to put everything in the array.

In my original email I don't think I made it clear why I was looking at only
four drives.  Four 2 or 3 TB drives is about the best I can do with my current
budget.

Based on the opinions here and some additional reading, I'm leaning towards
four 2 TB drives in a RAID-5 array.  Since this is a home system the drives
don't see the level of activity they might in a commercial setting, even when
you account for large video recordings.  So, I think I can probably safely go
with RAID-5.  Also, my system as currently configured performs a "data-check"
about every 20 days.  For a 1.5 TB array this check takes ~130 minutes (at
least, the most recent one did).  Even though normal I/O isn't putting a huge
load on the array, this periodic check does exercise the drives.  I would
guess that this check has been performed at least 100 times since the array
was built.  It managed to survive all of that so living through a (hopefully
never occurring) rebuild seems like an okay bet.

Doubtless, once I build the new array I'll have drives failing left and right
and I'll really wish I had decided otherwise...


-- 
--John Gruenenfelder    Systems Manager, MKS Imaging Technology, LLC.
Try Weasel Reader for PalmOS  --  http://weaselreader.org
"This is the most fun I've had without being drenched in the blood
of my enemies!"
        --Sam of Sam & Max