Mirroring files over servers was Re: [OT]: bash or any other $favourite{shell}? [was: Wish list]

[paddy]

On Fri, Jun 16, 2006 at 05:50:10PM +0100, Steve Mynott wrote:
> On Mon, Jun 12, 2006 at 03:01:51PM +0100, Dirk Koopman typed:
> 
> > I suppose what I am looking for is some kind of RAID 5 effort, but
> > distributed across a few machines.

Dirk,

okay I nearly bit last time, but I couldn't spare the time to find the
resources I wanted to quote.

by now you've read the obligatory slahdot article, yes ? ;-)

and you're aware of old stuff like NOW, as well as solutions like AFS, etc.

we still don't know what you actually want to do ??

but if I could turn it up I once read an article, that seemed pretty 
convincing at the time, that argued that there are inherent limitations 
with the kind of RAID5 over network kind of thing you seem to be asking 
for, that make it good for less things than you might at first think.

Naturally, I don't recall the details or the link ;-)

hassle me if you can't find it on google :-)

now ...

Steve,

> I evaluated DRBD for a customer who wanted mirrored servers about a year
> back which was claimed to be network RAID-1 (mirror).
> 
> http://www.drbd.org/
> 
> It's not that easy to configure, but does work 

disclaimer: I am using drbd (in a small way)

it's not how it works that is important. it's how it breaks ;-)

> and but
 
make your mind up ;-)

> I particularly didn't
> like the way the remote mirrored filesystem wasn't viewable in normal use (you
> can't mount the remote node readonly), so you just had to trust that the
> kernel module had done its work.  

you places your bets ... and you takes your chances ;-)

I understand that drbd8 (which is in some sort of beta right now) will do the 
gfs-nbd-type-thing of supporting access from both sides.  while it seems it
is a natural evolution, in some ways it is a completely different thing.

> And I have to confess to a prejudice against
> all networked file systems.  So I ended up doing it the old school way and
> running rsync every 15 mins together with Mysql replication to the remote
> server and local hardware RAID-5 disks, which seemed more risk adverse.

so these are not "networked filesystems" ?

what are "networked filesystems" to you ? for example, to me drbd is an nbd,
not an nfs. (well, compared to pavel's nbd more like nbd + raid1 over network,
yes)  please do take the time to reply to this bit, as I imagine you have
something very interesting to say (not kidding, need to try to learn :-) )

FWIW, I nearly used Mysql replication for my app, but I wanted to be as close
to synchronous as possible, and mysql cluster couldn't do it at the time
(plus there were complications around ssl licensing and builds, and other
similar requirements besides).  bottom line: as an admin I'd like to have
a single synchronous replication service that's simple to deploy and 
maintain, but in my heart I believe that many such problems (and their
solutions) _need_ to be informed by the application to be efficient.
In the short term at least, clustering at the application level will
often tend to make more sense, where it is available.

equally, for most setups, it would generally make good sense, simply cost-wise,
to already have per node redundancy like the local RAID5 you describe, 
before you consider admin-intensive options like clustering.

drbd doesn't really make sense, to me at least, if your requirement is 
asynchronous (although there are synchronous-but-winging-it settings ;-)

The asymetric aspect is just a (they haven't written that bit yet) 
optimisation :-)

> Actually one problem with the whole idea of mirroring is the possibility of
> losing files on one disk and this file loss being synced over to the remote
> (eg. if you are using rsync's dreaded --delete option).  

Agreed, and IIRC interactions between drbd and LVM (specifically snapshots)
are not trivial, which is a shame.  No doubt this will get sorted out in
time.

Nevertheless, the difference between current-moment redundancy and
checkpointing, these are two different things ...

... do both :-)

(when it makes sense, naturally)

> For this reason, and the possibility of rolling back changes, I now use
> "rdiff-backup" and can recommend "backupninja" -- "a silent flower blossom
> death strike to lost data".  However, you might be better off renaming that
> one if presenting it as a solution to a PHB as "Enterprise Backup Solution
> Pro" or similar.
> 
> SSHFS also looks interesting:
> 
> http://fuse.sourceforge.net/sshfs.html

thanks for the links, will take a look.

Regards,
Paddy
-- 
Perl 6 will give you the big knob. -- Larry Wall