Sharding, and all that

[Simon Wistow]

On Thu, Dec 18, 2008 at 10:19:06AM +0000, Mark Fowler said:
> What's the collective group think on these?  Obviously (for this list, at 
> least) the app that will be accessing the database is Perl based, so I'm 
> only considering compatible technologies at the moment.

We use Data::ObjectDriver which has built in support for sharding and 
memcache (somewhat unsuprisingly) plus MySQL with replication.

The main advantages are that it's conceptually simple, resilient, 
flexible (especially with the ability to move entities between shards) 
and cheap to scale. 

The big question is whether to use a hashing scheme for shard selection 
(keywords: Consistent Hashing, libketama, DHTs) or whether to use a 
master lookup device (which is what I believe Flickr uses). 

As always, YMMV.