Arbyte Slides

[Simon Wistow]

On Mon, Dec 01, 2008 at 12:16:45PM +0000, Alistair MacLeod said:
> Work continues on making it CPAN ready.

Some comments on Gearman and TheSchwartz which (kind of obviously) used 
quite a lot here at 6A.

You're right about Gearman in that it's "Not Reliable" but potentially 
that's a poor choice of words - "Not Guaranteed" is probably a better 
way of putting it and it's deliberately designed that way - it's for use 
as a one shot tasks that you don't mind if they fail. For example I use 
it for firing off $n web request jobs simultaneously which update a 
cache. If after a timeout they've updated the cache then fine, otherwise 
I just use the old cache version.

As for it not working - ping me off list or add an RT ticket (if you 
haven't already) and I'll take a look.

As for the TheSchwartz you say it's not easily scalable - it uses 
Data::ObjectDriver and hence has inbuilt support for sharding. And trust 
me - it's scalable. We use it for everything here which, until fairly 
recently, included all of TypePad and LiveJournal simultaneously 
(including sending out all the email) :)

Also, you say it doesn't have batching after submission - unless I'm 
misunderstanding you that's not actually true. If you look in the docs 
for TheSchwartz::Job you'll find the coalesce param to new()

http://search.cpan.org/~bradfitz/TheSchwartz-1.07/lib/TheSchwartz/Job.pm#coalesce

Which allows batching.

Anyway, I've been thinking of writing something very similar to Arbyte 
so I'm looking forward to it. I notice you mentioned something about Jo 
bRunner::Simple that fork()s - one of the things I wnated was something 
that ran The various parts of Gearman (the injector, the Geamand and a 
number of workers) or TheSchwartz (the ibjector, the DB and a number of 
workers) all within the same process for testing purposes - is that the 
same kind of functionality that you're looking at providing?

Thanks!

Simon