Perl-friendly message queue-like system

[Mark Blackman]

On 22 Sep 2011, at 09:15, Martin A. Brooks wrote:

> 
> I have a bunch of servers doing mail ilftering.  I would like them to send tiny messages about the results of said filtering to a central point.  I would then like something at the central point to pop messages off the queue and update a database.  Think of it as sort of a centralised collection point for statistical data.  If the odd message gets eaten, it really doesn't matter.
> 
> 
[snip]
> 
> My Google-fu is failing me, what Perl-friendly system might I use for such a thing?



I'd consider using Spread, even though the Perl bindings for it are a
bit unloved, see the review.

http://search.cpan.org/~jesus/Spread-3.17.4.4/

The underlying Spread toolkit is a very reliable and well-designed
*group messaging* framework. Regrettably the Perl bindings are for an
entire major version behind the latest Spread toolkit release,  but I've
had very good results using Spread 3.17 for a reliable, general-purpose
log aggregation tool for well over 5 years and in particular, it can
handle widely distributed nodes over TCP, rather than the easy case of
everything in the same broadcast domain.

I would guess it works extremely well for the case you're describing.
It's not a message queueing system in that it's *not* intended to queue
messages indefinitely, but it's good for aggregating messages reliably,
which sounds like your main goal.

A key point to remember is that message *writers* don't need to join
groups to write to them. If your app joins a group, your inbound message
buffers will fill up very quickly if you don't read them and writers,
in this context, don't care about reading from other nodes.

You're meant to run a Spread daemon on every machine in the group and local 
clients connect to the local daemon which then manages the message delivery
to the other Spread daemons on the other hosts, which in turn manage
the message delivery to their local clients.

- Mark