Perl-friendly message queue-like system

[Martin A. Brooks]

Hi Bruce,


----- Original Message -----
> From: "Bruce Richardson" <itsbruce at workshy.org>
> To: "london pm" <london.pm at london.pm.org>
> Sent: Thursday, 22 September, 2011 5:08:02 PM
> Subject: Re: Perl-friendly message queue-like system
> 
> MQ-style solutions don't magically give you
> any greater protection from duplicate data if you fail to identify
> the data uniquely from the start.  And how do you intend to 
> replay the data you're passing into MQ?  I guess you'd have to log it...

Well that was entirely the point really.  I don't actually care if a little bit of data gets dropped on the floor and I would never expect or want to replay data.  A message is emitted, stuffed into a queue and then processed.  If any part of this fails for 1 message in every few thousand (we're handling something like 120,000 mail connections per day), meh.  It's not billing data, it really doesn't matter.

As it happens, the data is already handled locally in a transactional way.  Each of my mail filtering nodes runs a postgres database that exim logs to.  Rather than trying to merge N databases to get some pretty stats, I figured just having each node fling crumbs of data, monkey-like, off to some server would be a nice way of seeing that _something_ has happened.


-- 
Martin A. Brooks
http://antibodyMX.net/ - antispam & antivirus email filtering.