Perl-friendly message queue-like system

[Bruce Richardson]

On Thu, Sep 22, 2011 at 04:38:24PM +0100, Martin A. Brooks wrote:
> 
> The problem is not the database I store stuff in, it's getting stuff to the database in the first place.  The final data store will almost certainly be postgres, I see no reason to use anything else.
> 
> I don't want to use any kind of remote syslog as something then has to
> parse potentially quite a lot of unstructured data

Perl, meet data; data, Perl.

> and then be
> idempotent if fed the same log again for some reason.

It should be possible to mark each datum uniquely at the first stage of
logging, via some combination of hostname, datestamp and (if necessary)
application name or process id.  That should be plenty; any script
feeding a database should be capable of not re-inserting the same data
if it is already present.  MQ-style solutions don't magically give you
any greater protection from duplicate data if you fail to identify the
data uniquely from the start.  And how do you intend to replay the data
you're passing into MQ?  I guess you'd have to log it...

-- 
Bruce

Vajazzle - giving new meaning to the phrase "I'll scratch your eyes
out".