Perl-friendly message queue-like system

[Peter Edwards]

>
> The problem is not the database I store stuff in, it's getting stuff to the
> database in the first place.  The final data store will almost certainly be
> postgres, I see no reason to use anything else.
>
>
If volume is not a problem, sure.
We already had 10GB in our PostgreSQL db and didn't want any more coming
from generating rolling 5 minute, 15 minutes, 30 minutes, hourly (etc.)
stats -> real time graphs for our monitoring screens with historic period
comparison across a range of measures .

In case it's of use for someone else here's an easy way to do it.

With RRD you create a logging database file like

$ rrdtool create transfer.rrd \
  --start N --step 300 \
  DS:transfer_in_secs:GAUGE:1200:0:600 \
  DS:files_transferred:GAUGE:1200:0:20 \
  DS:transfer_errors:GAUGE:1200:0:20 \
  DS:average_errors:COMPUTE:transfer_errors,files_transferred,/,100,* \
  RRA:AVERAGE:0.5:1:3600 \
  RRA:MAX:0.5:1:3600 \
  RRA:MIN:0.5:1:3600 \
  RRA:AVERAGE:0.5:4:9600 \
  RRA:AVERAGE:0.5:24:6000

that logs 3 data points and computes one more



# in your app's logging configuration set up a Log4perl logger to write RRD
entries

use Log::Log4perl;

my $transfer_rrd_db_path = "./transfer.rrd";

Log::Log4perl->init(
   ...
   "log4perl.category.TRANSFERRRD" => "INFO, TransferRRD",
   "log4perl.appender.TransferRRD" => "Log::Log4perl::Appender::RRDs",
   "log4perl.appender.TransferRRD.dbname" => $transfer_rrd_db_path,
   "log4perl.appender.TransferRRD.layout" =>
"Log::Log4perl::Layout::PatternLayout",
   "log4perl.appender.TransferRRD.layout.ConversionPattern" => "N:%m",
  ...
);


# now in your code when you do some work

get_logger('TRANSFERRRD')->info(
    $time_elapsed . ':' .
    $files_transferred . ':' .
    $total_errors
    );



then you can create a graph (if that's what you want) with
$ rrdtool graph ./transfer.rrd \
  --start "-1day" --title="Average transfer time"  --vertical-label "secs" \
  --height 200 --width 400 \
  DEF:var_transfer_in_secs=transfer.rrd:transfer_in_secs:AVERAGE \
  DEF:var_files_transferred=transfer.rrd:files_transferred:AVERAGE \
  DEF:var_transfer_errors=transfer.rrd:transfer_errors:AVERAGE \
  LINE1:var_transfer_in_secs#ff0000:"transfer time in seconds" \
  AREA:var_files_transferred#33BB22:"files transferred" \
  AREA:var_transfer_errors#A1A1A1:"transfer errors"


More at
http://oss.oetiker.ch/rrdtool/doc/rrdtool.en.html


Regards, Peter
http://perl.dragonstaff.co.uk