Optimising perl

[Minty]

On 6/23/06, alex at owal.co.uk <alex at owal.co.uk> wrote:
> Can anyone suggest some reading for optimising perl? (This is inherited
> code so dont blame me for lazy algorythms used :-)

http://www.ccl4.org/~nick/P/Fast_Enough/

http://search.cpan.org/~timb/DBI_AdvancedTalk_2004/
and the associated http://www.oreilly.com/catalog/perldbi/

http://www.oreilly.com/catalog/maperl/

My personal fav is throwing in Storable as a caching layer, but that
might not help if you are hitting memory limits.

> We are building huge hashes in memory which we might tie to disk, or (my
> preference) tie to a memcached database. This gets us around the 4Gb
> memory limit in the perl binary we are using by spreading the memory
> across two processes.

Hashes have a (not tiny, when used for lots of data, or complex nested
structures) overhead - you are optimizing for fast lookups.  Play
around (and do some simple tests).  Eg, having a much smaller hash
with nothing other than a lookup key -> some kinda pointer/key, which
you can then use to grab the real data from disk (optionally with a
(Storable) disk access cache layer).

Also, have you considered using the DB (more) directly?  ie, less
in-perl-custom-caching, and more handing off the hard part to the
RDBMS.