web crawling in perl

David Cantrell david at cantrell.org.uk
Mon May 22 20:36:57 BST 2006

On Mon, May 22, 2006 at 07:37:55PM +0100, Sam Smith wrote:

> What do people think is the "best" perl (or possibly
> otherwise if it's much better) module/script for crawling
> remote websites?

I've been using WWW::Robot for some time and it Just Works.  I do
vaguely recall having to patch it to cope with a change in some other
module that it depends on, but the patch was an obvious one-liner.

David Cantrell | Hero of the Information Age

Eye have a spelling chequer / It came with my pea sea
It planely marques four my revue / Miss Steaks eye kin knot sea.
Eye strike a quay and type a word / And weight for it to say
Weather eye am wrong oar write / It shows me strait a weigh.

More information about the london.pm mailing list