web crawling in perl

[David Cantrell]

On Mon, May 22, 2006 at 07:37:55PM +0100, Sam Smith wrote:

> What do people think is the "best" perl (or possibly
> otherwise if it's much better) module/script for crawling
> remote websites?

I've been using WWW::Robot for some time and it Just Works.  I do
vaguely recall having to patch it to cope with a change in some other
module that it depends on, but the patch was an obvious one-liner.

-- 
David Cantrell | Hero of the Information Age

Eye have a spelling chequer / It came with my pea sea
It planely marques four my revue / Miss Steaks eye kin knot sea.
Eye strike a quay and type a word / And weight for it to say
Weather eye am wrong oar write / It shows me strait a weigh.