Web scraping frameworks?

[Jérôme Étévé]

Web::Scraper is great to hack something together quickly.

I use it regularly to do some quick ad-hoc data scraping.

For more heavy work, I prefer a combination of the following tools:

- Curl (Net::Curl or its LWP style incarnation LWP::Curl). I found it
to be more resilient than LWP against dodgy http server responses.

- For the page data scraping itself, LibXML (with its load_html and in
recover mode) + XPath. Again, for its resilience against crap HTML. We
all know correct HTML is the exception rather than the norm on the big
bad web.

- For queuing jobs, I'm a big fan of Gearman. It's light, very stable
and very simple.

Of course, it's only a toolbox. I doubt you can find a ready made
"framework" that fits your specific business needs out of the box.

J.



On 4 March 2014 22:55, Pierre M <piemas25 at gmail.com> wrote:
> I love using
>       Web::Scraper
> It's so simple and intuitive to use!
> But it only "goes down" (unless if I've missed something), and it doesn't
> allow to interact with the page (fill forms, click buttons, etc) so it
> doesn't handle complex scraping scenarios. For these, I like
>       Mojo::UserAgent
> which gives me more control. An example here:
>
> http://blog.kraih.com/post/43198036449/mojolicious-hack-of-the-day-web-scraping-with



-- 
Jerome Eteve
+44(0)7738864546
http://www.eteve.net/