Perl threads and libwww wierdness (Leo Lapworth)
ash+perl at best-scarper.co.uk
Wed Dec 14 14:48:03 GMT 2011
> Message: 1
> Date: Wed, 14 Dec 2011 08:44:37 +0000
> From: Leo Lapworth <leo at cuckoo.org>
> Subject: Re: Perl threads and libwww wierdness
> To: "London.pm Perl M[ou]ngers" <london.pm at london.pm.org>
> <CA+rMUP49P3Y6V8Vmzy03i26QwgoTZMtVANd+EOpZG2fGjMP8_Q at mail.gmail.com
> Content-Type: text/plain; charset=ISO-8859-1
> I still think it's your webserver....
> On 14 December 2011 01:16, Toby Wintermute <tjc at wintrmute.net> wrote:
> > I'm hitting some really odd behaviour, infrequently, with libwww and
> > mechanize under a highly-threaded Perl.
> > I have encountered a situation where I see unusual 404 errors - in
> > between 0.03% to 0.10% of requests.
> > Errors are randomly spaced on random pages
> > The 404 errors are reported on the distant webserver as well, for URLs
> > that are definitely not 404. (as the identical URL is being requested
> > successfully many times in the same period).
> My logic would be, if the webserver is reporting intermittent 404's for a
> specific URL, then it's the webserver that's generating it.
In general, I'd agree with Leo, although there may be wrinkles in the HTTP
request you aren't seeing (cache headers).
Have you tried a detailed dump of each outbound request?
If the service owner can't give you a reason for the 404s on a valid URL,
then you might consider a proxy to get more detailed diagnostics on the
> > Scale: This is typically running around 40 threads, all going flat-out
> > on an 8-core system; issues show up whenever you get over ~6 threads
> > though.
> So the webserver can't cope with the traffic generated when you use
> more than 6 threads, so too many requests.
> > The only reason I don't think this is a problem with the network or
> > webserver is that the problems don't show up if I use fork() instead
> > of threads. (On otherwise identical code; and the same overall
> > throughput rates are reached. However the fork() version is just for
> > that bit of code for testing this; it misses some functionality.)
> Does the fork() submit the same number of requests in the same
> time period? - if it's less than your thread version then that
> would point to the webserver, not the Perl code.
> I'd look into if your webserver (you don't mention what software it
> is, apache/starman/IIS?) has some sort of "arrrrgggg, I can't cope,
> throw a 404" setting or bug.
> Good luck!
More information about the london.pm