Perl threads and libwww wierdness (Leo Lapworth)

[Ashley Hindmarsh]

>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 14 Dec 2011 08:44:37 +0000
> From: Leo Lapworth <leo at cuckoo.org>
> Subject: Re: Perl threads and libwww wierdness
> To: "London.pm Perl M[ou]ngers" <london.pm at london.pm.org>
> Message-ID:
>        <CA+rMUP49P3Y6V8Vmzy03i26QwgoTZMtVANd+EOpZG2fGjMP8_Q at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> I still think it's your webserver....
>
> On 14 December 2011 01:16, Toby Wintermute <tjc at wintrmute.net> wrote:
> > I'm hitting some really odd behaviour, infrequently, with libwww and
> > mechanize under a highly-threaded Perl.
>
> > I have encountered a situation where I see unusual 404 errors - in
> > between 0.03% to 0.10% of requests.
> > Errors are randomly spaced on random pages
> >
> > The 404 errors are reported on the distant webserver as well, for URLs
> > that are definitely not 404. (as the identical URL is being requested
> > successfully many times in the same period).
>
> My logic would be, if the webserver is reporting intermittent 404's for a
> specific URL, then it's the webserver that's generating it.
>

In general, I'd agree with Leo, although there may be wrinkles in the HTTP
request you aren't seeing (cache headers).

Have you tried a detailed dump of each outbound request?

If the service owner can't give you a reason for the 404s on a valid URL,
then you might consider a proxy to get more detailed diagnostics on the
request/response.

  Ash

>
> > Scale: This is typically running around 40 threads, all going flat-out
> > on an 8-core system; issues show up whenever you get over ~6 threads
> > though.
>
> So the webserver can't cope with the traffic generated when you use
> more than 6 threads, so too many requests.
>
> > The only reason I don't think this is a problem with the network or
> > webserver is that the problems don't show up if I use fork() instead
> > of threads. (On otherwise identical code; and the same overall
> > throughput rates are reached. However the fork() version is just for
> > that bit of code for testing this; it misses some functionality.)
>
> Does the fork() submit the same number of requests in the same
> time period? - if it's less than your thread version then that
> would point to the webserver, not the Perl code.
>
> I'd look into if your webserver (you don't mention what software it
> is, apache/starman/IIS?) has some sort of "arrrrgggg, I can't cope,
> throw a 404" setting or bug.
>
> Good luck!
>
> Leo
>
>