Perl threads and libwww wierdness

[Leo Lapworth]

I still think it's your webserver....

On 14 December 2011 01:16, Toby Wintermute <tjc at wintrmute.net> wrote:
> I'm hitting some really odd behaviour, infrequently, with libwww and
> mechanize under a highly-threaded Perl.

> I have encountered a situation where I see unusual 404 errors - in
> between 0.03% to 0.10% of requests.
> Errors are randomly spaced on random pages
>
> The 404 errors are reported on the distant webserver as well, for URLs
> that are definitely not 404. (as the identical URL is being requested
> successfully many times in the same period).

My logic would be, if the webserver is reporting intermittent 404's for a
specific URL, then it's the webserver that's generating it.

> Scale: This is typically running around 40 threads, all going flat-out
> on an 8-core system; issues show up whenever you get over ~6 threads
> though.

So the webserver can't cope with the traffic generated when you use
more than 6 threads, so too many requests.

> The only reason I don't think this is a problem with the network or
> webserver is that the problems don't show up if I use fork() instead
> of threads. (On otherwise identical code; and the same overall
> throughput rates are reached. However the fork() version is just for
> that bit of code for testing this; it misses some functionality.)

Does the fork() submit the same number of requests in the same
time period? - if it's less than your thread version then that
would point to the webserver, not the Perl code.

I'd look into if your webserver (you don't mention what software it
is, apache/starman/IIS?) has some sort of "arrrrgggg, I can't cope,
throw a 404" setting or bug.

Good luck!

Leo