Perl threads and libwww wierdness

[Toby Wintermute]

On 14 December 2011 19:44, Leo Lapworth <leo at cuckoo.org> wrote:
> I still think it's your webserver....

See below..

> On 14 December 2011 01:16, Toby Wintermute <tjc at wintrmute.net> wrote:
>> I'm hitting some really odd behaviour, infrequently, with libwww and
>> mechanize under a highly-threaded Perl.
>
>> I have encountered a situation where I see unusual 404 errors - in
>> between 0.03% to 0.10% of requests.
>> Errors are randomly spaced on random pages
>>
>> The 404 errors are reported on the distant webserver as well, for URLs
>> that are definitely not 404. (as the identical URL is being requested
>> successfully many times in the same period).
>
> My logic would be, if the webserver is reporting intermittent 404's for a
> specific URL, then it's the webserver that's generating it.

That would be my logic too, but (a) I can't work out why it would drop
one in a thousand requests only, and (b) the non-threaded Perl scripts
don't see ANY 404 errors.

>> Scale: This is typically running around 40 threads, all going flat-out
>> on an 8-core system; issues show up whenever you get over ~6 threads
>> though.
>
> So the webserver can't cope with the traffic generated when you use
> more than 6 threads, so too many requests.

I start seeing errors appear before the webserver is getting fully
loaded up though. The machine serving the requests is quite powerful,
and there's only one machine making all the requests.

>> The only reason I don't think this is a problem with the network or
>> webserver is that the problems don't show up if I use fork() instead
>> of threads. (On otherwise identical code; and the same overall
>> throughput rates are reached. However the fork() version is just for
>> that bit of code for testing this; it misses some functionality.)
>
> Does the fork() submit the same number of requests in the same
> time period? - if it's less than your thread version then that
> would point to the webserver, not the Perl code.

That's the thing -- the fork-based version will generate just as many
hits, but with zero 404 errors.
So the webserver can handle that level of traffic.

Eg. I can pull more than 260/sec off the webserver with either fork or
thread based app. But the thread-based app starts seeing errors creep
in once it gets over around 180-200/sec, and by 260/sec, it's up to
0.1% of all requests. (ie. once every few seconds)

> I'd look into if your webserver (you don't mention what software it
> is, apache/starman/IIS?) has some sort of "arrrrgggg, I can't cope,
> throw a 404" setting or bug.

We've now replicated the issue on both Apache 2.2 and nginx, and also
against an older machine which coped with less overall hitrate, but
still similar error rates.

I'm just annoyed I haven't managed to replicate the issue on
stand-alone code I can demonstrate on a desktop class machine just yet
:/

Thanks,
Toby

-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world