Perl threads and libwww wierdness

[Toby Wintermute]

2011/12/16 Peter Vereshagin <peter at vereshagin.org>:
> Hello.
>
> 2011/12/14 12:16:04 +1100 Toby Wintermute <tjc at wintrmute.net> => To London.pm Perl M[ou]ngers :
> TW> I'm hitting some really odd behaviour, infrequently, with libwww and
> TW> mechanize under a highly-threaded Perl.
>
> TW> Error rates initially increase with the number of simultaneous
> TW> threads, but seem to top off at .1%. (ie. One in a thousand requests)
>
> (i)Threaded perl5 ( 'use threads' ) doesn't seem to be recommended for
> production environments.

I know it certainly wasn't recommended back in the days of 5.6 or 5.8,
but I thought things had improved since then..

> Forks are the standard IPC method for perl5. Event-driven stuff (AE/EV) is a
> more advanced way, but for 40 parallel requests the forks are just ok.

I'm not sure IPC means what you think it means - or I'm
misunderstanding you - It's Inter-process communication, right?
It'd say that forking, threading, and event-driven, are all methods of
doing simultaneous processing - but don't specify anything about IPC.

Which is the reason I was using threads - it's easy to do IPC between
them by sharing access to some data structures.
Whereas with forking you need to set up pipes, posix shared memory,
sockets, or some other means.

In the end, I bit the bullet yesterday and did rewrite the code to use
fork(), and then used Net::STOMP::Client with RabbitMQ to perform the
IPC.

Total throughput is actually slightly faster than the threaded
version, it uses quite a bit less memory, and critically, doesn't get
any spurious errors.

-Toby

-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world