Multi platform, high volume data recording

[Lyle - CosmicPerl.com]

nigel at turbo10.com wrote:
> Hi Lyle,
>
>> http://www.cosmicperl.com/design1.swf
>>
>
>     Do you really need realtime reporting? Google Analytics, for 
> example, works fine with a short delay. The reason I ask is that this 
> will relieve the pressure to scale.
>
>     If the requirement is a "would like to have" I would solve this 
> one when my system was starting to handle squillions of hits --- this 
> kind of a problem is a nice one to have --- in the meantime create a 
> system that doesn't give you too much grief and can evolve into its 
> right technical/economic niche.

Hi Nige,
  It's not real time, there is the delay of X minutes from the the data 
logging servers get the data, and later input it into the main DB 
cluster. The software will be designed to run like this, but also be 
able to run on a single machine.

For the type of users the client has, things need to be as real time as 
possible.

What's worrying me at the moment is the emailing side of things. There 
could potentially be a newsletter going out daily to a mailing list of 
over 90,000. I've only every used sendmail or SMTP server (win32), but 
after looking at how mysociety are doing it, I'm guessing splitting it 
into batches and running a cron job ever 20 minutes or so isn't going to 
cut it.


Lyle