Multi platform, high volume data recording

[Simon Wilcox]

Hi Lyle,

Lyle - CosmicPerl.com wrote:
> I'd appreciate some heavy criticism (without swearing or personal 
> insults) and suggestions. Although some things will not be changed, for 
> instance...

Flash is an interesting choice of presentation medium. I particularly 
loath that I can't link directly to any of your slides. A great example 
of why Flash sucks from a usability point of view.

Anyway, regarding the content I'd make four comments:

1. Multiple A records. We've been doing some testing on a setup just 
like this over the last couple of weeks and so far we've found that 
"automatic" is something of a misnomer. The current browser crop do NOT 
appear to automatically retry the 2nd IP address if the first goes down. 
Some data is always lost but when the user clicks refresh the browser 
does pick up the second address. So it appears to be good for automatic 
failover but it's not seamless and not error free. It's certainly better 
than not having a second datacentre though.

2. Fault tolerance. By having just one logging server receiving and 
processing data at one time you're introducing a single point of 
failure. If it goes down all the data collected goes with it.

3. DB scalability. MySQL only supports 1 master per database and given 
that this is primarily a data writing application you will need to 
consider partitioning your data into multiple databases so that you can 
write to multiple servers. You will also need to consider how you 
migrate the write master to another server if the primary server goes 
down. Load balancers are quite good for this too.

4. Commercial reality. Your diagram shows 12 servers in 2 datacentres. 
That's what, maybe a £30-40k spend on hardware, more if you virtualise 
with shared storage and £2k a month in rack rental and transit. Plus the 
admin cost of running & supporting that which is another £6-7k a month. 
I don't know what you're planning to charge for this software but if I 
was on the buying side, and I have been, there's no way we would spend 
that kind of money on a one-man band almost no matter how good the 
software was. People spending that kind of money tend to be cautious.

Or is this your plan for the hosted solution, in which case you *will* 
need to hire a sysadmin to help run that lot. You alone will not be able 
to write the software, build those servers, run the system and, most 
importantly, sell the solution no matter how good you are. Been there, 
done that. You *will* fail if you try it.

BTW - out of curiosity, what's your USP compared to Google Analytics and 
WebTrends ?

Good luck with the project, it's certainly "ambitious" as Sir Humphrey 
would say :-)

Simon.