[OT]: Syslog issues?

[paddy@panici.net]

On Wed, Mar 21, 2007 at 03:43:28PM +0000, Dirk Koopman wrote:
> Luis Motta Campos wrote:
> >  Dear M[ou]ngers
> >
> >  I'm facing a funny problem at work.
> >
> >  We have a very busy apache farm producing 40Gb of log files everyday, 
> >and no centralized logging facility.
> >  We're considering using syslog (or better, syslog-ng) to concentrate 
> >all logs on a single spot, so we can handle, parse, and summarize all 
> >information for human and machine consumption.
> 
> Don't think any of the syslog daemons could reliably handle that sort of 
> data. Remember that syslog's job is to try to store important messages 
> that might allow you to diagnose a problem. Therefore they use slow disc 
> writing methods (syncing etc) to make sure that stuff is written.
>
> Syslogd isn't expected to handle 40Gb/Day (~500KBytes or ~2250 lines per 
> sec).
> 
> On some machines that would constitute a DoS attack!

If he's gonna log 40GB/day over the network, then he needs the network
capacity for that.

Is writing 500KB/s to disk such a big issue ?

I would have thought the analysis part would be a bigger consideration.

I've never used any of them, but IIRC there are few implementations out
there to log from apache over the network to something like mysql or
perhaps postgres.

Regards,
Paddy