Domesticating applications, OpenBSD style

When I say logs, I'm talking about the stuff generated by operating systems and appliances into syslog + the logs that the applications write (sometimes to syslog, more frequently to local log files that then have to be scraped to be gathered). This includes things like webserver logs (which I find to be a significant percentage of the overall logs, but only ~1/3)

I do add some additional data to the log stream, but it's low volume compared to the basic logs I refer to above (A few log messages/min per server)

Also, keep in mind that when I talk about sizing a logging system, most of the time I'm talking about the peak data rate. What it takes to keep up with the logs at the busiest part of the busiest day.

I want to have a logging system that can process all logs within about 2 min of when they are generated. This is about the limit as far as I've found for having the system react to log entries or having changes start showing up in graphs.

There is also the average volume of logs per day. This comes into play when you are sizing your storage.

so when I talk about 100K logs/sec or 1Gb of logs being delivered, this is the peak time.

100K logs/sec @256 bytes/log = 25MB/sec (1.5GB/min, 90GB/hour) If you send this logging traffic to four destinations (archive, search, alerting, reporting), you are at ~100MB/sec of the theoretical 125MB/sec signalling rate that gig-E gives you. In practice this is right about at or just above the limit of what you can do with default network settings (without playing games like jumbo frames, compressing the network stream, etc). The talk I posted the link to earlier goes into the tricks for supporting this sort of thing.

But it's important to realize that this data rate may only be sustained for a few min per day on a peak day, so the daily volume of logs can be <1TB/day on a peak day (which compresses to ~10GB), and considerably less on off-peak days. Over a year, this may average out to 500GB/day (since I'm no longer there I can't lookup numbers, but these are in the ballpark)

This was at a company providing banking services for >10M users.

now, the company that I recently started at is generating 50GB of windows eventlog data per day most weekdays, (not counting application logs, firewall logs, IDS logs, etc) from a couple hundred production systems. I don't yet have a feed of those logs, so I can't break it down any more than that yet, but the type of business that we do is very cyclical, so I would expect that peak days of the month/year the windows eventlog log volume will easily be double/triple that.

If you parse the log files early and send both the parsed data and the raw data, the combined parsed data can easily be 2x-3x the size of the original raw data (metadata that you gather just adds to this)

As a result, ElasticSearch needs to be sized to handle somewhere around 500G/day (3*3*50GB/day) for it's retention period to handle the peak period with a little headroom.