The most common approach to determining how much log data will be generated is to use Events per Second (EPS). EPS is exactly what it is called, the number of log or system events that are generated by a device every second. But, why is EPS important and how is it used? Using EPS will help you scope or determine:
An appropriate LM or SIEM – since many LMs or SIEMs are rated or licensed based on EPS or amount of logged data, it is critical that you have an accurate estimate of your EPS or else you risk oversizing (paying too much) or under sizing (losing data) your solution.
Your online and offline storage requirements – if you have compliance requirements then you will have some type of retention policy. Your retention policy along with the amount of log data generated will determine your storage requirement.
Your daily storage management – Storage costs money and you don’t want to spend more than you have to, however, you do not want to run out of storage either. Understanding your EPS will better allow you to manage and plan your log data storage needs.
There are two EPS metrics that need to be factored into your planning and analysis: Normal Events per Second (NEx), and Peak Events per Second (PEx).
NEx, just as its name implies, represents the normal number of events per second while PEx, represents the peak number of events that are caused by abnormal activities such as a security attack. While PEx is a theoretical, albeit impractical, measurement, it does need to be factored in as it could impact the performance of your SIEM/LM solution as well as your storage requirements.
Why should you be concerned about PEx? Quite simply, a single security incident such as a worm, virus or DOS may fire off thousands of events per second from the firewall, IPS, router, or switch at a single gateway. Multiply this by your multiple subnets and it can quickly spiral out of control.
Now that we understand our EPS, we can estimate the amount of log data that is being generated per second and per day based on the following formulas:
Some SIEM and LM solutions in the market license by the amount of log data collected, or indexed, on a daily basis. This calculation will allow you to estimate the size of the license required under that model. In addition, by applying the above calculation to your data retention policies, you can estimate the amount of storage required for your log data.
|1||Windows Servers - HIGH EPS (~50 eps)||50|
|2||Windows Servers - MED EPS (~3 eps)||3|
|3||Windows Servers - LOW EPS (~1 eps)||1|
|5||Windows AD Servers||10|
|7||IBM AIX Unix Servers||2|
|8||HP-UX Unix Servers||2|
|9||Sun Solaris Unix Servers||2|
|10||IBM Mainframe / Midrange||2|
|13||Network Switches (Netflow)||30|
|14||Network Wireless LAN||5|
|15||Network Load balancers||5|
|17||Other Network Devices||5|
|18||Network Firewalls (Check Point - Internal)||10|
|19||Network Firewalls (Check Point - DMZ)||50|
|20||Network Firewalls (Cisco - Internal)||10|
|21||Network Firewalls (Cisco - DMZ)||30|
|25||Network Web Proxy||15|
|26||Other Security Devices||10|
|27||Web Servers (IIS, Tomcat, Apache)||1|
|28||Database (MSSQL, Oracle, Sybase - # of instances)||1|
|29||Email Servers (Exchange, Sandmail, etc)||2|
|30||AntiVirus Server (indicate number of AV clients)||5|
|31||Other Applications (Email, DB, AV, etc)||5|
|Result||Per Second||Per Day||Per Month|
|Generated Log File Size after Compression|
[[ Assuming]] 1. There will be 3 peaks per day (morning logins, lunchtime web surfing, evening logoffs/backups) 2. Each Peak event will last approximately 1 hour 3. Diviation in the Peak hours will be 3 times Normal
Aspire Tech Provides 10:1 Compression Ration. The calculator takes that into account when showing result