In searching for the next flow collection system that will support your organization, most results will turn up solutions claiming to support the consumption of millions of flows/second. For the most part, I think there is some truth to these claims but, when the system is trialed, the limitations become obvious.
Enterprise NetFlow Collections
It’s one thing to collect high volume NetFlow and quite another to report on it and that’s really where the rub lies. The amount of engineering necessary to collect millions of flows per second and write them to some type of backend really isn’t all that hard for a seasoned developer. Getting the data back out of the storage in a timely manner, is where expert design comes into play. And really, how fast is fast? Well, Google sort of sets that bar and they are generally pretty much instant with results. Drawing comparisons to Google however, really isn’t all that fare. First of all, the Google architecture is supported by thousands of servers. Most enterprise NetFlow collection consumers don’t have the deep pockets necessary to build out “a similar to Google” architecture. They want it all on one system or at least perceived as one system.
Distributed Collection Vs Single Collector
A single system however, is not always the best choice. Sometimes sending all the flows back from a remote location over a WAN can cause congestion. In these situations, distributed collectors makes the most sense. Enterprise NetFlow systems will provide single interface for viewing the data across dozens of collectors all while keeping the distribution transparent to the user. In other situations, sending the flows up to the cloud makes the most sense where a cluster of servers can support the collection. The vendor can help you decide on which strategy is the best fit for your environment.
Missed Flow Sequence Numbers (MFSNs)
If you speak with the vendor, some will tell you that their systems scales to millions of flows per second while dropping nothing. The question is: how do you know if they are or aren’t dropping flows? To reliably answer this, the system should be counting and reporting on the flow sequence numbers. If the collector is missing flow sequence numbers (MFSNs) from a specific router and only one router, the problem is likely the router or the network. If the collector is seeing MFSNs across all routers that are sending flows, then the problem starts to look like the collector not keeping up. Without monitoring for MFSNs, we can’t be sure if the collector is collecting everything we expect it to.
Filtering
When problems strike, a fast responsive interface is great but, it needs to be combined with ease of filtering. The ability to quickly include this and exclude that really speeds up the investigative process. All vendors claim the best interface. Make sure you try it out for yourself and watch any short tutorials they may have to help you appreciate the vendor specific features.
