Advanced NetFlow Traffic Analysis - Network traffic monitoring Archives

Building a NetFlow Cache: Exporting IPFIX

2013-03-12T20:28:49Z

Most engineers implementing NetFlow or IPFIX know how to get started. Where they sometimes stumble is in the area of a properly structured export with well thought out relationships between the templates. Today I want to provide an good example.

This post on building a NetFlow Cache and exporting IPFIX is pretty deep. For this reason, my prior post on Exporting NetFlow or IPFIX really should be reviewed first. A flow cache entry in a router or switch is built using the first packet between two hosts and the cache table is maintained for all active connections (i.e. flows). When a packet comes into the device, its tuple is compared to existing entries in the cache table. A match of the key fields triggers a flow entry update where packet, byte counts and perhaps other fields are incremented and updated. Packets that don’t match a flow entry are compared to policy (e.g. firewall or ACL rules) and are ultimately dropped or used to create new cache entries. Flow entries are exported to a flow collector periodically based on timers (I.e. Active Timeout) or flow behaviors.
When talking with a vendor about building out a flow cache, it is very important that we first establish what we want to export. For most vendor implementations, this decision is easy and they copy NetFlow v5. Some software developers however, want to take their flow export to another level and provide the very best in Network Traffic Monitoring metrics. To do this, a list of all the details that need to be exported should be created.

All of the data available in NetFlow v5
VoIP details on caller ID, Codec, Jitter and packet loss
Cloud service details on round trip time, URLs and deep packet inspection details that identify tricky applications like Skype, Webex and BitTorrent.
Interface names and speeds to avoid the reporting tools dependency on SNMP
Syslog and trap message details

The next step is to layout the IPFIX elements needed for each template. We have to use IPFIX because in all likelihood your company name isn’t Cisco and even if it is, the list includes elements that will end up being variable length fields such as URL and syslog details. These are, for all practical reasons, difficult to export using NetFlow v9. Let’s get started on IPFIX template creation and take the list above one item at a time. Each element name is preceded by an IANA or a dot separated vendor specific IE.

Be sure to take notice that IANA defined IEs are always the preference over vendor specific IEs. The vendor specific IEs are followed by the vendor Private Enterprise Number (PEN) (e.g. .32473).

A) Template: All of the data available in NetFlow v5/v9. This template is easy to define because all of the information necessary has been defined by IANA:

1 octetDeltaCount
2 packetDeltaCount
4 protocolIdentifier
5 ipClassOfService
6 tcpControlBits

Sent as 0 for non TCP traffic

7 sourceTransportPort
8 sourceIPv4Address
9 sourceIPv4PrefixLength
10 ingressInterface
11 destinationTransportPort
12 destinationIPv4Address
13 destinationIPv4PrefixLength
14 egressInterface
15 ipNextHopIPv4Address
16 bgpSourceAsNumber
17 bgpDestinationAsNumber
21 flowEndSysUpTime
22 flowStartSysUpTime
95 Application ID

This element causes another template below

148 Flow ID

This element is used to link to other templates

The above template is used for all protocols not sent by a more specific template. See below.

B) Template: VoIP details on caller ID, Codec, Jitter and packet loss. These elements are exported using a new template that contains meta data related to the flows exported above for the UDP real-time protocol (RTP). It contains the following IDs:

148 Flow ID

Links to the flow in A)

400.32473 Caller ID
401.32473 Codec
402.32473 Jitter
403.32473 Packet Loss

C) Template: Cloud service details on round trip time, URLs and deep packet inspection details that identify tricky applications like Skype and BitTorrent. These elements are exported using a meta data template similar to the one created in b). In this case however, the template is only used for TCP traffic and includes the following IDs:

148 Flow ID

Links to the flow in a)

501.32473 Round Trip Time
503.32473 URL

D) Template: Interface names and speeds to avoid the reporting tools dependency on SNMP:
- 10 ingressInterface
- 82 interfaceName
- 83 interfaceDescription
- 82.32473 ifSpeed

IMPORTANT: Always check with IANA for the IE before creating one with the company PEN! This helps guarantee wider acceptance from collectors built by different vendors and improves cross vendor reporting.

E) Template: Syslog and trap message details:
- 322 observationTimeSeconds
- 700.32473 Facility
- 701.32473 Severity
- 8 sourceIPv4Address
- 10 ingressInterface
- 12 destinationIPv4Address
- 704.32473 Message

F) Template: Application Name - This template provides the correlation between the application ID and the actual name of the application. There is an effort underway to standardize this export which would allow consistency across vendors.

95 Application ID

Links to the flow in a)

96 Application Name

With the templates above defined, it is time to find an IPFIX software solution that will export the data from your appliance. There are plenty of open source projects on the Internet that can do this. One favorite is IPFIXify which also takes care of the template refresh (e.g. every 1-5 minutes) that has to be reported periodically. IPFIXify also exports the RFC 5610 details. To learn more about the above process, consider purchasing my book on NetFlow and IPFIX titled Unleashing the Power of NetFlow and IPFIX.

Amazon EC2 Monitoring: Network Performance

2013-01-26T03:20:55Z

We recently did a cost analysis where we considered outsourcing to Amazon’s EC2 (Elastic Computing Cloud) service and the topic of network performance monitoring among other issues came up. We considered the amount of bandwidth we would use as well as how we would monitor the quality of service our customers were gaining through our use of EC2 and the final decision was that Amazon EC2 was not of us.

Amazon EC2 services allow companies to create a server hosted by Amazon and manipulate it pretty much as if it were your own. Once configured, the server can either sit on the internet for remote access or a VPN can be setup to allow the Amazon hosted server to sit on your internal network. It is a great concept. To monitor network performance to and from the hosted server, we planned on installing the nProbe to monitor TCP connection times to all of our customers. This would allow us to trend metrics on round trip times (RTT) and gain metrics on the Cisco IP SLA monitors used to verify acceptable connection times.

The nProbe, similar to Cisco’s performance monitoring Flexible NetFlow technology for medianets, watches TCP handshakes to provide metrics on RTT. Understandably this technology is not ideal for long lived TCP connections as a 2nd TCP hand shake doesn’t occur during a long lived connection. In our implementation however, nearly all connections are short lived hence a technology leveraging TCP handshakes to calculate a RTT works fine if not ideal because the RTT measured is of the actual end user connection and not a synthetic transaction like when using IP SLA monitors. There is a great white paper on calculating latency with NetFlow and TCP flags.

Imagine leveraging a cloud service like salesforce.com and being able to gain metrics on every TCP connection. You would be able to gain insight on which remote offices, subnets and individuals within those groups are seeing the worse connection times. Being a network traffic monitoring company, we have grown accustom to always having this type of insight at our finger tips. The more we looked at the Amazon EC2 service offering the more attractive it became until we started crunching the numbers.

Because we maintain a network traffic baseline of current traffic patterns, we were able to calculate the Amazon EC2 costs times our expected bandwidth use for an entire year. The final figures were in favor of us buying a bigger server and having it hosted at our local service provider. We learned that although Amazon’s Elastic Computing Cloud is an attractive offering for many applications, it was not ideal for our specific use. The bandwidth our server would use made the switch to Amazon EC2 cost prohibitive.

We will certainly consider this service in the future and you can bet when we do, the use of Cisco Performance Monitoring and or the nProbe will be part of the solution. Learn more by watching the Network Performance Monitoring webcast.

Migrating to Flexible NetFlow : BEST PRACTICES

2012-03-08T12:12:24Z

Migrating to Flexible NetFlow (FnF) is a simple and for some of us, an exciting process. It is exciting because of the deeper and improved insight into network traffic monitoring. If your company is interested in migrating to FnF from traditional NetFlow, there are a few things to consider.
The 5 steps to make this change are easy.

1)    List what business applications you want to monitor? Ask yourself: Do I want to measure performance to Cloud services such as salesforce.com, Skype or in house applications such as SAP?
2)    Decide what metrics you want to export. FnF can export MAC addresses, VLAN details and depending on the hardware, you can export entire packets, Cisco TrustSec, CoS, Jitter, packet loss, TCP latency, NAT details, NBAR, IP SLA, etc. Make sure your NetFlow Analyzer can filter and report on these details else, by our NetFlow collector!

3)    Make sure you are running the proper IOS version. Cisco IOS 15.2(2)T or more recent for the most capable Flexible NetFlow exports which combine Cisco NBAR and Cisco Medianet Performance Monitoring.
4)    Telnet to the router and disable traditional NetFlow and remove all of the commands from the router. Now try to forget about how to setup NetFlow. Commands like ip route-cache flow aren’t going to work anymore.
5)    Then, just setup Flexible NetFlow

Keep in mind that performing step 5 and configuring flexible netflow prior to step 4 will cause some duplicate flow exports. This is not something you want as it will over state utilization in the NetFlow reporting and it can also potentially cause problems for network security monitoring. Either way, it’s really a simple process and a huge value add to your overall network monitoring efforts. Don't forget to setup the new Cisco Performance Routing Flexible NetFlow exports. Call us if you need help.

Monitoring Video Performance with NetFlow

2012-02-19T07:41:55Z

Three years ago I was listening to John Chambers - CEO of Cisco Systems, proclaim that video was going to be the rage. I snickered and though it would be long time before anyone will be monitoring video performance with NetFlow. Well, here we are and John Chambers was right. My daughter is the one that made me realize why video vs. only voice will continue to grow.

Reporting on Skype with NetFlow
During a Skype connection with my daughter who was in my wife's car, my daughter wanted to show me her sneakers and then her book:

I noticed a big difference from when we just talked over the telephone. Seeing my face made her realize that I was fully engaged in what she had to say. She then put her jacket and shoes on and took the mobile phone outside to show me the fort she had built using scraps of wood. I couldn't believe it. She moved the camera in close for me to see things. She then brought be inside and put the phone in front of the dog so that I could say hello to 'Charlie'.

I have to admit, I liked the video especially since I was in London, England and my daughter was in Maine. What I didn't like was the jitter. I'm glad there are tools in our NetFlow traffic analyzer called Scrutinizer to monitor this.

Three years later at Cisco Live 2012 in London I was listening to Chief Cisco Futurist David Evans about the future of networking. I learned that video and data in general over the internet will continue to explode. This time I BELIEVE!

Cisco Performance Monitoring
Plixer was the first Cisco NetFlow Partner to become certified for Cisco Medianet Performance monitoring reports. Check out the VoIP jitter or lost packets in the network monitoring report below.

The above is VoIP with our Asterisk server. Skype traffic uses both TCP and UDP. We can measure the TCP latency during the connection setup with NetFlow Performance monitoring to look at Skype traffic as well. Today, customers can monitor cloud services with NetFlow. The example report below is filtering for the Cisco NBAR detected application: Skype.

Next Generation NetFlow
Keep in mind that these reports require the use of Flexible NetFlow which doesn't use the command ip route-cache flow. Make sure you are running IOS 15.2(2)T or more recent for the most capable Flexible NetFlow exports. The latest version provides even more network latency details than what is displayed above. I'm talking about Cisco IP SLA.

Performance Routing NetFlow
Cisco Performance Routing (PfR) can export IP SLA details using Flexible NetFlow. When a router determines that a connection is a bit congested, it will evaluate existing flows and reroute traffic over different connections ensuring priority to time sensitive traffic. By using PfR and Cisco Performance Monitoring together with Flow Hopper, administrators gain end to end network visibility on a link by link, hop by hop basis all with NetFlow.

The bottom line: Network traffic monitoring with NetFlow is at a whole new level from just two years ago. Join NetFlow Developments on Linkedin and stay on top of the future of NetFlow.