Technical Insights > Network Visibility + Security

Cool Packet Broker Tricks - Deduplication

2020-11-17  |  5 min read 

One of the dirty secrets of networking is that there is usually a fair amount of duplicate traffic on a network, particularly when network visibility solutions are in play. This may seem counterintuitive as network visibility solutions are intended to solve visibility challenges, not create them, but the sad fact of the matter is that some degree of duplication is probably unavoidable in most networks. The good news is that it is possible to dedupe traffic and that doing so will not only help prevent issues with tools that struggle with duplicate traffic but can also help you get a lot more out of those tools by reducing the traffic (by eliminating dupes) that you send.

Where do dupes come from?

Duplicate Hipsters in the livingroom - a problem that unlike duplicate packets on the network, a network packet broker will not help
While useful when dealing with duplicate packets on the network, a network packet broker
will be less successful in dealing with duplicate hipsters in your living room

There are a number of ways that you can get duplicate packets in a network. One way is to have taps in multiple locations in the network. For example, you may tap on the edge and tap on the core. Traffic traversing both will be duplicated. Same for traffic across multiple VLANs. SPAN is another source of duplication. Typically you will see both traffic coming in and traffic going out of a particular switch getting duplicated in a SPAN port scenario, meaning that most traffic by default will be duplicated. There may be other causes as well.

The Importance of Deduplication Performance

We recently ran some tests with Tolly where we compared the Vision X with the Gigamon GigaVUE HC-3. One of the surprises we saw was that the Gigamon network packet broker dropped traffic when asked to dedupe. Considering that the entire purpose of network packet brokers is to enhance visibility and not create blind spots, this was a surprise.

Tested Deduplication Performance - Gigamon Gigavue HC-3 vs Keysight Vision X

The Right Hardware Architecture

Due to the importance of network visibility, we took care when doing the hardware architecture and design of the Vision X, leveraging decades of experience as the source of truth in network performance for the biggest network equipment makers and service providers in the business. With Vision X a couple of the more important moves we made here were to have our modules share pipelines. Both Vision X and the GigaVUE HC-3 use the Broadcom Tomahawk II chipset – a great choice for high performance. Both support four pipelines and four 32 port modules. One of the first big differences is that we have each module share two pipelines – giving better scalability and burstability. Another things we did was to include FPGAs. Just like the folks building superfast switches for financial markets, we figure performance of a packet broker is too important to leave to chance and that it was worth the effort and expense to build hardware acceleration into the platform. Picture the difference in performance between a business laptop and a gaming rig with a high end graphics card.

More on Dupes

The Keysight product management team has put together a great set of Visibilty Tech Tip videos – which we encourage you to visit. More specific to the situation here with dupes, our own Patrick Mccabe has done a couple vids:

Deduplication with Vision Network Packet Brokers – Part 1, Theory

Deduplication with Vision Network Packet Brokers – Part 2, Configuration