Reading NSX VXLAN Encapsulated Frames

A question that frequently comes up in conversations with customers is the "loss of visibility" when moving from a traditional network implementation to an overlay model using VXLAN. This is a very valid concern and having been a network operator in my past, a concern I can really appreciate. Many times the need to do packet level analysis is required to help resolve an issue. Usually it ends up being a "See where your application rejects the syntax sent to it" more than a legitimate network issue, but since the network has the tools and visibility, the responsibility falls to them.

One of the more common tools is Wireshark so that's what we'll use in this example. We have a very simple topology that looks like this. The DC1-CentOS-01 machine is connected to a NSX provided logical switch, which uses VXLAN for the transport. We have a Distributed Logical Router (DLR) running OSPF to then connect to a NSX Edge that also uses OSPF to connect to a Cisco Catalyst 4948 and from there, the rest of the world.

I setup a SPAN session from the interface where the VXLAN traffic on the ESX host hits the network to my laptop. Nothing magical here, just the usual SPAN session.

I fired up Wireshark and selected my wired interface. As this is a trunk from my ESXi server and I use NFS for my file system, I saw a ton of traffic. I used the filtering capability in Wireshark to display the traffic with a source IP of 192.168.11.18, which is the VXLAN vmk of the ESXi host where the CentOS VM is running. Here's what we see - at first glance, promising!

Let's expand the headers so we can see the traffic encapsulated in VXLAN.

Oh, it must be in the data section.

Ummmm, now what?  Have no fear, the data is there, we just need to tell Wireshark to decode it properly.  Click on Analyze ----> Decode As

Click on the + in the lower left corner and let's fill in the blanks.

Now when you click on OK, it'll take you back to the trace file and check this out....we see a totally different view of the world.

I mentioned earlier we are running OSPF and there it is.  So what about traffic from the CentOS-01 VM?  Well, let's start something and see what we see.

That looks better. Yes, I know it's a cop out that I am just pinging Google - it's easy and still illustrates the point. Let's look at the headers now.

We can expand the VXLAN header and see the VNI assigned by NSX for that logical segment.

We can also see the original L2 frame and IP header.

Pretty cool, isn't it?

So with just a few clicks you are able to see inside VXLAN frames and not lose visibility for packet capture. Hopefully this was helpful.

The pcap file can be found here.

For completeness I used this version of Wireshark.