Thursday, June 15, 2017

Reading NSX VXLAN Encapsulated Frames in Wireshark

A question that frequently comes up in conversations with customers is the "loss of visibility" when moving from a traditional network implementation to an overlay model using VXLAN. This is a very valid concern and having been a network operator in my past, a concern I can really appreciate. Many times the need to do packet level analysis is required to help resolve an issue. Usually it ends up being a "See where your application rejects the syntax sent to it" more than a legitimate network issue, but since the network has the tools and visibility, the responsibility falls to them.

One of the more common tools is Wireshark so that's what we'll use in this example. We have a very simple topology that looks like this. The DC1-CentOS-01 machine is connected to a NSX provided logical switch, which uses VXLAN for the transport. We have a Distributed Logical Router (DLR) running OSPF to then connect to a NSX Edge that also uses OSPF to connect to a Cisco Catalyst 4948 and from there, the rest of the world.
I setup a SPAN session from the interface where the VXLAN traffic on the ESX host hits the network to my laptop. Nothing magical here, just the usual SPAN session.

I fired up Wireshark and selected my wired interface. As this is a trunk from my ESXi server and I use NFS for my file system, I saw a ton of traffic. I used the filtering capability in Wireshark to display the traffic with a source IP of 192.168.11.18, which is the VXLAN vmk of the ESXi host where the CentOS VM is running. Here's what we see - at first glance, promising!


Let's expand the headers so we can see the traffic encapsulated in VXLAN.

Oh, it must be in the data section.

Ummmm, now what?  Have no fear, the data is there, we just need to tell Wireshark to decode it properly.  Click on Analyze ----> Decode As

Click on the + in the lower left corner and let's fill in the blanks.

Now when you click on OK, it'll take you back to the trace file and check this out....we see a totally different view of the world.


I mentioned earlier we are running OSPF and there it is.  So what about traffic from the CentOS-01 VM?  Well, let's start something and see what we see.
That looks better. Yes, I know it's a cop out that I am just pinging Google - it's easy and still illustrates the point. Let's look at the headers now.
We can expand the VXLAN header and see the VNI assigned by NSX for that logical segment.

We can also see the original L2 frame and IP header.

Pretty cool, isn't it?

So with just a few clicks you are able to see inside VXLAN frames and not lose visibility for packet capture. Hopefully this was helpful.

The pcap file can be found here.

For completeness I used this version of Wireshark.

Wednesday, June 14, 2017

Chicago Super and Hurricane Heat 2017

Last weekend my son and I participated in the Spartan Race weekend in Chicago.
Our original goal was to run the Super, the mid-length race, on our journey to a double trifecta this year. I mentioned in my Ohio race post, my doctor recommended I not participate in the 4 hour endurance event called the Hurricane Heat due to an injury. Fast forward a few weeks and I feel much better so I signed us up for the one in Chicago. My son had a good time at the Ohio Hurricane Heat but I’ll be honest, I was a bit skeptical about how much “funishment” I could take. For example, in Ohio, they ended their event with doing 110 burpees as they were Hurricane Heat class 110. I was hoping for a different ending for our class, 113, and oh boy did we get it. 

We were up early Saturday to head to the Richmond Hunt Club where Spartan Race had taken over for the weekend. It doesn’t take long in the morning of a race to get cleaned up since you know you’re going to be sweating and muddy soon enough. The hotel said breakfast was open at 6:30 but we waited until 6:40 with no movement from the door to open and instead headed to Cracker Barrel. After a good breakfast with some carbs and coffee, we made our way to the parking area about 30 minutes away. Spartan had arranged for off-site parking at a nearby farm so we headed there, parked and rode a school bus to the race venue.

We went through registration, turned in our death waivers, and headed to the biggest team tent where our team, Corn Fed Spartans, were located. We dropped our gear and said Hi to a few people we knew and then headed to the merchandise tent to see if they have a venue specific shirt and a venue specific delta. These usually sell our quickly and we were glad we scored them. We stashed our purchases and wandered the venue to see what obstacles we would be facing and getting excited for the start of the race. We had registered for this event late last year and as such were in the first Open wave at 9AM, which was good since it was predicted to be a warm day.


We made our way to the starting line and if you have read my other posts, you know I dread the first wall. Nothing different for me in Chicago as my knees started to get weak and my head played games with me about failing to get over it on my own. Just as I was doubting myself, I tried and failed. This is getting pretty old and once again my son gave me a quick boost and I was over. I was pretty angry with myself and tried to change my focus to the 8 miles ahead of me. After the usual start line warm up and pep talk we were off.

My son took off and I headed out with the rest of the corral. Lo and behold, the first obstacle we came to were walls just like the one I failed. I ran up to the first one, jumped and was over in a flash and then proceeded to do the same for the 2nd one. This means it’s all in my head – I have to get that starting corral monkey off my back. Anyways, it was a classic Spartan course and my training throughout the year, and dropping 50lbs of weight so far, really paid off as I was running and having a ton of fun with the obstacles as they came up. The first part of the course was very dry and fast. The
heavy carries and leg strength obstacles were high points with upper arm obstacles providing more reminders to focus on them more otherwise, this’ll be a long summer. I helped people get up and over the high hurdles, 7’ wall and Stairway to Spartan and in turn took a boost from them as well. As usual, everyone was helpful and I hope I helped encourage the lady who was deathly afraid of the vertical cargo net.

Portions of the course were muddy, but compared to what I heard last year’s event had, we had it easy. The mud and water didn’t get too bad until closer to the end in what we were calling the Swamp. That was mid shin to knee deep black, smelly mud that was trying to rip your shoes off.  Some of the new obstacles like Bender, Olympus and Twister were great opportunities to practice my burpees as I failed each of them. The last stretch of obstacles was particularly challenging with Spear Throw (more burpees), A frame (no burpees), rope climb (burpees), multi-rig (burpees) and slip wall (no burpees) before the fire jump.

All in all, I was happy with my time as it was more than 4.5h faster than my Asheville Super time which is the only other Super I have done.  Granted, Asheville was a Championship Series race and had mountains, while this course was very, very flat. I collected my medal and met my son who had finished more than 1 hour earlier. Oh the joys of youth….


We hosed off the big chunks of dirt, took some pictures and talked with more Corn Fed friends before starting to think about lunch before the Hurricane Heat. The race venue is out in the country so not a lot of food options close by but we did find a Subway and I had the best Italian BMT ever made.  We also made sure we focused on hydration as the next event would be 4+ hours and it was a warm day.  We went back to the parking area and sat in the car for a while and I took a quick cat nap, as best you can in a car.

It was time to go back to the venue for the Hurricane Heat (HH). The HH is not a race but an event focused on teamwork and endurance. It’s led by Spartan Race employees called Krypteia who lead the event. Prior to the event they provide an equipment list you need to bring with you as well as any special items they might want. We had the standard gear list with our specialty items being a 5 gallon bucket and a regular towel. We brought our towels from home and on a weird side note, they are older than my son as I got them when I was leaving home to go to college. They seemed like they would be perfect ones to get muddy and use at an event like this. We had stopped at Lowe’s the night before and bought two 5 gallon buckets as we didn’t want to fly with them. In the email and video message before the event they also stress bringing your signed waiver along with your gear and be ready to roll before the official start. Being late, forgetting gear, or as we found, failing tasks, results in escalations in severity of the event.

We met at the Spartan Endurance rally point and watched as the group grew to 60+ people who had signed up. Our lead Krypteia was Kyoul Cha, a former Hot Shot (wilderness firefighter, all around bad ass and nice guy) and a secondary Krypteia, Danielle Rieck, a bad ass in her own right and Spartan Agoge finisher, full Delta finisher and is also very nice. We had a great leadership team and knew the event would be special.  We started by turning in our forms and getting checked in as well as learning the Warrior Ethos.

The Warrior Ethos has 4 points:
1.       I will always place the mission first
2.       I will never accept defeat
3.       I will never quit
4.       I will never leave a fallen comrade behind

These sound like platitudes / “rah-rah” talk at 4:30PM but become very real for me in a few hours.
Our Krypteia had a full event planned and we started with a warm up to one of Kyoul’s favorite musicians, Steve Aoki and Afrojack – video here. It was a good warm up for the rest of the event as the lyrics mention “I’m not afraid” multiple times.  We then proceeded to part of the festival area where we participated in a few team exercises like “Tunnel of Love” and “Conveyor Belt 2.0.” We really seemed to struggle with following instructions and working together as a team so the we hit escalations which is a nice way of saying that they event got harder. We also had people show up 45 minutes late and an astonishing 75 minutes late. We also had people show up without the right gear and after the event when watching some of the video on Facebook I learned we were the first class Kyoul has led that hit escalations as quickly as we did and as frequently as we did. I guess we were good at something, but it wasn’t working together.  😉

We shifted gears after “Conveyor Belt 2.0” and made our way to the pond for some fun with water. We did “Pyramid Pirouette” as well as bucket drags with our towels and 3 person lunges with buckets of water. We also did an activity where we passed buckets of water continuously in a circle. Neither of these were shining examples of team work so they escalated again where the last time we were passing buckets while in a squatting position. I will say the smaller group I was in for some of the events did a really good job, but the one my son was in sounded pretty shaky.  I’ll leave it at that and if you keep reading, you’ll know why I don’t want to throw stones while living in a glass house.

Our final event was called “Couples Therapy” where  we used our towels to tie ourselves together at the foot and wrist. I was paired with a woman I had been standing next to during the bucket passing activity. She has an interesting job and I’ll not use her name to help provide privacy which could be important for her safety. Once bound together we were to bear crawl an undetermined distance. I really struggle with bear crawling between poor upper body strength and achy knees it’s not a favorite for any distance. We did pretty good for a while but as my shoulders were getting gassed I had to resort to crawling on my hands and knees. This is a dirt and gravel road so it was pretty painful but I didn’t want to give up (see the Warrior’s Ethos above). We were making progress but it was clear we were falling behind. My partner was very supportive and encouraging while I literally crawled along with her. Danielle noticed we were falling behind and told us to get up and go be the pacers for the team.  This was a boon and a curse at the same time for me. It was great my partner and I didn’t have to crawl to the front but it also meant the entire HH saw us go up front and then have to be slowed down by me.

Getting into my a head for a minute (I know – It’s scary in here but I won’t take you too far) this was one of the most humbling experiences I have had. Knowing that I was the reason the team was being forced to move slow which just prolongs everyone else’s pain is *way* outside my comfort zone. I am used to doing a good job at whatever I do and then to absolutely suck at this task was distressing. I have thought a lot about this since Saturday night and this had the opportunity to be not a humbling experience, which in general, can be a good thing, but rather a humiliating experience. This is where Warrior’s Ethos #4 comes into play. My partner was not berating me for being horrible at bear crawling but rather being very supportive. Words of encouragement and understanding from her and the rest of the team were beyond helpful. It was a crazy mix of frustration, embarrassment, humble pie and plenty of pain for me. I am sure some people were cursing me, but I didn’t hear them so it doesn’t count.  😉

The sun was dipping into the horizon when Kyoul told us all to get up and line up for a group photo. We had survived the Hurricane Heat!  We took the team photos and made out way back to the rally point, which ironically is where we were bear crawling to, but didn’t make it that far. We lined up again according to t-shirt size and were given our dog tags and Spartan Endurance shirts. My bear crawl partner and one of my bucket partners all took a selfie together and they both again were very supportive. I honestly figured they’d never want to see me again, but they are the exact examples of why I love this sport of Obstacle Course Racing.  My son rocked the event and had a great time even with some of the team challenges he had.

We boarded the bus and headed back to the parking lot to then drive back to the hotel. Showers were the order of the day before some pizza and “Talladega Nights” on TV.  We didn’t make it very far before sleep took over and we knew we had to get up to fly home the next morning.

One of the things I was hoping to determine from the HH was my readiness for the 12-hour Hurricane Heat. My son is signed up for the one in Palmerton, PA in July and I was kicking around joining him. I’ve waffled back and forth on this since Saturday and have decided I am physically not ready for the 12-hour event – yet. I will do one but I need to ramp my fitness levels beyond what they are today. I am gearing up for the Ultra Beast in Dallas and this was a solid gut-check that I need to get focused.
Would I do another HH again, yes indeed!  Every event is different and Kyoul said this was one of the most challenging one he has led so I am proud I survived both physically and mentally. Do it even if you think you are not ready – test yourself and push.


My next Spartan Race is in July at Fort Knox and I have some unfinished business with a starting wall.  Hope to see you there!

I will put some HH pics up once we get them, should be full of fun and laughs. 

Twitter Delicious Facebook Digg Stumbleupon Favorites More