Thursday, June 15, 2017

Reading NSX VXLAN Encapsulated Frames in Wireshark

A question that frequently comes up in conversations with customers is the "loss of visibility" when moving from a traditional network implementation to an overlay model using VXLAN. This is a very valid concern and having been a network operator in my past, a concern I can really appreciate. Many times the need to do packet level analysis is required to help resolve an issue. Usually it ends up being a "See where your application rejects the syntax sent to it" more than a legitimate network issue, but since the network has the tools and visibility, the responsibility falls to them.

One of the more common tools is Wireshark so that's what we'll use in this example. We have a very simple topology that looks like this. The DC1-CentOS-01 machine is connected to a NSX provided logical switch, which uses VXLAN for the transport. We have a Distributed Logical Router (DLR) running OSPF to then connect to a NSX Edge that also uses OSPF to connect to a Cisco Catalyst 4948 and from there, the rest of the world.
I setup a SPAN session from the interface where the VXLAN traffic on the ESX host hits the network to my laptop. Nothing magical here, just the usual SPAN session.

I fired up Wireshark and selected my wired interface. As this is a trunk from my ESXi server and I use NFS for my file system, I saw a ton of traffic. I used the filtering capability in Wireshark to display the traffic with a source IP of, which is the VXLAN vmk of the ESXi host where the CentOS VM is running. Here's what we see - at first glance, promising!

Let's expand the headers so we can see the traffic encapsulated in VXLAN.

Oh, it must be in the data section.

Ummmm, now what?  Have no fear, the data is there, we just need to tell Wireshark to decode it properly.  Click on Analyze ----> Decode As

Click on the + in the lower left corner and let's fill in the blanks.

Now when you click on OK, it'll take you back to the trace file and check this out....we see a totally different view of the world.

I mentioned earlier we are running OSPF and there it is.  So what about traffic from the CentOS-01 VM?  Well, let's start something and see what we see.
That looks better. Yes, I know it's a cop out that I am just pinging Google - it's easy and still illustrates the point. Let's look at the headers now.
We can expand the VXLAN header and see the VNI assigned by NSX for that logical segment.

We can also see the original L2 frame and IP header.

Pretty cool, isn't it?

So with just a few clicks you are able to see inside VXLAN frames and not lose visibility for packet capture. Hopefully this was helpful.

The pcap file can be found here.

For completeness I used this version of Wireshark.

Wednesday, June 14, 2017

Chicago Super and Hurricane Heat 2017

Last weekend my son and I participated in the Spartan Race weekend in Chicago.
Our original goal was to run the Super, the mid-length race, on our journey to a double trifecta this year. I mentioned in my Ohio race post, my doctor recommended I not participate in the 4 hour endurance event called the Hurricane Heat due to an injury. Fast forward a few weeks and I feel much better so I signed us up for the one in Chicago. My son had a good time at the Ohio Hurricane Heat but I’ll be honest, I was a bit skeptical about how much “funishment” I could take. For example, in Ohio, they ended their event with doing 110 burpees as they were Hurricane Heat class 110. I was hoping for a different ending for our class, 113, and oh boy did we get it. 

We were up early Saturday to head to the Richmond Hunt Club where Spartan Race had taken over for the weekend. It doesn’t take long in the morning of a race to get cleaned up since you know you’re going to be sweating and muddy soon enough. The hotel said breakfast was open at 6:30 but we waited until 6:40 with no movement from the door to open and instead headed to Cracker Barrel. After a good breakfast with some carbs and coffee, we made our way to the parking area about 30 minutes away. Spartan had arranged for off-site parking at a nearby farm so we headed there, parked and rode a school bus to the race venue.

We went through registration, turned in our death waivers, and headed to the biggest team tent where our team, Corn Fed Spartans, were located. We dropped our gear and said Hi to a few people we knew and then headed to the merchandise tent to see if they have a venue specific shirt and a venue specific delta. These usually sell our quickly and we were glad we scored them. We stashed our purchases and wandered the venue to see what obstacles we would be facing and getting excited for the start of the race. We had registered for this event late last year and as such were in the first Open wave at 9AM, which was good since it was predicted to be a warm day.

We made our way to the starting line and if you have read my other posts, you know I dread the first wall. Nothing different for me in Chicago as my knees started to get weak and my head played games with me about failing to get over it on my own. Just as I was doubting myself, I tried and failed. This is getting pretty old and once again my son gave me a quick boost and I was over. I was pretty angry with myself and tried to change my focus to the 8 miles ahead of me. After the usual start line warm up and pep talk we were off.

My son took off and I headed out with the rest of the corral. Lo and behold, the first obstacle we came to were walls just like the one I failed. I ran up to the first one, jumped and was over in a flash and then proceeded to do the same for the 2nd one. This means it’s all in my head – I have to get that starting corral monkey off my back. Anyways, it was a classic Spartan course and my training throughout the year, and dropping 50lbs of weight so far, really paid off as I was running and having a ton of fun with the obstacles as they came up. The first part of the course was very dry and fast. The
heavy carries and leg strength obstacles were high points with upper arm obstacles providing more reminders to focus on them more otherwise, this’ll be a long summer. I helped people get up and over the high hurdles, 7’ wall and Stairway to Spartan and in turn took a boost from them as well. As usual, everyone was helpful and I hope I helped encourage the lady who was deathly afraid of the vertical cargo net.

Portions of the course were muddy, but compared to what I heard last year’s event had, we had it easy. The mud and water didn’t get too bad until closer to the end in what we were calling the Swamp. That was mid shin to knee deep black, smelly mud that was trying to rip your shoes off.  Some of the new obstacles like Bender, Olympus and Twister were great opportunities to practice my burpees as I failed each of them. The last stretch of obstacles was particularly challenging with Spear Throw (more burpees), A frame (no burpees), rope climb (burpees), multi-rig (burpees) and slip wall (no burpees) before the fire jump.

All in all, I was happy with my time as it was more than 4.5h faster than my Asheville Super time which is the only other Super I have done.  Granted, Asheville was a Championship Series race and had mountains, while this course was very, very flat. I collected my medal and met my son who had finished more than 1 hour earlier. Oh the joys of youth….

We hosed off the big chunks of dirt, took some pictures and talked with more Corn Fed friends before starting to think about lunch before the Hurricane Heat. The race venue is out in the country so not a lot of food options close by but we did find a Subway and I had the best Italian BMT ever made.  We also made sure we focused on hydration as the next event would be 4+ hours and it was a warm day.  We went back to the parking area and sat in the car for a while and I took a quick cat nap, as best you can in a car.

It was time to go back to the venue for the Hurricane Heat (HH). The HH is not a race but an event focused on teamwork and endurance. It’s led by Spartan Race employees called Krypteia who lead the event. Prior to the event they provide an equipment list you need to bring with you as well as any special items they might want. We had the standard gear list with our specialty items being a 5 gallon bucket and a regular towel. We brought our towels from home and on a weird side note, they are older than my son as I got them when I was leaving home to go to college. They seemed like they would be perfect ones to get muddy and use at an event like this. We had stopped at Lowe’s the night before and bought two 5 gallon buckets as we didn’t want to fly with them. In the email and video message before the event they also stress bringing your signed waiver along with your gear and be ready to roll before the official start. Being late, forgetting gear, or as we found, failing tasks, results in escalations in severity of the event.

We met at the Spartan Endurance rally point and watched as the group grew to 60+ people who had signed up. Our lead Krypteia was Kyoul Cha, a former Hot Shot (wilderness firefighter, all around bad ass and nice guy) and a secondary Krypteia, Danielle Rieck, a bad ass in her own right and Spartan Agoge finisher, full Delta finisher and is also very nice. We had a great leadership team and knew the event would be special.  We started by turning in our forms and getting checked in as well as learning the Warrior Ethos.

The Warrior Ethos has 4 points:
1.       I will always place the mission first
2.       I will never accept defeat
3.       I will never quit
4.       I will never leave a fallen comrade behind

These sound like platitudes / “rah-rah” talk at 4:30PM but become very real for me in a few hours.
Our Krypteia had a full event planned and we started with a warm up to one of Kyoul’s favorite musicians, Steve Aoki and Afrojack – video here. It was a good warm up for the rest of the event as the lyrics mention “I’m not afraid” multiple times.  We then proceeded to part of the festival area where we participated in a few team exercises like “Tunnel of Love” and “Conveyor Belt 2.0.” We really seemed to struggle with following instructions and working together as a team so the we hit escalations which is a nice way of saying that they event got harder. We also had people show up 45 minutes late and an astonishing 75 minutes late. We also had people show up without the right gear and after the event when watching some of the video on Facebook I learned we were the first class Kyoul has led that hit escalations as quickly as we did and as frequently as we did. I guess we were good at something, but it wasn’t working together.  😉

We shifted gears after “Conveyor Belt 2.0” and made our way to the pond for some fun with water. We did “Pyramid Pirouette” as well as bucket drags with our towels and 3 person lunges with buckets of water. We also did an activity where we passed buckets of water continuously in a circle. Neither of these were shining examples of team work so they escalated again where the last time we were passing buckets while in a squatting position. I will say the smaller group I was in for some of the events did a really good job, but the one my son was in sounded pretty shaky.  I’ll leave it at that and if you keep reading, you’ll know why I don’t want to throw stones while living in a glass house.

Our final event was called “Couples Therapy” where  we used our towels to tie ourselves together at the foot and wrist. I was paired with a woman I had been standing next to during the bucket passing activity. She has an interesting job and I’ll not use her name to help provide privacy which could be important for her safety. Once bound together we were to bear crawl an undetermined distance. I really struggle with bear crawling between poor upper body strength and achy knees it’s not a favorite for any distance. We did pretty good for a while but as my shoulders were getting gassed I had to resort to crawling on my hands and knees. This is a dirt and gravel road so it was pretty painful but I didn’t want to give up (see the Warrior’s Ethos above). We were making progress but it was clear we were falling behind. My partner was very supportive and encouraging while I literally crawled along with her. Danielle noticed we were falling behind and told us to get up and go be the pacers for the team.  This was a boon and a curse at the same time for me. It was great my partner and I didn’t have to crawl to the front but it also meant the entire HH saw us go up front and then have to be slowed down by me.

Getting into my a head for a minute (I know – It’s scary in here but I won’t take you too far) this was one of the most humbling experiences I have had. Knowing that I was the reason the team was being forced to move slow which just prolongs everyone else’s pain is *way* outside my comfort zone. I am used to doing a good job at whatever I do and then to absolutely suck at this task was distressing. I have thought a lot about this since Saturday night and this had the opportunity to be not a humbling experience, which in general, can be a good thing, but rather a humiliating experience. This is where Warrior’s Ethos #4 comes into play. My partner was not berating me for being horrible at bear crawling but rather being very supportive. Words of encouragement and understanding from her and the rest of the team were beyond helpful. It was a crazy mix of frustration, embarrassment, humble pie and plenty of pain for me. I am sure some people were cursing me, but I didn’t hear them so it doesn’t count.  😉

The sun was dipping into the horizon when Kyoul told us all to get up and line up for a group photo. We had survived the Hurricane Heat!  We took the team photos and made out way back to the rally point, which ironically is where we were bear crawling to, but didn’t make it that far. We lined up again according to t-shirt size and were given our dog tags and Spartan Endurance shirts. My bear crawl partner and one of my bucket partners all took a selfie together and they both again were very supportive. I honestly figured they’d never want to see me again, but they are the exact examples of why I love this sport of Obstacle Course Racing.  My son rocked the event and had a great time even with some of the team challenges he had.

We boarded the bus and headed back to the parking lot to then drive back to the hotel. Showers were the order of the day before some pizza and “Talladega Nights” on TV.  We didn’t make it very far before sleep took over and we knew we had to get up to fly home the next morning.

One of the things I was hoping to determine from the HH was my readiness for the 12-hour Hurricane Heat. My son is signed up for the one in Palmerton, PA in July and I was kicking around joining him. I’ve waffled back and forth on this since Saturday and have decided I am physically not ready for the 12-hour event – yet. I will do one but I need to ramp my fitness levels beyond what they are today. I am gearing up for the Ultra Beast in Dallas and this was a solid gut-check that I need to get focused.
Would I do another HH again, yes indeed!  Every event is different and Kyoul said this was one of the most challenging one he has led so I am proud I survived both physically and mentally. Do it even if you think you are not ready – test yourself and push.

My next Spartan Race is in July at Fort Knox and I have some unfinished business with a starting wall.  Hope to see you there!

I will put some HH pics up once we get them, should be full of fun and laughs. 

Friday, May 26, 2017

Ohio Beast/Sprint Weekend 2017

In keeping with what I want to be become a habit, here’s a bit of a rundown of last weekend (5/19-21) and what we did at the Ohio Beast/Sprint weekend. We have mapped out our race schedule early in the year and even though this is our “home” race we decided to mix it up a bit and not run but rather explore new things. This meant the Kid’s Race for our 3 that are not old enough to participate in the full Spartan races, volunteering at the race and then the Spartan Endurance event, Hurricane Heat.

For the TL;DR crowd – we had fun and got very muddy. 10/10 would do it again.  😉

The whole weekend revolved around Spartan Race and driving back and forth to Chandlersville, OH which is about 1h45m from home. My son and I signed up to be volunteers and this was new for us. We received the email notifying us so sign up for times to volunteer a few weeks before the race. I was hoping we could volunteer to help with course setup as I think it would be neat to see some of the logistics behind a Spartan Race but they require you to be 18. I meet that requirement (by a few years!) but my son does not.  We tossed around different ideas and settled on the Course Marshal position where we would be recording the Elite athletes and verifying their burpee penalty if they failed the obstacle. We could easily do this job Sunday AM and then participate in the Hurricane Heat later that afternoon so it sounded like a perfect opportunity!

One caveat for the Course Marshal position is that we needed to attend training Friday afternoon, but you get a free race *and* a hoodie. We made the trek to the race site and met up with Alex who showed us the camera and how to use it. It was simple to do and seemed easy enough. All in all, training took maybe 20 minutes and much of that was standing around chit-chatting with the other volunteers. We drove home and were excited about the weekend ahead.

Saturday came and today was all about the kids. We packed into the van and headed to the course. For our youngest 3, this would be their first Spartan Race. They have never even been to a course because there really isn’t much for a spectator to do while your racers are out and about. It was foggy and overcast with the threat of rain all weekend but it’s a Spartan Race so mud is part of the fun. The youngest two have done plenty of 5Ks and the oldest of the three has also done some 10k trail races so they were well prepared.  We checked in and walked around the festival area while waiting for their wave to go out onto the course.  We watched the Beast runners start a few waves and then walked over to some of the other obstacles and watched people take a crack at them.

Once it was finally time, they loaded into their starting corral (complete with wall to climb over!!) and waited the last few minutes. This is where the local Spartan Race team really did a good job. The Beast runners were maybe 20 feet away from the Kid’s Race starting corral so the starter for the Beast had them “coach” the kids on how to start and say AROO. The kids really liked this and helped make them feel like they were getting ready to do something epic (which they were). They got to listen to the usual Spartan starting speech (check it out here if you’ve not heard it) and watch the 10AM beast wave go out. Additionally, this was the official Corn Fed Spartans wave and there must have been 100 Corn Fed in the corral and our kids were sporting their jerseys.  Jen from Spartan Race, was the starter for the Kid’s Race and she got them loose and warmed up with some light exercises before sending them off on their 1 mile race.

Spartan Race does a great job of making the Kid’s Race similar to the regular races and they had an A-Frame, inverted wall, regular wall, spear throw, mud pits, mud crawl, balance beams and more around the 1 mile, undulating course. Our kids had a great time and did amazing with helping each other over the obstacles and being true Corn Fed Spartans. They finished the race and collected their finisher medals and enjoyed a post-race snack. Afterwards we walked around a lot more and went to watch some of the new obstacles like Olympus and some classics like the multi-rig and sandbag carry.  We called it a day and drove home.

The next day my son and I were up at 3:45AM to head back to the course to start our volunteer shift at 5:30AM. We made great time as there wasn’t a lot of traffic on the roads for some reason. Ha!  We checked in at the volunteer tent to get our assignment which was spear throw. Spear throw is one of the most failed obstacles on the course and is considered by many to be a burpee machine. Our job would be to record all of the Elite male racers, record their bib number if they failed and then later, count their burpees. We would stop counting the men once the first women Elite showed up and then record 30 of them before packing up and heading back to review the footage.

We hopped in the ATV that would take us out to our obstacle and away we went. The course was super muddy from the rain and pounding that a few thousand Beast runners gave it and we were slipping and sliding all over.  During the ride my cell phone fell out of my pocket but we luckily found it.  I was afraid it fell into some of the soupy mud and was gone for good. We got dropped off at the spear throw and mounted the camera and got it in position and then killed some time working on our spear throw technique. It’s not often you get a chance to practice on a real setup as you get one shot during the race and if you miss – 30 burpees.  We also walked down to one of the new obstacles, Twister. This one looks like a burpee machine to me as well!

As it got closer to race time we went back to our station and awaited the first runners. It didn’t take too long for us to hear them pounding down the trail and the leaders all stuck their throws – whoo hoo!  After that, chaos ensued as groups of runners arrived, some missing and some hitting. We got the runner’s bib number either as they went by or while they were doing their burpees. I would write their number down on a white board, show it to the camera and get the next one. It was crazy busy and at times we had 9 or 10 runners grinding out their penalties.  It was maybe half an hour before the first ladies came through. As with the men, the first runners made their throws and then the burpees stacked up.  We stopped collecting men’s bib numbers and focused on the women and started counting them as they came through. Once we hit 30, we turned off the camera and hitched a ride back to base camp.

We then started the tedious process of counting each burpee to make sure the runners completed their penalty. This wasn’t too bad when there were only a few but as the number of people entering and leaving the burpee zone increased it became confusing.  We had to rewind and review multiple times since the penalty for not doing 30 burpees is a 30 second penalty for each one missed!  Ouch! The Elites are running for money and points in the standings so it is no joke and we took our job very seriously. All in all we only had to report 3 men for penalties – the women were all legit at 30 or more.  
We finished our shift around 10:00 AM and we needed to kill some time until the 2PM start of the Hurricane Heat. We walked over to the Biggest Team tent, which was awarded to the Crazy Mudder Muckers and they were kind enough to share with the Corn Fed Spartans. It was nice to talk with people as they prepared to go out for their race or as they came in.  The Hurricane Heat (HH) is a 4+ hour Spartan Endurance event that is not a race. Its focus is on teamwork and the Warrior Ethos. Both my son and I were signed up for it, but after a visit to the Dr. Friday morning, I had to sit it out due to a pulled abdominal muscle. I was bumming that I couldn’t get in on the fun, but knew it made sense for the long term as we have a busy race schedule now that summer is here.  I figured while I was there and my son was doing the HH I would see if I could pickup another volunteer shift.

Part of the HH is designed to be a bit of mystery and being late to the event results in group punishment (50 burpees for the 1st late person, 100 for the 2nd and so on) so the HH participants started to gather around 12:30 for gear check and to make sure everyone was ready. My son was eager to join the group so he went to have some fun in the mud and I went to the volunteer tent. I’ve heard there is no shortage of need for volunteers and this held true so I signed back in to work at bag check.
Bag check was a lot more fun than I thought it would be as we got to see many of the racers as they came off the course and congratulate them. For quite a few it was their first Spartan Race ever and they loved it and for a very small few, they were convinced they’d never do one again. When we were not getting people’s bags we stood around in the rain and talked with the other volunteers. I talked with a lady from New Hampshire who had competed in the “Tough Guy” OCR in England as well as 3 or 4 years of various Spartan Races.  Another volunteer was in her first year of racing and everything in between.  It was fun to hear what other people have done and what they do for a day job.  OCR has all kinds of people that participate in it.

As the racers found their way through the course, we were asked to help tear down the festival area. This is a pretty daunting task but as the saying goes, many hands make light work.  We all worked well as a team and tore our area down. As we were finishing up, we were asked to go to the finish line to cheer on the final group of racers. It was awesome to see this group of 8 people help and push each other through the last few obstacles and then cross the finish together.  They were having a great time and they had a lot of volunteers cheering them on.  Once they got their medals it was back to work for us. We finished the final tear down just in time to see the HH group come back from their fun in the woods and do 110 burpees (it was HH class 110 so that’s where the number came from).  They were caked in mud and smiles all around when they were finished. 

My son and I stood around for a bit and talked with some Corn Fed who were still there before walking to the car and heading home.  It was quite a weekend and different from one in where we’ve raced but it was nice to see Spartan from a different perspective. We met a lot of really neat people and met many of the people we talk with on social media about OCR as well.  I am looking forward to the next race in Chicago where I’ll be running for my first Super of 2017 on my way to a double trifecta.  Aroo!

Tuesday, April 25, 2017

Adventures in Getting Started With PowerShell

This might be a bit below many of you and if so, feel free to laugh at my newness to PowerShell. I am beginning my journey in becoming “API-enabled” with a focus on VMware products, specifically VMware NSX, and from talking with customers and peers, PowerShell seemed like a great place to start.  I saw on Twitter that we have released a new PowerCLI this week so I figured let’s start with that. 

My home lab has a Windows 2012R2 server that I use for my admin/jumpbox so I started a RDP session, opened a PowerCLI window and started to follow along on the blog.  After failing with the first command, I knew I was in for a great learning experience and started this blog post.

A quick search online and I see I may not really have PowerShell installed or it is woefully out of date.  I’ve downloaded the RTM and installed it per the instructions on this MSDN article.  So after a reboot, let’s try it again. Much better!

Next, I needed to import the VMware PowerCLI module and it asked me to participate in the Customer Experience Improvement Program (CEIP).

I chose to participate as I am just a guy blonking around on a keyboard and not playing with production or customer workloads.  I also followed the instructions to create a shortcut on my desktop that would automatically load the PowerCLI module every time I open the window.  Simple, right?  Like I said, I am getting started so apologize if this comes off as a “Color by Numbers” approach – it’s what I need.

So now that I have PowerCLI installed on my host, where do I go next?  Well, I want PowerNSX so I start at GitHub and read this post.  I follow along and paste the string to install it and it’s obvious from the long, complex string I have a lot of things to learn (Good!)

You can see there is a bit of an issue, probably related to the new installation of PowerCLI, so let’s open an Issue.  Lo and behold, my colleague in the UK has already found this issue and the
PowerNSX team have already provided a new one-liner to fix it.

Armed with the new string, it seems much happier.

Now I have PowerNSX and am ready to be dangerous.  We’ll save the fun stuff for the next post!

Note this was originally posted on 

Thursday, March 30, 2017

The Adventures of Miles the United Polaris Bear

I'm on vacation and thought I'd have some fun with a stuffed bear I got when I flew in United's new business class service, called Polaris.  The service itself was amazing and the crew was fantastic. I couldn't have asked for a better 15+ hr flight.  That said, I already let the crew know and even emailed the 1K team to have them pass on the kudos so they get some recognition.

With the help of Blake Krone from Twitter, the bear has been dubbed Miles and these are his adventures in Sydney, New Zealand and who knows where else.

Tuesday, March 28, 2017

VMworld 2017 Session Voting Is Live

It's that time of year again, where you get to play a unique role in deciding what content and speakers you want to see at VMworld 2017. This is one of the most unique attributes VMworld offers in my opinion and allows you, the customer and attendee, to help shape your conference experience.

Wednesday, March 22, 2017

vRNI Next Steps - Adding Data Sources

A few weeks months ago I did a blog post on how to install VMware vRealize Network Insight 3.2. It has been a busy beginning of the year for me and I am now ready to share the next step in getting vRNI setup , adding Data Sources.  Data Sources in vRNI are where vRNI starts to gain intelligence about the topology, workloads and traffic.

Twitter Delicious Facebook Digg Stumbleupon Favorites More