Sunday, March 20, 2011

OTV Deep Dive - Part 3

After a long delay, let's pick up where we left off last with our OTV deep dive. This post will focus on a key concept with OTV that is critical to understand. We'll examine how we localize our First Hop Redundancy Protocols (FHRPs). These protocols are Host Standby Routing Protocol (HSRP v1 and v2) Virtual Router Redundancy Protocol (VRRP), and Gateway Load Balancing Protocol (GLBP). These protocols allow two network devices to share a common IP address to be used as the default gateway on a subnet and provide redundancy and load balancing to clients in that subnet.
Before we can discuss FHRP localization, let's review why this might be significant to our design. Typically with FHRPs the members of the group are local to each other both logically and physically. Depending on the FHRP there is load balancing or redirection between the devices to the "active" member to handle traffic. This works well when considered locally and most of us use it without a second thought.
When we start to stretch or extend our VLANs across distances, latency is introduced. While a 1ms one-way latency may not sound significant, when accumulated over a complete flow or transaction, it can become quite detrimental to performance. This is exacerbated if the two devices are both in the same location, but have default gateways in another data center. Sub optimal switching and routing at its finest. This effect is referred to as tromboning traffic and is illustrated below where device A needs to talk with device B and the default gateway resides across a stretched VLAN.











We address this with OTV by implementing filters to prevent the FHRP peers in each opposite data centers from seeing each other and therefore becoming localized. There are two approaches to do this, one using a MAC access list which we won't cover, and the other, recommended one is via an IP ACL that is applied as a VLAN ACL (VACL). To be fair, both work equally well in my experience, but he IP ACL is easier to operationalize and I am a staunch believer in making network easier to maintain and avoiding what I refer to as Science Fair Projects. We've all worked on, inherited or (hopefully not!) created a Science Fair Project - let's avoid that. ;)

The configuration for the IP ACL looks like this:

ip access-list HSRP_IP
10 permit udp any 224.0.0.2/32 eq 1985
20 permit udp any 224.0.0.102/32 eq 1985

This access list matches the multicast addresses for HSRPv1, and HSRPv2, though can be modified for VRRP and GLBP.
This access-list is then applied as a VACL to filter the FHRP hellos from entering the OTV through the internal interfaces. The VACL looks like below where we’ll filter HSRP on VLAN 31-33.

vlan access-map HSRP_Local 10
match ip address HSRP_IP
action drop
vlan access-map HSRP_Local 20
match ip address ALL
action forward
vlan filter HSRP_Local vlan-list 16,23

If you are like me and want to verify your VACL is applied and matching, the steps are not as easy we’d like them to be but the capability does exist. *NOTE* that I am not responsible for you monkeying around with any of the other commands available when you attach to the module. You’ve been warned. :)
The first thing to do is attach to the module where your internal interfaces physically are. In the example below, it’s module 1. If your OTV is configured in a non-default VDC, you’ll need to set the parser to use that VDC as below.

champs1# attach mod 1
Attaching to module 1 ...
To exit type 'exit', to abort type '$.'
module-1# vdc 3
module-1# show system internal access-list input statistics
VLAN 16 :
=========
Tcam 1 resource usage:
----------------------
Label_b = 0x806
Bank 0
------
IPv4 Class
Policies: VACL(HSRP_Local) [Merged]
Entries:
[Index] Entry [Stats]
---------------------
[0013] deny udp 0.0.0.0/0 224.0.0.102/32 eq 1985 [1863]
[0014] deny udp 0.0.0.0/0 224.0.0.2/32 eq 1985 [4121]

[0015] permit ip 0.0.0.0/0 0.0.0.0/0 [1766386]

VLAN 23 :
=========
Tcam 1 resource usage:
----------------------
Label_b = 0x806
Bank 0
------
IPv4 Class
Policies: VACL(HSRP_Local) [Merged]
Entries:
[Index] Entry [Stats]
---------------------
[0013] deny udp 0.0.0.0/0 224.0.0.102/32 eq 1985 [1863]
[0014] deny udp 0.0.0.0/0 224.0.0.2/32 eq 1985 [4121]
[0015] permit ip 0.0.0.0/0 0.0.0.0/0 [1766386]


With this configuration, the FHRP in each data center will be locally active and mitigate the tromboning we mentioned earlier. This has a significant impact in that now we only send traffic across the Data Center Interconnect (DCI) that needs to go across as the local routers in each site can service the traffic.

Note that is technique is useful for optimizing egress traffic but does nothing to help draw or “attract” traffic into the right data center. Other technologies that provide that functionality will be the topic of future blogs. ;)

One last step to undertake when performing FHRP isolation is to exclude the FHRP MAC addresses from being advertised by OTV. You might be thinking OTV won't know about the FHRP MACs becuase of the VACL, right? Wrong. :) Due to the nature of MAC address learning, OTV will learn about the MAC addresses before the VACL drops them so we need to tell OTV to not advertise them. This is a three part process where we'll define the mac access list, add it to a route-map and then apply it to the OTV ISIS process as shown below.

mac-list OTV_HSRP seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00
mac-list OTV_HSRP seq 11 deny 0000.0c9f.f000 ffff.ffff.ff00
mac-list OTV_HSRP seq 15 deny 0100.5e00.0000 ffff.ffff.ff00
mac-list OTV_HSRP seq 20 permit 0000.0000.0000 0000.0000.0000


route-map OTV_HSRP_filter permit 10
match mac-list OTV_HSRP

otv-isis default
vpn Overlay0
redistribute filter route-map OTV_HSRP_filter


We’ll cover AED election, and some other fun topics in the next post (hopefully sooner rather than later.

As always, your comments and feedback are appreciated!

19 comments:

  1. The only issue I see with this is that it seems to prevent you from running FHRP within each data center itself. In what will be our implementation, we plan on having 2 7ks in each DC.

    ReplyDelete
  2. Hi Tony,
    First, thank you for reading the blog! As the VACL is applied on the internal interface of the OTV device or VDC, FHRP hellos continue to work locally within the data center. The device or VDC where the SVI L2/L3 boundary exists does not change and the SVIs can form a FHRP partnership. Hope that helps!
    Ron

    ReplyDelete
  3. Ron,

    Great post on otv. I get that the ip acl prevents the gateways on each side from discovering each other which results in the localization. I thought however that you still needed to do the Mac acl to prevent the virtual Mac addresses from being learned across OTV. Thoughts?

    --Rich

    ReplyDelete
  4. LOL! Good eyes, Rich - glad you were paying attention in the CPOC. :) I've updated the blog with the information - must have been a cust and paste faux pas from Word to blogger. My bad.

    Ron

    ReplyDelete
  5. I'm really glad I have visited this blog, I have learned relevant information that is very useful for my field of study. Thanks for sharing those valuable information here.

    ReplyDelete
  6. Good morning,

    I can't wait for your studies of this weekess link that you have identified, which is:
    "but does nothing to help draw or “attract” traffic into the right data center. Other technologies that provide that functionality will be the topic of future blogs. "

    I am reading about LISP (http://en.wikipedia.org/wiki/Locator/Identifier_Separation_Protocol) and mac to bgp (http://tools.ietf.org/html/draft-raggarwa-mac-vpn-01)
    But this is quite in beta/dream.

    danv

    ReplyDelete
  7. Ron,

    Thanks for the great writeup!

    Like DanV I wish LISP was a reality. I know your next post is supposed to cover the techniques to address this but I'm wondering, can host routes be used to attract ingress traffic? Is there a way to use policy routing to make it dynamic?

    ReplyDelete
  8. Ron,

    mac-list OTV_HSRP seq 15 deny 0100.5e00.0000 ffff.ffff.ff00

    What's the purpose of the above entry in the mac-list? I know we did this in our testing but looking over the otv guide on CCO it doesn't mention it. Let me know. Thanks.

    --Rich

    ReplyDelete
  9. It worked perfectly, thank you very much.

    ReplyDelete
  10. The VLAN-List command above filters on VLANS 16 and 23 not 31-33 correct?

    ReplyDelete
  11. If I can do FHRP filtering at the data center ... why would I want to implement OTV if I have L2 connectivity between DCs ?

    99 % of the "trombone" effect will go away with filtering and leave a far more simple topology not to save a bunch of money.

    ReplyDelete
  12. Hi Anonymous (from April 2)
    You can do FHRP configuration without OTV but you'll miss the benefits of STP domain separation(this is HUGE), ARP neighbor discovery to minimize ARP traffic across the network and simplified multi-homing for more than 2 sites. What you propose is certainly an option, but many customers enjoy the benefits of OTV for the reasons I mention.

    Thanks for reading.
    Ron

    ReplyDelete
  13. Hi Anonymous (from March 25),
    Yes, the VACL is applied only to VLANs 16 and 23.

    Thanks for reading.
    Ron

    ReplyDelete
  14. Ron,
    Great post!
    So I am trying to do something similar but not really!
    I help a client install nexus 5K's as a L3 core. They are replacing a 4500 core. In the 4500 core they leveraged HSRP v1 to do a DCI to a small DR site. So basically the small DR site had a 3750 and it was in the same HSRP group as the main DC 4500's. It was a lower priority so it would come up as listen. The customer did this so that in case of a disaster, the 3750 would be the L3 for the DR. Now that we are doing the Nexus boxes at the core I am 99% sure I cannot mimic that setup with HSRP at the DR. So I am going to do a poor man's OTV and just leave the DR hsrp as active for the DR and the Nexus 5K's as active at the DC. Any tips on creating HSRP multicast filtering and MAC filtering on the Nexus side and the 3750 side? Thanks,

    ReplyDelete
  15. Hi
    Ron
    I have question regarding the OTV use case.
    My Concern is all use cases for the OTV that big data center where most of the traffic for specific datacenter is north south , my assumption here is east-west traffic is confined in the same datacenter & when you have plan for the disaster( tornado warning or preplan for the outage in one datacenter) you can use this DCI for vmotion & other application.
    Is my assumption is valid for this OTV use case or there are more EAST-WEST traffic everyday between Data Cetner 1 & Data Center 2 for all the design guide.

    Sincerely
    Viral Patel

    ReplyDelete
  16. Hi

    Thank you for the time and effort to post this. I have just gone through it all and find it very useful. I Have stumbled on an issue though. I get the below error on our Core. To give a high level of the topology:
    VDC-Core to VDC-OTV to VDC-WAN to other site with mirrored config.

    As soon as I deploy the filter list, I get the below repeatedly.

    %ARP-3-DUP_VADDR_SRC_IP: arp [6825] Source address of packet received from 0000.0c07.ac01 on Vlan20(Ethernet3/3) is duplicate of local virtual ip, 10.42.0.254

    Can you shed any light please?

    Anthony

    ReplyDelete
  17. Hi

    Great post but I'm missing something here that I hope you can help with. Instead of creating one HSRP group that spans both locations, why would I not just create seperate HSRP groups with only the 2 local 7Ks in each group? This way I don't have to worry about blocking hello's over OTV and mac learning?

    ReplyDelete
  18. Hi Marvin,

    I think that if you do that you will have a virtual ip duplicate. I mean if your HSRP virtual IP is xxx.xxx.xxx.1 on both side even with different HSRP group xxx.xxx.xxx.1 will still existe two times on this span vlan.

    ReplyDelete
  19. Hi Ron ,

    Nice article . Any article produced which you mention above : Other technologies that provide that functionality will be the topic of future blogs. ;) waiting for that blogs

    ReplyDelete