Saturday, February 12, 2011

Nexus 7000 + Fabric Extenders = Scalable Access Layer



One of the most difficult components in any data center architecture to design and plan for is the access layer. In a traditional network hierarchy the access layer is where the most dynamic and changing requirements exist. Myriad technologies abound and can tell a history of the data center as new technologies were introduced with the progression from 100Mb Ethernet to 1G to 10G and the emergence of Unified Fabric (FCoE). Scaling these access layers has been a black art at times because of the changing pace of technology. What if you could have an access layer that meets your current 100/1G Ethernet needs today as well as 10G, provided a reduction in management points and helps tame the Spanning Tree beast? Enter the Nexus 7000 with support for Nexus 2000 Fabric Extenders (FEX).


The Nexus 7000s have been shipping for close to 3 years now and have a well established install base, mature software and have proven themselves as scalable Data Center platforms. The Nexus 2000 has been shipping for over 2 years and has been solving access layer challenges for customers very well when paired with the Nexus 5000 switch. Combining the two technologies provides similar benefits for the traditional FEX architectures only on a larger scale. Today the Nexus 5000 series support up to a maximum of 16 FEX while the Nexus 7000 supports 32 with current code and plans for more in the future. Let’s dig into the details.

First, what are the requirements for FEX support on the Nexus 7000? Three primary requirements must be met:
1. NX-OS 5.1(1) or higher must be installed on the Nexus 7000
2. 32 port M1 10GE modules (part number)
3. EPLD must be current to support VNTag

Once these requirements are met we can connect the FEX to the Nexus 7000. The options supported include traditional 10G Short Reach (SR), 10G Long Reach (LR) optics and Fabric Extender Transceiver (FET) for the M1 32 port card. The M1 32 “L” card add support for active Twinax cables which currently are available in 7 and 10M lengths. In our example, we’ll be using SR optics.


Let’s start by verifying we meet the requirements.
We see below we are running NX-OS 5.1(2) so we’re good to go there.
cmhlab-dc2-sw2-agg1# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Software
BIOS: version 3.22.0
kickstart: version 5.1(2)
system: version 5.1(2)
BIOS compile time: 02/20/10
kickstart image file is: bootflash:///n7000-s1-kickstart.5.1.2.bin
kickstart compile time: 12/25/2020 12:00:00 [12/18/2010 09:55:20]
system image file is: bootflash:///n7000-s1-dk9.5.1.2.bin
system compile time: 11/29/2010 12:00:00 [12/18/2010 11:02:00]


We also have the correct modules installed
cmhlab-dc2-sw2-agg1# show mod
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
2 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
3 32 10 Gbps Ethernet XL Module N7K-M132XP-12L ok
4 32 1/10 Gbps Ethernet Module N7K-F132XP-15 ok
5 0 Supervisor module-1X N7K-SUP1 active *
6 0 Supervisor module-1X N7K-SUP1 ha-standby
8 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
9 48 1000 Mbps Optical Ethernet Modul N7K-M148GS-11 ok
10 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok

Now let’s check the EPLD
*NOTE* This must be done from the default VDC and if an EPLD upgrades is required, it is disruptive so plan accordingly.
cmhlab-dc2-sw2-otv1# install all epld bootflash:n7000-s1-epld.5.1.1.img

Compatibility check:
Module Type Upgradable Impact Reason
------ ---- ---------- ---------- ------
2 LC Yes disruptive Module Upgradable
3 LC Yes disruptive Module Upgradable
4 LC Yes disruptive Module Upgradable
5 SUP Yes disruptive Module Upgradable
6 SUP Yes disruptive Module Upgradable
8 LC Yes disruptive Module Upgradable
9 LC Yes disruptive Module Upgradable
10 LC Yes disruptive Module Upgradable
1 Xbar Yes disruptive Module Upgradable
2 Xbar Yes disruptive Module Upgradable
3 Xbar Yes disruptive Module Upgradable
4 Xbar Yes disruptive Module Upgradable
5 Xbar Yes disruptive Module Upgradable
1 FAN Yes disruptive Module Upgradable
2 FAN Yes disruptive Module Upgradable
3 FAN Yes disruptive Module Upgradable
4 FAN Yes disruptive Module Upgradable

Copy complete, now saving to disk (please wait)...
Retrieving EPLD versions... Please wait.

Images will be upgraded according to following table:
Module Type EPLD Running-Version New-Version Upg-Required
------ ---- ------------- --------------- ----------- ------------
2 LC Power Manager 4.008 4.008 No
2 LC IO 1.016 1.016 No
2 LC Forwarding Engine 1.006 1.006 No
2 LC FE Bridge(1) 186.006 186.006 No
2 LC FE Bridge(2) 186.006 186.006 No
2 LC Linksec Engine(1) 2.006 2.006 No
2 LC Linksec Engine(2) 2.006 2.006 No
2 LC Linksec Engine(3) 2.006 2.006 No
2 LC Linksec Engine(4) 2.006 2.006 No
2 LC Linksec Engine(5) 2.006 2.006 No
2 LC Linksec Engine(6) 2.006 2.006 No
2 LC Linksec Engine(7) 2.006 2.006 No
2 LC Linksec Engine(8) 2.006 2.006 No
3 LC Power Manager 4.008 4.008 No
3 LC IO 1.016 1.016 No
3 LC Forwarding Engine 1.006 1.006 No
3 LC FE Bridge(1) 186.006 186.006 No
3 LC Linksec Engine(1) 2.006 2.006 No

4 LC Power Manager 1.000 1.000 No
4 LC IO 0.045 0.045 No
5 SUP Power Manager 3.009 3.009 No
5 SUP IO 3.028 3.028 No
5 SUP Inband 1.008 1.008 No
5 SUP Local Bus CPLD 3.000 3.000 No
5 SUP CMP CPLD 6.000 6.000 No
6 SUP Power Manager 3.009 3.009 No
6 SUP IO 3.028 3.028 No
6 SUP Inband 1.008 1.008 No
6 SUP Local Bus CPLD 3.000 3.000 No
6 SUP CMP CPLD 6.000 6.000 No
8 LC Power Manager 5.006 5.006 No
8 LC IO 2.014 2.014 No
8 LC Forwarding Engine 1.006 1.006 No
9 LC Power Manager 4.008 4.008 No
9 LC IO 1.006 1.006 No
9 LC Forwarding Engine 1.006 1.006 No
9 LC SFP 1.004 1.004 No
10 LC Power Manager 5.006 5.006 No
10 LC IO 2.014 2.014 No
10 LC Forwarding Engine 1.006 1.006 No
1 Xbar Power Manager 2.010 2.010 No
2 Xbar Power Manager 2.010 2.010 No
3 Xbar Power Manager 2.010 2.010 No
4 Xbar Power Manager 2.010 2.010 No
5 Xbar Power Manager 2.010 2.010 No
1 FAN Fan Controller (1) 0.007 0.007 No
1 FAN Fan Controller (2) 0.007 0.007 No
2 FAN Fan Controller (1) 0.007 0.007 No
2 FAN Fan Controller (2) 0.007 0.007 No
3 FAN Fan Controller (1) 0.007 0.007 No
3 FAN Fan Controller (2) 0.007 0.007 No
4 FAN Fan Controller (1) 0.007 0.007 No
4 FAN Fan Controller (2) 0.007 0.007 No
All Modules are up to date.
cmhlab-dc2-sw2-otv1#


So we’re in good shape there, too. It’s like I’ve done this before….. :)
Now that we’re ready, we’ve cabled the FEX to the switch via port e3/1-4 and we’ll be creating a topology that looks like this.






















First, we need to install the FEX feature set. This is a bit different than what we’ve done with features in the past and must be done from the default VDC.
cmhlab-dc2-sw2-otv1# show run | i fex
cmhlab-dc2-sw2-otv1# confi t
Enter configuration commands, one per line. End with CNTL/Z.
cmhlab-dc2-sw2-otv1(config)# install feature-set fex
cmhlab-dc2-sw2-otv1(config)# show run | i fex
install feature-set fex
allow feature-set fex
allow feature-set fex
allow feature-set fex
allow feature-set fex
cmhlab-dc2-sw2-otv1(config)#

Note that each VDC now has a config for allow feature-set fex.
Next, we’ll go to our VDC where we want the FEX configured and get it setup.
cmhlab-dc2-sw2-agg1# confi
Enter configuration commands, one per line. End with CNTL/Z.
cmhlab-dc2-sw2-agg1(config)# feature-set fex

Then we’ll define the FEX and specify the model. While this isn’t required because the FEX will identify itself to the Nexus switch, I think it makes the config more readable and is somewhat self documenting.

cmhlab-dc2-sw2-agg1(config)# fex 150
cmhlab-dc2-sw2-agg1(config-fex)# type n2248T
cmhlab-dc2-sw2-agg1(config-fex)# description FEX150-for-Agg1-VDC

Now we’ll configure the physical ports the FEX is connected into.

cmhlab-dc2-sw2-agg1(config-fex)# int e3/1-4
cmhlab-dc2-sw2-agg1(config-if-range)# desc FEX 150
cmhlab-dc2-sw2-agg1(config-if-range)# switchport
cmhlab-dc2-sw2-agg1(config-if-range)# switchport mode fex-fabric
cmhlab-dc2-sw2-agg1(config-if-range)# fex associate 150
cmhlab-dc2-sw2-agg1(config-if-range)# channel-group 150

Now that we’ve told the switch to treat the ports as fex-fabric ports and created a port channel, let’s bring it up.

cmhlab-dc2-sw2-agg1(config-if-range)# int po150
cmhlab-dc2-sw2-agg1(config-if)# desc Port Channel to FEX 150
cmhlab-dc2-sw2-agg1(config-if)# no shut

cmhlab-dc2-sw2-agg1(config-if)# int e3/1-4
cmhlab-dc2-sw2-agg1(config-if-range)# shut
cmhlab-dc2-sw2-agg1(config-if-range)# no shut
cmhlab-dc2-sw2-agg1(config-if-range)#
cmhlab-dc2-sw2-agg1(config-if-range)# 2011 Feb 12 18:08:23 cmhlab-dc2-sw2-agg1 %FEX-2-NOHMS_ENV_FEX_ONLINE: FEX-150 On-line (Serial Number JAF1440BDFR)


It’s that simple.

cmhlab-dc2-sw2-agg1# show fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
150 FEX150-for-Agg1-VDC Online N2K-C2248TP-1GE JAF1440BDFR
cmhlab-dc2-sw2-agg1#
cmhlab-dc2-sw2-agg1# show fex 150
FEX: 150 Description: FEX150-for-Agg1-VDC state: Online
FEX version: 5.1(2) [Switch version: 5.1(2)]
Extender Model: N2K-C2248TP-1GE, Extender Serial: JAF1440BDFR
Part No: 73-12748-05
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth3/1
Fabric interface state:
Po150 - Interface Up. State: Active
Eth3/1 - Interface Up. State: Active
Eth3/2 - Interface Up. State: Active
Eth3/3 - Interface Up. State: Active
Eth3/4 - Interface Up. State: Active
cmhlab-dc2-sw2-agg1#

If we look at the port channel we created, it looks like any other port channel.

cmhlab-dc2-sw2-agg1# show int po150
port-channel150 is up
Hardware: Port-Channel, address: c471.feee.c924 (bia c471.feee.c924)
Description: Port Channel to FEX 150
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is fex-fabric
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth3/1, Eth3/2, Eth3/3, Eth3/4
Last clearing of "show interface" counters never
30 seconds input rate 124432 bits/sec, 12 packets/sec
30 seconds output rate 23272 bits/sec, 20 packets/sec

With the FEX being on-line, we now have 48 additional interfaces available to configure.

cmhlab-dc2-sw2-agg1# show int brief | i 150
mgmt0 -- up 10.0.2.13 1000 1500
Eth3/1 1 eth fabric up none 10G(S) 150
Eth3/2 1 eth fabric up none 10G(S) 150
Eth3/3 1 eth fabric up none 10G(S) 150
Eth3/4 1 eth fabric up none 10G(S) 150
Po150 1 eth fabric up none a-10G(S) none
Eth150/1/1 1 eth access down Administratively down auto(D) --
Eth150/1/2 1 eth access down Administratively down auto(D) --
Eth150/1/3 1 eth access down Administratively down auto(D) --
Eth150/1/4 1 eth access down Administratively down auto(D) --
Eth150/1/5 1 eth access down Administratively down auto(D) --
Eth150/1/6 1 eth access down Administratively down auto(D) --
Eth150/1/7 1 eth access down Administratively down auto(D) --
Eth150/1/8 1 eth access down Administratively down auto(D) --
Eth150/1/9 1 eth access down Administratively down auto(D) --
Eth150/1/10 1 eth access down Administratively down auto(D) --
Eth150/1/11 1 eth access down Administratively down auto(D) --
Eth150/1/12 1 eth access down Administratively down auto(D) --
Eth150/1/13 1 eth access down Administratively down auto(D) --
Eth150/1/14 1 eth access down Administratively down auto(D) --
Eth150/1/15 1 eth access down Administratively down auto(D) --
Eth150/1/16 1 eth access down Administratively down auto(D) --
Eth150/1/17 1 eth access down Administratively down auto(D) --
Eth150/1/18 1 eth access down Administratively down auto(D) --
Eth150/1/19 1 eth access down Administratively down auto(D) --
Eth150/1/20 1 eth access down Administratively down auto(D) --
Eth150/1/21 1 eth access down Administratively down auto(D) --
Eth150/1/22 1 eth access down Administratively down auto(D) --
Eth150/1/23 1 eth access down Administratively down auto(D) --
Eth150/1/24 1 eth access down Administratively down auto(D) --
Eth150/1/25 1 eth access down Administratively down auto(D) --
Eth150/1/26 1 eth access down Administratively down auto(D) --
Eth150/1/27 1 eth access down Administratively down auto(D) --
Eth150/1/28 1 eth access down Administratively down auto(D) --
Eth150/1/29 1 eth access down Administratively down auto(D) --
Eth150/1/30 1 eth access down Administratively down auto(D) --
Eth150/1/31 1 eth access down Administratively down auto(D) --
Eth150/1/32 1 eth access down Administratively down auto(D) --
Eth150/1/33 1 eth access down Administratively down auto(D) --
Eth150/1/34 1 eth access down Administratively down auto(D) --
Eth150/1/35 1 eth access down Administratively down auto(D) --
Eth150/1/36 1 eth access down Administratively down auto(D) --
Eth150/1/37 1 eth access down Administratively down auto(D) --
Eth150/1/38 1 eth access down Administratively down auto(D) --
Eth150/1/39 1 eth access down Administratively down auto(D) --
Eth150/1/40 1 eth access down Administratively down auto(D) --
Eth150/1/41 1 eth access down Administratively down auto(D) --
Eth150/1/42 1 eth access down Administratively down auto(D) --
Eth150/1/43 1 eth access down Administratively down auto(D) --
Eth150/1/44 1 eth access down Administratively down auto(D) --
Eth150/1/45 1 eth access down Administratively down auto(D) --
Eth150/1/46 1 eth access down Administratively down auto(D) --
Eth150/1/47 1 eth access down Administratively down auto(D) --
Eth150/1/48 1 eth access down Administratively down auto(D) --
cmhlab-dc2-sw2-agg1#


Note that today we cannot have a FEX multi-homed into two Nexus 7000s like we can on the Nexus 5000. Look for that capability in a future release along with support for additional FEX platforms.

When you think of the scale – 32 FEX x 48 ports = 1,536, that’s pretty impressive. Being able to take advantage of the cable savings with localized, in –rack cabling without the challenges of increased STP diameter, the FEX and Nexus 7000 make a powerful impact on the data center topology.

As always, I welcome your comments and feedback.

10 comments:

  1. Ron,

    Great article, can you clarify which FEXs are currently supported on the 7k?

    Thanks,
    Jason

    ReplyDelete
  2. The only FEX supported today is the 2248. Look for the support of the 2232 in a future release.

    ReplyDelete
  3. Hi Jason and Ryan,
    Thanks for reading! Like Ryan said, the 2232 will be added in a future release as will the 2224.

    ReplyDelete
  4. Thanks, again great post.

    ReplyDelete
  5. do we know if the F series 10G cards in the 7K will support direct fex connectivity in the future too?

    ReplyDelete
  6. The F1 module will not support FEX as they do not have the required ASICs.

    ReplyDelete
  7. I have a few question about this.
    1) Now, FEX support Dual-homed to Nexus 7K ?
    2) Host Port-Channel to one FEX ?
    3) Host vPC to two FEX ?

    Thanks

    ReplyDelete
  8. VIA,
    Thanks for reading the blog! Fast answer to all of your questions is that these capabilities are coming in the future. More specific information on when they will available can be obtained from your Cisco account team or partner engineer.

    ReplyDelete
  9. Ron,
    thx for the write up, exactly what I am looking for. I am glad to stumble upon this blog. Going into my favorite bookmark

    ben

    ReplyDelete
  10. Very soon this web site will be famous among all blog viewers, due to it's pleasant articles

    ReplyDelete