Cisco Nexus 2000: A Love/Hate Relationship

My feelings towards the Nexus 2000 Fabric Extender (FEX) are hardly a secret. The myriad of design choices and platform limitations present engineers with some rather difficult decisions. Because of this, I’ve seen a handful of engineers reverse course on their current design due to limitations. It’s because of this that I have harsh feelings for the FEX.

One of the issues plaguing the topic around FEX design is that misunderstanding about what a FEX truly is. The one thing you have to remember that the FEX in and of itself is not a switch. It is the quintessential line card for the Nexus 5000. The uplink ports should be only considered as switching fabric. Since FEXs do not switch traffic locally, all traffic is sent upstream across the “fabric” to the Nexus 5000/7000. This switch provides a centralized forwarding and policy enforcement. Yes, this means that even host-to-host traffic between two ports on the same FEX on the same VLAN will flow through the parent switch. Thinking of them as line cards may help you reconsider some of your design choices, which will provide you with more available features.

Design Choices

Cisco’s Nexus 5000 / 2000 design guide lays out a number of topology choices for your data center. Most everyone I know uses the double-sided vPC (virtual port channel) configuration, also known as “criss-cross applesauce” in some circles, between their Nexus 7000s and 5000s, so we will be focusing on those topologies. The first one is dubbed the single-homed or straight-through vPC configuration. This topology not only supports a vPC to a host from (2) FEXs, and FCOE (on the 2232), but also static pinning (for more information on static pinning please see the Nexus 5000/2000 Design Guide linked above). The main advantage here truly being that you’re able to build redundant teaming configurations from the servers to the fabric extenders.

The second configuration has been donned the dual-homed or active-active configuration. In this scenario, each FEX has a dual-connection to the 5Ks via a vPC. This configuration has a couple of downsides. First off, the port configuration needs to match on both 5Ks. Also, you’re no longer able to configure a port-channel to the host device from two different fabric extenders. You are however still able do either active-standby NIC teaming from servers to two different fabric extenders, or configure a port-channel that terminates on a single FEX (not on the 2148). You also lose the ability to do FCoE and static pinning.

Other Caveats

• The 2148 is not capable of doing a standard port-channel using 2 or more ports on the FEX. This is apparently a hardware limitation in the ASICs used on the 2148.
• A port-channel with (2) 2148s (also called Single Homed Fabric Extender vPC Topology) is limited to 2 ports. Again, see above.
• The 2148 only supports 1000BaseT… The 2248 has of course brought us 10/100/1000, and the 2232 has 1/10gb port options.
• The 2148 also runs BPDU Guard… permanently. Recent code updates have allowed us to disable BPDU Guard on the 2248, but no such luck on the 2148 line.
• FEXs cannot be used as a SPAN destination port… a rather handy feature if you happen to have Nexus 5510s, which only support 10Gb.
• No distributed forwarding on the FEX itself… unlike what a modern line card would offer.

Why am I still deploying FEX?

Well, there are obviously a number of issues I see with the architecture, but overall there are a few key points that I feel continue to make it worthwhile to consider.

• Simplifies network management by consolidating your edge network. This minimizes the amount of touch points required when provisioning new services.
• Simplifies your network topology by not only reducing your spanning tree domain, but also by allowing you to take full advantage of uplink bandwidth using vPCs.
• Flexibility, from the same management point, I can manage multiple switches with a variety of connectivity options.

Conclusion

With all of the gotchas and caveats, making a design choice can be difficult. But if you get in on the ground level and do your research, the Nexus 2000 can be a great asset for your data center. While I’ve certainly had my share frustrations with them, in the long run I have learned my lessons and I even plan to deploy more in the future. As for those of you looking for advice on a new infrastructure build-out, for most situations, I’m recommending to my customers and colleagues that they consider deploying Nexus 2248s in a single-homed configuration. This provides the best of both worlds in regards to redundancy and available features.

This article was originally posted on the PacketPushers Blog. I am cross posting it here now for posterity, with a 6 months after the original article.