Recently I’ve been lucky enough to be challenged with learning a bit about Fibre Channel Switching, but I’m even luckier in that I’m getting to know it on a set of MDS switches running NX-OS (previously referred to as SAN-OS). So far, I’ve learned the basics of getting things to work, but nothing really beyond that. As the SAN world has always been a mystery to me, I figured I would share what I’ve learned with other engineers that are at least looking for a baseline look into the storage network.
New Terminology
First, lets familureize ourselves with a few terms that we may run into when dealing with the very basics of FC switching…
- WWN: World Wide Name, think 8-byte MAC address. Also pWWN/sWWN (Port/Switch WWN) — This is the addressing of the Fibre Channel world. All of our configs are going to use pWWNs (Port World Wide Names, which actually refer to the node, or N_port)
- vSAN: A vSAN is a virtual collection of ports, sort of like a VRF, or even a vDC (but within the same management plane) — Each port can only be a member of one vSANs. — From my understanding, This is a Cisco specific technology typically used to create at least one unique vSAN per switch. This defines the two (or more) distinct fabric paths.
- Zone: a Zone is a grouping of ports inside a vSAN used to control which devices can speak with other devices. Devices can be members of multiple zones. Devices in different Zones cannot speak to each other. — Think VLAN.
- N_port: Node Port — Could be a Host, or Storage device.
- F_port: Fabric port — Connects to an N_port
- FLOGI: Fibre Channel Logins — Used to exchange device information. Including WWNs
Configuration
First, we’re going to define the group of ports we’re using. Descriptions are set, and since the MDS is licensed per port, you have to enable the license for that port. I also ensure that each of these ports are not shutdown.
interface fc1/1 switchport description Controller A port-license acquire no shutdown interface fc1/2 switchport description Controller B port-license acquire no shutdown interface fc1/15 switchport description Blade Server Fabric A port-license acquire no shutdown interface fc1/16 switchport description Blade Server Fabric B port-license acquire no shutdown interface fc1/17 switchport description DMZ Server Fabric A port-license acquire no shutdown interface fc1/18 switchport description DMZ Server Fabric B port-license acquire no shutdown
Next up we’re going to configure the vSAN for this switch, and add the ports/interfaces to it. (Almost brings me back to configuring VLANS on HP switches… )
vsan database vsan 10 name "FABRIC_A" vsan 10 interface fc1/1 vsan 10 interface fc1/2 vsan 10 interface fc1/15 vsan 10 interface fc1/16 vsan 10 interface fc1/17 vsan 10 interface fc1/18
Now that we have our ports inside the vSAN we can discover their PWWNs using the FLOGI database. We will need these for our fcalias groups below.
mds# sh flogi database -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/1 10 0x0f0100 50:0a:09:81:8d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca fc1/2 10 0x0f0200 50:0a:09:81:9d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca fc1/15 10 0x0f0300 20:41:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1 fc1/15 10 0x0f0301 20:00:00:25:b5:0b:00:df 20:00:00:25:b5:00:00:8f fc1/15 10 0x0f0302 20:00:00:25:b5:0b:00:ef 20:00:00:25:b5:00:00:9f fc1/15 10 0x0f0304 20:00:00:25:b5:0b:00:bf 20:00:00:25:b5:00:00:6f fc1/16 10 0x0f0000 20:42:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1 fc1/16 10 0x0f0002 20:00:00:25:b5:0b:00:9f 20:00:00:25:b5:00:00:2f fc1/16 10 0x0f0004 20:00:00:25:b5:0b:00:6f 20:00:00:25:b5:00:00:3f fc1/16 10 0x0f0004 20:00:00:25:b5:0b:00:7f 20:00:00:25:b5:00:00:4f fc1/17 10 0x0f0300 20:41:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1 fc1/17 10 0x0f0301 20:00:00:25:b5:0a:00:ef 20:00:00:25:b5:00:00:8f fc1/17 10 0x0f0302 20:00:00:25:b5:0a:00:df 20:00:00:25:b5:00:00:9f fc1/17 10 0x0f0304 20:00:00:25:b5:0a:00:cf 20:00:00:25:b5:00:00:6f fc1/18 10 0x0f0000 20:42:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1 fc1/18 10 0x0f0002 20:00:00:25:b5:0a:00:5f 20:00:00:25:b5:00:00:2f fc1/18 10 0x0f0004 20:00:00:25:b5:0a:00:bf 20:00:00:25:b5:00:00:3f fc1/18 10 0x0f0004 20:00:00:25:b5:0a:00:8f 20:00:00:25:b5:00:00:4f
Here we’re defining logical groups of devices and identifying their PWWNs. Nothing too scary here.
fcalias name STORAGE vsan 10
member pwwn 50:0a:09:81:8d:8f:f8:ca
member pwwn 50:0a:09:81:9d:8f:f8:ca
fcalias name BLADE_SERVERS vsan 10
member pwwn 20:00:00:25:b5:0b:00:df
member pwwn 20:00:00:25:b5:0b:00:ef
member pwwn 20:00:00:25:b5:0b:00:bf
member pwwn 20:00:00:25:b5:0b:00:9f
member pwwn 20:00:00:25:b5:0b:00:6f
member pwwn 20:00:00:25:b5:0b:00:7f
fcalias name DMZ_SERVERS vsan 10
member pwwn 20:00:00:25:b5:0a:00:ef
member pwwn 20:00:00:25:b5:0a:00:df
member pwwn 20:00:00:25:b5:0a:00:cf
member pwwn 20:00:00:25:b5:0a:00:5f
member pwwn 20:00:00:25:b5:0a:00:bf
member pwwn 20:00:00:25:b5:0a:00:8f
And finally, this is where the proverbial “magic” happens. We’re defining a zone called SERVERS_TO_STORAGE inside vsan 10 and adding our fcalias groups as members of the zone. This allows these group members to communicate with each other. This zone is segmented from our DMZ_SERVERS_TO_STORAGE zone, so that there is no change of communication between our DMZ_SERVERS and BLADE_SERVERS.
zone name BLADE_SERVERS_TO_STORAGE vsan 10
member fcalias STORAGE
member fcalias BLADE_SERVERS
zone name DMZ_SERVERS_TO_STORAGE vsan 10
member fcalias STORAGE
member fcalias DMZ_SERVERS
Verification
From what I understand the only practical verification you can do is checking your host to ensure you can see the LUNS presented by your storage array. With any luck, everything should show up. If not, double check the PWWNS used in the fcalias groups.
Conclusion
While I have given you little more than a 30-second education on fibre channel, I hope it’s enough that when your storage engineer rambles on endlessly about zoning, you at least have a clue as to what he is actually talking about. And as long as your infrastructure isn’t too complicated, I’m sure you could figure out how to connect your new UCS Blades to their storage without their help. That line is getting awful blurred these days, you might find yourself having to learn more about storage because of it.
Who writes this crap?
Tony Mattke is a network engineer for a financial institution in Indiana. In the past he has worked for ISPs, data centers, networking manufactures, and the occasional enterprise. For feedback, please leave a comment on the article in question. For everything else including fan mail or death threats, contact him via twitter.
{ 11 comments… read them below or add one }
Tony,
Nice post.
On VSAN/Zones, VSANs (rather than Zones) are similar in concepts to VLANs. Every VSAN runs a separate instance of FC services (such as FC Name Server, Zone Server, FLOGI Server, etc.). VSAN allows a physical switch to be divided into multiple virtual switches – you can have a production VSAN, a Q/A VSAN, and so on. And for communication between devices in different VSANs, you would need to use IVR – Inter-VSAN Routing (not very commonly used though).
Zoning is about access control, which devices can talk to each other.
-Ashish
Great post! Thanks for sharing.
Great 45 second primer. When did Cisco drop SAN-OS in favor of NX-OS? Makes sense, just missed that memo.
Looks like around June of 2009… http://www.cisco.com/en/US/products/ps5989/prod_r…
Great post Tony. I remember my first imlementation of two fabrics based on MDS 9148. This license system is sometimes tricky, when you have acquire or auto on a port and must add another port :) Let's say my friend has a bad experience with that ;)
Mirek
To my understanding,
- Zones are like Private VLANs (mostly community Vlans)
- VSANs are like VLANs
Also Brocade does routing between different fabrics. But Brocade does not have vSAN concept. Brocade follows a different path as Fiber Channel Routing – FCR.
Looks like you forgot to put zoneset for vsan 10. Without it zones configured will not take effect
The ability to think like that shows you're an expert
FYI
Device alias is preferred in a true MDS environment as to allow easier use of additional features.
Also, you should add the zones to a zoneset and activate.
Good Article! Clear and easier to understand and configure!
what about "Fabric B"? are those devices all lumped into the one vsan, or should they get segregated into two?