Recently I’ve been lucky enough to be challenged with learning a bit about Fibre Channel Switching, but I’m even luckier in that I’m getting to know it on a set of MDS switches running NX-OS (previously referred to as SAN-OS). So far, I’ve learned the basics of getting things to work, but nothing really beyond that. As the SAN world has always been a mystery to me, I figured I would share what I’ve learned with other engineers that are at least looking for a baseline look into the storage network.
First, lets familureize ourselves with a few terms that we may run into when dealing with the very basics of FC switching…
- WWN: World Wide Name, think 8-byte MAC address. Also pWWN/sWWN (Port/Switch WWN) — This is the addressing of the Fibre Channel world. All of our configs are going to use pWWNs (Port World Wide Names, which actually refer to the node, or N_port)
- vSAN: A vSAN is a virtual collection of ports, sort of like a VRF, or even a vDC (but within the same management plane) — Each port can only be a member of one vSANs. — From my understanding, This is a Cisco specific technology typically used to create at least one unique vSAN per switch. This defines the two (or more) distinct fabric paths.
- Zone: a Zone is a grouping of ports inside a vSAN used to control which devices can speak with other devices. Devices can be members of multiple zones. Devices in different Zones cannot speak to each other. — Think VLAN.
- N_port: Node Port — Could be a Host, or Storage device.
- F_port: Fabric port — Connects to an N_port
- FLOGI: Fibre Channel Logins — Used to exchange device information. Including WWNs
First, we’re going to define the group of ports we’re using. Descriptions are set, and since the MDS is licensed per port, you have to enable the license for that port. I also ensure that each of these ports are not shutdown.
interface fc1/1 switchport description Controller A port-license acquire no shutdown interface fc1/2 switchport description Controller B port-license acquire no shutdown interface fc1/15 switchport description Blade Server Fabric A port-license acquire no shutdown interface fc1/16 switchport description Blade Server Fabric B port-license acquire no shutdown interface fc1/17 switchport description DMZ Server Fabric A port-license acquire no shutdown interface fc1/18 switchport description DMZ Server Fabric B port-license acquire no shutdown
Next up we’re going to configure the vSAN for this switch, and add the ports/interfaces to it. (Almost brings me back to configuring VLANS on HP switches… )
vsan database vsan 10 name "FABRIC_A" vsan 10 interface fc1/1 vsan 10 interface fc1/2 vsan 10 interface fc1/15 vsan 10 interface fc1/16 vsan 10 interface fc1/17 vsan 10 interface fc1/18
Now that we have our ports inside the vSAN we can discover their PWWNs using the FLOGI database. We will need these for our fcalias groups below.
mds# sh flogi database -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/1 10 0x0f0100 50:0a:09:81:8d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca fc1/2 10 0x0f0200 50:0a:09:81:9d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca fc1/15 10 0x0f0300 20:41:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1 fc1/15 10 0x0f0301 20:00:00:25:b5:0b:00:df 20:00:00:25:b5:00:00:8f fc1/15 10 0x0f0302 20:00:00:25:b5:0b:00:ef 20:00:00:25:b5:00:00:9f fc1/15 10 0x0f0304 20:00:00:25:b5:0b:00:bf 20:00:00:25:b5:00:00:6f fc1/16 10 0x0f0000 20:42:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1 fc1/16 10 0x0f0002 20:00:00:25:b5:0b:00:9f 20:00:00:25:b5:00:00:2f fc1/16 10 0x0f0004 20:00:00:25:b5:0b:00:6f 20:00:00:25:b5:00:00:3f fc1/16 10 0x0f0004 20:00:00:25:b5:0b:00:7f 20:00:00:25:b5:00:00:4f fc1/17 10 0x0f0300 20:41:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1 fc1/17 10 0x0f0301 20:00:00:25:b5:0a:00:ef 20:00:00:25:b5:00:00:8f fc1/17 10 0x0f0302 20:00:00:25:b5:0a:00:df 20:00:00:25:b5:00:00:9f fc1/17 10 0x0f0304 20:00:00:25:b5:0a:00:cf 20:00:00:25:b5:00:00:6f fc1/18 10 0x0f0000 20:42:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1 fc1/18 10 0x0f0002 20:00:00:25:b5:0a:00:5f 20:00:00:25:b5:00:00:2f fc1/18 10 0x0f0004 20:00:00:25:b5:0a:00:bf 20:00:00:25:b5:00:00:3f fc1/18 10 0x0f0004 20:00:00:25:b5:0a:00:8f 20:00:00:25:b5:00:00:4f
Here we’re defining logical groups of devices and identifying their PWWNs. Nothing too scary here.
fcalias name STORAGE vsan 10 member pwwn 50:0a:09:81:8d:8f:f8:ca member pwwn 50:0a:09:81:9d:8f:f8:ca fcalias name BLADE_SERVERS vsan 10 member pwwn 20:00:00:25:b5:0b:00:df member pwwn 20:00:00:25:b5:0b:00:ef member pwwn 20:00:00:25:b5:0b:00:bf member pwwn 20:00:00:25:b5:0b:00:9f member pwwn 20:00:00:25:b5:0b:00:6f member pwwn 20:00:00:25:b5:0b:00:7f fcalias name DMZ_SERVERS vsan 10 member pwwn 20:00:00:25:b5:0a:00:ef member pwwn 20:00:00:25:b5:0a:00:df member pwwn 20:00:00:25:b5:0a:00:cf member pwwn 20:00:00:25:b5:0a:00:5f member pwwn 20:00:00:25:b5:0a:00:bf member pwwn 20:00:00:25:b5:0a:00:8f
And finally, this is where the proverbial “magic” happens. We’re defining a zone called SERVERS_TO_STORAGE inside vsan 10 and adding our fcalias groups as members of the zone. This allows these group members to communicate with each other. This zone is segmented from our DMZ_SERVERS_TO_STORAGE zone, so that there is no change of communication between our DMZ_SERVERS and BLADE_SERVERS.
zone name BLADE_SERVERS_TO_STORAGE vsan 10 member fcalias STORAGE member fcalias BLADE_SERVERS zone name DMZ_SERVERS_TO_STORAGE vsan 10 member fcalias STORAGE member fcalias DMZ_SERVERS
From what I understand the only practical verification you can do is checking your host to ensure you can see the LUNS presented by your storage array. With any luck, everything should show up. If not, double check the PWWNS used in the fcalias groups.
While I have given you little more than a 30-second education on fibre channel, I hope it’s enough that when your storage engineer rambles on endlessly about zoning, you at least have a clue as to what he is actually talking about. And as long as your infrastructure isn’t too complicated, I’m sure you could figure out how to connect your new UCS Blades to their storage without their help. That line is getting awful blurred these days, you might find yourself having to learn more about storage because of it.
Network engineer turned management currently servicing the enterprise data center market. I started working on networks in the ’90s and still feel like that was just a few years ago. Jack of all trades, master of none; I love to learn about everything. Feel free to ask me about photography, woodworking, nhra, watches, or even networking! — For feedback, please leave a comment on the article in question, and I’ll respond as soon as I can. For everything else including fan mail or death threats, contact me via twitter.