MDS Fibre Channel Switching Basics for Network Engineers

Recently I’ve been lucky enough to be challenged with learning a bit about Fibre Channel Switching, but I’m even luckier in that I’m getting to know it on a set of MDS switches running NX-OS (previously referred to as SAN-OS). So far, I’ve learned the basics of getting things to work, but nothing really beyond that. As the SAN world has always been a mystery to me, I figured I would share what I’ve learned with other engineers that are at least looking for a baseline look into the storage network.

New Terminology

First, lets familureize ourselves with a few terms that we may run into when dealing with the very basics of FC switching…

• WWN: World Wide Name, think 8-byte MAC address. Also pWWN/sWWN (Port/Switch WWN) — This is the addressing of the Fibre Channel world. All of our configs are going to use pWWNs (Port World Wide Names, which actually refer to the node, or N_port)
• vSAN: A vSAN is a virtual collection of ports, sort of like a VRF, or even a vDC (but within the same management plane) — Each port can only be a member of one vSANs. — From my understanding, This is a Cisco specific technology typically used to create at least one unique vSAN per switch. This defines the two (or more) distinct fabric paths.
• Zone: a Zone is a grouping of ports inside a vSAN used to control which devices can speak with other devices. Devices can be members of multiple zones. Devices in different Zones cannot speak to each other. — Think VLAN.
• N_port: Node Port — Could be a Host, or Storage device.
• F_port: Fabric port — Connects to an N_port
• FLOGI: Fibre Channel Logins — Used to exchange device information. Including WWNs

Configuration

First, we’re going to define the group of ports we’re using. Descriptions are set, and since the MDS is licensed per port, you have to enable the license for that port. I also ensure that each of these ports are not shutdown.

interface fc1/1
  switchport description Controller A
  port-license acquire
  no shutdown

interface fc1/2
  switchport description Controller B
  port-license acquire
  no shutdown

interface fc1/15
  switchport description Blade Server Fabric A
  port-license acquire
  no shutdown

interface fc1/16
  switchport description Blade Server Fabric B
  port-license acquire
  no shutdown

interface fc1/17
  switchport description DMZ Server Fabric A
  port-license acquire
  no shutdown

interface fc1/18
  switchport description DMZ Server Fabric B
  port-license acquire
  no shutdown

Next up we’re going to configure the vSAN for this switch, and add the ports/interfaces to it. (Almost brings me back to configuring VLANS on HP switches… )

vsan database
  vsan 10 name "FABRIC_A" 
  vsan 10 interface fc1/1
  vsan 10 interface fc1/2
  vsan 10 interface fc1/15
  vsan 10 interface fc1/16
  vsan 10 interface fc1/17
  vsan 10 interface fc1/18

Now that we have our ports inside the vSAN we can discover their PWWNs using the FLOGI database. We will need these for our fcalias groups below.

mds# sh flogi database 
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME       
--------------------------------------------------------------------------------
fc1/1            10    0x0f0100  50:0a:09:81:8d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca
fc1/2            10    0x0f0200  50:0a:09:81:9d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca
fc1/15           10    0x0f0300  20:41:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1
fc1/15           10    0x0f0301  20:00:00:25:b5:0b:00:df 20:00:00:25:b5:00:00:8f
fc1/15           10    0x0f0302  20:00:00:25:b5:0b:00:ef 20:00:00:25:b5:00:00:9f
fc1/15           10    0x0f0304  20:00:00:25:b5:0b:00:bf 20:00:00:25:b5:00:00:6f
fc1/16           10    0x0f0000  20:42:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1
fc1/16           10    0x0f0002  20:00:00:25:b5:0b:00:9f 20:00:00:25:b5:00:00:2f
fc1/16           10    0x0f0004  20:00:00:25:b5:0b:00:6f 20:00:00:25:b5:00:00:3f
fc1/16           10    0x0f0004  20:00:00:25:b5:0b:00:7f 20:00:00:25:b5:00:00:4f
fc1/17           10    0x0f0300  20:41:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1
fc1/17           10    0x0f0301  20:00:00:25:b5:0a:00:ef 20:00:00:25:b5:00:00:8f
fc1/17           10    0x0f0302  20:00:00:25:b5:0a:00:df 20:00:00:25:b5:00:00:9f
fc1/17           10    0x0f0304  20:00:00:25:b5:0a:00:cf 20:00:00:25:b5:00:00:6f
fc1/18           10    0x0f0000  20:42:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1
fc1/18           10    0x0f0002  20:00:00:25:b5:0a:00:5f 20:00:00:25:b5:00:00:2f
fc1/18           10    0x0f0004  20:00:00:25:b5:0a:00:bf 20:00:00:25:b5:00:00:3f
fc1/18           10    0x0f0004  20:00:00:25:b5:0a:00:8f 20:00:00:25:b5:00:00:4f

Here we’re defining logical groups of devices and identifying their PWWNs. Nothing too scary here.

fcalias name STORAGE vsan 10
    member pwwn 50:0a:09:81:8d:8f:f8:ca
    member pwwn 50:0a:09:81:9d:8f:f8:ca

fcalias name BLADE_SERVERS vsan 10
    member pwwn 20:00:00:25:b5:0b:00:df
    member pwwn 20:00:00:25:b5:0b:00:ef
    member pwwn 20:00:00:25:b5:0b:00:bf
    member pwwn 20:00:00:25:b5:0b:00:9f
    member pwwn 20:00:00:25:b5:0b:00:6f
    member pwwn 20:00:00:25:b5:0b:00:7f

fcalias name DMZ_SERVERS vsan 10
    member pwwn 20:00:00:25:b5:0a:00:ef
    member pwwn 20:00:00:25:b5:0a:00:df
    member pwwn 20:00:00:25:b5:0a:00:cf
    member pwwn 20:00:00:25:b5:0a:00:5f
    member pwwn 20:00:00:25:b5:0a:00:bf
    member pwwn 20:00:00:25:b5:0a:00:8f

And finally, this is where the proverbial “magic” happens. We’re defining a zone called SERVERS_TO_STORAGE inside vsan 10 and adding our fcalias groups as members of the zone. This allows these group members to communicate with each other. This zone is segmented from our DMZ_SERVERS_TO_STORAGE zone, so that there is no change of communication between our DMZ_SERVERS and BLADE_SERVERS.

zone name BLADE_SERVERS_TO_STORAGE vsan 10
    member fcalias STORAGE
    member fcalias BLADE_SERVERS

zone name DMZ_SERVERS_TO_STORAGE vsan 10
    member fcalias STORAGE
    member fcalias DMZ_SERVERS

Verification

From what I understand the only practical verification you can do is checking your host to ensure you can see the LUNS presented by your storage array. With any luck, everything should show up. If not, double check the PWWNS used in the fcalias groups.

Conclusion

While I have given you little more than a 30-second education on fibre channel, I hope it’s enough that when your storage engineer rambles on endlessly about zoning, you at least have a clue as to what he is actually talking about. And as long as your infrastructure isn’t too complicated, I’m sure you could figure out how to connect your new UCS Blades to their storage without their help. That line is getting awful blurred these days, you might find yourself having to learn more about storage because of it.