As many of you know my background isn’t in enterprise, but I currently fill that role in my $job. In order to succeed I’ve had to develop many new skills including learning Cisco Wireless, UCS, a little Fibre Channel, and of course Cisco ASA. While I have been using firewalls for many years, I’ve never used the ASA for anything more than a user firewall, or for supporting a small branch. So yes, my skills are lacking in the ASA market compared to other technologies, and when you get deep into the grind with any product you’re going to need some new tricks to aid in your troubleshooting. This is where ASA paacket captures come into place.

Define Interesting Traffic

As with any packet capture, or even log viewing the amount of noise involved generally dwarfs the data you actually want to find. In order to ease your pain Cisco has allowed us filter out packet capture using an ACL.

FW# access-list FOO line 1 extended permit ip any host 10.2.1.5 
FW# access-list FOO line 2 extended permit ip host 10.2.1.5 any 

Once you have your traffic defined, you need to setup your capture. Below I’ve defined a capture named appropriately, FOO, capturing headers from our outside interface that matches our ACL with a 100,000 byte buffer. In ASA 8.0 and above you can do the same capture without the ACL using the match option, which captures traffic bi-directionally.

FW# capture FOO type raw-data access-list CAP buffer 100000 interface outside headers-only
!
! 8.0 and above
FW# capture FOO type raw-data match tcp any host 10.2.1.5 buffer 100000 interface outside headers-only

Viewing your capture

Viewing your capture on the firewall is easy enough using the show capture command. You can do some basic filtering using the built in IOS | include / exclude as well.

FW# sh capture FOO 

1446 packets captured
-snip-
1062: 10:53:32.978983 10.2.1.5.443 > 10.5.3.7.50344: . 2979731571:2979732951(1380) ack 3788440994 win 8204 
1063: 10:53:32.978999 10.2.1.5.443 > 10.5.3.7.50344: . 2979732951:2979734331(1380) ack 3788440994 win 8204 
1064: 10:53:32.979014 10.2.1.5.443 > 10.5.3.7.50344: P 2979734331:2979734949(618) ack 3788440994 win 8204 
1065: 10:53:32.982691 10.5.3.7.50344 > 10.2.1.5.443: . ack 2979734949 win 16560 
1066: 10:53:34.712441 10.5.3.7.50348 > 10.2.1.5.9960: S 3576506984

For more detailed information you can use the detail or dump option with the show command.

While viewing the files on the firewall is handy in a pinch, it is far from your only option. You can also copy the capture off the firewall using tftp or ftp using the copy command copy /pcap capture:FOO tftp:. The firewall exports the capture in PCAP file format, which allows you to retrieve the file and open it up in Wireshark! Although if you don’t run a tftp or ftp server on your desktop, an even handier option may be to simply access the file in your web browser using the following url. https://FIREWALL_IP/capture/FOO/pcap

You need to allow http access from your workstation on the interface you’re entering

Another cool trick is setting up a circular buffer to continually watch interesting traffic. Something I’m using on one of my current projects is a capture that watches for flows that are denied via ACL.

FW# capture denied type asp-drop acl-drop buffer 4194304 circular-buffer headers-only 

Now, once you’re done tracking down what you broke, you probably want to stop the capture, and remove the files. This is done by simply issuing a no capture NAME command from the CLI.

Like I said, I’m not a firewall guru, but this can certainly go a long way to make you look like one to your systems guys when they forget to tell your their web app opens a java client that needs to communicate on TCP/9960….. Not that they’d actually do that or anything.

If any of you have any other cool tricks you can do with ASA Packet Captures please feel free to leave a comment below so I can update the article and give you credit!

Mid December 2012 Apple shut down the Messages Beta for Lion, soon after many hackintosh users started noticing issues with signing into iMessage. At some point in time, people far smarter than me managed to patch a little used bootloader called Clover to allow us to log into iMessage, but Clover is young and still full of random issues. Honestly, it never liked the system id on my partition, so I was never able to use it. But now, it seems that someone has patched our widely used Chameleon bootloader! I’ve tested it on my own hackintosh, and many users are also reporting success.

The instructions are simple enough, and should only take you 3 minutes + a reboot to implement and test!

1. Download the following files to your hackintosh
• FileNVRAM.dylib.zip
• Chameleon-2.2svn-r2170.zip
2. Execute the following commands
sudo mkdir /Extra/modules
cd /Extra/modules
sudo unzip ~/Downloads/FileNVRAM.dylib.zip
sudo rm -rf __MACOSX
sudo rm -rf ACPICodec.dylib

If you have ACPICodec.dylib in your /Extra/modules folder, you need to delete it.

3. Unzip the Chameleon installer, and run it — make sure you install to your boot disk
4. Reboot, and try to login to iMessage

Hopefully this will take care of your issues. If not, it was worth a try! I highly recommend the Insanely Mac Forum for researching any issues you may be having. After all, where do you think I learned how to do this?

Recently I was faced with an issue outside my normal expertise… those of you that know me realize I am anything but a security engineer. But in reality, you must always expand your horizons. One of the projects I’m working on involves migrating between two edge networks. Obviously, for a time there has to be traffic using both networks while you migrate services from one network to the other. This creates an issue from services that may be NAT’d from the inside of the network, where as the current (read: old) default route takes them out a different connection..
In order to solve this, you need to either change the default route, which may not be possible, or start NAT’ing the source address of your traffic. It took me a bit of time to get the details worked out, so I wanted to share what I found out.

Plain Jane Static NAT

Since 8.3, NAT has changed quite a bit. The most obvious change is the use of Object groups pretty much everywhere. In some ways, this simplifies the config. In others, not so much. Basic static NAT takes the form of a single object group that defines the inside host, and the static NAT statement.

object network SERVER
  host 10.72.14.29
  nat (OUTSIDE,INSIDE) static 12.98.16.42

NAT + PAT

The biggest thing that kept giving me issues was that I was attempting to reuse the normal static NAT config, and build a separate PAT config for the source address. Once I realized I was being a tool, the solution came rather quickly. This NAT statement is configured in the same manner as Identity NAT, used to prevent address translations to certain destinations. First you define two object groups, one for the NAT address, and one for the real inside address of the server.

object network SERVER_NAT
  host 12.98.16.42
!
object network SERVER_INSIDE
  host 10.72.14.29
!
nat (OUTSIDE,INSIDE) source dynamic any interface destination static SERVER_NAT SERVER_INSIDE

Next comes this slighly confusing NAT statement…

nat (OUTSIDE,INSIDE) — this is familiar, should make sense to most of us.

source dynamic any interface — this states that our source address is using dynamic NAT/PAT, the traffic could originate from anywhere, and should be NAT’d to the interface it leaves the firewall on. This is the key to our source NAT/PAT configuration…

destination static SERVER_NAT SERVER_INSIDE — finally, this just states the our destination is a static NAT statement, translating our object group SERVER_NAT, to the address in the object group SERVER_INSIDE.

fin

So, hopefully this makes sense… if not, please remember that RouterJockey.com provides no warranties or promises and that you’re just as hopelessly screwed as I am. Thanks for stopping by!

INE published a great info-graphic on the earning potential of Cisco’s certifications and I felt the need to share it here. It covers a range of topics from average salaries on all three tiers of proficiency, to top locations for positions, always remember that the information included is in no way meant as career advice.

In situations where service providers want to offer transparent LAN services that preserve a customers VLAN tags across your Layer-2 network, This amendment to the IEEE 802.1q standard allows us to use a single VLAN to transport multiple VLANS across the MAN or WAN. In doing so, we stack on an extra 802.1q tag to the customer’s traffic at the provider’s edge (PE). The original 802.1Q specification allows a single VLAN header to be inserted into an Ethernet frame. QinQ allows multiple VLAN headers to be inserted into a single frame, this is essential when implementing Metro Ethernet networks.

QinQ Configuration

First we need to ensure our transport switches can support the baby giant frames. To check the current MTU on the switch you can issue the command show system mtu and in global configuration mode, you can use system mtu 1504 to change the switches MTU to the recommended 1504 bytes..

SW1# show system mtu
System MTU size is 1500 bytes
SW1# configure terminal
SW1(config)# system mtu 1504
Changes to the System MTU will not take effect until the next reload is done.

The actual QinQ configuration takes place entirely on the transport switches, no modifications are required on the customer equipment. The first step is to configure the trunk between our two backbone switches. We’ll be using VLAN 101 for Customer-A, and 201 for Customer-B. As you will see, the configuration is rather quick and simple.

SW1(config)# interface fa1/0/48
SW1(config-if)# switchport trunk encapsulation dot1q
SW1(config-if)# switchport mode trunk
SW1(config-if)# switchport trunk allowed vlan 101,201

SW2(config)# interface fa1/0/48
SW2(config-if)# switchport trunk encapsulation dot1q
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 101,201

Now, on the provider edge (PE) ports, we need to assign the port to the appropriate VLAN, and configure the dot1q-tunnel. This tunnel is what allows us to transport a customer’s VLANtrunk across our network. The command l2protocol-tunnel allows transportation of Layer-2 protocols such as CDP, LLDP, STP, and VTP. I’m also turning off CDP here for sanity’s sake.

SW1(config)# interface fa1/0/10
SW1(config-if)# desc Customer-A
SW1(config-if)# switchport access vlan 101
SW1(config-if)# switchport mode dot1q-tunnel
SW1(config-if)# l2protocol-tunnel
SW1(config-if)# no cdp enable
SW1(config-if)# interface fa1/0/20
SW1(config-if)# desc Customer-B
SW1(config-if)# switchport access vlan 201
SW1(config-if)# switchport mode dot1q-tunnel
SW1(config-if)# l2protocol-tunnel
SW1(config-if)# no cdp enable

SW2(config)# interface fa1/0/10
SW2(config-if)# desc Customer-A
SW2(config-if)# switchport access vlan 101
SW2(config-if)# switchport mode dot1q-tunnel
SW2(config-if)# l2protocol-tunnel
SW2(config-if)# no cdp enable
SW2(config-if)# interface fa1/0/20
SW2(config-if)# desc Customer-B
SW2(config-if)# switchport access vlan 201
SW2(config-if)# switchport mode dot1q-tunnel
SW2(config-if)# l2protocol-tunnel
SW2(config-if)# no cdp enable

And that’s it. Each customer has tunneled connectivity between their sites using their own VLAN numbering all encapsulated within their own VLAN on the providers Layer-2 network. In the near future I plan on writting a bit on 802.1Q tunnel termination in regards to the Cisco 10000, aka the BFR. It’s been a few years since I’ve done it, but I can still remember the basics.

Since I’ve recently had some fun working with the Cisco 5585-X and the IPS blades, I wanted to document some of the information I learned while getting them online. Some of this came from various sources around the ‘net including the TAC IPS group, other parts are from consultants, or what we just figured out on our own while working with it. The first thing you need to understand is how traffic gets to / from the IPS. The easiest way to explain it, is that the IPS sits inside the firewall. It will be sent traffic, from a matched ACL, after the interface has performed it’s own inbound ACL filtering. In our situation, we simply defined an ACL that stated permit ip any any — but depending on the application, you may only want to filter traffic in certain directions. Below, we have an example that filters traffic with a destination defined in our object-group. The ACL you defined is matched in a class-map, which is matched in your global_policy policy-map. The command ips inline fail-open sensor vs0 actually sends the traffic (via the backplane) to the IPS blade. This command also states, that if the ASA detects an issue with the IPS, to bypass it, and continue to pass traffic as if it was never installed. To cease the flow of traffic to the sensor, remove this command from the policy-map… This would be rather usefull if a bad set of signatures were deployed and the IPS was erroneously dropping traffic.

object-group network IPS-RANGES
 network-object 10.0.0.0 255.0.0.0
 network-object 4.2.2.0 255.255.255.0
!
access-list IPS-ACL permit ip any object-group IPS-RANGES
!
class-map IPS-CLASSMAP
 match access-list IPS-ACL
!
policy-map global_policy
class IPS-CLASSMAP
 ips inline fail-open sensor vs0

Once traffic is flowing through the IPS, you should take a look at the following command. At the bottom you will see statistics of packets sent to, and receieved from the blade. As well as the number of drops.

ASA-FW# sh service-policy  

Global policy: 
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: dns preset_dns_map, packet 14803467, lock fail 0, drop 39, reset-drop 0
      Inspect: ftp, packet 1298420, lock fail 0, drop 0, reset-drop 0
      Inspect: h323 h225 _default_h323_map, packet 0, lock fail 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: h323 ras _default_h323_map, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: rsh, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: rtsp, packet 0, lock fail 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: esmtp _default_esmtp_map, packet 31147261, lock fail 2, drop 388, reset-drop 0
      Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: skinny , packet 0, lock fail 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: sunrpc, packet 0, lock fail 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: xdmcp, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: sip , packet 0, lock fail 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: netbios, packet 278, lock fail 0, drop 0, reset-drop 0
      Inspect: tftp, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: ip-options _default_ip_options_map, packet 0, lock fail 0, drop 0, reset-drop 0
      Inspect: icmp, packet 77268, lock fail 0, drop 832, reset-drop 0
      Inspect: ipsec-pass-thru _default_ipsec_passthru_map, packet 2533443, lock fail 0, drop 0, reset-drop 0
      Inspect: pptp, packet 0, lock fail 0, drop 0, reset-drop 0
     Class-map: IPS-CLASSMAP
      IPS: card status Up, mode inline fail-open, sensor vs0
        packet input 40316913, packet output 40316927, drop 7, reset-drop 0
    Class-map: class-default

      Default Queueing      Set connection policy:         drop 0
      Set connection decrement-ttl

Enabling and Configuring Automatic Updates

The simplest/easiest method to enable and configure the Auto/Cisco.com Update feature is via the sensor GUI (IDM). Once you login, head to Configuration > Sensor Management > Auto/Cisco.com Update. From there, make sure Enable Signature and Engine Updates from Cisco.com is checked, and head into the Cisco.com Server Settings. You must supply a valid CCO username/password, and configured a scheduled start time for the update.. Updates are released once every 1 – 2 weeks, so hourly checks are overkill. Updates happen over https, and the IPS must have the ability to connect without using a proxy. If you wish to perform a test, simply update the schedule to reflect a time in the near future. The current system time can be obtained from the CLI using the show clock command.

The default update URL https://198.133.219.25//cgi-bin/front.x/ida/locator/locator.pl is currently the only valid source. The double-forward slash following the IP address is expected and required.

After an update attempt has occurred, details can be found in Auto Update Statistics section of the show stat host command. Other than authentication failures, connectivity failures are the most common and are generally due to the IPS being unable to reach the update server on tcp port 443.

ASA-IPS# show stat host | begin Auto Update
Auto Update Statistics
   lastDirectoryReadAttempt = 12:20:00 GMT-05:00 Wed May 01 2011
    =   Read directory: http://user@72.163.7.55//swc/esd/guest/
    =   Success
   lastDownloadAttempt = 12:20:00 GMT-05:00 Wed May 01 2011
    =   Download: http://user@72.163.7.55//swc/esd/guest/IPS-sig-S563-req-E4.pkg
    =   Success
   lastInstallAttempt = 12:20:30 GMT-05:00 Wed May 01 2011
    =   IPS-sig-S563-req-E4: Update completed successfully
    =   Success
   nextAttempt = 12:20:00 GMT-05:00 Thu May 02 2011

Signatures

The nomenclature surrounding IPS Signatures is a bit confusing. For instance Enable/disable does NOT select/de-select signatures to be used by IPS, that functionality is called Retire/Unretire. When a signature is retired, it means the IPS will NOT compile that signature into memory for scanning. Unretiring a signature instructs the IPS to compile the signature into memory and to use the signature to scan traffic. Enable/disable is to enforce/disregard the action associated with the signatures by the IPS when packets or packet flows match the signature. Therefore in order to perform an action against a packet / packet flow a signature must be unretired AND enabled.

When customizing your signature base there are a number of things to keep in mind. Current IPS signature definition releases contain over 5,000 individual signatures, but only a limited amount are enabled/active by default. The signatures that are enabled/active by default is determined by Cisco’s IPS Signature development team and changes from release to release as new signatures are added and old signatures become obsolete.

A new user may be inclined to select and enable all or most of the signatures thinking this would provide the most protection possible, however this is not the case. Signatures may have been retired for any number of reasons including large numbers of false positives, the signature being replaced by a newer/better one, the signature being overly complex or causing noticeable performance/stability issues, the threat no longer being relevant, etc… As such, it is never a good idea to attempt to enable and unretire bulk signatures. Doing so will almost certainly result in some serious issues like legitimate traffic being denied, bad performance, sensor crashes, etc.. Signatures that are manually enabled and/or unretired should always be thoroughly researched and understood prior to doing so.

If you wish to return the sensor’s current signature policy to default, issue the following commands…

IPS# conf t
IPS(config)# service signature-definition sig0
IPS(config-sig)# default signatures
IPS(config-sig)# exit
Apply Changes?[yes]: yes

SNMP Traps

The idea behind trap-directed notification is that it is impractical for a server to poll or request information from every SNMP object on every device. The solution is for each device to notify the server of events without solicitation. It does this by sending a message known as a trap of the event. To enable global trap notification configure the following service replacing key bits as necessary.

sensor# conf t
sensor(config)# service notification
sensor(config-not)# trap-destination 192.168.69.10
sensor(config-not-tra)# trap-community-name public
sensor(config-not-tra)# trap-port 162
sensor(config-not-tra)# exit
sensor(config-not)# enable-notifications true
sensor(config-not)# exit
Apply Changes?[yes]: yes

By default traps are not generated for events, the Request SNMP Trap Action can be added to an EAO (Event Action Override) to add a trap as an Action whenever a signature fires.

In the example provided, all Alerts will generate traps you may instead specify specify a desired Risk Rating range. (100-90 for High, 89-70 for Medium, 69-1 for Low)

sensor# conf t
sensor(config)# service event-action-rules rules0
sensor(config-eve)# overrides request-snmp-trap
sensor(config-eve-ove)# override-item-status enabled
sensor(config-eve-ove)# risk-rating-range 1-100
sensor(config-eve-ove)# exit
sensor(config-eve)# exit
Apply Changes?[yes]: yes

If you wish these traps may be added to a specific signature (instead of the EAO) by using the Request SNMP Trap Action on a per-signature basis.

As a follow up to my blog post on the PacketPushers blog, I wanted to share with you another time saving tip for getting our jobs done not only quickly, but helping to remove one of the tedious steps in firewall maintenance. Today, I needed to add a large chunk of ACEs to my INSIDE-IN ACL (about 6 times larger than my example here, but spread across a couple devices..) Luckily, I used my head when it came time to start adjusting line numbers. The first thing I did was to write out my ACEs, and instead of filling in the line numbers, I just used XXX. (I would be deploying this in a few places, so I saved a couple versions as well..)

Sample output included below: IP Addresses were changed to protect the innocent…

access-list INSIDE-IN line XXX remark ~
access-list INSIDE-IN line XXX remark - Traffic to SOMETHING via HTTPS 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 1.16.32.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.23.86.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.31.90.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.24.10.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.14.32.0 255.255.254.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 2.51.10.0 255.255.252.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 2.33.10.0 255.255.254.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 1.16.32.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.23.86.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.31.90.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.24.10.0 255.255.255.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.14.32.0 255.255.254.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 2.51.10.0 255.255.252.0 eq 443 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 2.33.10.0 255.255.254.0 eq 443
access-list INSIDE-IN line XXX remark ~
access-list INSIDE-IN line XXX remark - Traffic to SOMETHING via SSH
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 1.16.32.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.23.86.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.31.90.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.24.10.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 6.14.32.0 255.255.254.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 2.51.10.0 255.255.252.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.10.10.10 2.33.10.0 255.255.254.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 1.16.32.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.23.86.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.31.90.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.24.10.0 255.255.255.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 6.14.32.0 255.255.254.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 2.51.10.0 255.255.252.0 eq 22 
access-list INSIDE-IN line XXX extended permit tcp host 10.20.10.10 2.33.10.0 255.255.254.0 eq 22

Next, I had to remember how to do an incremental substitution, luckily the Internet is a wonderful resource. We enter EX mode, and define a function named CountUp() — to do this, press : and enter the lines below.

 fun CountUp()
  let ret = g:i
  let g:i = g:i + 1
  return ret
endf

Now, we can perform a search and replacement using the function to fill in the data. We set i= to our first line number and pipe this into our search and replace statement.

:let i = 126 | %s/XXX/\=CountUp()/g

Instantly our line numbers are populated, and we’ve saved a substantial amount of time. I <3 Vim

access-list INSIDE-IN line 126 remark ~
access-list INSIDE-IN line 127 remark – Traffic to SOMETHING via HTTPS 
access-list INSIDE-IN line 128 extended permit tcp host 10.10.10.10 1.16.32.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 129 extended permit tcp host 10.10.10.10 6.23.86.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 130 extended permit tcp host 10.10.10.10 6.31.90.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 131 extended permit tcp host 10.10.10.10 6.24.10.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 132 extended permit tcp host 10.10.10.10 6.14.32.0 255.255.254.0 eq 443 
access-list INSIDE-IN line 133 extended permit tcp host 10.10.10.10 2.51.10.0 255.255.252.0 eq 443 
access-list INSIDE-IN line 134 extended permit tcp host 10.10.10.10 2.33.10.0 255.255.254.0 eq 443 
access-list INSIDE-IN line 135 extended permit tcp host 10.20.10.10 1.16.32.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 136 extended permit tcp host 10.20.10.10 6.23.86.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 137 extended permit tcp host 10.20.10.10 6.31.90.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 138 extended permit tcp host 10.20.10.10 6.24.10.0 255.255.255.0 eq 443 
access-list INSIDE-IN line 139 extended permit tcp host 10.20.10.10 6.14.32.0 255.255.254.0 eq 443 
access-list INSIDE-IN line 140 extended permit tcp host 10.20.10.10 2.51.10.0 255.255.252.0 eq 443 
access-list INSIDE-IN line 141 extended permit tcp host 10.20.10.10 2.33.10.0 255.255.254.0 eq 443
access-list INSIDE-IN line 142 remark ~
access-list INSIDE-IN line 143 remark – Traffic to SOMETHING via SSH
access-list INSIDE-IN line 144 extended permit tcp host 10.10.10.10 1.16.32.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 145 extended permit tcp host 10.10.10.10 6.23.86.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 146 extended permit tcp host 10.10.10.10 6.31.90.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 147 extended permit tcp host 10.10.10.10 6.24.10.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 148 extended permit tcp host 10.10.10.10 6.14.32.0 255.255.254.0 eq 22 
access-list INSIDE-IN line 149 extended permit tcp host 10.10.10.10 2.51.10.0 255.255.252.0 eq 22 
access-list INSIDE-IN line 150 extended permit tcp host 10.10.10.10 2.33.10.0 255.255.254.0 eq 22 
access-list INSIDE-IN line 151 extended permit tcp host 10.20.10.10 1.16.32.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 152 extended permit tcp host 10.20.10.10 6.23.86.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 153 extended permit tcp host 10.20.10.10 6.31.90.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 154 extended permit tcp host 10.20.10.10 6.24.10.0 255.255.255.0 eq 22 
access-list INSIDE-IN line 155 extended permit tcp host 10.20.10.10 6.14.32.0 255.255.254.0 eq 22 
access-list INSIDE-IN line 156 extended permit tcp host 10.20.10.10 2.51.10.0 255.255.252.0 eq 22 
access-list INSIDE-IN line 157 extended permit tcp host 10.20.10.10 2.33.10.0 255.255.254.0 eq 22

This is the most versitile solution I could find, as it allows for multiple substitutions per line, another solution that is easier to remember, and quicker to implement, but will only perform one solution per line follows below.

:let i=126 | g/XXX/s//\=i/ | let i=i+1

Recently I’ve been lucky enough to be challenged with learning a bit about Fibre Channel Switching, but I’m even luckier in that I’m getting to know it on a set of MDS switches running NX-OS (previously referred to as SAN-OS). So far, I’ve learned the basics of getting things to work, but nothing really beyond that. As the SAN world has always been a mystery to me, I figured I would share what I’ve learned with other engineers that are at least looking for a baseline look into the storage network.

New Terminology

First, lets familureize ourselves with a few terms that we may run into when dealing with the very basics of FC switching…

• WWN: World Wide Name, think 8-byte MAC address. Also pWWN/sWWN (Port/Switch WWN) — This is the addressing of the Fibre Channel world. All of our configs are going to use pWWNs (Port World Wide Names, which actually refer to the node, or N_port)
• vSAN: A vSAN is a virtual collection of ports, sort of like a VRF, or even a vDC (but within the same management plane) — Each port can only be a member of one vSANs. — From my understanding, This is a Cisco specific technology typically used to create at least one unique vSAN per switch. This defines the two (or more) distinct fabric paths.
• Zone: a Zone is a grouping of ports inside a vSAN used to control which devices can speak with other devices. Devices can be members of multiple zones. Devices in different Zones cannot speak to each other. — Think VLAN.
• N_port: Node Port — Could be a Host, or Storage device.
• F_port: Fabric port — Connects to an N_port
• FLOGI: Fibre Channel Logins — Used to exchange device information. Including WWNs

Configuration

First, we’re going to define the group of ports we’re using. Descriptions are set, and since the MDS is licensed per port, you have to enable the license for that port. I also ensure that each of these ports are not shutdown.

interface fc1/1
  switchport description Controller A
  port-license acquire
  no shutdown

interface fc1/2
  switchport description Controller B
  port-license acquire
  no shutdown

interface fc1/15
  switchport description Blade Server Fabric A
  port-license acquire
  no shutdown

interface fc1/16
  switchport description Blade Server Fabric B
  port-license acquire
  no shutdown

interface fc1/17
  switchport description DMZ Server Fabric A
  port-license acquire
  no shutdown

interface fc1/18
  switchport description DMZ Server Fabric B
  port-license acquire
  no shutdown

Next up we’re going to configure the vSAN for this switch, and add the ports/interfaces to it. (Almost brings me back to configuring VLANS on HP switches… )

vsan database
  vsan 10 name "FABRIC_A" 
  vsan 10 interface fc1/1
  vsan 10 interface fc1/2
  vsan 10 interface fc1/15
  vsan 10 interface fc1/16
  vsan 10 interface fc1/17
  vsan 10 interface fc1/18

Now that we have our ports inside the vSAN we can discover their PWWNs using the FLOGI database. We will need these for our fcalias groups below.

mds# sh flogi database 
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME       
--------------------------------------------------------------------------------
fc1/1            10    0x0f0100  50:0a:09:81:8d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca
fc1/2            10    0x0f0200  50:0a:09:81:9d:8f:f8:ca 50:0a:09:80:8d:8f:f8:ca
fc1/15           10    0x0f0300  20:41:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1
fc1/15           10    0x0f0301  20:00:00:25:b5:0b:00:df 20:00:00:25:b5:00:00:8f
fc1/15           10    0x0f0302  20:00:00:25:b5:0b:00:ef 20:00:00:25:b5:00:00:9f
fc1/15           10    0x0f0304  20:00:00:25:b5:0b:00:bf 20:00:00:25:b5:00:00:6f
fc1/16           10    0x0f0000  20:42:00:05:73:e3:56:c0 20:14:00:05:73:e3:56:c1
fc1/16           10    0x0f0002  20:00:00:25:b5:0b:00:9f 20:00:00:25:b5:00:00:2f
fc1/16           10    0x0f0004  20:00:00:25:b5:0b:00:6f 20:00:00:25:b5:00:00:3f
fc1/16           10    0x0f0004  20:00:00:25:b5:0b:00:7f 20:00:00:25:b5:00:00:4f
fc1/17           10    0x0f0300  20:41:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1
fc1/17           10    0x0f0301  20:00:00:25:b5:0a:00:ef 20:00:00:25:b5:00:00:8f
fc1/17           10    0x0f0302  20:00:00:25:b5:0a:00:df 20:00:00:25:b5:00:00:9f
fc1/17           10    0x0f0304  20:00:00:25:b5:0a:00:cf 20:00:00:25:b5:00:00:6f
fc1/18           10    0x0f0000  20:42:00:01:73:b3:26:a0 20:14:00:05:73:e3:56:c1
fc1/18           10    0x0f0002  20:00:00:25:b5:0a:00:5f 20:00:00:25:b5:00:00:2f
fc1/18           10    0x0f0004  20:00:00:25:b5:0a:00:bf 20:00:00:25:b5:00:00:3f
fc1/18           10    0x0f0004  20:00:00:25:b5:0a:00:8f 20:00:00:25:b5:00:00:4f

Here we’re defining logical groups of devices and identifying their PWWNs. Nothing too scary here.

fcalias name STORAGE vsan 10
    member pwwn 50:0a:09:81:8d:8f:f8:ca
    member pwwn 50:0a:09:81:9d:8f:f8:ca

fcalias name BLADE_SERVERS vsan 10
    member pwwn 20:00:00:25:b5:0b:00:df
    member pwwn 20:00:00:25:b5:0b:00:ef
    member pwwn 20:00:00:25:b5:0b:00:bf
    member pwwn 20:00:00:25:b5:0b:00:9f
    member pwwn 20:00:00:25:b5:0b:00:6f
    member pwwn 20:00:00:25:b5:0b:00:7f

fcalias name DMZ_SERVERS vsan 10
    member pwwn 20:00:00:25:b5:0a:00:ef
    member pwwn 20:00:00:25:b5:0a:00:df
    member pwwn 20:00:00:25:b5:0a:00:cf
    member pwwn 20:00:00:25:b5:0a:00:5f
    member pwwn 20:00:00:25:b5:0a:00:bf
    member pwwn 20:00:00:25:b5:0a:00:8f

And finally, this is where the proverbial “magic” happens. We’re defining a zone called SERVERS_TO_STORAGE inside vsan 10 and adding our fcalias groups as members of the zone. This allows these group members to communicate with each other. This zone is segmented from our DMZ_SERVERS_TO_STORAGE zone, so that there is no change of communication between our DMZ_SERVERS and BLADE_SERVERS.

zone name BLADE_SERVERS_TO_STORAGE vsan 10
    member fcalias STORAGE
    member fcalias BLADE_SERVERS

zone name DMZ_SERVERS_TO_STORAGE vsan 10
    member fcalias STORAGE
    member fcalias DMZ_SERVERS

Verification

From what I understand the only practical verification you can do is checking your host to ensure you can see the LUNS presented by your storage array. With any luck, everything should show up. If not, double check the PWWNS used in the fcalias groups.

Conclusion

While I have given you little more than a 30-second education on fibre channel, I hope it’s enough that when your storage engineer rambles on endlessly about zoning, you at least have a clue as to what he is actually talking about. And as long as your infrastructure isn’t too complicated, I’m sure you could figure out how to connect your new UCS Blades to their storage without their help. That line is getting awful blurred these days, you might find yourself having to learn more about storage because of it.

Our second visit on day 2 of Network Field day was Brocade, who incidentally supplied us with a great lunch! We spent a little time going through the expected marketing presentation, fortunately they kept it short and to the point… Next up was another short presentation from Jon Hudson, aka @the_socialist, who started things out with a overview of Brocade and their core product line. Fortunately for us, Jon had done his homework and cropped his presentation down to the essentials which aided in keeping our short attention-span on focus. All of this lead up to the surprise they had waiting for us. A live Brocade VCS lab. Yes, you read that correctly. A full, hands on lab. Not a demo, not a video.

For the lab we were broken up into 5 teams of two engineers. Each team had their own switch, with console access, and some OM3 fiber patch cables. They also gave us a packet of information with information about configuring VCS and a list of tasks to accomplish. The icing on the cake was that they wanted us to compete, and allowed us to sabbotage… Serioulsy? Quite a can of worms to open in a room full of the best and brightest engineers. Luckily enough, my partner was the illustrious Greg Ferro, aka the EtherealMind. Next to us, Tom Hollingsworth (NetworkingNerd), and Jeff Fry (FryGuy) joined forces. I knew right out of the gate that they would be our biggest rivals. Once we got started, Greg and I both noted that Brocade’s CLI was very similar to Cisco’s IOS. After some basic configuration tasks, we were ready to cable our switch to the core switches. Greg and I headed off with our OM3 patch cables into the data center right on the coat tails of Tom and Jeff. Once they were patched and headed back in, Greg got us patched in, while I trade out their cables… now, whether or not their LC connectors were slightly tampered with or not is a matter of opinion. Lets just say I was rather tempted to flip send/receive on one of the connectors. Unfortunatley, this is also where things started going downhill. Tom rebooted our switch a couple of times, and ended up changing the password. I chose to bide my time, waiting until we got the password back, and they were on their final configuration steps. Lets just say server was in the middle of a vMotion when it lost connectivity. :) By the time they got things sorted out, we were back up and had already vMotioned our server completing the list of tasks. It was at this point in time that I realized how much entertainment we were providing the team at Brocade, some of which were about to fall out of their chairs.

After the lab, we started up a Q&A with the Brocade team, to paraphrase a quote from @NetworkingNerd, Ivan Pepelnjak and Jon Hudson put on an impressive sparing match over their TRILL implementation (VCS). During most of this time I was having a realization of how powerful their entire presentation was… Starting from a quick overview of Brocade, to the lab, and then holding a Q&A… it was all a very well orchestrated power play on Brocade’s part. I realized how simple their VCS implementation is, the configuration on the fabric switches is just but a few lines, and can obviously be configured by a bunch of caffeine doped apes. (Please ignore my previous comment regarding the best and brightest engineers.) Brocade has done their homework in many ways… and it really made me think long and hard about Brocade and where their going. In a matter of speaking, I’ve had a Brocade wake-up call. (Hence the “This is Jack sleeping on a plane” gif at the top of this article.. ) They’ve made some bold moves in the Ethernet market that will get the attention of engineers around the globe.

The second day of Network Field Day 2 started early at the Juniper EBC, luckily Abner Germanow was prepared with breakfast for the weary and slightly hung over delegates. He gave us an overview of Juniper Networks as a whole including some back history of how they started innovating by putting routing code into ASICs. He quickly handed of to Dan Backman who started off by talking about how Junos has developed itself around workflows. He demonstrated the extensibility of Junos through tools like XML and API calls. Because of the way it was developed, they have the unique ability to provide powerful scripting and automation tools. Dan actually told us that the entire Junos back end is XML, which is VERY interesting. Next he brought up a live Juniper lab to show us the real power of their scripting/automation. This is the first time I’ve heard of Junos commit scripts, which I now wish I had in IOS. During this entire demonstration all of delegates really seemed to enjoy the flexibility Dan was demonstrating, by the end, he had us all drooling over it. And that was before he dropped the bombshell… his entire demonstration had been running inside of Junosphere. Before we were able to bombard him with questions about how to get access to it, he showed some a rather impressive demo using Cariden Mate, and an IS-IS db gathered from what appears to be the I2 backbone. Very cool stuff. Cariden was able to generate a topology from the database, and their plugin for Cariden was able to generate the appropriate Junosphere configuration/startup files. Several times during his presentation he made reference to there being “one more thing” or some secret he wanted to share. It wasn’t long before we learned they were going to give us access to Junosphere for testing! Be on the lookout for my Junosphere review once I’m able to check it out.

Next was “Shoeless” Dave Ward CTO of Juniper’s Platform Division, previously with Cisco Systems for 12 years where he designed little things like the CRS-1, CRS-3, and the ASR 9000 series.. definitely not a light weight. David was also one of the presenters at the Tech Field Day Open Flow Symposium earlier in the week, and as it happens he was back to talk about just that. This is a rather deep dive into Juniper’s approach for Programmable Networking, and I’m not sure I truly do it justice. Juniper has developed their controller inside a orchestration platform they’re calling Junos Space. From the sounds of it, Space is a layer that interacts with a fully comparable OpenFlow controller, David says they even have gear from Big Switch Networks running on their Space platform. He went on to talk about how Juniper has begun to market Service Engineered Paths (SEP), which are simply OpenFlow enabled Traffic Engineered (TE) paths can be programmed with permit/deny statements in front of them for particular service flows. This enables selective traffic path redirection based upon ephemeral data, such as LSPs being requested/configured via the OpenFlow software. This is very cool stuff, auto-magic end-to-end soft MPLS PVCs dynamically configured by the controller! Make sure you watch the video, it’s definitely worth the time.

Next up was a presentation from one of Juniper’s acquisitions, Altor Networks and their virtual firewall appliance, vGW. This is very similar to Cisco’s VSG in that it enables you to firewall all communications within the VMware cluster including those flows going between VMs. Since this isn’t exactly anything new, I’m not going to spend a lot of time covering it. In it’s simplest form, vGW is a high-performance hypervisor based stateful firewall with an integrated IDS, and AntiVirus protection. Update: Apparently I missed some details during this discussion. Luckily enough, Ivan has come out with another great post letting us know the significance of what Juniper has done with their vGW.

Last but certainly not least was a presentation on QFabric. Juniper’s answer to Cisco’s Fabric Path, Brocade’s VCS, and of course the industry standard TRILL. It is of course a proprietary solution, but in some cases that can be a non-issue. Of course, at around 5:20 into the vidoe, Juniper presented us with their own version of “the problem“, a series of slides that by this point in time we have seen over and over again… Although once Tony Bourke mentioned that we were well aware of “the problem” he recovered quickly and moved on. (Kudos for Tony for quite a tactful engagement and remediation of the situation!) Again, there isn’t a lot of ground breaking going on during this talk so I wont spend much time here… But if you’re dying for more information check out the artciles at IOShints and the Packet Pushers Blog / Podcast.

The last stop was their proof of concept lab which for a bunch of networking geeks is a great way to earn extra credit. Who doesn’t love a data center? Even better, the entire data center was a lab! Here are a few photos from the EBC Lab.