Network Duct Tape Gone Wrong

As many of you may know, I’m in the middle of a huge network redesign, last week our new firewalls finally arrived and it became time for us to start migrating services onto the edge network I’ve been building for the past couple months. Unfortunately, the first thing they wanted to move was a group of new proxy servers. Since they were already re-addressed and ready for the new infrastructure we figured some policy based routing (PBR) would do the trick until we were ready to flip the switch and change our default route to point out the new edge network.

I spent about a couple minutes writing up the configuration that we would need, nothing too complex. Since we planned out pointing the whole Vlan out the new infrastructure, I wrote 2 ACLs and a route-map that should’ve accomplished the following.

1. Match packets destined for our internal network and exclude them from having their next-hop changed.
2. Match packets with our subnet’s source address and change their next-hop.
3. Make things work happily forever after, or something like that.

no ip access-list PBR-Internal
ip access-list PBR-Internal
  permit ip any 10.0.0.0/8 
  permit ip any 172.16.0.0/12 
  permit ip any 192.168.0.0/16 
!
no ip access-list DMZ-PBR-SRC
ip access-list DMZ-PBR-SRC
  permit ip 10.255.255.0/24 any 
!
no route-map DMZ-PBR
route-map DMZ-PBR deny 10
  match ip address PBR-Internal
route-map DMZ-PBR permit 20
  match ip address DMZ-PBR-SRC 
  set ip next-hop 10.255.184.246 
!
int vlan 1255
   ip policy route-map DMZ-PBR

Unfortunately what I found when I applied this configuration was that I was no longer able to speak with anything in that subnet. I assumed that there was a bug in the code, but I wasn’t sure how to work around it, not without making our firewall guy hairpin all of that traffic that is. I emailed TAC who informed me that the Nexus has a hardware limitation that prevented deny statements in PBR route-maps… odd, I knew that IOS wouldn’t let you put a deny statement in an ACL, but I never heard such a thing about NX-OS.

A few minutes later I came up with an idea that will make some of you laugh, others will puke… Since I now needed an ACL that would match EVERYTHING on the internet, except RFC1918 addresses I decided to write just that. (TAC offered to write one, and I let them, but I figured mine would be done first… and it was) Luckily NX-OS lets me use CIDR addresses inside ACLs.

no ip access-list DMZ-PBR
ip access-list DMZ-PBR
  permit ip 10.255.255.0/24 0.0.0.0/5
  permit ip 10.255.255.0/24 8.0.0.0/7
  permit ip 10.255.255.0/24 11.0.0.0/8
  permit ip 10.255.255.0/24 12.0.0.0/6
  permit ip 10.255.255.0/24 16.0.0.0/4
  permit ip 10.255.255.0/24 32.0.0.0/3
  permit ip 10.255.255.0/24 64.0.0.0/3
  permit ip 10.255.255.0/24 96.0.0.0/2
  permit ip 10.255.255.0/24 128.0.0.0/3
  permit ip 10.255.255.0/24 160.0.0.0/5
  permit ip 10.255.255.0/24 168.0.0.0/6
  permit ip 10.255.255.0/24 172.0.0.0/12
  permit ip 10.255.255.0/24 172.32.0.0/11
  permit ip 10.255.255.0/24 172.64.0.0/10
  permit ip 10.255.255.0/24 172.128.0.0/9
  permit ip 10.255.255.0/24 173.0.0.0/8
  permit ip 10.255.255.0/24 174.0.0.0/7
  permit ip 10.255.255.0/24 176.0.0.0/4
  permit ip 10.255.255.0/24 192.0.0.0/9
  permit ip 10.255.255.0/24 192.128.0.0/11
  permit ip 10.255.255.0/24 192.160.0.0/13
  permit ip 10.255.255.0/24 192.169.0.0/16
  permit ip 10.255.255.0/24 192.170.0.0/15
  permit ip 10.255.255.0/24 192.172.0.0/14
  permit ip 10.255.255.0/24 192.176.0.0/12
  permit ip 10.255.255.0/24 192.192.0.0/10
  permit ip 10.255.255.0/24 193.0.0.0/8
  permit ip 10.255.255.0/24 194.0.0.0/7
  permit ip 10.255.255.0/24 196.0.0.0/6
  permit ip 10.255.255.0/24 200.0.0.0/5
  permit ip 10.255.255.0/24 208.0.0.0/4
  permit ip 10.255.255.0/24 224.0.0.0/3
!
no route-map DMZ-PBR
route-map DMZ-PBR permit 10
  match ip address DMZ-PBR 
  set ip next-hop 10.255.184.246 
!
int vlan 1255
   ip policy route-map DMZ-PBR

Yes, its big, and ugly, but it works. Some of you may notice that 127/8 and 224/8 are in there, I’m not too worried about it for now. After all, the only thing using it are a couple Websense proxy severs… What’s hilarious is the reason I beat TAC writing this by a few hours… their ACL was 607 lines long. They apparently refused to do any subnets larger than a /8… If you have time to leave a comment, I’d love to hear your stories about network duct tape gone wrong.