As many of you may know, I’m in the middle of a huge network redesign, last week our new firewalls finally arrived and it became time for us to start migrating services onto the edge network I’ve been building for the past couple months. Unfortunately, the first thing they wanted to move was a group of new proxy servers. Since they were already re-addressed and ready for the new infrastructure we figured some policy based routing (PBR) would do the trick until we were ready to flip the switch and change our default route to point out the new edge network.
I spent about a couple minutes writing up the configuration that we would need, nothing too complex. Since we planned out pointing the whole Vlan out the new infrastructure, I wrote 2 ACLs and a route-map that should’ve accomplished the following.
- Match packets destined for our internal network and exclude them from having their next-hop changed.
- Match packets with our subnet’s source address and change their next-hop.
- Make things work happily forever after, or something like that.
no ip access-list PBR-Internal ip access-list PBR-Internal permit ip any 10.0.0.0/8 permit ip any 172.16.0.0/12 permit ip any 192.168.0.0/16 ! no ip access-list DMZ-PBR-SRC ip access-list DMZ-PBR-SRC permit ip 10.255.255.0/24 any ! no route-map DMZ-PBR route-map DMZ-PBR deny 10 match ip address PBR-Internal route-map DMZ-PBR permit 20 match ip address DMZ-PBR-SRC set ip next-hop 10.255.184.246 ! int vlan 1255 ip policy route-map DMZ-PBR
Unfortunately what I found when I applied this configuration was that I was no longer able to speak with anything in that subnet. I assumed that there was a bug in the code, but I wasn’t sure how to work around it, not without making our firewall guy hairpin all of that traffic that is. I emailed TAC who informed me that the Nexus has a hardware limitation that prevented deny statements in PBR route-maps… odd, I knew that IOS wouldn’t let you put a deny statement in an ACL, but I never heard such a thing about NX-OS.
A few minutes later I came up with an idea that will make some of you laugh, others will puke… Since I now needed an ACL that would match EVERYTHING on the internet, except RFC1918 addresses I decided to write just that. (TAC offered to write one, and I let them, but I figured mine would be done first… and it was) Luckily NX-OS lets me use CIDR addresses inside ACLs.
no ip access-list DMZ-PBR ip access-list DMZ-PBR permit ip 10.255.255.0/24 0.0.0.0/5 permit ip 10.255.255.0/24 18.104.22.168/7 permit ip 10.255.255.0/24 22.214.171.124/8 permit ip 10.255.255.0/24 126.96.36.199/6 permit ip 10.255.255.0/24 188.8.131.52/4 permit ip 10.255.255.0/24 184.108.40.206/3 permit ip 10.255.255.0/24 220.127.116.11/3 permit ip 10.255.255.0/24 18.104.22.168/2 permit ip 10.255.255.0/24 22.214.171.124/3 permit ip 10.255.255.0/24 126.96.36.199/5 permit ip 10.255.255.0/24 188.8.131.52/6 permit ip 10.255.255.0/24 184.108.40.206/12 permit ip 10.255.255.0/24 220.127.116.11/11 permit ip 10.255.255.0/24 18.104.22.168/10 permit ip 10.255.255.0/24 22.214.171.124/9 permit ip 10.255.255.0/24 126.96.36.199/8 permit ip 10.255.255.0/24 188.8.131.52/7 permit ip 10.255.255.0/24 184.108.40.206/4 permit ip 10.255.255.0/24 192.0.0.0/9 permit ip 10.255.255.0/24 220.127.116.11/11 permit ip 10.255.255.0/24 18.104.22.168/13 permit ip 10.255.255.0/24 22.214.171.124/16 permit ip 10.255.255.0/24 126.96.36.199/15 permit ip 10.255.255.0/24 188.8.131.52/14 permit ip 10.255.255.0/24 184.108.40.206/12 permit ip 10.255.255.0/24 220.127.116.11/10 permit ip 10.255.255.0/24 18.104.22.168/8 permit ip 10.255.255.0/24 22.214.171.124/7 permit ip 10.255.255.0/24 126.96.36.199/6 permit ip 10.255.255.0/24 188.8.131.52/5 permit ip 10.255.255.0/24 184.108.40.206/4 permit ip 10.255.255.0/24 220.127.116.11/3 ! no route-map DMZ-PBR route-map DMZ-PBR permit 10 match ip address DMZ-PBR set ip next-hop 10.255.184.246 ! int vlan 1255 ip policy route-map DMZ-PBR
Yes, its big, and ugly, but it works. Some of you may notice that 127/8 and 224/8 are in there, I’m not too worried about it for now. After all, the only thing using it are a couple Websense proxy severs… What’s hilarious is the reason I beat TAC writing this by a few hours… their ACL was 607 lines long. They apparently refused to do any subnets larger than a /8… If you have time to leave a comment, I’d love to hear your stories about network duct tape gone wrong.
Network engineer turned management currently servicing the enterprise data center market. I started working on networks in the ’90s and still feel like that was just a few years ago. Jack of all trades, master of none; I love to learn about everything. Feel free to ask me about photography, woodworking, nhra, watches, or even networking! — For feedback, please leave a comment on the article in question, and I’ll respond as soon as I can. For everything else including fan mail or death threats, contact me via twitter.