Welcome back! I missed you.
For the last year or so, we've been on a long-term plan to eliminate single points of failure (SPOF) in our network. I've blogged about several of them – I'll list a few posts below that you might enjoy reading.
One of the last pieces of the puzzle – Internet Redundancy – fell into place yesterday afternoon.
We now have a fully redundant BGP setup – with 2x identical burstable AT&T EaMIS 100meg Internet connections, in two locations 7 miles apart – and connected via Gigaman (Gigabit Metro Ethernet).
I thought it would be helpful to share our router configs and some of the internal HSRP setup as well. These are scrubbed with only the relevant info – so don't bother looking for fun stuff 🙂
HSRP – Internal Layer 3 VLANs
So, since we have internal redundancy (and have for a while) I thought I'd share that setup. The purpose and goal is to make sure that each affected VLAN has a "shared" gateway, so if one side dies, the other takes over.
Here's an example. Our data vlan 3 is for some "typical" enduser clients. The subnet is the standard 192.168.1.0/24 class C. The "gateway" is 192.168.1.254.
Side A – Primary – "data vlan 3" – Cisco 6509
interface Vlan3
ip address 192.168.1.252 255.255.255.0
ip helper-address 10.5.1.71
standby 3 ip 192.168.1.254
standby 3 priority 200
standby 3 preempt
end
Side B – Secondary – "data vlan 3" – Cisco 4507R
interface Vlan3
ip address 192.168.1.253 255.255.255.0
ip helper-address 10.5.1.71
standby 3 ip 192.168.1.254
end
So, if you follow along – Side A has the .252 IP. Side B has the .253 IP. Both "share" the .254 IP. Side A has the highest priority – and "preempts" side B – which means, that Side A should ALWAYS be IP .254 unless it fails, then Side B takes over. The "preempt" means that as soon as Side A comes available again, it will take back over. So, it's active/standby failover, and it will "fix itself" if it does happen to failover.
HSRP – Public Internet
Very similarly to our Internal Layer 3 VLANs, I also use HSRP to handle the "inside" part of our Public Internet routers. We utilize two public /24 (Class C) spaces. One is subnet 211 and one is subnet 212. Both subnets are advertised and have public HSRP setup.
Side A – Primary – "subnet 211" – Secondary – "subnet 212" – Cisco 3825
interface GigabitEthernet0/0
description Ethernet to LAN
ip address x.x.212.253 255.255.255.0 secondary
ip address x.x.211.252 255.255.255.0
no ip redirects
no ip proxy-arp
duplex auto
speed auto
media-type rj45
no keepalive
no cdp enable
standby 211 ip x.x.211.1
standby 211 priority 200
standby 211 preempt
standby 212 ip x.x.212.1
end
Side B – Secondary – "subnet 211" – Primary – "subnet 212" – Cisco 2821
interface GigabitEthernet0/0
description Ethernet to LAN
ip address x.x.211.253 255.255.255.0 secondary
ip address x.x.212.252 255.255.255.0
no ip redirects
no ip proxy-arp
duplex auto
speed auto
no keepalive
traffic-shape rate 1288000 128000 128000 4096
no cdp enable
no mop enabled
standby 211 ip x.x.211.1
standby 212 ip x.x.212.1
standby 212 priority 200
standby 212 preempt
end
Make sense? Notice that both Interfaces have IPs, and I have 2x (two) HSRP groups – one handling each subnet. Also, I've traffic shaped Side B – we have 100megs, but pay for a simple T1 CIR, so, I want to make sure we aren't surprised by any major traffic. I can adjust that in an emergency, but, I keep it traffic shaped in a "normal" situation.
BGP – Public Internet
This is the piece that was a little more challenging. I won't go into the details of BGP, but I will tell you that HSRP will give our "internal" network it's proper gateway – so – HSRP will get me to the Internet. But, BGP is what gets the "internet" back to me. Both were needed.
In my environment, I was both subnet 211 and 212 to be available at all times, so, both routers will "Advertise" both subnets. Let's take a look at the setup.
Side A – Primary – "subnet 211" – Secondary – "subnet 212" – Cisco 3825
interface GigabitEthernet0/1
description Ethernet to WAN / EaMIS
ip address x.x.x.106 255.255.255.252
duplex full
speed 100
media-type rj45
no keepalive
!
router bgp 2386
no synchronization
bgp log-neighbor-changes
network x.x.211.0 mask 255.255.255.0
network x.x.212.0 mask 255.255.255.0
neighbor x.x.x.105 remote-as 7018
neighbor x.x.x.105 prefix-list defaultonly in
neighbor x.x.x.105 route-map 211primary out
no auto-summary
!
ip bgp-community new-format
!
ip prefix-list defaultonly seq 5 permit 0.0.0.0/0
!
access-list 1 permit x.x.211.0
access-list 2 permit x.x.212.0
!
route-map 211primary permit 10
match ip address 1
!
route-map 211primary permit 20
match ip address 2
set as-path prepend 2386 2386 2386
Side B – Secondary – "subnet 211" – Primary – "subnet 212" – Cisco 2821
interface GigabitEthernet0/1
description Ethernet to WAN / EaMIS
ip address x.x.x.94 255.255.255.252
duplex full
speed 100
no keepalive
traffic-shape rate 1288000 128000 128000 4096
!
router bgp 2386
no synchronization
bgp log-neighbor-changes
network x.x.211.0 mask 255.255.255.0
network x.x.212.0 mask 255.255.255.0
neighbor x.x.x.93 remote-as 7018
neighbor x.x.x.93 prefix-list defaultonly in
neighbor x.x.x.93 route-map 212primary out
no auto-summary
!
ip bgp-community new-format
!
ip prefix-list defaultonly seq 5 permit 0.0.0.0/0
!
access-list 1 permit x.x.212.0
access-list 2 permit x.x.211.0
!
route-map 212primary permit 10
match ip address 1
!
route-map 212primary permit 20
match ip address 2
set as-path prepend 2386 2386 2386
Did you follow along? Gig0/1 is the "public" / WAN interface. The BGP ASN (Autonomous System Number) we're using is 2386 – that was given to us from AT&T and is a "common" ASN used by customers that are multi-homed and doing BGP. The BGP ASN 7018 is the AT&T ASN. I had to tell my router who my "neighbor" was, and then had to "advertise" both my 211 and 212 networks. That's what you see in the "router bgp 2386" command.
The ""prefix-list" shows that instead of holding all routing tables, I'm only interested in my "next hop" route. I just want to know how to get from my router into the AT&T backbone. That's it.
The two route-maps help me tell my routers which subnet (211 or 212) is the "primary" subnet out that router. You'll notice a "set as-path prepend 2386 2386 2386" command. Basically, when using the prepend command, I'm telling my router that THAT path is secondary and not as desirable. That makes sense, right? For example on my Cisco 3825, it's "primary" for subnet 211, so the route-map prepends for subnet 212.
Anyway, here's a small Visio PDF that shows what this looks like.
Hope it helps. It's hard to find "exact" or "production" BGP setups. Ours is not complicated for what we want to do – and it does work – this is not theory – this is real life production – right now.