Troubleshooting BGP

Troubleshooting BGP
Philip Smith
<philip@nsrc.org>
APRICOT 2016
22nd-26th February 2016
Auckland, New Zealand
Last updated 21st February 2016

Presentation Slides
p  Will be available on
n  http://bgp4all.com/ftp/seminars/
APRICOT2016-Troubleshooting-BGP.pdf
n  And on the APRICOT2016 website
p  Feel free to ask questions any time

Assumptions
p  Presentation assumes working knowledge
of BGP
n  Beginner and Intermediate experience of
protocol
p  Knowledge of Cisco CLI
n  Hopefully you can translate concepts into your
own router CLI
p  If in any doubt, please ask!

Agenda
p  Fundamentals of Troubleshooting
p  Local Configuration Problems
p  Internet Reachability Problems

Fundamentals:
Problem Areas
p  First step is to recognise what usually
causes problems
p  Possible Problem Areas:
n  Misconfiguration
p  Configuration errors caused by bad documentation,
misunderstanding of concepts, poor communication
between colleagues or departments
n  Human error
p  Typos, using wrong commands, accidents, poorly
planned maintenance activities

Fundamentals:
Problem Areas
p  More Possible Problem Areas:
n  “feature behaviour”
p  Or – “it used to do this with Release X.Y(a) but
Release X.Y(b) does that”
n  Interoperability issues
p  Differences in interpretation of RFC1771 and its
developments (RFC4271 replaces RFC1771)
n  Those beyond your control
p  Upstream ISP or peers make a change which has an
unforeseen impact on your network

Fundamentals:
Working on Solutions
p  Next step is to try and fix the problem
n  And this is not about diving into network and
trying random commands on random routers,
just to “see what difference this makes”
p  The best procedure for “unfamiliar
problems” is to
n  Start at one place,
n  Deal with one symptom, and learn more about
it

Fundamentals:
Working on Solutions
p  Remember! Troubleshooting is about:
n  Not panicking
n  Creating a checklist
n  Working to that checklist
n  Starting at the bottom and working up

Fundamentals:
Checklists
p  This presentation will have references in
the later stages to checklists
n  They are the best way to work to a solution
n  They are what many NOC staff follow when
diagnosing and solving network problems
n  It may seem daft to start with simple tests
when the problem looks complex
p  But quite often the apparently complex can be solved
quite easily

Fundamentals:
Tools
p  Use system and network logs as an aid
p  Record keeping:
n  Good and detailed system logs
n  Last known good configuration
n  History trail of working configurations and all
intermediate changes
n  Record of commands entered on routers and
other network devices

Fundamentals:
Tools
p  Familiarise yourself with the router’s tools:
n  Is logging of the BGP process enabled?
p  (And is it captured/recorded off the router?)
n  Are you familiar with the BGP debug process
and commands (if available)
p  Check vendor documentation before switching on full
BGP debugging – you might get fewer surprises

Fundamentals:
Tools
p  Traffic and traffic flow measurement in the
network
n  Unexplained change in traffic levels on an
interface, a connection, a peering,…
n  Correlation of customer feedback on network
or connectivity issues…

Agenda
p  Fundamentals
p  Local Configuration Problems
p  Internet Reachability Problems

Local Configuration Problems
p  Peer Establishment
p  Missing Routes
p  Inconsistent Route Selection
p  Loops and Convergence Issues

Peer Establishment
p  Routers establish a TCP session
n  Port 179 – Permit in interface filters
n  IP connectivity (route from IGP)
p  OPEN messages are exchanged
n  Peering addresses must match the
TCP session
n  Local AS configuration parameters

Common Problems
p  Sessions are not established
n  No IP reachability
n  Incorrect configuration
p  Peers are flapping
n  Layer 2 problems

Peer Establishment:
Diagram
R2#sh run | begin ^router bgp
router bgp 1
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 1
AS 1
AS 2
R1
iBGP
eBGP
1.1.1.1 2.2.2.2
3.3.3.3
?
?
R2
R3

R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
1.1.1.1 4 1 0 0 0 0 0 never Active
3.3.3.3 4 2 0 0 0 0 0 never Idle
Peer Establishment:
Symptoms
p  Both peers are having problems
n  State may change between Active, Idle and Connect

R2#
router bgp 1
Local AS
eBGP Peer
iBGP Peer
Peer Establishment
p  Is the Local AS configured correctly?
p  Is the remote-as assigned correctly?
p  Verify with your diagram or other documentation!

p  Assume that IP connectivity has been checked
p  Check TCP to find out what connections we are accepting
p  We Are Listening for TCP Connections for Port 179 for the
Configured Peering Addresses Only!
p  Remote Is Trying to Open the Session from 4.4.4.4 Address…
R2#show tcp brief all
TCB Local Address Foreign Address (state)
005F2934 *.179 3.3.3.3.* LISTEN
0063F3D4 *.179 1.1.1.1.* LISTEN
R2#debug ip tcp transactions
TCP special event debugging is on
R2#
TCP: sending RST, seq 0, ack 2500483296
TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179
Peer Establishment:
iBGP

R2#debug ip bgp
BGP debugging is on
R2#
BGP: 1.1.1.1 open active, local address 4.4.4.5
BGP: 1.1.1.1 open failed: Connection refused by remote host
R2#sh ip route 1.1.1.1
Routing entry for 1.1.1.1/32
Known via "static", distance 1, metric 0 (connected)
* directly connected, via Serial1
Route metric is 0, traffic share count is 1
R2#show ip interface brief | include Serial1
Serial1 4.4.4.5 YES manual up up
Peer Establishment:
iBGP
p  What about us?
p  We are trying to open the session from 4.4.4.5 address…

R2#
router bgp 1
neighbor 1.1.1.1 update-source Loopback0
Peer Establishment:
iBGP
p  Source address is the outgoing interface towards
the destination but peering in this case is using
loopback interfaces!
p  Force both routers to source from the correct
interface
p  Use “update-source” to specify the loopback
when loopback peering

Peer Establishment:
iBGP – Summary
p  Assume that IP connectivity has been checked
n  Including IGP reachability between peers
p  Check TCP to find out what connections we are
accepting
n  Check the ports and source/destination addresses
n  Do they match the configuration?
p  Common problem:
n  iBGP is run between loopback interfaces on router (for
stability), but the configuration is missing from the
router ⇒ iBGP fails to establish
n  Remember that source address is the IP address of the
outgoing interface unless otherwise specified

AS 1
AS 2
R1
iBGP
eBGP
1.1.1.1 2.2.2.2
3.3.3.3
?
R2
R3
Peer Establishment:
Diagram
p  R1 is established now
p  The eBGP session is still having trouble!

R2#ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
Peer Establishment:
eBGP
p  Trying to load-balance over multiple links to the
eBGP peer
p  Verify IP connectivity
n  Check the routing table
n  Use ping/trace to verify two way reachability
p  Routing towards destination is correct, but…

R2#ping ip
Target IP address: 3.3.3.3
Extended commands [n]: y
Source address or interface: 2.2.2.2
.....
Success rate is 0 percent (0/5)
Peer Establishment:
eBGP
p  Use extended pings to test loopback to loopback
connectivity
p  R3 does not have a route to our loopback, 2.2.2.2

p  Assume R3 added a route to 2.2.2.2
p  Still having problems…
R2#sh ip bgp neigh 3.3.3.3
BGP neighbor is 3.3.3.3, remote AS 2, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle
Last read 00:00:04, hold time is 180, keepalive interval is 60 seconds
Received 0 messages, 0 notifications, 0 in queue
Sent 0 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Default minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 1, neighbor version 0
Index 2, Offset 0, Mask 0x4
0 accepted prefixes consume 0 bytes
Prefix advertised 0, suppressed 0, withdrawn 0
Connections established 0; dropped 0
Last reset never
External BGP neighbor not directly connected.
No active TCP connection
Peer Establishment:
eBGP

p  eBGP peers are normally directly connected
n  By default, TTL is set to 1 for eBGP peers
n  If not directly connected, specify ebgp-multihop
p  At this point, the session should come up
R2#
router bgp 1
neighbor 3.3.3.3 ebgp-multihop 2
Peer Establishment:
eBGP

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
3.3.3.3 4 2 10 26 0 0 0 never Active
Peer Establishment:
eBGP
p  Still having trouble!
n  Connectivity issues have already been checked and
corrected

R2#debug ip bgp events
14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.2
14:06:37: BGP: 3.3.3.3 went from Active to OpenSent
14:06:37: BGP: 3.3.3.3 sending OPEN, version 4
14:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 0001
14:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT
14:06:37: BGP: service reset requests
14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle
14:06:37: BGP: 3.3.3.3 closing
Peer Establishment:
eBGP
p  If an error is detected, a notification is sent and the session
is closed
p  R3 is configured incorrectly
n  Has “neighbor 2.2.2.2 remote-as 10”
n  Should have “neighbor 2.2.2.2 remote-as 1”
p  After R3 makes this correction the session should come up

access-list 100 permit tcp host 3.3.3.3 eq 179 host 2.2.2.2
access-list 100 permit tcp host 3.3.3.3 host 2.2.2.2 eq 179
Peer Establishment:
eBGP – Summary
p  Remember to allow TCP/179 through edge filters
p  Be very careful with multihop eBGP
n  Check IP connectivity (local and remote routing tables)
n  Remember to source updates from loopback
n  Watch for filters anywhere in the path
n  TTL must be at least 2 for ebgp-multihop between
directly connected neighbours
p  Use TTL value carefully

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
3.3.3.3 4 2 10 26 0 0 0 never Active
Peer Establishment:
Passwords
p  Using passwords on iBGP and eBGP sessions
n  Link won’t come up
n  Been through all the previous troubleshooting steps

R2#
router bgp 1
neighbor 3.3.3.3 password 7 05080F1C221C
%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179
to 2.2.2.2:11272
to 2.2.2.2:11272
to 2.2.2.2:11272
Peer Establishment:
Passwords
p  Configuration on R2 looks fine!
p  Check the log messages – enable “log-neighbor-changes”

R3#
router bgp 2
Peer Establishment:
Passwords
p  Check configuration on R3
n  Password is missing from the eBGP configuration
p  Fix the R3 configuration
n  Peering should now come up!
n  But it does not

Peer Establishment:
Passwords
p  Let’s look at the log messages again for clues
R2#
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
p  We are getting invalid MD5 digest messages –
password mismatch!

p  We must have mis-typed the password on one of
the peering routers
n  Fix the password – best to re-enter password on both
routers
n  eBGP session now comes up
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11027
to 2.2.2.2:179
%BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up
Peer Establishment:
Passwords

Peer Establishment:
Passwords – Summary
p  Common problems:
n  Missing password – needs to be on both ends
n  Cut and paste errors – don’t!
n  Typographical & transcription errors
n  Capitalisation, extra characters, white space…
p  Common solutions:
n  Check for symptoms/messages in the logs
n  Re-enter passwords using keyboard, from
scratch – don’t cut&paste

p  Symptoms – the eBGP session flaps
p  eBGP peering establishes, then drops, re-
establishes, then drops,…
AS 2AS 1
Layer 2
eBGP R2R1
Flapping Peer:
Common Symptoms

R2#
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time
expired) 0 bytes
R2#show ip bgp neighbor 1.1.1.1 | include Last reset
Last reset 00:01:02, due to BGP Notification sent, hold time
expired
Flapping Peer
p  Ensure BGP neighbour logging is enabled
n  no logs ⇒ no clue what is going on
p  R1 and R2 are peering over some 3rd party L2
network
p  We are not receiving keepalives from the other
side!

2.2.2.2 4 2 53 284 10167 0 97 00:02:15 0
R1#show ip bgp summary | begin Neighbor
2.2.2.2 4 2 53 284 10167 0 98 00:03:04 0
Flapping Peer
p  Let’s take a look at our peer!
p  Hellos are stuck in OutQ behind update packets!
p  Notice that the MsgSent counter has not moved

R1#ping 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 ms
R1#ping ip
Target IP address: 2.2.2.2
Repeat count [5]:
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
.....
Success rate is 0 percent (0/5)
Flapping Peer
p  Normal pings work but a 1500byte ping fails?

Flapping Peer:
Diagnosis and Solution
p  Diagnosis
n  Keepalives get lost because they get stuck in the
router’s queue behind BGP update packets.
n  BGP update packets are packed to the size of the MTU –
keepalives and BGP OPEN packets are not packed to the
size of the MTU ⇒ Path MTU problems
n  Use ping with different size packets to confirm the above
– 100byte ping succeeds, 1500byte ping fails = MTU
problem somewhere
p  Solution
n  Pass the problem to the L2 folks – but be helpful, try
and pinpoint using ping where the problem might be in
the network

Flapping Peer:
Other Common Problems
p  Remote router rebooting continually (typical with
a 3-5 minute BGP peering cycle time)
p  Remote router BGP process unstable, restarting
p  Traffic Shaping & Rate Limiting parameters
p  MTU incorrectly set on links, PMTU discovery
disabled on router
p  For non-ATM/FR links, instability in the L2 point-
to-point circuits
n  Faulty MUXes, bad connectors, interoperability
problems, PPP problems, satellite or radio problems,
weather, etc. The list is endless – your L2 folks should
know how to solve them
n  For you, ping is the tool to use

Flapping Peer:
Fixed!
p  Large packets are ok now
p  BGP session is stable!
AS 2AS 1
Layer 2
eBGP R2R1
Small Packets
Large Packets

Quick Review
p  Once the session has been established,
UPDATEs are exchanged
n  All the locally known routes
n  Only the bestpath is advertised
p  Incremental UPDATE messages are
exchanged afterwards

Quick Review
p  Bestpath received from eBGP peer
n  Advertise to all peers
p  Bestpath received from iBGP peer
n  Advertise only to eBGP peers
n  A full iBGP mesh must exist
p  (Unless we are using Route Reflectors)

Missing Routes
p  Route Origination
p  UPDATE Exchange
p  Filtering
p  iBGP mesh problems

Missing Routes:
Route Origination
p  Common problem occurs when putting
prefixes into the BGP table
p  BGP table is NOT the RIB
n  (RIB = Routing Information Base – a.k.a the
Routing Table)
n  BGP table, as with OSPF table, ISIS table,
static routes, etc, is used to feed the RIB, and
hence the FIB
n  Each routing protocol has a different priority or
“distance”

Missing Routes:
Route Origination
p  To get a prefix into BGP, it must exist in
another routing process too, typically:
n  Static route pointing to customer (for customer
routes into your iBGP)
n  Static route pointing to Null (for aggregates
you want to put into your eBGP)

Route Origination:
Example I
p  Network statement
p  BGP is not originating the route???
p  Do we have the exact route?
R1# show run | include 200.200.0.0
network 200.200.0.0 mask 255.255.252.0
R1# show ip bgp | include 200.200.0.0
R1#
R1# show ip route 200.200.0.0 255.255.252.0
% Network not in table

p  Nail down routes you want to originate
p  Check the RIB
p  BGP originates the route!!
Route Origination:
Example I
ip route 200.200.0.0 255.255.252.0 Null0 254
R1# show ip route 200.200.0.0 255.255.252.0
200.200.0.0/22 is subnetted, 1 subnets
S 200.200.0.0 [1/0] via Null 0
R1# show ip bgp | include 200.200.0.0
*> 200.200.0.0/22 0.0.0.0 0 32768

p  Trying to originate an aggregate route
p  The RIB has a component but BGP does not
create the aggregate???
Route Origination:
Example II
aggregate-address 7.7.0.0 255.255.0.0 summary-only
R1# show ip route 7.7.0.0 255.255.0.0 longer
7.0.0.0/32 is subnetted, 1 subnets
C 7.7.7.7 [1/0] is directly connected, Loopback 0
R1# show ip bgp | i 7.7.0.0
R1#

p  Remember, to have a BGP aggregate you need a
BGP component, not a RIB component
p  Once BGP has a component route we originate
the aggregate
p  s means this component is suppressed due to the
“summary-only” argument
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
R1#
network 7.7.7.7 mask 255.255.255.255
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
*> 7.7.0.0/16 0.0.0.0 32768 i
s> 7.7.7.7/32 0.0.0.0 0 32768 i
Route Origination:
Example II

Troubleshooting Tips
p  BGP Network statement rules
n  Always need an exact route (RIB)
p  aggregate-address looks in the BGP
table, not the RIB
p  Showing RIB component routes:
show ip route x.x.x.x y.y.y.y longer
p  Showing BGP component routes:
show ip bgp x.x.x.x y.y.y.y longer

Missing Routes:
Update Exchange
p  Ah, Route Reflectors…
n  Such a nice solution to help scale iBGP
n  But why do people insist in breaking the rules
all the time?!
p  Common issues
n  Clashing router IDs
n  Clashing cluster IDs

R1
R3
R2
R4
Missing Routes:
Example I
p  Two RR clusters
p  R1 is a RR for R3
p  R2 is a RR for R4
p  R4 is advertising
7.0.0.0/8
n  R2 has the route
n  R1 and R3 do not

p  First, did R2 advertise the route to R1?
p  Did R1 receive it?
R2# show ip bgp neighbors 1.1.1.1 advertised-routes
BGP table version is 2, local router ID is 2.2.2.2
Network Next Hop Metric LocPrf Weight Path
*>i7.0.0.0 4.4.4.4 0 100 0 i
R1# show ip bgp neighbors 2.2.2.2 routes
Total number of prefixes 0
Missing Routes:
Example I

p  Time to debug!!
p  Tell R2 to resend his UPDATEs
p  R1 shows us something interesting
p  Cannot accept an update with our Router-ID as
the ORIGINATOR_ID. Another means of loop
detection in BGP
Missing Routes:
Example I
access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
R2# clear ip bgp 1.1.1.1 out
*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE w/ attr:
nexthop 4.4.4.4, origin i, localpref 100, metric 0,
originator 100.1.1.1, clusterlist 2.2.2.2, path , community ,
extended community
*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE about
7.0.0.0/8 -- DENIED due to: ORIGINATOR is us;

p  R1 is not accepting the route when R2 sends it on
from its client, R4
n  R1 and R4 have the same router ID!
n  If R1 sees its own router ID in the originator attribute in
any received prefix, it will reject that prefix
p  This is how a route reflector attempts to avoid routing
loops
p  Solution
n  Do NOT set the router ID by hand unless you have a
very good reason to do so and have a very good plan for
deployment
n  Router-ID is usually calculated automatically by router
Missing Routes:
Example I – Summary

R1#show run | include cluster
bgp cluster-id 10
R2#show run | include cluster
bgp cluster-id 10
R1
R3
R2
R4
Missing Routes:
Example II
p  One RR cluster
p  R1 and R2 are RRs
p  R3 and R4 are RRCs
p  R4 is advertising
7.0.0.0/8
n  R2 has the route
n  R1 and R3 do not

p  Same troubleshooting steps as for the previous
example!
p  Did R2 advertise it to R1?
p  Did R1 receive it?
R2# show ip bgp neighbors 1.1.1.1 advertised-routes
Origin codes: i - IGP, e - EGP, ? – incomplete
*>i7.0.0.0 4.4.4.4 0 100 0 i
R1# show ip bgp neighbor 2.2.2.2 routes
Missing Routes:
Example II

p  Time to debug!!
p  Tell R2 to resend his UPDATEs
p  R1 shows us something interesting
p  Remember, all RRCs must peer with all RRs in a
cluster; allows R4 to send the update directly to R1
Missing Routes:
Example II
access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
R2# clear ip bgp 1.1.1.1 out
Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE w/ attr:
nexthop 4.4.4.4, origin i, localpref 100, metric 0, originator
4.4.4.4, clusterlist 0.0.0.10, path , community , extended
community
Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE about
7.0.0.0/8 -- DENIED due to: reflected from the same cluster;

p  R1 is not accepting the route when R2 sends it on
n  If R1 sees its own router ID in the cluster-ID attribute in
any received prefix, it will reject that prefix
p  How a route reflector avoids redundant information
p  Reason
n  Early documentation claimed that RRC redundancy
should be achieved by dual route reflectors in the same
cluster
n  This is fine and good, but then ALL clients must peer
with both RRs, otherwise examples like this will occur
p  Solution
n  Use overlapping Route Reflector Clusters for
redundancy, stay with defaults
Missing Routes:
Example II – Summary

p  The list of NLRI you sent a peer:
n  show ip bgp neighbor x.x.x.x advertised
n  Note: The attribute values shown are taken from the
BGP table; attribute modifications by outbound route-
maps will not be shown
p  Display the routes sent to us by neighbour x.x.x.x
after processing by our inbound filters:
n  show ip bgp neighbor x.x.x.x routes
p  Display the routes sent to us by neighbour x.x.x.x
prior to processing by our inbound filters
n  show ip bgp neighbor x.x.x.x received
n  Can only use if Soft-Reconfiguration is enabled

p  Ideal for troubleshooting problems with inbound
filters and attributes
p  show ip bgp neighbor x.x.x.x routes
p  show ip bgp neighbor x.x.x.x received
alpha#sh ip bgp neigh 192.168.12.1 routes
*>i1.0.0.0 192.168.12.1 0 50 0 i
*>i222.222.0.0/19 192.168.5.1 200 0 3 4 i
alpha#sh ip bgp neigh 192.168.12.1 received-routes
* i1.0.0.0 192.168.12.1 0 100 0 i
* i169.254.0.0 192.168.5.1 0 100 0 3 i
* i222.222.0.0/19 192.168.5.1 100 0 3 4 i
“soft-reconfiguration”

Update Filtering
p  Type of filters
n  Prefix filters
n  AS_PATH filters
n  Community filters
n  Route-maps
p  Applied incoming and/or outgoing

Missing Routes
Update Filters
p  Determine which filters are applied to the
BGP session
n  show ip bgp neighbors x.x.x.x
n  show run | include neighbor x.x.x.x
p  Examine the route and pick out the
relevant attributes
n  show ip bgp x.x.x.x
p  Compare the attributes against the filters

p  Missing 10.0.0.0/8 in R1 (1.1.1.1)
p  Not received from R2 (2.2.2.2)
R1#show ip bgp neigh 2.2.2.2 routes
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing Routes
Update Filters

R2#show ip bgp neigh 1.1.1.1 advertised-routes
R2#show ip bgp 10.0.0.0
BGP routing table entry for 10.0.0.0/8, version 1660
Paths: (1 available, best #1)
Not advertised to any peer
Local
0.0.0.0 from 0.0.0.0 (2.2.2.2)
Origin IGP, metric 0, localpref 100, weight 32768, valid,
sourced, local, best
p  R2 originates the route
p  Does not advertise it to R1
Missing Routes
Update Filters

R2#show run | include neighbor 1.1.1.1
neighbor 1.1.1.1 filter-list 1 out
R2#sh ip as-path 1
AS path access list 1
permit ^$
Missing Routes
Update Filters
p  Time to check filters!
p  ^ matches the beginning of a line
p  $ matches the end of a line
p  ^$ means match any empty AS_PATH
p  Filter “looks” correct

R2#show ip bgp filter-list 1
R2#show ip bgp regexp ^$
Status codes: s suppressed, d damped, h history, * valid,
> best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
*> 10.0.0.0 0.0.0.0 0 32768 i
Missing Routes
Update Filters
p  Nothing matches the filter-list???
p  Re-typing the regexp gives the expected output

p  Copy and paste the entire regexp line from the
configuration
p  There is a trailing white space at the end
p  It is considered part of the regular expression
Nothing matches again! Let’s use the up arrow key to see where the
cursor stops
End of Line Is at the Cursor
Missing Routes
Update Filters

p  Force R2 to resend the update after the filter-list
correction
p  Then check R1 to see if it has the route
p  R1 still does not have the route
p  Time to check R1’s inbound policy for R2
R2#clear ip bgp 1.1.1.1 out
Missing Routes
Update Filters

R1#show run | include neighbor 2.2.2.2
neighbor 2.2.2.2 route-map POLICY in
R1#show route-map POLICY
route-map POLICY, permit, sequence 10
Match clauses:
ip address (access-lists): 100 101
as-path (as-path filter): 1
Set clauses:
Policy routing matches: 0 packets, 0 bytes
R1#show access-list 100
Extended IP access list 100
permit ip host 10.0.0.0 host 255.255.0.0
permit ip 200.1.1.0 0.0.0.255 host 255.255.255.0
R1#show ip as-path 1
permit ^12$
Missing Routes
Update Filters

p  Confused? Let’s run some debugs
Standard IP access list 99
permit 10.0.0.0
R1#debug ip bgp 2.2.2.2 update 99
BGP updates debugging is on for access list 99 for neighbor 2.2.2.2
R1#
4d00h: BGP(0): 2.2.2.2 rcvd UPDATE w/ attr: nexthop 2.2.2.2,
origin i, metric 0, path 12
4d00h: BGP(0): 2.2.2.2 rcvd 10.0.0.0/8 -- DENIED due to: route-map;
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing Routes
Update Filters

R1#sh run | include neighbor 2.2.2.2
neighbor 2.2.2.2 route-map POLICY in
R1#sh route-map POLICY
route-map POLICY, permit, sequence 10
Match clauses:
ip address (access-lists): 100 101
as-path (as-path filter): 1
Set clauses:
Policy routing matches: 0 packets, 0 bytes
R1#sh access-list 100
R1#sh access-list 101
permit ip 200.1.1.0 0.0.0.255 host 255.255.255.0
R1#sh ip as-path 1
permit ^12$
Missing Routes
Update Filters

Missing Routes
Update Filters
p  Wrong mask! Needs to be /8 and the ACL allows a /16 only!
access list 100
p  Should be
access list 100
p  Use prefix-list instead, more difficult to make a mistake
ip prefix-list my_filter permit 10.0.0.0/8
p  What about ACL 101?
n  Multiple matches on the same line are ORed
n  Multiple matches on different lines are ANDed
p  ACL 101 does not matter because ACL 100 matches
which satisfies the OR condition

Update Filtering:
Summary
p  If you suspect a filtering problem, become
familiar with the router tools to find out
what BGP filters are applied
p  Tip: don’t cut and paste!
n  Many filtering errors and diagnosis problems
result from cut and paste buffer problems on
the client, the connection, and even the router

Update Filtering:
Common Problems
p  Typos in regular expressions
n  Extra characters, missing characters, white
space, etc
n  In regular expressions every character
matters, so accuracy is highly important
p  Typos in prefix filters
n  Watch the router CLI, and the filter logic – it
may not be as obvious as you think, or as
simple as the manual makes out
n  Watch netmask confusion, and 255 profusion –
easy to muddle 255 with 0 and 225!

p  Missing 10.0.0.0/8 in R1 (1.1.1.1)
p  Not received from R2 (2.2.2.2)
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing Routes
Community Problems

Local
0.0.0.0 from 0.0.0.0 (2.2.2.2)
Origin IGP, metric 0, localpref 100, weight 32768, valid,
sourced, local, best
Missing Routes
Community Problems
p  R2 originates the route
p  But the community is not set
n  Would be displayed in the “show ip bgp” output

p  Fix the configuration so community is set
Local
0.0.0.0 from 0.0.0.0 (2.2.2.2)
Origin IGP, metric 0, localpref 100, weight 32768,
valid, sourced, local, best
Community 2:2 1:50
R2#show run | begin bgp
router bgp 2
network 10.0.0.0 route-map set-community
...
route-map set-community permit 10
set community 2:2 1:50
Missing Routes
Community Problems

p  R2 now advertises prefix with community to R1
p  But R1 still doesn’t see the prefix
n  R1 insists there is nothing wrong with their configuration
p  Configuration verified on R2
p  No filters blocking announcement on R2
p  So what’s wrong?
Missing Routes
Community Problems

p  Check R2 configuration again!
p  Looks okay – filters okay, route-map okay
p  But forgotten “neighbor 1.1.1.1 send-community”
n  Cisco IOS does NOT send communities by default
router bgp 2
network 10.0.0.0 route-map set-community
neighbor 1.1.1.1 prefix-list my-agg out
neighbor 1.1.1.1 prefix-list their-agg in
!
ip prefix-list my-agg permit 10.0.0.0/8
ip prefix-list their-agg permit 20.0.0.0/8
!
route-map set-community permit 10
set community 2:2 1:50
Missing Routes
Community Problems

p  R2 now advertises prefix with community to R1
p  But R1 still doesn’t see the prefix
n  Nothing wrong on R2 now, so turn attention to R1
router bgp 1
neighbor 2.2.2.2 route-map R2-in in
neighbor 2.2.2.2 route-map R1-out out
!
ip community-list 1 permit 1:150
!
route-map R2-in permit 10
match community 1
set local-preference 150
Missing Routes
Community Problems

Missing Routes
Community Problems
p  Community match on R1 expects 1:150 to be set
on prefix
p  But R2 is sending 1:50
n  Typo or miscommunication between operations?
p  R2 is also using the route-map to filter
n  If the prefix does not have community 1:150 set, it is
dropped – there is no next step in the route-map
n  Watch the route-map rules in Cisco IOS – they are
basically:
p  if <match> then <set> and exit route-map
p  else if <match> then <set> and exit route-map
p  else if <match> then <set> etc…
n  Blank route-map line means match everything, set
nothing

p  Fix configuration on R2 to set community 1:150 on
announcements to R1
p  Fix configuration on R1 to also permit prefixes not matching
the route-map – troubleshooting is easier with prefix-filters
doing the filtering
* 10.0.0.0 2.2.2.2 0 0 2 i
R1#show run | begin ^route-map
match community 1
set local-preference 150
Missing Routes
Community Problems

Missing Routes
Community Problems
p  Watch route-maps
n  Route-map rules often catch out operators when they
are used for filtering
n  Absence of an appropriate match means the prefix will
be discarded
p  Remember to configure all routers to send BGP
communities
n  Include it in your default template for iBGP
p  It should be iBGP default in a Service Provider Network
n  Remember that it is required to send communities for
eBGP too

Missing Routes:
Common Community Problems
p  Each router implementation has different defaults
for when communities are sent
n  Some don’t send communities
n  Others do for iBGP and not for eBGP
n  Others do for both iBGP and eBGP peers
p  Watch how your implementation handles
communities
n  There may be implicit filtering rules
p  Each ISP has different community policies
n  Never assume that because communities exist that
people will use them, or pay attention to the ones you
send

Missing Routes:
General Problems
p  Make and then Stick to simple policy rules:
n  Most router implementations have particular rules for
filtering of prefixes, AS-paths, and for manipulating BGP
attributes
n  Try not to mix these rules
p  Rules for manipulating attributes can also be
used for filtering prefixes and ASNs
n  These can be very powerful, but can also become very
confusing

Missing Routes
iBGP Example I
p  Symptom: prefixes seen across network,
but no connectivity
n  Prefixes learned from eBGP peer are passed
across iBGP mesh
n  But no connectivity to those prefixes

AS 1
AS 3
iBGP eBGP
1.1.1.1 2.2.2.2
3.3.3.3
4.4.4.4
A
B
AS 2
eBGP
R2R1
R5
R4
R3
10.10.0.0/24
Missing Routes
iBGP Example I
p  R3 customers can reach AS2
p  No other customers connected
to AS1 or AS3 can reach AS2

p  Looking at R3
p  Looking at R4
R3#show ip bgp
Status codes: * valid, > best, i - internal,
*> 3.0.0.0 10.10.10.10 0 2 5 i
*> 4.0.0.0 10.10.10.10 0 2 5 i
*> 10.10.0.0/24 10.10.10.10 0 2 i
*> 10.20.0.0/16 10.10.10.10 0 2 i
R4#show ip bgp
* i3.0.0.0 10.10.10.10 100 0 2 5 i
* i4.0.0.0 10.10.10.10 100 0 2 5 i
* i10.10.0.0/24 10.10.10.10 100 0 2 i
* i10.20.0.0/16 10.10.10.10 100 0 2 i
Missing Routes
iBGP Example I

Missing Routes:
iBGP Example I
p  Notice that R3 reports the prefixes learned
from AS2
n  Paths are valid (*) and best (>)
p  Notice that R4 reports the prefixes learned
from R3
n  Paths are valid (*) and internal (i)
n  But no best path
n  This is the clue…

R4#sh ip bgp 10.10.0.0/24
Paths: (1 available, no best path)
2, (received & used)
10.10.10.10 (inaccessible) from 3.2.1.2 (3.3.3.3)
Origin IGP, metric 0, localpref 100, valid, internal
R4#sh ip route 10.10.0.0 255.255.255.0
R4#sh ip route 10.10.10.10
The clues
Missing Routes:
iBGP Example I
p  Look at the BGP table entry:
p  Look at the Routing Table entry
p  The next hop?

Missing Routes:
iBGP Example I – Diagnosis
p  R4 does not use the 10.10.0.0/24 destination
because there is no valid next-hop
p  Configuration on R3 has:
n  Either no routing information on how to reach the
10.10.10.10/30 point to point link
p  By forgetting to put the link into the IGP
n  Or not excluded external next-hops from the internal
network
p  By forgetting to set itself as the next-hop for all externally
learned prefixes on the iBGP session with R4

Missing Routes:
iBGP Example I – Solution
p  Make sure that all the BGP NEXT_HOPs are
known by the IGP
n  (whether OSPF/ISIS, static or connected routes)
n  If NEXT_HOP is also in iBGP, ensure the iBGP distance is
longer than the IGP distance
n  —or—
p  Don’t carry external NEXT_HOPs in your network
n  Replace eBGP next_hop with local router address on all
the edge BGP routers
n  (Cisco IOS “next-hop-self”)

R4#show ip bgp
*>i3.0.0.0 3.3.3.3 100 0 2 5 i
*>i4.0.0.0 3.3.3.3 100 0 2 5 i
*>i10.10.0.0/24 3.3.3.3 100 0 2 i
*>i10.20.0.0/16 3.3.3.3 100 0 2 i
Missing Routes
iBGP Example I – Solution
p  R3 now includes the missing “next-hop-self”
configuration
p  Looking at R4 now:

Missing Routes
iBGP Example II
p  Symptom: customer complains about
patchy Internet access
n  Can access some, but not all, sites connected
to backbone
n  Can access some, but not all, of the Internet

AS 1
AS 3
iBGP eBGP
1.1.1.1 2.2.2.2
3.3.3.3
4.4.4.4
A
B
AS 2
eBGP
R2R1
R5
R4
R3
10.10.0.0/24
Missing Routes
iBGP Example II
p  Customer connected to R1 can
see AS3, but not AS2
p  Also complains about not being
able to see sites connected to R5
p  No complaints from other
customers

Missing Routes
iBGP Example II
p  Diagnosis: This is the classic iBGP mesh problem
n  The full mesh isn’t complete – how do we know this?
p  Customer is connected to R1
n  Can’t see AS2 ⇒ R3 is somehow not passing routing
information about AS2 to R1
n  Can’t see R5 ⇒ R5 is somehow not passing routing
information about sites connected to R5
n  But can see rest of the Internet ⇒ his prefix is being
announced to some places, so not an iBGP origination
problem

p  BGP summary shows that the peering with
router R1 is down
n  Up/Down is 3 days 10 hours, yet active
n  Which means it was last up 3 days and 10
hours ago
n  So something has broken between R1 and R3
R3#sh ip bgp sum | begin ^Neigh
1.1.1.1 4 1 200 20 32 0 0 3d10h Active
2.2.2.2 4 1 210 25 32 0 0 3d16h 15
4.4.4.4 4 1 213 22 32 0 0 3d16h 12
5.5.5.5 4 1 215 19 32 0 0 3d16h 0
10.10.10.10 4 2 2501 2503 32 0 0 3d16h 100
R3#
Missing Routes
iBGP Example II

R1#sh conf | b bgp
router bgp 1
neighbor iBGP-ipv4-peers peer-group
neighbor iBGP-ipv4-peers remote-as 1
neighbor iBGP-ipv4-peers update-source Loopback0
neighbor iBGP-ipv4-peers send-community
neighbor iBGP-ipv4-peers prefix-list ibgp-prefixes out
neighbor 2.2.2.2 peer-group iBGP-ipv4-peers
Missing Routes
iBGP Example II
p  Now check configuration on R1
p  Where is the peering with R3?
p  Restore the missing line, and the iBGP with R3
comes back up

1.1.1.1 4 1 200 20 32 0 0 00:00:50 8
2.2.2.2 4 1 210 25 32 0 0 3d16h 15
4.4.4.4 4 1 213 22 32 0 0 3d16h 12
5.5.5.5 4 1 215 19 32 0 0 3d16h 0
10.10.10.10 4 2 2501 2503 32 0 0 3d16h 100
R3#
Missing Routes
iBGP Example II
p  BGP summary shows that no prefixes are
being heard from R5
n  This could be due to inbound filters on R3 on
the iBGP with R5
p  But there were no filters in the configuration on R3
n  This must be due to outbound filters on R5 on
the iBGP with R3

R5#sh conf | b neighbor 3.3.3.3
neighbor 3.3.3.3 update-source loopback0
neighbor 3.3.3.3 prefix-list ebgp-filters out
neighbor 4.4.4.4 update-source loopback0
neighbor 4.4.4.4 prefix-list ibgp-filters out
!
ip prefix-list ebgp-filters permit 20.0.0.0/8
ip prefix-list ibgp-filters permit 10.0.0.0/8
Missing Routes
iBGP Example II
p  Now check configuration on R5
p  Error in prefix-list in R3 iBGP peering
n  ebgp-filters has been used instead of ibgp-filters
n  Typo — another advantage of using peer-groups!

1.1.1.1 4 1 200 20 32 0 0 00:01:53 8
2.2.2.2 4 1 210 25 32 0 0 3d16h 15
4.4.4.4 4 1 213 22 32 0 0 3d16h 12
5.5.5.5 4 1 215 19 32 0 0 3d16h 6
10.10.10.10 4 2 2501 2503 32 0 0 3d16h 100
R3#
Missing Routes
iBGP Example II
p  Fix the prefix-list on R5
p  Check the iBGP again on R3
n  Peering with R1 is up
n  Peering with R5 has prefixes
p  Confirm that all is okay with customer

p  Watch the iBGP full mesh
n  Use peer-groups both for efficiency and to
avoid making policy errors within the iBGP
mesh
n  Use route reflectors to avoid accidentally
missing iBGP peers, especially as the mesh
grows in size
p  Watch the next-hop for external paths

Inconsistent Route Selection
p  Two common problems with route selection
n  Inconsistency
n  Appearance of an incorrect decision
p  RFC1771 defined the decision algorithm
p  Every vendor has tweaked the algorithm
n  http://www.cisco.com/warp/public/459/25.shtml
p  Route selection problems can result from
oversights by RFC1771
p  RFC1771 was made obsolete by RFC4271 in
January 2006
n  Hopefully compliance with RFC4271 will help avoid
future issues

Inconsistent Route Selection:
Example I
p  RFC1771 said that MED is not always compared
p  As a result, the ordering of the paths can effect
the decision process
p  For example, the default in Cisco IOS is to
compare the prefixes in order of arrival (most
recent to oldest)
n  This can result in inconsistent route selection
n  Symptom is that the best path chosen after each BGP
reset is different

Example I
p  Inconsistent route selection may cause problems
n  Routing loops
n  Convergence loops—i.e. the protocol continuously sends
updates in an attempt to converge
n  Changes in traffic patterns
p  Difficult to catch and troubleshoot
n  In Cisco IOS, the deterministic-med configuration
command is used to order paths consistently
p  Recommend enabling on all the routers in the AS
n  The bestpath is recalculated as soon as the command
is entered

p  RouterA will have three paths
p  MEDs from AS 3 will not be compared
with MEDs from AS 1
p  RouterA will sometimes select the path
from R1 as best and may also select the
path from R3 as best
AS 3
AS 2
AS 1
AS 10
10.0.0.0/8
MED 20
MED 30
MED 0
R2
R3
R1
Symptom I:
Diagram
A
B
C

RouterA#sh ip bgp 10.0.0.0
Paths: (3 available, best #3, advertised over iBGP, eBGP)
3 10
2.2.2.2 from 2.2.2.2
3 10
3.3.3.3 from 3.3.3.3
Origin IGP, metric 30, valid, external
1 10
1.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, internal, best
Example I
p  Initial State
n  Path 1 beats Path 2 – Lower MED
n  Path 3 beats Path 1 – Lower Router-ID

1 10
1.1.1.1 from 1.1.1.1
3 10
2.2.2.2 from 2.2.2.2
3 10
3.3.3.3 from 3.3.3.3
Origin IGP, metric 30, valid, external, best
Example I
p  1.1.1.1 bounced so the paths are re-ordered
n  Path 1 beats Path 2 – Lower Router-ID
n  Path 3 beats Path 1 – External vs Internal

Deterministic MED:
Operation
p  The paths are ordered by Neighbour AS
p  The bestpath for each Neighbour AS group is
selected
p  The overall bestpath results from comparing the
winners from each group
p  The bestpath will be consistent because paths will
be placed in a deterministic order

1 10
1.1.1.1 from 1.1.1.1
3 10
2.2.2.2 from 2.2.2.2
3 10
3.3.3.3 from 3.3.3.3
Origin IGP, metric 30, valid, external
Deterministic MED:
Result
p  Path 1 is best for AS 1
p  Path 2 beats Path 3 for AS 3 – Lower MED
p  Path 1 beats Path 2 – Lower Router-ID

Deterministic MED:
Summary
p  Always use “bgp deterministic-med”
p  Need to enable throughout entire network at
roughly the same time
p  If only enabled on a portion of the network
routing loops and/or convergence problems may
become more severe
p  As a result, default behaviour cannot be changed
so the knob must be configured by the user

p  RouterA will have three paths
p  RouterA will consistently select the
path from R1 as best!
AS 3
AS 2
AS 1
AS 10
10.0.0.0/8
MED 20
MED 30
MED 0
R2
R3
R1
Solution – Diagram
A
B
C

10 100
1.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, external
20 100
2.2.2.2 from 2.2.2.2
Origin IGP, metric 0, localpref 100, valid, external, best
R3
AS 10 AS 20
R1 R2
Example II
p  The bestpath
changes every time
the peering is reset

20 100
2.2.2.2 from 2.2.2.2
Origin IGP, metric 0, localpref 100, valid, external
10 100
1.1.1.1 from 1.1.1.1
Example II
p  The “oldest” external is the bestpath
n  All other attributes are the same
n  Stability enhancement!!—CSCdk12061—Integrated in 12.0(1)
p  “bgp bestpath compare-router-id” will disable this
enhancement—CSCdr47086—Integrated in 12.0(11)S and
12.1(3)

R1#sh ip bgp 11.0.0.0
100
1.1.1.1 from 1.1.1.1
Origin IGP, localpref 120, valid, internal
100
2.2.2.2 from 2.2.2.2
Example III
p  Path 1 has higher localpref but path 2 is better???
p  This appears to be incorrect…

Example III
p  Path is from an internal peer which means the
path must be synchronized by default
p  Check to see if synchronization is on or off
p  Sync is still enabled, check for IGP path:
p  CSCdr90728 “BGP: Paths are not marked as not
synchronized”—Fixed in 12.1(4)
p  Path 1 is not synchronized
p  Router made the correct choice
R1# show run | include sync
R1#
R1# show ip route 11.0.0.0

Inconsistent Path Selection
p  Summary:
n  RFC1771 wasn’t prefect when it came to path
selection – years of operational experience
have shown this
n  Vendors and ISPs have worked to put in
stability enhancements, now reflected in
RFC4271
n  But these can lead to interesting problems
n  And of course some defaults linger much
longer than they ought to – so never assume
that an out of the box default configuration will
be perfect for your network

Route Oscillation
p  One of the most common problems
p  Main symptom is that traffic exiting the network
oscillates every minute between two exit points
n  This is almost always caused by the BGP NEXT_HOP
being known only by BGP
n  Common problem in ISP networks – but if you have
never seen it before, it can be a nightmare to debug and
fix
p  Other symptom is high CPU utilisation for the BGP
router process

p  R3 prefers routes via AS 4 one minute
p  BGP scanner runs then R3 prefers routes via AS 12
p  The entire table oscillates every 60 seconds
AS 3
AS 12
AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:
Diagram

267 network entries and 272 paths using 34623 bytes of memory
R3#sh ip route summary | begin bgp
bgp 3 4 6 520 1400
External: 0 Internal: 10 Local: 0
internal 5 5800
Total 10 263 13936 43320
Route Oscillation:
Diagnosis
p  Watch for:
n  Table version number incrementing rapidly
n  Number of networks/paths or external/internal
routes changing

p  Pick a route from the RIB that has changed within the last
minute
p  Monitor that route to see if it changes every minute
R3#show ip route 156.1.0.0
Known via "bgp 3", distance 200, metric 0
Routing Descriptor Blocks:
* 1.1.1.1, from 1.1.1.1, 00:00:53 ago
AS Hops 2, BGP network version 474
Advertised to non peer-group peers:
2.2.2.2
4 12
1.1.1.1 from 1.1.1.1 (1.1.1.1)
Origin IGP, localpref 100, valid, internal, best
12
142.108.10.2 (inaccessible) from 2.2.2.2 (2.2.2.2)
Route Oscillation:
Troubleshooting

R3#sh ip route 156.1.0.0
* 142.108.10.2, from 2.2.2.2, 00:00:27 ago
R3#sh ip bgp 156.1.0.0
1.1.1.1
4 12
1.1.1.1 from 1.1.1.1 (1.1.1.1)
12
142.108.10.2 from 2.2.2.2 (2.2.2.2)
Route Oscillation:
Troubleshooting
p  Check again after bgp_scanner runs
p  bgp_scanner runs every 60 seconds and validates
reachability to all nexthops

R3#show ip route 142.108.10.2
* 142.108.10.2, from 2.2.2.2, 00:00:50 ago
R3#show ip bgp 142.108.10.2
1.1.1.1
4 12
1.1.1.1 from 1.1.1.1 (1.1.1.1)
12
142.108.10.2 from 2.2.2.2 (2.2.2.2)
Route Oscillation:
Troubleshooting
p  Lets take a closer look at the nexthop

R3#sh debug
BGP events debugging is on
BGP updates debugging is on
IP routing debugging is on
R3#
BGP: scanning routing tables
BGP: nettable_walker 142.108.0.0/16 calling revise_route
RT: del 142.108.0.0 via 142.108.10.2, bgp metric [200/0]
BGP: revise route installing 142.108.0.0/16 -> 1.1.1.1
RT: add 142.108.0.0/16 via 1.1.1.1, bgp metric [200/0]
Route Oscillation:
Troubleshooting
p  BGP nexthop is known via BGP
p  Illegal recursive lookup
p  Scanner will notice and install the other path in the RIB

p  Route to the nexthop is now valid
p  Scanner will detect this and re-install the other path
p  Routes will oscillate forever
R3#
BGP: scanning routing tables
BGP: ip nettable_walker 142.108.0.0/16 calling revise_route
BGP: nettable_walker 156.1.0.0/16 calling revise_route
Route Oscillation:
Troubleshooting

p  R3 naturally prefers routes from AS 12
p  R3 does not have an IGP route to 142.108.10.2 which is the next-
hop for routes learned via AS 12
p  R3 learns 142.108.0.0/16 via AS 4 so 142.108.10.2 becomes
reachable
AS 3
AS 12
AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:
Step by Step

Route Oscillation:
Step by Step
p  R3 then prefers the AS 12 route for
142.108.0.0/16 whose next-hop is 142.108.10.2
p  This is an illegal recursive lookup
p  BGP detects the problem when scanner runs and
flags 142.108.10.2 as inaccessible
p  Routes through AS 4 are now preferred
p  The cycle continues forever…

Route Oscillation:
Solution
p  Make sure that all the BGP NEXT_HOPs are
known by the IGP
n  (whether OSPF/ISIS, static or connected routes)
n  If NEXT_HOP is also in iBGP, ensure the iBGP distance is
longer than the IGP distance
—or—
p  Don’t carry external NEXT_HOPs in your network
n  Replace eBGP next_hop with local router address on all
the edge BGP routers
n  (Cisco IOS “next-hop-self”)

p  R3 now has IGP route to AS 12 next-hop or
R2 is using next-hop-self
p  R3 now prefers routes via AS 12 all the time
p  No more oscillation!!
AS 3
AS 12
AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:
Solution

p  High CPU utilisation in the BGP process is
normally a sign of a convergence problem
p  Find a prefix that changes every minute
p  Troubleshoot/debug that one prefix

p  BGP routing loop?
n  First, check for IGP routing loops to the BGP NEXT_HOPs
p  BGP loops are normally caused by
n  Not following physical topology in RR environment
n  Multipath with confederations
n  Lack of a full iBGP mesh
p  Get the following from each router in the loop
path
n  The routing table entry
n  The BGP table entry
n  The route to the NEXT_HOP

Convergence Problems
p  Route reflector
with 250 route
reflector clients
p  100k routes
p  BGP will not
converge
RR

RR# show ip bgp summary
20.3.1.160 4 100 10 5416 9419 0 0 00:00:12 Closing
20.3.1.161 4 100 11 4418 8055 0 335 00:10:34 0
20.3.1.162 4 100 12 4718 8759 0 128 00:10:34 0
20.3.1.163 4 100 9 3517 0 1 0 00:00:53 Connect
20.3.1.164 4 100 13 4789 8759 0 374 00:10:37 0
20.3.1.165 4 100 13 3126 0 0 161 00:10:37 0
20.3.1.166 4 100 9 5019 9645 0 0 00:00:13 Closing
20.3.1.167 4 100 9 6209 9218 0 350 00:10:38 0
RR#show log | i BGP
*May 3 15:27:16: %BGP-5-ADJCHANGE: neighbor 20.3.1.118 Down— BGP Notification sent
*May 3 15:27:16: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.118 4/0 (hold time expired) 0 byt
*May 3 15:28:10: %BGP-5-ADJCHANGE: neighbor 20.3.1.52 Down— BGP Notification sent
*May 3 15:28:10: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.52 4/0 (hold time expired) 0 byte
p  Have been trying to converge for 10 minutes
p  Peers keep dropping so we never converge?
p  Check the log to find out why

p  We are either missing hellos or our peers are not sending
them
p  Check for interface input drops
p  72k drops will definitely cause a few peers to go down
p  We are missing hellos because the interface input queue is
very small
p  A rush of TCP Acks from 250 peers can fill 75 spots in a
hurry
p  Increase the size of the queue
RR# show interface gig 2/0 | include drops
Output queue 0/40, 0 drops; input queue 0/75, 72390 drops
RR#
RR# show run interface gig 2/0
interface GigabitEthernet 2/0
ip address 7.7.7.156 255.255.255.0
hold-queue 2000 in

p  Let’s start over and give BGP another chance
p  No more interface input drops
p  Our peers are stable!!
RR# show log | include BGP
RR#
RR# show interface gig 2/0 | include input drops
Output queue 0/40, 0 drops; input queue 0/2000, 0 drops
RR#
RR# clear ip bgp *
RR#

p  BGP converged in 25 minutes
p  Still seems like a long time
p  What was TCP doing?
RR#show tcp stat | begin Sent:
Sent: 1666865 Total, 0 urgent packets
763 control packets (including 5 retransmitted)
1614856 data packets (818818410 bytes)
39992 data packets (13532829 bytes) retransmitted
6548 ack only packets (3245 delayed)
1 window probe packets, 2641 window update packets
RR#show ip bgp neighbor | include max data segment
Datagrams (max data segment is 536 bytes):

p  1.6 Million packets is high
p  536 is the default MSS (max segment size) for a TCP
connection
p  Very small considering the amount of data we need to
transfer
p  Enable path mtu discovery
p  Sets MSS to max possible value
RR#show run | include tcp
ip tcp path-mtu-discovery
RR#

p  Restart the test one more time
p  MSS looks a lot better
RR# clear ip bgp *
RR#

p  TCP sent 1 million fewer packets
p  Path MTU discovery helps reduce overhead by
sending more data per packet
p  BGP converged in 15 minutes!
p  More respectable time for 250 peers and 100k
routes
RR# show tcp stat | begin Sent:
Sent: 615415 Total, 0 urgent packets
0 control packets (including 0 retransmitted)
602587 data packets (818797102 bytes)
9609 data packets (7053551 bytes) retransmitted
2603 ack only packets (1757 delayed)
0 window probe packets, 355 window update packets

Summary/Tips
p  Use ACLs when enabling debug commands
p  Ensure that BGP logging is switched on
p  Ensure that deterministic MED’s are
enabled
p  If the entire table is having problem pick
one prefix and troubleshoot it

Internet Reachability Problems
p  BGP Attribute Confusion
n  To Control Traffic in → Send MEDs and AS-
PATH prepends on outbound announcements
n  To Control Traffic out → Attach local-
preference to inbound announcements
p  Troubleshooting of multihoming and
transit is often hampered because the
relationship between routing information
flow and traffic flow is forgotten

BGP Path Selection Process
p  Each vendor has “tweaked” the path
selection process
n  Know it for your router equipment – saves
time later
n  Especially applies with networks with more
than one BGP implementation present
n  Best policy is to use supplied “knobs” to ensure
consistency – and avoid steps in the process
which can lead to inconsistency

MED Confusion
p  Default MED on Cisco IOS is ZERO
n  It may not be this on your router, or your
peer’s router
p  Best not to rely on MEDs for multihoming
on multiple links to upstream
n  Their default might be 232-1 resulting in your
hoped for best path being their worst path
n  “Workaround”, i.e. current good practice, is to
use communities rather than MEDs

Community Confusion I
p  set community in a route-map does just
that – it overwrites any other community
set on the prefix
n  Use additive keyword to add community to
existing list
p  Use Internet format for community
(AS:xx) not the 32-bit IETF format
n  32-bit format is harder for humans to
comprehend
n  Whereas AS:xx format is more intuitive/
recognisable

Community Confusion II
p  Cisco IOS never sends community by default
n  Some implementations send community by default for
iBGP peerings
n  Some implementations also send community by default
for eBGP peerings
p  Never assume that your neighbouring AS will
honour your no-export community – ask first!
n  If you leak iBGP prefixes to your upstream for
loadsharing purposes, this could result in your iBGP
prefixes leaking to the Internet

AS-PATH prepending
p  20 prepends will not lessen the priority of
your path any more than 10 prepends will
– check it out at a Looking Glass
n  The Internet is on average only 5 ASes deep,
maximum AS prepend most ISPs have to use
is around this too
n  Know you BGP path selection algorithm
p  Some ISPs limit AS-path lengths
n  For example, to drop prefixes with AS-paths
longer than 15 ASNs:
n  bgp maxas-limit 15

Private ASNs
p  Private ASes should not ever appear in the
Internet
p  Cisco IOS remove-private-AS command does
not remove every instance of a private AS
n  e.g. won’t remove private AS appearing in the middle of
a path surrounded by public ASNs
n  www.cisco.com/warp/public/459/32.html
p  Apparent non-removal of private-ASNs may not
be a bug, but a configuration error somewhere
else

AS 3AS 1
R3R1
R2
AS 2
192.168.1.0/24
Troubleshooting Connectivity
Example I
p  Symptom: AS1 announces 192.168.1.0/24 to
AS2 but AS3 cannot see the network

We are checking eBGP filters on R1 and R2.
Remember that R2 access will require cooperation
and assistance from your peer
We are checking iBGP across AS2’s network
(unneeded step in this case, but usually the next
consideration). Quite often iBGP is misconfigured, lack
of full mesh, problems with RRs, etc.
Example I
p  Checklist:
n  AS1 announces, but does AS2 see it?
n  Does AS2 see it over entire network?

We are checking eBGP configuration on R2. There may be a
configuration error with as-path filters, or prefix-lists, or
communities such that only local prefixes get out
We are checking eBGP configuration on R3. Maybe AS3 does
not know to expect prefixes from AS1 in the peering with AS2,
or maybe it has similar errors in as-path or prefix or community
filters
Example I
p  Checklist:
n  Does AS2 send it to AS3?
n  Does AS3 see all of AS2’s originated prefixes?

Example I
p  Troubleshooting connectivity beyond
immediate peers is much harder
n  Relies on your peer to assist you – they have
the relationship with their BGP peers, not you
n  Quite often connectivity problems are due to
the private business relationship between the
two neighbouring ASNs

AS 3AS 1
R3R1
202.173.147.0
The Internet
Example II
p  Symptom: AS1 announces 202.173.147.0/24 to
its upstreams but AS3 cannot see the network

We are checking eBGP filters on R1 and upstreams.
Remember that upstreams will need to be able to help
you with this
We are checking if the upstreams are announcing
the network to anywhere on the Internet. See next
slides on how to do this.
Example II
p  Checklist:
n  AS1 announces, but do its upstreams see it?
n  Is the prefix visible anywhere on the Internet?

Example II
p  Help is at hand – the Looking Glass
p  Many networks around the globe run Looking
Glasses
n  These let you see the BGP table and often run simple
ping or traceroutes from their sites
n  www.traceroute.org and www.bgp4.as/looking-glasses
p  Some ISPs, especially those with large and
diverse networks, run their own internal Looking
Glass to aid internal troubleshooting
p  Next slides have some examples of a typical
looking glass in action

Example II
p  Hmmm….
p  Looking Glass can see 202.173.144.0/21
n  This includes 202.173.147.0/24
n  So the problem must be with AS3, or AS3’s
upstream
p  A traceroute confirms the connectivity

Example II
p  Help is at hand – RouteViews
p  The main RouteViews router has BGP
feeds from around 60 peers
n  www.routeviews.org explains the project
n  Gives access to a real router, and allows any
provider to find out how their prefixes are seen
in various parts of the Internet
n  Complements the Looking Glass facilities
p  Anyway, back to our problem…

We are checking eBGP configuration on AS3’s upstream.
There may be a configuration error with as-path filters, or
prefix-lists, or communities such that only local prefixes get
out. This needs AS3’s assistance
We are checking eBGP configuration on R3. Maybe AS3
does not know to expect the prefix from AS1 in the peering
with its upstream, or maybe it has some errors in as-path or
prefix or community filters
Example II
p  Checklist:
n  Does AS3’s upstream send it to AS3?
n  Does AS3 see any of AS1’s originated prefixes?

Example II
p  Troubleshooting across the Internet is
harder
n  But tools are available
p  Looking Glasses, offering traceroute, ping
and BGP status are available all over the
globe
n  Most connectivity problems seem to be found
at the edge of the network, rarely in the transit
core
n  Problems with the transit core are usually
intermittent and short term in nature

AS 3AS 2
R2
The Internet
R1
AS 1
R3
Example III
p  Symptom: AS1 is trying to loadshare between
its upstreams, but has trouble getting traffic
through the AS2 link

Example III
p  Checklist:
n  What does “trouble” mean?
p  Is outbound traffic loadsharing okay?
n  Can usually fix this with selectively rejecting prefixes,
and using local preference
n  Generally easy to fix, local problem, simple application
of policy
p  Is inbound traffic loadsharing okay?
n  Bigger problem if not…
n  Need to do some troubleshooting if configuration with
communities, AS-PATH prepends, MEDs and selective
leaking of subprefixes don’t seem to help

We are checking eBGP filters on R1 and R2.
Remember that R2 access will require cooperation
and assistance from your peer
We are checking iBGP across AS2’s network. Quite
often iBGP is misconfigured, lack of full mesh,
problems with RRs, etc.
Example III
p  Checklist:
n  AS1 announces, but does AS2 see it?
n  Does AS2 see it over entire network?

Example III
p  Checklist:
n  Does AS2 send it to its upstream?
n  Does the Internet see all of AS2’s originated
prefixes?
We are checking eBGP configuration on R2. There may
be a configuration error with as-path filters, or prefix-lists,
or communities such that only local prefixes get out
We are checking eBGP configuration on other Internet
routers. This means using looking glasses. And trying to
find one as close to AS2 as possible.

Example III
p  Checklist:
n  Repeat all of the above for AS3
p  Stopping here and resorting to a huge
prepend towards AS3 won’t solve the
problem
p  There are many common problems – listed
on next slide
n  And tools to help decipher the problem

Example III
p  No inbound traffic from AS2
n  AS2 is not seeing AS1’s prefix, or is blocking it
in inbound filters
p  A trickle of inbound traffic
n  Switch on NetFlow (if the router has it) and
check the origin of the traffic
n  If it is just from AS2’s network blocks, then is
AS2 announcing the prefix to its upstreams?
n  If they claim they are, ask them to ask their
upstream for a BGP RIB dump showing the
relevant prefixes – or use a Looking Glass to
check

Example III
p  A light flow of traffic from AS2, but 50%
less than from AS3
n  Looking Glass comes to the rescue
p  LG will let you see what AS2, or AS2’s upstreams are
announcing
p  AS1 may choose this as primary path, but AS2
relationship with their upstream may decide
otherwise
n  NetFlow comes to the rescue
p  Allows AS1 to see what the origins are, and with the
LG, helps AS1 to find where the prefix filtering culprit
might be

AS 3AS 2
R2
The Internet
R1
AS 1
R3
Example IV
p  Symptom: AS1 is loadsharing between its upstreams, but
the traffic load swings randomly between AS2 and AS3

Example IV
p  Checklist:
n  Assume AS1 has done everything in this
tutorial so far
n  L2 problem? Route Flap Damping?
All the configurations look fine, the Looking Glass
outputs look fine, life is wonderful… Apart from those
annoying traffic swings every hour or so
Since BGP is configured fine, and the net has been
stable for so long, can only be an L2 problem, or
Route Flap Damping side-effect

Example IV
p  L2 – upstream somewhere has poor
connectivity between themselves and the
rest of the Internet
n  Only real solution is to impress upon upstream
that this isn’t good enough, and get them to fix
it
n  Or change upstreams

Example IV
p  Route Flap Damping
n  RIPE-378 describes impact of route flap damping on
Internet
p  www.ripe.net/ripe/docs/ripe-378
p  Strongly discouraged in its current form
n  RIPE-580 & RFC7196 suggest improvements
p  www.ripe.net/ripe/docs/ripe-580
p  www.rfc-editor.org/rfc/rfc7196.txt
n  Many ISPs still implement route flap damping
n  Many ISPs simply use the vendor defaults
p  Vendor defaults are too severe
p  Again Looking Glasses come to the operator’s
assistance

Example IV
p  Several Looking Glasses allow the
operators to check the flap or damped
status of their announcements
n  Many oscillating connectivity issues are usually
caused by L2 problems
n  Route flap damping will cause connectivity to
persist via alternative paths even though
primary paths have been restored
n  Quite often, the exponential back off of the flap
damping timer will give rise to bizarre routing
p  Common symptom is that bizarre routing will often
clear away by itself

Troubleshooting Summary
p  Most troubleshooting is about:
p  Experience
n  Recognising the common problems
p  Not panicking
p  Logical approach
n  Check configuration first
n  Check locally first before blaming the peer
n  Troubleshoot layer 1, then layer 2, then layer
3, etc

Troubleshooting Summary
p  Most troubleshooting is about:
p  Using the available tools
n  The debugging tools on the router hardware
n  Internet Looking Glasses
n  Colleagues and their knowledge
n  Public mailing lists where appropriate

Closing Comments
p  Tutorial has covered the most common
troubleshooting techniques used by ISPs
today
p  Once these have been mastered, more
complex or arcane problems are easier to
solve
p  Feedback and input for future
improvements is encouraged and very
welcome

Troubleshooting BGP
Philip Smith
<philip@nsrc.org>
APRICOT 2016
22nd – 26th February 2016
Auckland, New Zealand

Troubleshooting BGP

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Troubleshooting BGP (20)

More from APNIC (20)

Recently uploaded (20)

Troubleshooting BGP