Troubleshooting BGP Adjacency

In this post I will walk you through some steps you can take to troubleshoot BGP neighbor adjacency. These steps become even more helpful when you have access to only one side on the link (in the case where you are trying to run BGP with a service provider). We will focus in this post on some of the reasons that may prevent two BGP routers from forming a relationship and will demonstrate along the way how the BGP state machine moves from Idle to Established.

I’m using here two Cisco CSR1000v routers as my BGP speakers but the tips below apply in general to any router from any vendor.

In the diagram above I have two routers, R1 and R2, with two parallel physical links between them. The two routers want to peer using the loopback addresses via BGP which is a common way to do load sharing between two routers. However the BGP adjacency is not coming up and stuck in Idle state as you can see from the output below:

R2#sh ip bgp sum

BGP router identifier 2.2.2.2, local AS number 200

BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

1.1.1.1         4          100       0       0        1    0    0 never    Idle

Follow the steps below to verify that your configurations are complete and that there are no connectivity issues between the two routers:

1- First test and verify that R1 is reachable from R2 and vice versa. Issue a ping command from R2 sourcing your ping from the loopback0 interface with R2’s loopback0 interface as the destination as shown below:

R2#ping 1.1.1.1 source loopback 0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

Packet sent with a source address of 2.2.2.2

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

If your ping fails then you have a connectivity problem and you need to fix that before continuing this process. You want to make sure that the static routes are there on each router and that each router is able to ARP for the other router’s IP address.

2- You also need to verify that there is no firewall between R1 and R2 blocking TCP port 179 which is the port BGP uses to establish the connection. A quick way to ensure whether there is no firewall between R1 and R2 is to use the telnet command with port 179 as the destination port. Perform this test from both routers and don’t forget to source the traffic from the loopback interface:

R2#telnet 1.1.1.1 179 /source-interface loopback 0

Trying 1.1.1.1, 179 …

% Connection refused by remote host

As you can see from the output above, I got a “Connection refused by remote host” response when I tried to telnet from R2 to R1. This simply means that there is no device in the middle blocking traffic and R1 is rejecting the request obviously because 179 is not a standard port for telnet. If there was a firewall in the middle blocking traffic then you would get a “Unable to connect to remote host” response instead.

3- Now that we have verified that there are no connectivity problems, let’s focus on the BGP configurations. I will turn on “debug ip bgp x.x.x.x” on the router which shows me that the router is failing to establish a TCP connection with its peer. A good show command to use at this point is “show ip bgp neighbor x.x.x.x”

R2#show ip bgp nei 1.1.1.1

BGP neighbor is 1.1.1.1,  remote AS 100, external link

  BGP version 4, remote router ID 0.0.0.0

 Address tracking is enabled, the RIB does have a route to 1.1.1.1

  Connections established 0; dropped 0

  Last reset never

  External BGP neighbor not directly connected.

  Transport(tcp) path-mtu-discovery is enabled

  Graceful-Restart is disabled

  No active TCP connection

The output above tells me two important things. First that the RIB does have a route to reach the peer which confirms that the router has static routes needed to reach its peer’s loopback address.

The second important thing this output shows is this line: “External BGP neighbor not directly connected”. By default only directly connected eBGP peers are allowed to establish relationship.In order to change this default behavior, I have to add the “neighbor disable-connected-check” on both routers.

4- Even after disabling the direct-connected check, BGP relationship is still not coming up so my next step is to enable “debug ip tcp transactions” to see if that tells me why the TCP connection is failing:

R1#deb ip tcp transactions

TCP special event debugging is on

*Mar 13 05:53:52.522: Reserved port 0 in Transport Port Agent for TCP IP type 0

*Mar 13 05:53:52.522: TCP: connection attempt to port 179

*Mar 13 05:53:52.522: TCP: sending RST, seq 0, ack 4262843216

*Mar 13 05:53:52.522: TCP: sent RST to 192.168.32.20:27730 from 1.1.1.1:179

The last line in the output above is interesting. It is showing that R1 (1.1.1.1:179) is sending a TCP reset to R2 (192.168.32.20). Which means that it was R2 who initiated the TCP session. What this reveals also is that R2 sourced the connection request from its physical interface which is the default behavior in eBGP. But since I want the routers to peer using the loopback addresses and each router is expecting to receive a connection request from its peer loopback address, then i need to add the “neighbor update-source” to BGP on both ends

5- Now if I look at “debug ip bgp“, I can see that the TCP session is getting established and BGP is transitioning from the Idle -> Connect -> OpenSent -> OpenConfirm as shown below:

*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive open to 2.2.2.2

*Mar 13 06:49:28.979: BGP: Fetched peer 1.1.1.1 from tcb

*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from Idle to Connect

*Mar 13 06:49:28.979: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Receive OPEN

*Mar 13 06:49:28.979: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Send OPEN

*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from Connect to OpenSent

*Mar 13 06:49:28.979: BGP: 1.1.1.1 passive went from OpenSent to OpenConfirm

*Mar 13 06:49:28.980: %BGP-3-NOTIFICATION: received from neighbor 1.1.1.1 passive 2/2 (peer in wrong AS) 2 bytes 00C8

*Mar 13 06:49:28.980: BGP: ses global 1.1.1.1 (0x7F028066E270:0) pas Receive NOTIFICATION 2/2 (peer in wrong AS) 2 bytes 00C8

*Mar 13 06:49:28.980: %BGP-5-NBR_RESET: Neighbor 1.1.1.1 *Mar 13 06:49:28.980: BGP: 1.1.1.1 passive went from OpenConfirm to Closing

*Mar 13 06:49:28.980: BGP: 1.1.1.1 passive went from Closing to Idle

When BGP is in the OpenConfirm state it’s one step away from reaching its final state (ESTABLISHED) and while in the OpenConfirm state BGP waits to hear a KEEPALIVE from its peer before it moves to the Established state. As you can see from the output above, after reaching OpenConfirm BGP instead closes the connection and transitions back to Idle because it receives an error (peer in wrong AS).

This is a clear indication that the AS number on R1 is wrong so I will fix that and issue a “clear ip bgp *” command to restart the process.

And now after I corrected AS number in the configs on R1, the BGP state machine transitions to Established as you see below and the two peers can start exchanging routing updates and keepalives.

R2#sh ip bgp neighbors 1.1.1.1

BGP neighbor is 1.1.1.1,  remote AS 100, external link

  BGP version 4, remote router ID 1.1.1.1

  BGP state = Established, up for 00:00:42

Obviously that’s not everything, and there are other reasons that could prevent BGP relationship from being established but I wanted to discuss the most common ones that I have seen in the field. Do you have something to share? Please respond and share below.

Here is the final configs for R1 and R2 for your reference.

R1#sh run

interface Loopback0

 ip address 1.1.1.1 255.255.255.255

!

interface GigabitEthernet4

 ip address 192.168.32.10 255.255.255.0

 negotiation auto

!

interface GigabitEthernet5

 ip address 192.168.116.10 255.255.255.0

 negotiation auto

!

router bgp 100

 bgp log-neighbor-changes

 neighbor 2.2.2.2 remote-as 200

 neighbor 2.2.2.2 disable-connected-check

 neighbor 2.2.2.2 update-source Loopback0

!

no ip http secure-server

ip route 0.0.0.0 0.0.0.0 192.168.32.1

ip route 2.2.2.2 255.255.255.255 192.168.32.20

ip route 2.2.2.2 255.255.255.255 192.168.116.20

!

R2#sh run

Building configuration…

!

hostname R2

!

!

interface Loopback0

 ip address 2.2.2.2 255.255.255.255

!

interface GigabitEthernet4

 ip address 192.168.32.20 255.255.255.0

 negotiation auto

!

interface GigabitEthernet5

 ip address 192.168.116.20 255.255.255.0

 negotiation auto

!

router bgp 200

 bgp log-neighbor-changes

 neighbor 1.1.1.1 remote-as 100

 neighbor 1.1.1.1 disable-connected-check

 neighbor 1.1.1.1 update-source Loopback0

!

ip route 0.0.0.0 0.0.0.0 192.168.32.1

ip route 1.1.1.1 255.255.255.255 192.168.32.10

ip route 1.1.1.1 255.255.255.255 192.168.116.10