Troubleshooting Layer 3—IP services

ADVPN issues

ADVPN tunnel flapping (scenario without IPsec protection)

Symptom

In scenarios where ADVPN tunnel messages are not protected by an IPsec profile, the ADVPN tunnel experiences flapping, with the tunnel interface status alternating between Success and Dumb.

Common causes

The following are the common causes of this type of issue:

·     The configurations of the VAM server and VAM client are modified or deleted.

·     The tunnel configuration for ADVPN are modified or deleted.

·     The physical interface of the ADVPN tunnel is malfunctioning.

·     Route flapping occurs between the VAM server and VAM client.

·     Route flapping occurs between the two ends of the ADVPN tunnel.

·     The intermediate network between both the two end of the ADVPN tunnel is unstable.

Troubleshooting flow

Figure 1 shows the troubleshooting flowchart.

Figure 1 Flowchart for troubleshooting ADVPN tunnel flapping

 

Solution

1.     Identify whether the configuration of the VAM server is modified or deleted.

a.     Identify whether the VAM server in the specified ADVPN domain is disabled. Disabling the VAM server will delete the ADVPN tunnels in the domain.

<Sysname> display vam server statistics advpn-domain 1

ADVPN domain name      : 1

Server status          : Enabled

Holding time           : 0H 1M 47S

Registered spoke number: 98

Registered hub number  : 2

…

If the Server status field is Disabled, execute the vam server enable advpn-domain domain-name command in system view to re-enable the VAM server function for the specified ADVPN domain.

b.     Identify whether the private network address for the specified ADVPN tunnel's hub is modified. If the private network address of the hub is deleted, the ADVPN tunnels established with the hub will also be deleted.

In the specified hub group view, execute the display this command to identify whether the configured hub private IPv4 address is correct. If not, modify it in hub group view.

<Sysname> system-view

[Sysname] vam server advpn-domain 1

[Sysname-vam-server-domain-1] hub-group 1

[Sysname-vam-server-domain-1-hub-group-1] display this

#

 hub-group 1

  hub private-address 10.1.1.1 advpn-port 3333

#

c.     Identify whether the configured listening port number is modified. The listening port number of the VAM server must be the same as the port number specified on the VAM client. Otherwise, the protocol messages between the VAM client and VAM server cannot be transmitted properly. If the VAM client does not receive keepalive responses from the VAM server for an extended period, it proactively switches the ADVPN tunnel state to Dumb.

Execute the display this command in system view to identify whether the configured listening port number is correct. If not, execute the vam server listen-port command in system view to modify it.

<Sysname> system-view

[Sysname] vam server listen-port 10000

d.     After completing the checks above, if you have adjusted the configuration, execute the reset vam server address-map command to make the VAM client come online again. If the issue persists, go to step 2.

2.     Identify whether the VAM client configuration is modified or deleted.

a.     Identify whether the VAM client is disabled. Disabling the VAM client will delete established ADVPN tunnels.

<Sysname> display vam client fsm

Client name      : abc

Status           : Enabled

ADVPN domain name: 1

  Primary server: abc.com (28.1.1.23)

    Private address: 10.0.0.12

…

If the Status field is Disabled, execute the vam client enable name client-name command in system view to re-enable the VAM client.

b.     Identify whether the address of the primary VAM server is modified or deleted . If the address of the primary VAM server is deleted on the VAM client, the existing ADVPN tunnels are deleted and the VAM client registers with the new VAM server.

<Sysname> display vam client fsm

Client name      : abc

Status           : Enabled

ADVPN domain name: 1

  Primary server: abc.com (28.1.1.23)

    Private address: 10.0.0.12

…

If the primary server address in the displayed information is incorrect, execute the server primary command in VAM client view to reconfigure the primary VAM server address.

<Sysname> system-view

[Sysname] vam client name abc

[Sysname-vam-client-abc] server primary ip-address 1.1.1.1 port 2000

Make sure the port number specified on the VAM client is the same as the listening port number configured on the VAM server using the vam server listen-port command.

After completing the checks above, if you have adjusted the configuration, execute the reset vam client fsm command to make the VAM client come online again. If the issue persists, go to step 3.

3.     Identify whether the ADVPN tunnel configuration is modified or deleted.

a.     Enter the specified ADVPN tunnel interface view. Execute the display this command to identify whether the tunnel interface's private network address, source port , and bound VAM client are correct.

<Sysname> system-view

[Sysname] interface tunnel 1 mode advpn udp

[Sysname-Tunnel1] display this

#

interface Tunnel2 mode advpn udp

 ip address 10.1.1.1 255.255.255.0

 advpn source-port 3333

 vam client abc

#

If traffic on the same ADVPN tunnel interface must be processed on the same slot, use the service slot/service chassis chassis-number slot slot-number command on the tunnel interface to configure the slot handling the interface traffic. Otherwise, packet loss might occur.

b.     If you need to adjust the configuration, complete the relevant settings in tunnel interface view. Then, execute the reset advpn session interface tunnel number command in user view to trigger the re-establishment of the corresponding ADVPN tunnel. If the issue persists, go to step 4.

4.     Identify whether the physical interface of the ADVPN tunnel is functioning properly.

Execute the display ip interface brief command to check the Physical field of the physical interface where the ADVPN tunnel endpoint resides. If the Physical field consistently displays down or alternates between up and down, it indicates a physical interface issue. You should first troubleshoot the physical interface. If the physical interface of the tunnel is functioning properly but the issue persists, go to step 5.

5.     Check for route flapping between the VAM server and VAM client.

Execute the display ip routing-table command on both the VAM server and VAM client to identify whether there are routes to reach each other. If the related route information keeps switching between appearing and disappearing, it indicates route flapping.

¡     If there are no routes between the VAM server and VAM client, or if there is route flapping, check the routing configuration and refer to the route flapping troubleshooting manual of the relevant protocol to make sure the routing is normal.

¡     If there are no routing issues between the VAM server and VAM client, go to step 6.

6.     Identify whether route flapping exists at the two ends of the ADVPN tunnel.

Execute the display ip routing-table command at the two ends of the ADVPN tunnel to check for the presence of routes to each other. If the related route information keeps switching between appearing and disappearing, it indicates route flapping.

¡     If there are no routes between the two ends of the ADVPN tunnel, or if there is route flapping, see the route flapping troubleshooting manual of the relevant protocol to make sure the routing is normal.

¡     If there are no routing issues at the two ends of the ADVPN tunnel, go to step 7.

7.     Identify whether the intermediate network at the two ends of the ADVPN tunnel are stable.

Execute the display vam client statistics command on the local end to identify whether the number of sent keepalive packets is the same as the number of received keepalive packets.

<Sysname> display vam client statistics

Client name: abc

 Status     : Enabled

  Primary server: abc.com

    Packets sent:

      Initialization request        : 1

      Initialization complete       : 1

      Register request              : 1

      Authentication information    : 1

      Address resolution request    : 9

      Network registration request  : 0

      Update request                : 0

      Logout request                : 0

      Hub information response      : 0

      Data flow information response: 0

      Keepalive                     : 35

      Error notification            : 0

    Packets received:

      Initialization response      : 1

      Initialization complete      : 1

      Authentication request       : 1

      Register response            : 1

      Address resolution response  : 9

      Network registration response: 0

      Update response              : 0

      Hub information request      : 0

      Data flow information request: 0

      Logout response              : 0

      Keepalive                    : 35

      Error notification           : 0

      Unknown                      : 0

…

If not, it indicates an unstable intermediate network. Troubleshoot the intermediate network issues hop by hop under the guidance of a technician. If yes, go to step 8.

8.     Collect the following information and contact the support:

¡     Results of each step.

¡     Configuration file.

¡     Debugging information for events and errors related to the VAM server, VAM client, and ADVPN. As a best practice, avoid enabling packet debugging to prevent excessive debugging messages from interfering with normal operations and information browsing on the user interface (UI).

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

ADVPN tunnel flapping (scenario with IPsec protection)

Symptom

In scenarios where an IPsec profile protects ADVPN tunnel messages, the ADVPN tunnel experiences flapping, with the tunnel status continuously changing between Success and Dumb.

Common causes

The following are the common causes of this type of issue:

·     The configuration of the VAM server, VAM client, or ADVPN tunnel is modified or deleted.

·     The physical interface of the ADVPN tunnel is malfunctioning.

·     Route flapping occurs between VAM server and VAM client, or between VAM clients.

·     IPsec issues exist, such as IKE DPD probe packet loss and IPsec renegotiation failure.

Analysis

Figure 2 shows the troubleshooting flowchart.

Figure 2 Flowchart for troubleshooting ADVPN tunnel flapping

 

Solution

1.     Identify whether there are routes between both the two ends of the ADVPN tunnel.

Execute the display ip routing-table command on the two ends of the ADVPN tunnel to check for routes to each other.

¡     If there are no routes between the two ends of the ADVPN tunnel, check the route configuration.

¡     If there are routes between the two ends of the ADVPN tunnel, go to step 2.

2.     Identify whether the IPsec configuration is cause.

Execute the undo tunnel protection ipsec profile command on the tunnel interfaces at the two ends of the ADVPN tunnel to remove the applied IPsec profile, and then identify whether the issue is resolved.

¡     If the issue persists after IPsec protection is removed, execute the troubleshooting steps in ADVPN tunnel flapping (scenario without IPsec protection).

¡     If removing IPsec protection immediately resolves the issue, IPsec services caused the fault. Go to step 3.

3.     Identify whether the IKE SA and IPsec SA are successfully negotiated.

Execute the display ike sa and display ipsec sa commands at the two ends of the ADVPN tunnel to determine which phase of the SA negotiation was unsuccessful.

a.     If the display ike sa command does not display an IKE SA or state of the displayed IKE SA is Unknown, the phase 1 IKE SA negotiation fails. Identify whether the IKE profile configurations used by IPsec profiles applied at the two ends of the tunnel match. If the IKE configurations match but the issue persists, go to step 4.

b.     If the display ipsec sa command does not display an IPsec SA, it indicates that the phase two IPsec SA negotiation is unsuccessful. Identify whether the IPsec proposal configurations used by IPsec profiles applied at the two ends of the tunnel are correct.

-     The used IPsec proposals must include the same security protocol, authentication algorithm, encryption algorithm, and packet encapsulation mode.

-     The encapsulation modes used in the IPsec proposals must be tunnel mode.

If the IPsec proposals are configured correctly but the issue persists, go to step 4.

4.     Collect the following information and contact the support:

¡     Results of each step.

¡     Configuration file.

¡     Debugging information for events and errors related to IKE, IPsec, VAM server, VAM client, and ADVPN. As a best practice, avoid enabling packet debugging to prevent excessive debugging messages from interfering with normal operations and information browsing on the user interface (UI).

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Failed to ping the ADVPN tunnel peer (scenario without IPsec protection)

Symptom

In scenarios where the ADVPN tunnel messages are not protected by an IPsec profile, the IP address of the remote tunnel interface cannot be pinged.

Common causes

The following are the common causes of this type of issue:

·     No route to the peer exists.

·     The ADVPN tunnel fails to be created.

·     The length of the ADVPN tunnel packet exceeds the MTU of the tunnel interface.

Analysis

Figure 3 shows the troubleshooting flowchart.

Figure 3 Fault Diagnosis Flowchart for Inability to Ping ADVPN Tunnel Endpoint (Non-IPsec Scenario)

 

Solution

1.     Identify whether there is a route between both the two ends of the ADVPN tunnel.

Execute the display ip routing-table command at the two ends of the ADVPN tunnel to check for the presence of routes to each other.

¡     If no route to the peer exists, check the route configuration and configure a correct route.

¡     If a route to the peer exists, go to step 2.

2.     Identify whether the ADVPN tunnel is created successfully.

Execute the display advpn session command at the two ends of the ADVPN tunnel to identify whether the specified ADVPN tunnel exists.

<Sysname> display advpn session interface tunnel 1 private-address 10.0.1.3

Private address  Public address              Port  Type  State      Holding time

10.0.0.3         192.168.180.136             1139  H-S   Success    5H 38M 8S

¡     If the ADVPN tunnel to the destination private network address does not exist or the tunnel status is Dumb, it indicates that the ADVPN tunnel has not been successfully established. Check the configurations of the VAM server, VAM client, and ADVPN tunnel in sequence and ensure their correctness.

-     Execute the display vam server address-map command on the VAM server. If you do not see the IPv4 private-public address mapping information for VAM clients registered with the VAM server, check the configurations of both the VAM server and VAM client and make sure they are correct. For information about configuring the VAM server and VAM client, see the ADVPN configuration guide.

-     On the VAM server, execute the display vam server address-map command. If you see the IPv4 private-public address mapping information for VAM clients registered with the VAM server, identify whether the ADVPN tunnel configuration is correct. For information about configuring the ADVPN tunnel, see the ADVPN configuration guide.

¡     If an ADVPN tunnel to the peer private network address exists, go to step 3.

3.     Identify whether the ADVPN tunnel packet length is less than the interface's MTU value.

Execute the display interface tunnel command on the two ends of the ADVPN tunnel to check the value of the Maximum transmission unit field.

<Sysname> display interface tunnel 1

Tunnel1

Current state: UP

Line protocol state: UP

Description: Tunnel1 Interface

Bandwidth: 64kbps

Maximum transmission unit: 1476

...

Next, execute the ping –s packet-size –a local-address peer-address command to test the transmission of ping packets with different packet sizes through the tunnel interface. Identify a critical packet length value that results in ping failures, which represents the maximum length of ADVPN tunnel packets that the transmission link can allow.

¡     If the MTU value of the tunnel interface is less than the packet length of the ADVPN tunnel, execute the mtu size command in tunnel interface view to adjust the MTU value of the tunnel interface.

¡     If the MTU value of the tunnel interface is greater than the packet length of the ADVPN tunnel, go to step 4.

4.     Collect the following information and contact the support:

¡     Results of each step.

¡     Configuration file.

¡     Debugging information for events and errors related to the VAM server, VAM client, and ADVPN. As a best practice, avoid enabling packet debugging to prevent excessive debugging messages from interfering with normal operations and information browsing on the user interface (UI).

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Failed to ping the ADVPN tunnel peer (scenario with IPsec protection)

Symptom

In scenarios where the ADVPN tunnel messages are protected by an IPsec profile, the IP address of the remote tunnel interface cannot be pinged.

Common causes

The following are the common causes of this type of issue:

·     ADVPN tunnel packets fail be to encrypted or decrypted by IPsec.

·     The length of the ADVPN tunnel packet exceeds the MTU of the tunnel interface.

·     No route to the peer exists.

·     IPsec issues exist, such as unsuccessful establishment of IPsec tunnels and IPsec anti-replay packet loss.

Analysis

Figure 4 shows the troubleshooting flowchart.

Figure 4 Fault diagnosis flowchart for failed pings to the ADVPN tunnel peer (IPsec scenario)

 

Solution

1.     Identify whether there is a route between both the two ends of the ADVPN tunnel.

Execute the display ip routing-table command at the two ends of the ADVPN tunnel to check for the presence of routes to each other.

¡     If no route to the peer exists, check the route configuration and configure a correct route.

¡     If a route to the peer exists, go to step 2.

2.     Identify whether the IPsec configuration is cause.

Execute the undo tunnel protection ipsec profile command on the tunnel interfaces at the two ends of the ADVPN tunnel to remove the applied IPsec profile, and then identify whether the issue is resolved.

¡     If the issue persists after IPsec protection is removed, go to step 3.

¡     If removing IPsec protection immediately resolves the issue, IPsec services caused the fault. Go to step 5.

3.     Identify whether the ADVPN tunnel is created successfully.

Execute the display advpn session command at the two ends of the ADVPN tunnel to identify whether the specified ADVPN tunnel exists.

<Sysname> display advpn session interface tunnel 1 private-address 10.0.1.3

Private address  Public address              Port  Type  State      Holding time

10.0.0.3         192.168.180.136             1139  H-S   Success    5H 38M 8S

¡     If the ADVPN tunnel to the destination private network address does not exist or the tunnel status is Dumb, it indicates that the ADVPN tunnel has not been successfully established. Check the configurations of the VAM server, VAM client, and ADVPN tunnel in sequence and ensure their accuracy.

-     Execute the display vam server address-map command on the VAM server. If you do not see the IPv4 private-public address mapping information for VAM clients registered with the VAM server, check the configurations of both the VAM server and VAM client and make sure they are correct. For information about configuring the VAM server and VAM client, see the ADVPN configuration guide.

-     On the VAM server, execute the display vam server address-map command. If you see the IPv4 private-public address mapping information for VAM clients registered with the VAM server, identify whether the ADVPN tunnel configuration is correct. For information about configuring the ADVPN tunnel, see the ADVPN configuration guide.

¡     If an ADVPN tunnel to the peer private network address exists, go to step 4.

4.     Identify whether the ADVPN tunnel packet length is less than the interface's MTU value.

Execute the display interface tunnel command on the two ends of the ADVPN tunnel to check the value of the Maximum transmission unit field.

<Sysname> display interface tunnel 1

Tunnel1

Current state: UP

Line protocol state: UP

Description: Tunnel1 Interface

Bandwidth: 64kbps

Maximum transmission unit: 1476

...

Next, execute the ping –s packet-size –a local-address peer-address command to test the transmission of ping packets with different packet sizes through the tunnel interface. Identify a critical packet length value that results in ping failures, which represents the maximum length of ADVPN tunnel packets that the transmission link can allow.

¡     If the MTU value of the tunnel interface is less than the packet length of the ADVPN tunnel, execute the mtu size command in tunnel interface view to adjust the MTU value of the tunnel interface.

¡     If the MTU value of the tunnel interface is greater than the ADVPN tunnel packet length mentioned above, go to step 7.

5.     Identify whether the IKE SA and IPsec SA are successfully negotiated.

Execute the display ike sa and display ipsec sa commands at the two ends of the ADVPN tunnel to determine which phase of the SA negotiation was unsuccessful.

¡     If the display ike sa command does not display an IKE SA or state of the displayed IKE SA is Unknown, the phase 1 IKE SA negotiation fails. Identify whether the IKE profile configurations used by IPsec profiles applied at the two ends of the tunnel match. If the IKE configurations match but the issue persists, go to step 4.

¡     If the display ipsec sa command does not display an IPsec SA, it indicates that the phase two IPsec SA negotiation is unsuccessful. Identify whether the IPsec proposal configurations used by IPsec profiles applied at the two ends of the tunnel are correct.

-     The used IPsec proposals must include the same security protocol, authentication algorithm, encryption algorithm, and packet encapsulation mode.

-     The encapsulation modes used in the IPsec proposals must be tunnel mode.

If the IPsec proposals are configured correctly but the issue persists, go to step 6.

6.     Identify whether IPsec processing causes packet loss.

Execute the display ipsec statistic command at the two ends of the ADVPN tunnel to identify the reasons for packet loss.

¡     If there are packet losses in the IPsec message statistics, troubleshoot the issue based on the packet loss statistics.

<Sysname> display ipsec statistics

  IPsec packet statistics:

    Received/sent packets: 47/64

    Received/sent bytes: 3948/5208

    Received/sent packet rate: 5/5 packets/sec

    Received/sent byte rate: 290/290 bytes/sec

    Dropped packets (received/sent): 0/45

 

    Dropped packets statistics

      No available SA: 0

      Wrong SA: 0

      Invalid length: 0

      Authentication failure: 0

      Encapsulation failure: 0

      Decapsulation failure: 0

      Replayed packets: 0

      ACL check failure: 45

      MTU check failure: 0

      Loopback limit exceeded: 0

      Crypto speed limit exceeded: 0

      Responder only limitation: 0

¡     If no packet loss appears in the IPsec packet statistics, go to step 7.

7.     Collect the following information and contact the support:

¡     Results of each step.

¡     Configuration file.

¡     Debugging information for events and errors related to IKE, IPsec, VAM server, VAM client, and ADVPN. As a best practice, avoid enabling packet debugging to prevent excessive debugging messages from interfering with normal operations and information browsing on the user interface (UI).

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A