I ran into an issue yesterday with a 2960X IOS based Cisco switch, and being unable to connect over SSH. Everytime I connected, it prompted me for my credentials, but as soon as I entered them it instantly kicked me out. I asked my coworker to try his and he had the same experience.
I went over to the switch and consoled in to the master one of the stack, and was able to get right in. I checked the logs, but nothing abnormal. I checked AAA settings, but nothing was wrong there either. The configuration hadn't been modified all day, and I remember having successfully logged into this switch just earlier that morning.
Then I checked the VTY lines at the bottom of the running configuration.
Switch#sh run | b line vty
line vty 0 4
access-class 10 in
exec-timeout 0 0
logging synchronous
transport input ssh
line vty 5 15
access-class 10 in
no exec
transport input ssh
I thought this was really oddly configured. The access-list 10 is to restrict certain users from logging in, but that was configured correctly. But the exec-timeout set to "0 0" meant unlimited timeout. In other words, anyone could connect over SSH and as long as they didn't disconnect, they would never be forced to. By default, this setting is set to "exec-timeout 10 0", or 10 minutes and 0 seconds.
Additionally, the "no exec" under line vty 5 15 meant that once the first 5 vty lines filled up, a user could connect but wouldn't be allowed to go to the enable prompt. Instead, they'd be disconnected.
Next, I checked all active SSH connections.
Switch#sh ssh
Connection Version Mode Encryption Hmac State Username
0 2.0 IN aes256-cbc hmac-sha1 Session started user1
0 2.0 OUT aes256-cbc hmac-sha1 Session started user1
1 2.0 IN aes256-cbc hmac-sha1 Session started user1
1 2.0 OUT aes256-cbc hmac-sha1 Session started user1
2 2.0 IN aes256-cbc hmac-sha1 Session started user2
2 2.0 OUT aes256-cbc hmac-sha1 Session started user2
3 2.0 IN aes256-cbc hmac-sha1 Session started user3
3 2.0 OUT aes256-cbc hmac-sha1 Session started user3
4 2.0 IN aes256-cbc hmac-sha1 Session started user1
4 2.0 OUT aes256-cbc hmac-sha1 Session started user1
Aha! I found the issue! There were 5 active SSH connections, 3 of which from the same user. Therefore, when I tried to connect, I was assigned vty line 5, but because of the "no exec" command, I was immediately kicked out.
It turned out the above users had SSH'd into this switch at some point, but never manually disconnected (via the "exit" command). Most likely, they closed their Putty/Terminal session without properly exiting, but with the timeout set to infinite, the connection remained established.
A few commands later, and I was able to SSH in.
Switch#disconnect ssh vty 0
Switch#disconnect ssh vty 1
Switch#disconnect ssh vty 2
Switch#disconnect ssh vty 3
Switch#disconnect ssh vty 4
I then reset the vty config settings back to default to prevent this from happening again.
Switch#config t
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#line vty 0 15
Switch(config-line)#exec-timeout 10 0
Switch(config-line)#exec
Switch(config-line)#end
Switch#sh run | b line vty
line vty 0 4
access-class 10 in
logging synchronous
transport input ssh
line vty 5 15
access-class 10 in
transport input ssh
Network Engineering
Thursday, February 4, 2016
Thursday, February 28, 2013
AnyConnect VPN dropping connection after 1-2 minutes
I ran into an issue recently regarding an unstable AnyConnect VPN connection from a laptop, through a DSL router (not mine), terminating at a Cisco ASA running IOS 9.1. The laptop, running Windows 8 (yes, I know), had AnyConnect 3.1 installed on it. The problem was the AnyConnect connection would connect every time, but after about 1 or 2 minutes, it would disconnect.
The logs in the ASA indicated that the user session was being terminated on the remote end:
%ASA-5-722037: Group User IP SVC closing connection: User Requested.
%ASA-6-716002: Group User IP WebVPN session terminated: User Requested.
I couldn't figure out why, as the laptop wasn't being manually disconnected by the user, and the logs on the laptop didn't provide any reason why either. Also, I forgot to mention that the laptop would stay connected when plugged in to any other internet connection, whether my home high speed connection,
Running the "show vpn-sessiondb anyconnect" command on the ASA would show the following output below. Note the top one was my testing laptop that worked, the bottom connection was the problematic one, with the output caught within the 1-2 minute window before it disconnected.
ASA/pri/act#show vpn-sessiondb anyconnect
Session Type: AnyConnect
Username : USERA Index : 229
Assigned IP : x.x.x.x Public IP : x.x.x.x
Protocol : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License : AnyConnect Essentials
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)DES DTLS-Tunnel: (1)DES
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1 DTLS-Tunnel: (1)SHA1
Bytes Tx : 699061 Bytes Rx : 245379
Group Policy : GroupPolicy_VPN Tunnel Group : VPN
Login Time : 10:37:45 EST Tue Feb 26 2013
Duration : 0h:41m:04s
Inactivity : 0h:00m:00s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
Username : USERB Index : 236
Assigned IP : x.x.x.x Public IP : x.x.x.x
Protocol : AnyConnect-Parent SSL-Tunnel
License : AnyConnect Essentials
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)DES
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1
Bytes Tx : 59422 Bytes Rx : 36304
Group Policy : GroupPolicy_VPN Tunnel Group : VPN
Login Time : 11:15:31 EST Tue Feb 26 2013
Duration : 0h:01m:18s
Inactivity : 0h:00m:00s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
The logs in the ASA indicated that the user session was being terminated on the remote end:
%ASA-5-722037: Group
ASA/pri/act#show vpn-sessiondb anyconnect
Session Type: AnyConnect
Username : USERA Index : 229
Assigned IP : x.x.x.x Public IP : x.x.x.x
Protocol : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License : AnyConnect Essentials
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)DES DTLS-Tunnel: (1)DES
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1 DTLS-Tunnel: (1)SHA1
Bytes Tx : 699061 Bytes Rx : 245379
Group Policy : GroupPolicy_VPN Tunnel Group : VPN
Login Time : 10:37:45 EST Tue Feb 26 2013
Duration : 0h:41m:04s
Inactivity : 0h:00m:00s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
Username : USERB Index : 236
Assigned IP : x.x.x.x Public IP : x.x.x.x
Protocol : AnyConnect-Parent SSL-Tunnel
License : AnyConnect Essentials
Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)DES
Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1
Bytes Tx : 59422 Bytes Rx : 36304
Group Policy : GroupPolicy_VPN Tunnel Group : VPN
Login Time : 11:15:31 EST Tue Feb 26 2013
Duration : 0h:01m:18s
Inactivity : 0h:00m:00s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
This command actually led me to figuring out the solution. The part where it says "DTLS-Tunnel" on the working connection, but not the bottom connection, was where the issue was. See, the DSL router I was going through had an outbound filter setup to allow port 443/tcp only, NOT 443/udp.
After reading online about this, I learned that the AnyConnect SSL VPN connection first tries to connect over 443/tcp (TLS), then if successful, transitions over to 443/udp (DTLS). The problem was the connection would establish, but since 443/udp was blocked, it would time out and terminate the connection.
In the ASA, I figured out the following command to run in order to disable the DTLS part of the connection, and force it to only use TLS, since that was what was open to me.
group-policy GroupPolicy_VPN attributes
webvpn
anyconnect ssl dtls none
Once I turned DTLS off in my group policy, the connection established and stayed up correctly after that.
The following text I found somewhere on Cisco's support forum site, linked here, which helped me figure out the issue.
The SSL-Tunnel is the TCP tunnel that is first created to the ASA. When it is fully established, the client will then try to negotiate a UDP DTLS-Tunnel. While the DTLS-Tunnel is being established, data can pass over the SSL-Tunnel. When the DTLS-Tunnel is fully established, all data now moves to the DTLS-tunnel and the SSL-tunnel is only used for occasional control channel traffic. If something should happen to UDP, the DTLS-Tunnel will be torn down and all data will pass through the SSL-Tunnel again.
The decision of how to send the data is very dynamic. As each network bound data packet is processed there is a point in the code where the decision is made to use either the SSL connection or the DTLS connection. If the DTLS connection is healthy at that moment, the packet is sent via the DTLS connection. Otherwise it is sent via the SSL connection.
The SSL connection is established first and data is passed over this connection while attempting to establish a DTLS connection. Once the DTLS connection has been established, the decision point in the code described above just starts sending the packets via the DTLS connection instead of the SSL connection. Control packets, on the other hand, always go over the SSL connection.
The key point is if the connection is considered healthy. If DTLS, an unreliable protocol, is in use and the DTLS connection has gone bad for whatever reason, the client does not know this until Dead Peer Detection (DPD) occurs. Therefore, data will be lost over the DTLS connection during that short period of time because the connection is still considered healthy. Once DPD occurs, data will immediately be set via the SSL connection and a DTLS reconnect will happen.
The ASA will send data over the last connection it received data on. Therefore, if the client has determined that the DTLS connection is not healthy, and starts sending data over the SSL connection, the ASA will reply on the SSL connection. The ASA will resume use of the DTLS connection when data is received on the DTLS connection."
The following text I found somewhere on Cisco's support forum site, linked here, which helped me figure out the issue.
The SSL-Tunnel is the TCP tunnel that is first created to the ASA. When it is fully established, the client will then try to negotiate a UDP DTLS-Tunnel. While the DTLS-Tunnel is being established, data can pass over the SSL-Tunnel. When the DTLS-Tunnel is fully established, all data now moves to the DTLS-tunnel and the SSL-tunnel is only used for occasional control channel traffic. If something should happen to UDP, the DTLS-Tunnel will be torn down and all data will pass through the SSL-Tunnel again.
The decision of how to send the data is very dynamic. As each network bound data packet is processed there is a point in the code where the decision is made to use either the SSL connection or the DTLS connection. If the DTLS connection is healthy at that moment, the packet is sent via the DTLS connection. Otherwise it is sent via the SSL connection.
The SSL connection is established first and data is passed over this connection while attempting to establish a DTLS connection. Once the DTLS connection has been established, the decision point in the code described above just starts sending the packets via the DTLS connection instead of the SSL connection. Control packets, on the other hand, always go over the SSL connection.
The key point is if the connection is considered healthy. If DTLS, an unreliable protocol, is in use and the DTLS connection has gone bad for whatever reason, the client does not know this until Dead Peer Detection (DPD) occurs. Therefore, data will be lost over the DTLS connection during that short period of time because the connection is still considered healthy. Once DPD occurs, data will immediately be set via the SSL connection and a DTLS reconnect will happen.
The ASA will send data over the last connection it received data on. Therefore, if the client has determined that the DTLS connection is not healthy, and starts sending data over the SSL connection, the ASA will reply on the SSL connection. The ASA will resume use of the DTLS connection when data is received on the DTLS connection."
Subscribe to:
Posts (Atom)