Thursday, February 4, 2016

Instantly disconnected from a switch after authenticating over SSH

I ran into an issue yesterday with a 2960X IOS based Cisco switch, and being unable to connect over SSH.  Everytime I connected, it prompted me for my credentials, but as soon as I entered them it instantly kicked me out.  I asked my coworker to try his and he had the same experience.

I went over to the switch and consoled in to the master one of the stack, and was able to get right in.  I checked the logs, but nothing abnormal.  I checked AAA settings, but nothing was wrong there either.  The configuration hadn't been modified all day, and I remember having successfully logged into this switch just earlier that morning.

Then I checked the VTY lines at the bottom of the running configuration.

Switch#sh run | b line vty
line vty 0 4
 access-class 10 in
 exec-timeout 0 0
 logging synchronous
 transport input ssh
line vty 5 15
 access-class 10 in
 no exec
 transport input ssh


I thought this was really oddly configured.  The access-list 10 is to restrict certain users from logging in, but that was configured correctly.  But the exec-timeout set to "0 0" meant unlimited timeout.  In other words, anyone could connect over SSH and as long as they didn't disconnect, they would never be forced to.  By default, this setting is set to "exec-timeout 10 0", or 10 minutes and 0 seconds.

Additionally, the "no exec" under line vty 5 15 meant that once the first 5 vty lines filled up, a user could connect but wouldn't be allowed to go to the enable prompt.  Instead, they'd be disconnected.

Next, I checked all active SSH connections.

Switch#sh ssh
Connection Version Mode Encryption  Hmac         State                 Username
0          2.0     IN   aes256-cbc  hmac-sha1    Session started       user1
0          2.0     OUT  aes256-cbc  hmac-sha1    Session started       user1
1          2.0     IN   aes256-cbc  hmac-sha1    Session started       user1
1          2.0     OUT  aes256-cbc  hmac-sha1    Session started       user1
2          2.0     IN   aes256-cbc  hmac-sha1    Session started       user2
2          2.0     OUT  aes256-cbc  hmac-sha1    Session started       user2
3          2.0     IN   aes256-cbc  hmac-sha1    Session started       user3
3          2.0     OUT  aes256-cbc  hmac-sha1    Session started       user3
4          2.0     IN   aes256-cbc  hmac-sha1    Session started       user1
4          2.0     OUT  aes256-cbc  hmac-sha1    Session started       user1


Aha!  I found the issue!  There were 5 active SSH connections, 3 of which from the same user.  Therefore, when I tried to connect, I was assigned vty line 5, but because of the "no exec" command, I was immediately kicked out.

It turned out the above users had SSH'd into this switch at some point, but never manually disconnected (via the "exit" command).  Most likely, they closed their Putty/Terminal session without properly exiting, but with the timeout set to infinite, the connection remained established.

A few commands later, and I was able to SSH in.

Switch#disconnect ssh vty 0
Switch#disconnect ssh vty 1
Switch#disconnect ssh vty 2
Switch#disconnect ssh vty 3
Switch#disconnect ssh vty 4

 I then reset the vty config settings back to default to prevent this from happening again.

Switch#config t        
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#line vty 0 15
Switch(config-line)#exec-timeout 10 0
Switch(config-line)#exec
Switch(config-line)#end
Switch#sh run | b line vty
line vty 0 4
 access-class 10 in
 logging synchronous
 transport input ssh
line vty 5 15
 access-class 10 in
 transport input ssh

Thursday, February 28, 2013

AnyConnect VPN dropping connection after 1-2 minutes

I ran into an issue recently regarding an unstable AnyConnect VPN connection from a laptop, through a DSL router (not mine), terminating at a Cisco ASA running IOS 9.1. The laptop, running Windows 8 (yes, I know), had AnyConnect 3.1 installed on it. The problem was the AnyConnect connection would connect every time, but after about 1 or 2 minutes, it would disconnect.

The logs in the ASA indicated that the user session was being terminated on the remote end:

%ASA-5-722037: Group User IP SVC closing connection: User Requested.
%ASA-6-716002: Group User IP WebVPN session terminated: User Requested.

I couldn't figure out why, as the laptop wasn't being manually disconnected by the user, and the logs on the laptop didn't provide any reason why either.  Also, I forgot to mention that the laptop would stay connected when plugged in to any other internet connection, whether my home high speed connection, 

Running the "show vpn-sessiondb anyconnect" command on the ASA would show the following output below.  Note the top one was my testing laptop that worked, the bottom connection was the problematic one, with the output caught within the 1-2 minute window before it disconnected.


ASA/pri/act#show vpn-sessiondb anyconnect 

Session Type: AnyConnect

Username     : USERA           Index        : 229
Assigned IP  : x.x.x.x             Public IP    : x.x.x.x
Protocol     : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License      : AnyConnect Essentials
Encryption   : AnyConnect-Parent: (1)none  SSL-Tunnel: (1)DES  DTLS-Tunnel: (1)DES
Hashing      : AnyConnect-Parent: (1)none  SSL-Tunnel: (1)SHA1  DTLS-Tunnel: (1)SHA1
Bytes Tx     : 699061                 Bytes Rx     : 245379
Group Policy : GroupPolicy_VPN  Tunnel Group : VPN
Login Time   : 10:37:45 EST Tue Feb 26 2013
Duration     : 0h:41m:04s
Inactivity   : 0h:00m:00s
NAC Result   : Unknown
VLAN Mapping : N/A                    VLAN         : none

Username     : USERB           Index        : 236
Assigned IP  : x.x.x.x            Public IP    : x.x.x.x
Protocol     : AnyConnect-Parent SSL-Tunnel
License      : AnyConnect Essentials
Encryption   : AnyConnect-Parent: (1)none  SSL-Tunnel: (1)DES
Hashing      : AnyConnect-Parent: (1)none  SSL-Tunnel: (1)SHA1
Bytes Tx     : 59422                  Bytes Rx     : 36304
Group Policy : GroupPolicy_VPN   Tunnel Group : VPN
Login Time   : 11:15:31 EST Tue Feb 26 2013
Duration     : 0h:01m:18s
Inactivity   : 0h:00m:00s
NAC Result   : Unknown
VLAN Mapping : N/A                    VLAN         : none


This command actually led me to figuring out the solution.  The part where it says "DTLS-Tunnel" on the working connection, but not the bottom connection, was where the issue was.  See, the DSL router I was going through had an outbound filter setup to allow port 443/tcp only, NOT 443/udp.

After reading online about this, I learned that the AnyConnect SSL VPN connection first tries to connect over 443/tcp (TLS), then if successful, transitions over to 443/udp (DTLS).  The problem was the connection would establish, but since 443/udp was blocked, it would time out and terminate the connection.

In the ASA, I figured out the following command to run in order to disable the DTLS part of the connection, and force it to only use TLS, since that was what was open to me.

group-policy GroupPolicy_VPN attributes
 webvpn
  anyconnect ssl dtls none

Once I turned DTLS off in my group policy, the connection established and stayed up correctly after that.


The following text I found somewhere on Cisco's support forum site, linked here, which helped me figure out the issue.

The SSL-Tunnel is the TCP tunnel that is first created to the ASA. When it is fully established, the client will then try to negotiate a UDP DTLS-Tunnel. While the DTLS-Tunnel is being established, data can pass over the SSL-Tunnel. When the DTLS-Tunnel is fully established, all data now moves to the DTLS-tunnel and the SSL-tunnel is only used for occasional control channel traffic. If something should happen to UDP, the DTLS-Tunnel will be torn down and all data will pass through the SSL-Tunnel again.

The decision of how to send the data is very dynamic. As each network bound data packet is processed there is a point in the code where the decision is made to use either the SSL connection or the DTLS connection. If the DTLS connection is healthy at that moment, the packet is sent via the DTLS connection. Otherwise it is sent via the SSL connection.

The SSL connection is established first and data is passed over this connection while attempting to establish a DTLS connection. Once the DTLS connection has been established, the decision point in the code described above just starts sending the packets via the DTLS connection instead of the SSL connection. Control packets, on the other hand, always go over the SSL connection.

The key point is if the connection is considered healthy. If DTLS, an unreliable protocol, is in use and the DTLS connection has gone bad for whatever reason, the client does not know this until Dead Peer Detection (DPD) occurs. Therefore, data will be lost over the DTLS connection during that short period of time because the connection is still considered healthy. Once DPD occurs, data will immediately be set via the SSL connection and a DTLS reconnect will happen.

The ASA will send data over the last connection it received data on. Therefore, if the client has determined that the DTLS connection is not healthy, and starts sending data over the SSL connection, the ASA will reply on the SSL connection. The ASA will resume use of the DTLS connection when data is received on the DTLS connection."