Incorrect firewall behavior in 12.3(T) series IOS

My three-day long search has come to an end ;) I couldn’t understand why a router was starting to drop some packets after enabling TCP inspection, causing a dramatic link speed degradation (depending on the system, down to 1-40 kB/s on a 3 Mbit/s link).

The IOS was showing Invalid Seq# as the reason for dropping the packets (A.B.C.D is the router’s public address, to which the hosts from LAN are translated):

CBAC* sis 816B0094 pak 815B8D90 SIS_OPEN/ESTAB TCP ACK 2746875679 SEQ 2017937301 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 pak 815B7A88 SIS_OPEN/ESTAB TCP PSH ACK 2746875679 SEQ 2017944541 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 pak 81426B88 SIS_OPEN/ESTAB TCP ACK 2017945989 SEQ 2746875679 LEN 0 (A.B.C.D:49167) => (156.17.193.37:1902)
CBAC* sis 816B0094 pak 815B93E8 SIS_OPEN/ESTAB TCP PSH ACK 2746875679 SEQ 2017945989 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 pak 81422F44 SIS_OPEN/ESTAB TCP ACK 2017947437 SEQ 2746875679 LEN 0 (A.B.C.D:49167) => (156.17.193.37:1902)
CBAC* sis 816B0094 pak 815BC6A8 SIS_OPEN/ESTAB TCP PSH ACK 2746875679 SEQ 2017947437 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 pak 814271E0 SIS_OPEN/ESTAB TCP ACK 2017948885 SEQ 2746875679 LEN 0 (A.B.C.D:49167) => (156.17.193.37:1902)
CBAC* sis 816B0094 pak 815B9A40 SIS_OPEN/ESTAB TCP ACK 2746875679 SEQ 2017948885 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 pak 815B7104 SIS_OPEN/ESTAB TCP PSH ACK 2746875679 SEQ 2017950333 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 L4 inspect result: DROP packet 815B7104 (156.17.193.37:1902) (192.168.1.2:49167) bytes 1448 ErrStr = Invalid Seq# tcp
CBAC* sis 816B0094 pak 81424BD0 SIS_OPEN/ESTAB TCP ACK 2017950333 SEQ 2746875679 LEN 0 (A.B.C.D:49167) => (156.17.193.37:1902)
CBAC* sis 816B0094 pak 815BAD48 SIS_OPEN/ESTAB TCP PSH ACK 2746875679 SEQ 2017951781 LEN 1448 (156.17.193.37:1902) <= (192.168.1.2:49167)
CBAC* sis 816B0094 L4 inspect result: DROP packet 815BAD48 (156.17.193.37:1902) (192.168.1.2:49167) bytes 1448 ErrStr = Invalid Seq# tcp

I’ve tried everything – from changing the inspect list, through enabling it on a different interface, all kinds of config tweaks like setting more or less paranoid ACLs, ending with changing MSS and MTU on the interfaces. Eventually, I came across bug CSCef65365. It seems that in 12.3(T) series IOS TCP connections may be very slow if you enable TCP window scaling mechanism on the systems taking part in transmission. In a nutshell, it’s all about a TCP option described in RFC 1323, which makes it possible to increase the TCP window size beyond 65536 bytes. It comes in handy to effectively send data in some specific types of networks or under some specific circumstances.

A workaround (a software upgrade wasn’t an option in this case) was to turn off window scaling on the LAN boxes. Below you may find instructions how to do it in a couple of systems:

  • Linux (confirmed, TCP window scaling enabled by default)
    echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
  • Windows Vista (confirmed, TCP window scaling enabled by default)
    netsh interface tcp set global autotuninglevel=disabled
  • Windows XP (unconfirmed, TCP window scaling disabled by default)
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters => Tcp1323Opts [REG_DWORD] = 0

In case anyone wanted to revert to previous settings:

  • Linux
    echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
  • Windows Vista
    netsh interface tcp set global autotuninglevel=normal
  • Windows XP
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters => remove key Tcp1323Opts [REG_DWORD]
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s