show crypto ipsec sa shows no (counter is zero all the time) packets are encapsulated or decapsulated. This is fine, but strange part is that send and receive error counters are also 0 all the time.
Group_member#show crypto ipsec sa
interface: GigabitEthernet0/0
Crypto map tag: getvpn-map, local addr 10.10.10.10
protected vrf: (none)
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
current_peer 0.0.0.0 port 848
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 10.10.10.10, remote crypto endpt.: 0.0.0.0
path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0x393A7B96(960134038)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0x393A7B96(960134038)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2045, flow_id: Onboard VPN:45, sibling_flags 80000040, crypto map: getvpn-map
sa timing: remaining key lifetime (sec): (4219)
IV size: 16 bytes
replay detection support: Y replay window size: 5
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x393A7B96(960134038)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2046, flow_id: Onboard VPN:46, sibling_flags 80000040, crypto map: getvpn-map
sa timing: remaining key lifetime (sec): (4219)
IV size: 16 bytes
replay detection support: Y replay window size: 5
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
CEF is turned off, so debug ip packet is able to show needed info.
Group_member#(config-if)#no ip route-cache
Packet debuging is turned on for problematic traffic flow, and also few other debugs.
Group_member#debug ip packet 100
IP packet debugging is on for access list 100
Group_member#debug crypto socket
Crypto secure socket events debugging is on
Group_member#debug cry ipsec error
Crypto IPSEC Error debugging is on
IP: s=10.10.1.1 (GigabitEthernet0/1), d=10.20.1.1, len 100, input feature, MCI Check(63), rtype 0, forus FALSE, sendself FALSE, mtu 0
IP: s=10.10.1.1 (GigabitEthernet0/1), d=10.20.1.1 (GigabitEthernet0/0), len 100, output feature, IPSec output classification(25), rtype 1, forus FALSE, sendself FALSE, mtu 0
Before encryption:
3F405990: 45000064 7A760000 E..dzv..
3F4059A0: FE01ABC6 0A0A4101 0A504101 0800FFD4 ~.+F..A..PA....T
3F4059B0: 026A0000 00000008 B93CC2C6 ABCDABCD .j......9BBF+M+M
3F4059C0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
3F4059D0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
3F4059E0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
3F4059F0: ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M
pak 48564F80 consumed in output feature , packet consumed, IPSec: to crypto engine(54), rtype 1, forus FALSE, sendself FALSE, mtu 0
Everything looks as it should.
Show crypto tech-support finally gives a trace to follow in its show cry engine accelerator statistic part, pkts dropped error counter and ppq full error counter:
Group_member#sh crypto tech-support
...
------------------ show cry engine accelerator statistic ------------------
Device: Onboard VPN
Location: Onboard: 0
:Statistics for encryption device since the last clear
of counters 86585 seconds ago
0 packets in 128 packets out
0 bytes in 9840 bytes out
0 paks/sec in 0 paks/sec out
0 Kbits/sec in 0 Kbits/sec out
0 packets decrypted 0 packets encrypted
0 bytes before decrypt 9840 bytes encrypted
0 bytes decrypted 0 bytes after encrypt
0 packets decompressed 0 packets compressed
0 bytes before decomp 0 bytes before comp
0 bytes after decomp 0 bytes after comp
0 packets bypass decompr 0 packets bypass compres
0 bytes bypass decompres 0 bytes bypass compressi
0 packets not decompress 0 packets not compressed
0 bytes not decompressed 0 bytes not compressed
1.0:1 compression ratio 1.0:1 overall
Last 5 minutes:
0 packets in 0 packets out
0 paks/sec in 0 paks/sec out
0 bits/sec in 0 bits/sec out
0 bytes decrypted 0 bytes encrypted
0 Kbits/sec decrypted 0 Kbits/sec encrypted
1.0:1 compression ratio 1.0:1 overall
Errors:
168 pkts dropped 168 ppq full
0 tx parts overflow 0 rx parts overflow
0 replenishment failure 0 zero len
0 flow inputs bad 0 cmd invalid
0 IPV4 len 0 IPV6 len
0 algor invalid
0 bad shadow particle 0 algor disabled
0 pre tx fail 0 dma error
0 dbit miss 0 pipeline abort
0 failsafe timeout 0 reserv
0 bad sz count 0 bad shdw
0 bad flow tx 0 spi mismatch
0 bad flow rx 0 auth fail
0 udm fs fail 0 pad fail
0 addr limit fixup fail 0 seq fail
0 quad fix sp 0 quad fix mp
0 quad fix cont
...
It seems that there is a problem when GETVPN is used with onboard crypto engine.
To confirm this assumption turn off accelerator (
no crypto engine accelerator):
Group_member(config)#no cry eng accelerator
...switching to SW crypto engine
And the problem is gone and packets are processed as they should.
It turns out that there is a problem with some 1800, 2800 and 3800 routers with onboard crypto when some 12.4T IOS versions are used. "
ppq full" counter going up in show crypto engine accelerator statistic identifies this issue. Upgrade to a version that resolves this issue is a complete solution, and
no crypto engine accelerator a temporary workaround.
To check whether onboard hardware VPN Module accelerator or software VPN implementation is used you can use show crypto engine config:
Group_member#sh cry eng config
crypto engine name: Virtual Private Network (VPN) Module
crypto engine type: hardware
State: Enabled
Location: onboard 0
Product Name: Onboard-VPN
FW Version: 01100200
Time running: 491 seconds
Compression: Yes
DES: Yes
3 DES: Yes
AES CBC: Yes (128,192,256)
AES CNTR: No
Maximum buffer length: 4096
Maximum DH index: 0300
Maximum SA index: 0300
Maximum Flow index: 0600
Maximum RSA key size: 2048
crypto lib version: 20.0.0
crypto engine in slot: 0
platform: VPN hardware accelerator
crypto lib version: 20.0.0
Group_member(config)#no cry eng accelerator
Group_member(config-if)#do sh cry eng config
crypto engine name: Cisco VPN Software Implementation
crypto engine type: software
serial number: FJ34208J
crypto engine state: installed
crypto engine in slot: N/A
platform: Cisco Software Crypto Engine
crypto lib version: 20.0.0
Short problem description:
GETVPN crypto engine accelerator ppq full errors
Packets are almost silently dropped while using hardware crypto engine. When crypto processing is switched to software (no cry eng accelerator), everything works fine.
Only place where we have seen trace of this error is in show crypto engine accelerator statistic command in "ppq full" statistics that are increasing by 1 for every packet that should be encrypted.
