802.1X Remote desktop (RDP) user authentication

When connecting through remote desktop (RDP) to machines that are connected to 802.1X enabled switch ports (for example in Cisco ISE installations), connectivity issues, and session failures and disconnection issues could happen. Most of these issues are related to the fact that native 802.1X supplicants don't authenticate and authorize users that connect through RDP. Access lists on port are left as you have machine with no user logged on it. In most cases these ACLs allow only traffic for logging user on domain etc., and remote desktop traffic is not allowed by these ACLs. We have resolved this issue by using Cisco AnyConnect Secure Mobility Client with Network Access Manager - NAM module, as it behaves better and eliminates this problem by authenticating and authorizing user that is connecting through RDP and granting all rights and expected ACLs.

Cisco ISE - duplicate IP address on Windows 7

Windows 7 sometimes reports IP address conflict (duplicate IP address warning message) when connected to 802.1X enabled ports while device tracking feature is used on switch. Device tracking is used in most Cisco ISE designs. You can check if you have device tracking on switch with:

switch#sh run | inc trackin
ip device tracking
switch#

This usually happens on Windows Vista, Windows 2008 and Windows 7 clients, when disconnecting and connecting again (or shut no shut on switch), after locking and unlocking, or logging off and then logging on. Reason behind this is that windows machines could send ARP packets at the same time the switch is probing for device status. Possible solutions are:
Solution 1 - Delay ip device tracking ARP probe for few seconds (you should have IOS version that supports this command).

Switch(config)# ip device tracking [probe {delay interval}]

ARP packet will be sent few seconds after link-up, and it will not happen at same time when windows 7 clients send first ARP packets. For example:

switch(config)#ip device tracking probe delay 5

Solution 2 - Disable gratuitous ARP in Windows registry [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] Set ArpRetryCount (32-bit DWORD) to 0 If you decide to use second method, please be sure to manage duplicate address problem with some other controls on switches or elsewhere.

With device tracking probe delay we had really good results, and believe there is no need for registry tuning (and problem with it is that when new machine with default registry settings is connected, you will again get same duplicate IP address problem).

VRF aware IPSEC with NAT using only one WAN (outside) interface

If you need to terminate all customers (or users, or any other remote parties) on only one WAN (outside) interface, for example when terminating IPSec VPN on Internet facing interface, method described in our previous post - VRF aware IPSEC with NAT, although quite useful and flexible, can't be used.
But with few changes it is possible. In configuration below a single crypto map with two entries is applied on WAN (outside facing) interface, so subinterfaces are no longer needed.
Although not directly related to this concept, additional NAT rule for translating inside LAN address on central site is also added. But this part is also quite useful, since it shows statefullness of new NAT feature. It is possible to reach central site from customers sites using both real, and natted addresses. If packets to real address are sent from customer sites, return traffic will keep real address (ip nat source static 10.10.2.1 192.168.192.1 vrf Central_VRF will not be used). If NATed address is used, NAT will of course do its job. So there is different behavior for same packet coming from central LAN towards customer sites, depending part of which session it is (it depends did the customer sent a packet to real or natted IP address).
Special care needs to be made with VRFs in many commands below. With any small misconfiguration it will not work.

Central site router:

ip vrf Central_VRF
 rd 100:100
!
ip vrf Customer_A_VRF
 rd 100:101
!
ip vrf Customer_B_VRF
 rd 100:102
!
ip vrf WAN
 rd 100:99
! 
crypto keyring Customer_A_CRYPTO_KEYRING vrf WAN
  pre-shared-key address 192.168.1.11 key abc123
crypto keyring Customer_B_CRYPTO_KEYRING vrf WAN
  pre-shared-key address 192.168.1.12 key abc123
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp profile Customer_A_ISAKMP_PROFILE
   vrf Customer_A_VRF
   keyring Customer_A_CRYPTO_KEYRING
   match identity address 192.168.1.11 255.255.255.255 WAN
crypto isakmp profile Customer_B_ISAKMP_PROFILE
   vrf Customer_B_VRF
   keyring Customer_B_CRYPTO_KEYRING
   match identity address 192.168.1.12 255.255.255.255 WAN
!
crypto ipsec transform-set Customer_A_TRANSFORM_SET esp-aes 256 esp-sha-hmac 
crypto ipsec transform-set Customer_B_TRANSFORM_SET esp-aes 256 esp-sha-hmac 
!
crypto map WAN_CRYPTO_MAP 10 ipsec-isakmp 
 set peer 192.168.1.11
 set transform-set Customer_A_TRANSFORM_SET 
 set isakmp-profile Customer_A_ISAKMP_PROFILE
 match address Customer_A_CRYPTO_ACL
crypto map WAN_CRYPTO_MAP 20 ipsec-isakmp 
 set peer 192.168.1.12
 set transform-set Customer_B_TRANSFORM_SET 
 set isakmp-profile Customer_B_ISAKMP_PROFILE
 match address Customer_B_CRYPTO_ACL
!
interface GigabitEthernet0/1
 description WAN
 ip vrf forwarding WAN
 ip address 192.168.1.2 255.255.255.240
 ip nat enable
 no ip route-cache cef
 no ip route-cache
 duplex auto
 speed auto
 crypto map WAN_CRYPTO_MAP
!
interface GigabitEthernet0/2
 description To_Central_LAN
 ip vrf forwarding Central_VRF
 ip address 10.10.1.1 255.255.255.0
 ip nat enable
 no ip route-cache cef
 no ip route-cache
 duplex auto
 speed auto
!
ip nat source static 10.10.2.1 192.168.192.1 vrf Central_VRF
ip nat source static 192.168.10.1 10.10.101.1 vrf Customer_A_VRF
ip nat source static 192.168.10.1 10.10.102.1 vrf Customer_B_VRF
ip route vrf Central_VRF 192.168.192.1 255.255.255.255 10.10.1.2
ip route vrf Central_VRF 10.10.2.1 255.255.255.255 10.10.1.2
ip route vrf Central_VRF 10.10.101.1 255.255.255.255 GigabitEthernet0/1 192.168.1.11
ip route vrf Central_VRF 10.10.102.1 255.255.255.255 GigabitEthernet0/1 192.168.1.12
ip route vrf Customer_A_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1 192.168.1.11
ip route vrf Customer_A_VRF 192.168.192.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
ip route vrf Customer_A_VRF 10.10.2.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
ip route vrf Customer_B_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1 192.168.1.12
ip route vrf Customer_B_VRF 192.168.192.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
ip route vrf Customer_B_VRF 10.10.2.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
!
ip access-list extended Customer_A_CRYPTO_ACL
 permit ip 10.10.2.0 0.0.0.255 192.168.10.0 0.0.0.255
 permit ip 192.168.192.0 0.0.0.255 192.168.10.0 0.0.0.255
ip access-list extended Customer_B_CRYPTO_ACL
 permit ip 10.10.2.0 0.0.0.255 192.168.10.0 0.0.0.255
 permit ip 192.168.192.0 0.0.0.255 192.168.10.0 0.0.0.255

VRF aware IPSEC with NAT

(If you need to terminate all customers (or users, or any other remote parties) on only one WAN (outside) interface, for example when terminating IPSec VPN on Internet facing interface, see also VRF aware IPSEC with NAT using only one WAN (outside) interface)

If it is required to terminate multiple IPsec connections from multiple customers on same router, and customer address spaces are overlapping, one solution is VRF aware IPsec together with VRF aware NAT functionality using new ip nat enable syntax.
Each customer is connected to central site using its own private link that is through 802.1q connected to separate subinterface dedicated to each customer on WAN interface on central router.
In this example both customer sites arrive to central site using same IP address from their LAN: 192.168.10.1
This IP address is NAT-ed to 10.10.101.1 for customer A, and to 10.10.102.1 for customer B.
This allows central site to differentiate packets sent from different customers with same source IP address, and to be able to correctly route packets back.

There are numerous parts in this setup to do wrong. Some of the possible errors are:
proxy identities not supported
IPSec policy invalidated proposal with error 32
phase 2 SA policy not acceptable

Jun 17 13:24:57.739: IPSEC(ipsec_process_proposal): proxy identities not supported
Jun 17 13:24:57.739: ISAKMP:(1009): IPSec policy invalidated proposal with error 32
Jun 17 13:24:57.739: ISAKMP:(1009): phase 2 SA policy not acceptable! (local 192.168.1.2 remote 192.168.1.1)

Profile discarded due to VRF mismatch
peer matches *none* of the profiles

Jun 18 09:24:55.763: ISAKMP:(0):: Profile User_A_ISAKMP_PROFILE discarded due to VRF mismatch * * *
Jun 18 09:24:55.763: ISAKMP:(0):: Have you put proper FVRF in "match id ip-address" command?
Jun 18 09:24:55.763: ISAKMP:(0):: peer matches *none* of the profiles

unroutable in debug ip packet output

Jun 18 13:03:31.107: IP: s=192.168.10.1 (GigabitEthernet0/1.102), d=10.10.2.1, len 100, unroutable

Important parts to make this all working are (bold in Central site router configuration below):

• Add vrf vrf name after crypto keyring commands.
• Add vrf name after match identity address in crypto isakmp profiles.
• Put frontend outside interfaces in FVRF using ip vrf forwarding commands.
• We have found 15.0-1.M7 to be more stable with this setup, but in the meantime it is possible that newer releases in 15.1 and 15.2 are also fine
• Use two routes for each customer, one in Central VRF (with addresses NAT-ed to addresses how customer addresses are seen on Central site, one in Customer VRF (with original non-NAT-ed customer addresses - since we are using VRFs and NAT, these addresses can (and in this example are) overlapping.

ip route vrf Central_site_VRF 10.10.101.1 255.255.255.255 GigabitEthernet0/1.101 192.168.1.1
ip route vrf Central_site_VRF 10.10.102.1 255.255.255.255 GigabitEthernet0/1.102 192.168.1.1
ip route vrf Customer_A_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1.101 192.168.1.1
ip route vrf Customer_B_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1.102 192.168.1.1

Here are working configurations:
Central site router:

ip vrf Central_site_VRF
 rd 100:100
!
ip vrf Customer_A_VRF
 rd 100:101
!
ip vrf Customer_B_VRF
 rd 100:102
! 
crypto keyring Customer_A_CRYPTO_KEYRING vrf Customer_A_VRF
  pre-shared-key address 192.168.1.1 key abc123
crypto keyring Customer_B_CRYPTO_KEYRING vrf Customer_B_VRF
  pre-shared-key address 192.168.1.1 key abc123
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp profile Customer_A_ISAKMP_PROFILE
   vrf Customer_A_VRF
   keyring Customer_A_CRYPTO_KEYRING
   match identity address 192.168.1.1 255.255.255.255 Customer_A_VRF
crypto isakmp profile Customer_B_ISAKMP_PROFILE
   vrf Customer_B_VRF
   keyring Customer_B_CRYPTO_KEYRING
   match identity address 192.168.1.1 255.255.255.255 Customer_B_VRF
!
crypto ipsec transform-set Customer_A_TRANSFORM_SET esp-aes 256 esp-sha-hmac 
crypto ipsec transform-set Customer_B_TRANSFORM_SET esp-aes 256 esp-sha-hmac 
!
crypto map Customer_A_CRYPTO_MAP 10 ipsec-isakmp 
 set peer 192.168.1.1
 set transform-set Customer_A_TRANSFORM_SET 
 set isakmp-profile Customer_A_ISAKMP_PROFILE
 match address Customer_A_CRYPTO_ACL
!
crypto map Customer_B_CRYPTO_MAP 10 ipsec-isakmp 
 set peer 192.168.1.1
 set transform-set Customer_B_TRANSFORM_SET 
 set isakmp-profile Customer_B_ISAKMP_PROFILE
 match address Customer_B_CRYPTO_ACL
!
interface GigabitEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description WAN Interface
 no ip address
 duplex auto
 speed auto
 !
!
interface GigabitEthernet0/1.101
 description To_Customer_A
 encapsulation dot1Q 101
 ip vrf forwarding Customer_A_VRF
 ip address 192.168.1.2 255.255.255.0
 ip nat enable
 crypto map Customer_A_CRYPTO_MAP
!
interface GigabitEthernet0/1.102
 description To_Customer_B
 encapsulation dot1Q 102
 ip vrf forwarding Customer_B_VRF
 ip address 192.168.1.2 255.255.255.0
 ip nat enable
 crypto map Customer_B_CRYPTO_MAP
!
interface GigabitEthernet0/2
 description To_Central_site_LAN
 ip vrf forwarding Central_site_VRF
 ip address 10.10.1.1 255.255.255.0
 ip nat enable
 duplex auto
 speed auto
!
ip forward-protocol nd
!
ip nat source static 192.168.10.1 10.10.101.1 vrf Customer_A_VRF
ip nat source static 192.168.10.1 10.10.102.1 vrf Customer_B_VRF
ip route vrf Central_site_VRF 10.10.2.1 255.255.255.255 10.10.1.2
ip route vrf Central_site_VRF 10.10.101.1 255.255.255.255 GigabitEthernet0/1.101 192.168.1.1
ip route vrf Central_site_VRF 10.10.102.1 255.255.255.255 GigabitEthernet0/1.102 192.168.1.1
ip route vrf Customer_A_VRF 10.10.2.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
ip route vrf Customer_A_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1.101 192.168.1.1
ip route vrf Customer_B_VRF 10.10.2.1 255.255.255.255 GigabitEthernet0/2 10.10.1.2
ip route vrf Customer_B_VRF 192.168.10.1 255.255.255.255 GigabitEthernet0/1.102 192.168.1.1
!
ip access-list extended Customer_A_CRYPTO_ACL
 permit ip 10.10.2.0 0.0.0.255 192.168.10.0 0.0.0.255
ip access-list extended Customer_B_CRYPTO_ACL
 permit ip 10.10.2.0 0.0.0.255 192.168.10.0 0.0.0.255

Customer A:

crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key abc123 address 0.0.0.0 0.0.0.0
!
crypto ipsec transform-set ts esp-aes 256 esp-sha-hmac 
!
crypto map CRYPTO_MAP 10 ipsec-isakmp 
 set peer 192.168.1.2
 set transform-set ts 
 match address CRYPTO_ACL
!
interface Loopback0
 ip address 192.168.10.1 255.255.255.255
!
interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.252
 speed auto
 crypto map CRYPTO_MAP
!
ip route 0.0.0.0 0.0.0.0 192.168.1.2
!
ip access-list extended CRYPTO_ACL
 permit ip 192.168.10.0 0.0.0.255 10.10.2.0 0.0.0.255

Customer B (same as Customer B - to cover overlapping addresses case. It is of course possible to have different configurations, even different router or firewall vendors at customer sites):

crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key abc123 address 0.0.0.0 0.0.0.0
!
!
crypto ipsec transform-set ts esp-aes 256 esp-sha-hmac 
!
crypto map CRYPTO_MAP 10 ipsec-isakmp 
 set peer 192.168.1.2
 set transform-set ts 
 match address CRYPTO_ACL
!         
!
!
interface Loopback0
 ip address 192.168.10.1 255.255.255.255
!
interface Loopback1
 ip address 192.168.10.3 255.255.255.255
!
interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.252
 speed auto
 crypto map CRYPTO_MAP
!
ip route 0.0.0.0 0.0.0.0 192.168.1.2
!
ip access-list extended CRYPTO_ACL
 permit ip 192.168.10.0 0.0.0.255 10.10.2.0 0.0.0.255

PIX, ASA, FWSM firewall - different number of connections on primary and secondary (active and standby)

When issuing commands show connections or show resource usage different number of connections is usually noticed on active and standby PIX or ASA appliance or FWSM module. Same is true no matter if single context or multiple contexts are used. These numbers, graphs or statistics are also present in management software such as Infovista, HP Openview and so on, since SNMP OID (and OIDs) are directly mapped from same statistics. So number of connections on active firewall is larger than number of connections on standby firewall.

ASA#show conn
71590 in use, 80688 most used
 Network Processor 1 connections 
UDP outside 10.10.10.1:33222 inside 192.168.1.1:23445 idle 0:02:27 Bytes 149 
 FLAGS - 
TCP outside 10.10.10.1:33222 inside 192.168.1.1:23445 idle 0:00:01 Bytes 8378 FLAGS - UOI


ASA#sh res usa
Resource              Current         Peak      Limit        Denied Context
SSH                         1            2          5             0 context1
Syslogs [rate]              4         9400       9400        608779 context1
Conns                   19471        80688      80000             0 context1
Xlates                  69659        90088      56901             0 context1
Hosts                   62848        85442      56901             0 context1
Conns [rate]              228        27689      66600             0 context1
Syslogs [rate]              0           10         00           956 test
Conns                       4          380       2000             0 test
Xlates                     21           22        242             0 test
Hosts                      21           22        242             0 test
Syslogs [rate]              0           10         00           956 production
Conns                       4          380       2000             0 production
Xlates                     21           22        242             0 production
Hosts                      21           22        242             0 production

Reason for this is that HTTP connections are not replicated by default. If needed, HTTP sessions replication can be turned on by issuing failover replication http command.

hostname(config)# failover replication http

GETVPN packets almost silently dropped when onboard VPN hardware is used

show crypto ipsec sa shows no (counter is zero all the time) packets are encapsulated or decapsulated. This is fine, but strange part is that send and receive error counters are also 0 all the time.

Group_member#show crypto ipsec sa
interface: GigabitEthernet0/0
    Crypto map tag: getvpn-map, local addr 10.10.10.10

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 0.0.0.0 port 848
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 10.10.10.10, remote crypto endpt.: 0.0.0.0
     path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x393A7B96(960134038)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x393A7B96(960134038)
        transform: esp-256-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2045, flow_id: Onboard VPN:45, sibling_flags 80000040, crypto map: getvpn-map
        sa timing: remaining key lifetime (sec): (4219)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x393A7B96(960134038)
        transform: esp-256-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2046, flow_id: Onboard VPN:46, sibling_flags 80000040, crypto map: getvpn-map
        sa timing: remaining key lifetime (sec): (4219)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 5
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:

CEF is turned off, so debug ip packet is able to show needed info.

Group_member#(config-if)#no ip route-cache   

Packet debuging is turned on for problematic traffic flow, and also few other debugs.

Group_member#debug ip packet 100
IP packet debugging is on for access list 100
Group_member#debug crypto socket
Crypto secure socket events debugging is on
Group_member#debug cry ipsec error
Crypto IPSEC Error debugging is on

IP: s=10.10.1.1 (GigabitEthernet0/1), d=10.20.1.1, len 100, input feature, MCI Check(63), rtype 0, forus FALSE, sendself FALSE, mtu 0
IP: s=10.10.1.1 (GigabitEthernet0/1), d=10.20.1.1 (GigabitEthernet0/0), len 100, output feature, IPSec output classification(25), rtype 1, forus FALSE, sendself FALSE, mtu 0
Before encryption:
3F405990:                   45000064 7A760000          E..dzv..
3F4059A0: FE01ABC6 0A0A4101 0A504101 0800FFD4  ~.+F..A..PA....T
3F4059B0: 026A0000 00000008 B93CC2C6 ABCDABCD  .j......9BBF+M+M
3F4059C0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
3F4059D0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
3F4059E0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
3F4059F0: ABCDABCD ABCDABCD ABCDABCD           +M+M+M+M+M+M 
pak 48564F80 consumed in output feature , packet consumed, IPSec: to crypto engine(54), rtype 1, forus FALSE, sendself FALSE, mtu 0

Everything looks as it should.

Show crypto tech-support finally gives a trace to follow in its show cry engine accelerator statistic part, pkts dropped error counter and ppq full error counter:

Group_member#sh crypto tech-support
...
------------------ show cry engine accelerator statistic ------------------



Device:   Onboard VPN
Location: Onboard: 0
        :Statistics for encryption device since the last clear
         of counters 86585 seconds ago
                      0 packets in                         128 packets out        
                      0 bytes in                          9840 bytes out          
                      0 paks/sec in                          0 paks/sec out       
                      0 Kbits/sec in                         0 Kbits/sec out      
                      0 packets decrypted                    0 packets encrypted  
                      0 bytes before decrypt              9840 bytes encrypted    
                      0 bytes decrypted                      0 bytes after encrypt
                      0 packets decompressed                 0 packets compressed 
                      0 bytes before decomp                  0 bytes before comp  
                      0 bytes after decomp                   0 bytes after comp   
                      0 packets bypass decompr               0 packets bypass compres
                      0 bytes bypass decompres               0 bytes bypass compressi
                      0 packets not decompress               0 packets not compressed
                      0 bytes not decompressed               0 bytes not compressed
                  1.0:1 compression ratio                1.0:1 overall
                Last 5 minutes:
                      0 packets in                           0 packets out        
                      0 paks/sec in                          0 paks/sec out       
                      0 bits/sec in                          0 bits/sec out       
                      0 bytes decrypted                      0 bytes encrypted    
                      0 Kbits/sec decrypted                  0 Kbits/sec encrypted
                  1.0:1 compression ratio                1.0:1 overall

 Errors:
                    168 pkts dropped                       168 ppq full           
                      0 tx parts overflow                    0 rx parts overflow  
                      0 replenishment failure                0 zero len           
                      0 flow inputs bad                      0 cmd invalid        
                      0 IPV4 len                             0 IPV6 len           
                      0 algor invalid      
                      0 bad shadow particle                  0 algor disabled     
                      0 pre tx fail                          0 dma error          
                      0 dbit miss                            0 pipeline abort     
                      0 failsafe timeout                     0 reserv             
                      0 bad sz count                         0 bad shdw           
                      0 bad flow tx                          0 spi mismatch       
                      0 bad flow rx                          0 auth fail          
                      0 udm fs fail                          0 pad fail           
                      0 addr limit fixup fail                0 seq fail           
                      0 quad fix sp                          0 quad fix mp        
                      0 quad fix cont 
...                   

It seems that there is a problem when GETVPN is used with onboard crypto engine.
To confirm this assumption turn off accelerator (no crypto engine accelerator):

Group_member(config)#no cry eng accelerator
...switching to SW crypto engine

And the problem is gone and packets are processed as they should.

It turns out that there is a problem with some 1800, 2800 and 3800 routers with onboard crypto when some 12.4T IOS versions are used. "ppq full" counter going up in show crypto engine accelerator statistic identifies this issue. Upgrade to a version that resolves this issue is a complete solution, and no crypto engine accelerator a temporary workaround.

To check whether onboard hardware VPN Module accelerator or software VPN implementation is used you can use show crypto engine config:

Group_member#sh cry eng config


        crypto engine name:  Virtual Private Network (VPN) Module
        crypto engine type:  hardware
                     State:  Enabled
                  Location:  onboard 0
              Product Name:  Onboard-VPN
                FW Version:  01100200
              Time running:  491 seconds
               Compression:  Yes
                       DES:  Yes
                     3 DES:  Yes
                   AES CBC:  Yes (128,192,256)
                  AES CNTR:  No
     Maximum buffer length:  4096
          Maximum DH index:  0300
          Maximum SA index:  0300
        Maximum Flow index:  0600
      Maximum RSA key size:  2048

        crypto lib version:  20.0.0

     crypto engine in slot:  0
                  platform:  VPN hardware accelerator
        crypto lib version:  20.0.0
     
Group_member(config)#no cry eng accelerator

Group_member(config-if)#do sh cry eng config

        crypto engine name:  Cisco VPN Software Implementation
        crypto engine type:  software
             serial number:  FJ34208J
       crypto engine state:  installed
     crypto engine in slot:  N/A
                  platform:  Cisco Software Crypto Engine
        crypto lib version:  20.0.0

Short problem description:
GETVPN crypto engine accelerator ppq full errors
Packets are almost silently dropped while using hardware crypto engine. When crypto processing is switched to software (no cry eng accelerator), everything works fine.
Only place where we have seen trace of this error is in show crypto engine accelerator statistic command in "ppq full" statistics that are increasing by 1 for every packet that should be encrypted.