# Acceleration - Hardware Acceleration

> Hardware cryptography acceleration in VyOS with Intel QAT and AES-NI - boost VPN and IPsec tunnel performance in cloud environments

Source: https://opennix.org/en/docs/vyos/system/vyos-acceleration/


## Overview

Hardware acceleration of cryptographic operations lets you significantly improve VyOS performance when working with VPN tunnels, IPsec connections, and other cryptographic workloads. VyOS supports the use of dedicated hardware to offload the CPU from resource-intensive encryption and decryption operations.

### Supported technologies

VyOS supports the following hardware acceleration technologies:

- **Intel Quick Assist Technology (QAT)** - a dedicated processor for cryptographic operations and data compression
- **AES-NI (Advanced Encryption Standard New Instructions)** - an Intel/AMD CPU instruction set for hardware AES encryption
- **Hardware cryptographic modules** in Intel Atom C3000 series processors and above

### Benefits

- **Higher performance** - up to 3x for IPsec tunnels when using Intel QAT
- **Lower CPU load** - cryptographic operations are handled by dedicated hardware
- **Scalability** - the ability to handle a larger number of VPN connections
- **Energy efficiency** - lower power consumption per encryption operation

### Use in cloud environments

**Yandex Cloud:**
In the virtualized Yandex Cloud environment, physical Intel QAT hardware is not available, but VyOS can still use:
- AES-NI instructions passed through by the hypervisor (available on most modern virtual machine types)
- Software cryptography with optimizations
- Using VM types with a high CPU frequency for VPN gateways is recommended

**VK Cloud (bare-metal):**
On VK Cloud bare-metal servers with Intel Xeon or Atom C3000 series processors, full hardware acceleration is available:
- Intel QAT for maximum IPsec performance
- Full support for all hardware acceleration features
- Recommended for high-load VPN gateways

## Intel Quick Assist Technology (QAT)

Intel QAT is a hardware acceleration technology that provides cryptographic operations and data compression with high performance and low power consumption.

### Supported hardware

Intel QAT is available on the following platforms:

- **Intel Atom Processor C3000 Series** (Denverton)
  - C3308, C3338, C3508, C3538, C3558, C3758, C3850, C3950, C3955
  - Built-in QuickAssist modules for cryptography and compression

- **Intel Xeon Scalable Processors** (2nd generation and above)
  - Xeon Silver, Gold, Platinum with QAT support
  - Up to 32 cryptographic accelerators in a system

- **Intel Xeon D Processors** (D-1500, D-2100 Series)
  - Built-in QAT support
  - Optimal for network appliances

- **Discrete Intel QuickAssist Adapters**
  - Intel QuickAssist Adapter 8950
  - Intel QuickAssist Adapter 8960
  - Intel QuickAssist Adapter 8970

### Checking for QAT

Before configuring, you need to verify whether QAT is supported by your hardware:

```bash
show system acceleration qat
```

**Example output when QAT is present:**

```
Intel QuickAssist Technology acceleration detected

Devices:
  qat_dev0: Ready
  qat_dev1: Ready

Status: Available
Driver Version: 4.19.0
Firmware Version: 4.11.0
```

**Example output when QAT is absent:**

```
Intel QuickAssist Technology not detected

No QAT devices found on this system.
AES-NI CPU instructions: Available (fallback to software crypto with hardware AES)
```

### Checking QAT status

To verify that the QAT devices are ready, use the command:

```bash
show system acceleration qat status
```

**Example output:**

```
QAT Device Status:

Device qat_dev0:
  State: up
  Node Id: 0
  Device Type: c3xxx
  Services: crypto;dc (Crypto and Compression)
  Instances: 3

Device qat_dev1:
  State: up
  Node Id: 0
  Device Type: c3xxx
  Services: crypto;dc
  Instances: 3

Total Devices: 2
Active Devices: 2
```

### Checking device configuration

To view the detailed configuration of a specific QAT device:

```bash
show system acceleration qat device qat_dev0 config
```

**Example output:**

```
[GENERAL]
ServicesEnabled = cy;dc

ConfigVersion = 2

[KERNEL]
NumberCyInstances = 1
NumberDcInstances = 0

# Crypto - Kernel instance
Cy0Name = "IPSec0"
Cy0IsPolled = 0
Cy0CoreAffinity = 0

[SSL]
NumberCyInstances = 1
NumberDcInstances = 1

# Crypto - User instance
Cy0Name = "SSL0"
Cy0IsPolled = 1
Cy0CoreAffinity = 0

# Data Compression - User instance
Dc0Name = "Dc0"
Dc0IsPolled = 1
Dc0CoreAffinity = 0
```

### Monitoring cryptographic flows

To view encryption counters and usage statistics:

```bash
show system acceleration qat device qat_dev0 flows
```

**Example output:**

```
QAT Device qat_dev0 Flow Statistics:

Crypto Operations:
  Encryption Requests: 1,456,892
  Decryption Requests: 1,445,321
  Authentication Requests: 2,902,213
  Total Operations: 5,804,426

  Encryption Bytes: 1.2 GB
  Decryption Bytes: 1.1 GB
  Total Bytes Processed: 2.3 GB

Compression Operations:
  Compression Requests: 0
  Decompression Requests: 0
  Total Bytes Compressed: 0 B

Performance:
  Operations/sec: 45,621
  Throughput: 778 Mbps
  Average Latency: 0.12 ms

Errors: 0
```

## Configuring hardware acceleration

### Enabling Intel QAT

To enable Intel QAT support, apply the following configuration:

```bash
configure
set system acceleration qat
commit
save
```

Once QAT is enabled, the system automatically starts using hardware acceleration for all supported cryptographic operations, including:

- IPsec VPN tunnels
- OpenVPN connections (with limitations)
- SSL/TLS operations
- Data compression operations

### Disabling QAT

If you need to fall back to software cryptography:

```bash
configure
delete system acceleration qat
commit
save
```

### Verifying that settings are applied

After enabling QAT, verify that cryptographic operations are using hardware acceleration:

```bash
show system acceleration qat
show system acceleration qat status
```

You can also check the load on the QAT devices:

```bash
show system acceleration qat device qat_dev0 flows
```

With active IPsec tunnels, you should see the encryption/decryption operation counters increasing.

## IPsec Hardware Offload

Intel QAT provides hardware acceleration for the following IPsec algorithms:

### Supported encryption algorithms

- **AES-CBC** (128, 192, 256 bit)
- **AES-CTR** (128, 192, 256 bit)
- **AES-GCM** (128, 192, 256 bit) - recommended
- **3DES-CBC**

### Supported authentication algorithms

- **HMAC-SHA1**
- **HMAC-SHA256** - recommended
- **HMAC-SHA384**
- **HMAC-SHA512**
- **AES-XCBC-MAC**
- **AES-GMAC**

### Supported Diffie-Hellman groups

- **Group 2** (MODP 1024-bit)
- **Group 5** (MODP 1536-bit)
- **Group 14** (MODP 2048-bit) - recommended
- **Group 15** (MODP 3072-bit)
- **Group 16** (MODP 4096-bit)
- **Group 19** (ECP 256-bit)
- **Group 20** (ECP 384-bit)
- **Group 21** (ECP 521-bit)

### Example IPsec configuration with QAT

Setting up a site-to-site VPN using algorithms optimized for QAT:

```bash
configure

# ESP group using AES-GCM (optimal for QAT)
set vpn ipsec esp-group QAT-ESP lifetime 3600
set vpn ipsec esp-group QAT-ESP mode tunnel
set vpn ipsec esp-group QAT-ESP pfs dh-group14
set vpn ipsec esp-group QAT-ESP proposal 1 encryption aes256gcm128
set vpn ipsec esp-group QAT-ESP proposal 1 hash sha256

# IKE group with modern parameters
set vpn ipsec ike-group QAT-IKE dead-peer-detection action restart
set vpn ipsec ike-group QAT-IKE dead-peer-detection interval 30
set vpn ipsec ike-group QAT-IKE dead-peer-detection timeout 120
set vpn ipsec ike-group QAT-IKE ikev2-reauth no
set vpn ipsec ike-group QAT-IKE key-exchange ikev2
set vpn ipsec ike-group QAT-IKE lifetime 28800
set vpn ipsec ike-group QAT-IKE proposal 1 dh-group 14
set vpn ipsec ike-group QAT-IKE proposal 1 encryption aes256gcm128
set vpn ipsec ike-group QAT-IKE proposal 1 hash sha256

# Site-to-site peer configuration
set vpn ipsec site-to-site peer 203.0.113.10 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer 203.0.113.10 authentication pre-shared-secret 'YourStrongPSK'
set vpn ipsec site-to-site peer 203.0.113.10 connection-type initiate
set vpn ipsec site-to-site peer 203.0.113.10 ike-group QAT-IKE
set vpn ipsec site-to-site peer 203.0.113.10 local-address 198.51.100.10
set vpn ipsec site-to-site peer 203.0.113.10 tunnel 1 esp-group QAT-ESP
set vpn ipsec site-to-site peer 203.0.113.10 tunnel 1 local prefix 10.0.1.0/24
set vpn ipsec site-to-site peer 203.0.113.10 tunnel 1 remote prefix 10.0.2.0/24

# Enable QAT acceleration
set system acceleration qat

commit
save
```

### Example for Yandex Cloud (without QAT, with AES-NI)

In the virtualized Yandex Cloud environment, use an optimal configuration for software cryptography with AES-NI:

```bash
configure

# ESP group for software cryptography
set vpn ipsec esp-group YC-ESP lifetime 3600
set vpn ipsec esp-group YC-ESP mode tunnel
set vpn ipsec esp-group YC-ESP pfs dh-group14
set vpn ipsec esp-group YC-ESP proposal 1 encryption aes256
set vpn ipsec esp-group YC-ESP proposal 1 hash sha256

# IKE group
set vpn ipsec ike-group YC-IKE dead-peer-detection action restart
set vpn ipsec ike-group YC-IKE dead-peer-detection interval 30
set vpn ipsec ike-group YC-IKE dead-peer-detection timeout 120
set vpn ipsec ike-group YC-IKE ikev2-reauth no
set vpn ipsec ike-group YC-IKE key-exchange ikev2
set vpn ipsec ike-group YC-IKE lifetime 28800
set vpn ipsec ike-group YC-IKE proposal 1 dh-group 14
set vpn ipsec ike-group YC-IKE proposal 1 encryption aes256
set vpn ipsec ike-group YC-IKE proposal 1 hash sha256

# Site-to-site between two VPCs in Yandex Cloud
set vpn ipsec site-to-site peer 10.128.0.10 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer 10.128.0.10 authentication pre-shared-secret 'YandexCloudVPNKey'
set vpn ipsec site-to-site peer 10.128.0.10 connection-type initiate
set vpn ipsec site-to-site peer 10.128.0.10 ike-group YC-IKE
set vpn ipsec site-to-site peer 10.128.0.10 local-address 10.129.0.10
set vpn ipsec site-to-site peer 10.128.0.10 tunnel 1 esp-group YC-ESP
set vpn ipsec site-to-site peer 10.128.0.10 tunnel 1 local prefix 10.129.1.0/24
set vpn ipsec site-to-site peer 10.128.0.10 tunnel 1 remote prefix 10.128.1.0/24

commit
save
```

### Example for VK Cloud (bare-metal with QAT)

On VK Cloud bare-metal servers with QAT support, go for maximum performance:

```bash
configure

# ESP group with ChaCha20-Poly1305 for maximum performance on QAT
set vpn ipsec esp-group VK-ESP-QAT lifetime 3600
set vpn ipsec esp-group VK-ESP-QAT mode tunnel
set vpn ipsec esp-group VK-ESP-QAT pfs dh-group14
set vpn ipsec esp-group VK-ESP-QAT proposal 1 encryption aes256gcm128
set vpn ipsec esp-group VK-ESP-QAT proposal 2 encryption aes256
set vpn ipsec esp-group VK-ESP-QAT proposal 2 hash sha256

# IKE group
set vpn ipsec ike-group VK-IKE-QAT dead-peer-detection action restart
set vpn ipsec ike-group VK-IKE-QAT dead-peer-detection interval 15
set vpn ipsec ike-group VK-IKE-QAT dead-peer-detection timeout 60
set vpn ipsec ike-group VK-IKE-QAT ikev2-reauth no
set vpn ipsec ike-group VK-IKE-QAT key-exchange ikev2
set vpn ipsec ike-group VK-IKE-QAT lifetime 28800
set vpn ipsec ike-group VK-IKE-QAT proposal 1 dh-group 14
set vpn ipsec ike-group VK-IKE-QAT proposal 1 encryption aes256gcm128
set vpn ipsec ike-group VK-IKE-QAT proposal 1 hash sha256

# Site-to-site VPN between data centers
set vpn ipsec site-to-site peer 185.185.185.10 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer 185.185.185.10 authentication pre-shared-secret 'VKCloudBareMetal2025'
set vpn ipsec site-to-site peer 185.185.185.10 connection-type respond
set vpn ipsec site-to-site peer 185.185.185.10 ike-group VK-IKE-QAT
set vpn ipsec site-to-site peer 185.185.185.10 local-address 185.185.186.10
set vpn ipsec site-to-site peer 185.185.185.10 tunnel 1 esp-group VK-ESP-QAT
set vpn ipsec site-to-site peer 185.185.185.10 tunnel 1 local prefix 192.168.10.0/24
set vpn ipsec site-to-site peer 185.185.185.10 tunnel 1 remote prefix 192.168.20.0/24

# Enable QAT acceleration
set system acceleration qat

commit
save
```

## AES-NI CPU Instructions

AES-NI (Advanced Encryption Standard New Instructions) is an Intel and AMD CPU instruction set for hardware acceleration of AES encryption operations.

### Checking AES-NI support

To check whether the CPU supports AES-NI instructions, run:

```bash
show system cpu info
```

**Example output with AES-NI support:**

```
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
Model name:          Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
CPU MHz:             2399.998
Hypervisor vendor:   KVM
Virtualization type: full
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
                     pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
                     constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq
                     pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt
                     tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm
                     abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle
                     avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
```

The **aes** flag in the Flags list indicates AES-NI support.

An alternative check from the command line:

```bash
run show cpu-info | grep aes
```

**Expected output:**

```
flags: ... aes ...
```

### Automatic use of AES-NI

VyOS automatically uses AES-NI instructions when they are available on the CPU. No additional configuration is required.

The cryptographic libraries (OpenSSL, strongSwan) automatically detect the presence of AES-NI and use hardware acceleration for:

- **AES-128-CBC, AES-192-CBC, AES-256-CBC**
- **AES-128-CTR, AES-192-CTR, AES-256-CTR**
- **AES-128-GCM, AES-192-GCM, AES-256-GCM**
- **AES-128-CCM, AES-192-CCM, AES-256-CCM**

### Verifying AES-NI use in OpenSSL

To compare AES performance with and without hardware acceleration:

```bash
# Test with hardware acceleration (default)
openssl speed -evp aes-256-gcm

# Force AES-NI off for comparison
OPENSSL_ia32cap="~0x200000200000000" openssl speed -evp aes-256-gcm
```

**Example results with AES-NI:**

```
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     285714.29k   857142.86k  2285714.29k  3657142.86k  4571428.57k  4685714.29k
```

**Example results without AES-NI (software implementation):**

```
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      71428.57k   214285.71k   571428.57k   914285.71k  1142857.14k  1171428.57k
```

The performance difference is roughly **4-5x** in favor of AES-NI.

## Performance testing

### IPsec benchmark without QAT

Testing IPsec tunnel performance on an Intel Atom C3558 processor **without** QAT enabled:

```bash
# On one side of the tunnel
iperf3 -s

# On the other side of the tunnel (through IPsec)
iperf3 -c 10.0.2.10 -t 60 -P 4
```

**Results without QAT:**

```
[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-60.00  sec  1.89 GBytes   270 Mbits/sec  124    sender
[SUM]   0.00-60.00  sec  1.88 GBytes   269 Mbits/sec         receiver

CPU Usage: 85-95% (strongSwan processes)
```

### IPsec benchmark with QAT

Enable QAT and run the test again:

```bash
configure
set system acceleration qat
commit
save
exit

# Restart IPsec to apply QAT
restart vpn ipsec
```

Repeat the test:

```bash
iperf3 -c 10.0.2.10 -t 60 -P 4
```

**Results with QAT:**

```
[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-60.00  sec  5.45 GBytes   778 Mbits/sec   18    sender
[SUM]   0.00-60.00  sec  5.44 GBytes   777 Mbits/sec         receiver

CPU Usage: 25-35% (strongSwan processes)
```

**Performance improvement:**
- Throughput: **+188%** (270 -> 778 Mbps)
- CPU load: **-65%** (90% -> 30%)
- Number of retransmissions: **-85%** (124 -> 18)

### Algorithm comparison table

Performance of various encryption algorithms on an Intel Atom C3558 with QAT (iperf3, 60 seconds):

| Algorithm | Without QAT | With QAT | Improvement |
|----------|---------|-------|-----------|
| AES-128-CBC + SHA1 | 320 Mbps | 850 Mbps | +166% |
| AES-256-CBC + SHA256 | 270 Mbps | 778 Mbps | +188% |
| AES-128-GCM | 380 Mbps | 920 Mbps | +142% |
| AES-256-GCM | 310 Mbps | 815 Mbps | +163% |
| 3DES-CBC + SHA1 | 145 Mbps | 420 Mbps | +190% |
| ChaCha20-Poly1305 | 450 Mbps | 450 Mbps | 0% (not supported by QAT) |

**Recommendations:**
- For maximum performance with QAT, use **AES-128-GCM**
- For a balance of security and performance, use **AES-256-GCM**
- ChaCha20-Poly1305 has high software performance but is not accelerated by QAT

### Real-time performance monitoring

To track QAT usage in real time, use a monitoring script:

```bash
# Create the monitoring script
cat > /config/scripts/qat-monitor.sh << 'EOF'
#!/bin/bash

while true; do
    clear
    echo "QAT Performance Monitor - $(date)"
    echo "========================================"

    show system acceleration qat status
    echo ""
    echo "Flow Statistics:"
    show system acceleration qat device qat_dev0 flows | grep -E "(Operations|Throughput|Errors)"

    echo ""
    echo "CPU Usage:"
    top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print "CPU Load: " 100 - $1"%"}'

    echo ""
    echo "IPsec Tunnels:"
    show vpn ipsec sa | grep -E "(peer|State)"

    sleep 5
done
EOF

chmod +x /config/scripts/qat-monitor.sh

# Start monitoring
/config/scripts/qat-monitor.sh
```

## Verification and diagnostics

### Comprehensive hardware acceleration check

Run the following commands for full diagnostics:

```bash
# 1. Check the CPU and AES-NI support
show system cpu info | grep -i aes

# 2. Check for QAT presence
show system acceleration qat

# 3. QAT device status
show system acceleration qat status

# 4. QAT configuration (if available)
show system acceleration qat device qat_dev0 config

# 5. Cryptographic operation counters
show system acceleration qat device qat_dev0 flows

# 6. IPsec tunnel status
show vpn ipsec sa

# 7. CPU load
show system processes

# 8. QAT driver version (from the system)
show version kernel
```

### Checking the logs

To identify QAT issues, check the system logs:

```bash
# QAT driver load logs
show log kernel | grep -i qat

# IPsec logs with QAT usage markers
show log vpn ipsec | grep -i qat

# General system logs
show log tail 100
```

**Example of successful QAT initialization:**

```
kernel: qat_c3xxx 0000:00:0b.0: enabling device (0000 -> 0002)
kernel: qat_c3xxx 0000:00:0b.0: Enabling MSI IRQ 24
kernel: qat_c3xxx 0000:00:0b.0: Successfully initialized qat_dev0
kernel: qat_c3xxx 0000:00:0b.0: qat_dev0 started 3 acceleration engines
kernel: QAT: Device qat_dev0 is ready for use
```

**Example of a QAT error:**

```
kernel: qat_c3xxx 0000:00:0b.0: Failed to initialize qat_dev0
kernel: qat_c3xxx 0000:00:0b.0: Firmware load failed
kernel: QAT: Device qat_dev0 is NOT available
```

### Checking cryptographic libraries

Verify that strongSwan (the IPsec daemon) is using QAT:

```bash
# Check strongSwan plugins
sudo ipsec statusall | grep -i plugin

# List loaded plugins
sudo ipsec listplugins
```

**Expected output when QAT is in use:**

```
Loaded plugins: charon aes des rc2 sha2 sha1 md5 mgf1 random nonce x509
                revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12
                pgp dnskey sshkey pem openssl kernel-netlink socket-default
                stroke vici updown eap-identity addrblock unity qat

qat plugin loaded: yes
```

### Testing individual algorithms

To test specific encryption algorithms:

```bash
# AES-256-GCM test
openssl speed -elapsed -evp aes-256-gcm

# AES-256-CBC test
openssl speed -elapsed -evp aes-256-cbc

# SHA256 test
openssl speed -elapsed sha256

# Comparison with the software implementation (AES-NI disabled)
OPENSSL_ia32cap="~0x200000200000000" openssl speed -elapsed -evp aes-256-gcm
```

## Troubleshooting

### QAT not detected

**Problem:** The `show system acceleration qat` command reports that QAT is not available.

**Solution:**

1. Check whether your hardware supports QAT:
   ```bash
   lspci | grep -i quickassist
   lspci | grep -i co-processor
   ```

2. Make sure QAT is enabled in the BIOS:
   - Enter the BIOS/UEFI
   - Find the "Advanced" or "Chipset Configuration" section
   - Enable the "Intel QuickAssist Technology" option

3. Check that the driver is loaded:
   ```bash
   lsmod | grep qat
   ```

4. If the module is not loaded, try loading it manually:
   ```bash
   sudo modprobe qat_c3xxx
   sudo modprobe intel_qat
   ```

### QAT device in the "Down" state

**Problem:** `show system acceleration qat status` shows the device in the "down" state.

**Solution:**

1. Check the error logs:
   ```bash
   show log kernel | grep -i qat | grep -i error
   dmesg | grep -i qat | grep -i error
   ```

2. Restart the QAT service:
   ```bash
   sudo systemctl restart qat
   # or
   sudo /etc/init.d/qat_service restart
   ```

3. Update the QAT firmware:
   ```bash
   # Check the current version
   show system acceleration qat status | grep Firmware

   # Update (platform-dependent)
   # Refer to the hardware vendor's documentation
   ```

4. Check the CPU temperature:
   ```bash
   show hardware sensors
   ```
   Overheating can cause QAT to shut down.

### IPsec is not using QAT

**Problem:** QAT is working, but the IPsec tunnels show no performance improvement.

**Solution:**

1. Check that QAT is enabled in the configuration:
   ```bash
   show configuration commands | grep acceleration
   ```

2. Restart IPsec after enabling QAT:
   ```bash
   restart vpn ipsec
   ```

3. Make sure supported algorithms are being used:
   ```bash
   show vpn ipsec sa detail
   ```
   Verify that the encryption and hashing algorithms are among those supported by QAT.

4. Check the QAT statistics during active data transfer:
   ```bash
   # Start transferring data through the tunnel
   # In another window, monitor the counters
   watch -n 1 'show system acceleration qat device qat_dev0 flows'
   ```
   The "Encryption Requests" and "Decryption Requests" counters should be increasing.

### Low performance with QAT

**Problem:** QAT is enabled, but performance does not meet expectations.

**Solution:**

1. Check the encryption algorithms:
   - Use AES-GCM instead of AES-CBC + SHA
   - AES-GCM delivers better performance on QAT

2. Check the DPD (Dead Peer Detection) settings:
   - Overly frequent checks can reduce performance
   - Recommended values: interval 30, timeout 120

3. Optimize the network parameters:
   ```bash
   configure
   set system option performance throughput
   commit
   save
   ```

4. Check the MTU and MSS:
   ```bash
   # Check the current MTU
   show interfaces ethernet eth0

   # Set an optimal MSS for IPsec
   configure
   set vpn ipsec site-to-site peer 203.0.113.10 tunnel 1 mtu 1400
   commit
   save
   ```

5. Make sure software offload is not being used:
   ```bash
   # Disable software offloading if enabled
   configure
   delete system option performance
   set system acceleration qat
   commit
   save
   ```

### Errors in the QAT logs

**Problem:** QAT error messages appear in the logs.

**Solution:**

1. **"qat_dev0: heartbeat failed"**
   - Indicates firmware problems
   - Solution: Update the firmware or reboot the system

2. **"qat_dev0: ring buffer overflow"**
   - QAT is overloaded with requests
   - Solution: Increase the number of QAT instances or distribute the load

3. **"qat_dev0: uncorrectable error detected"**
   - A hardware problem
   - Solution: Inspect the hardware; an RMA may be required

4. **"Intel QAT: failed to allocate memory"**
   - Not enough memory for QAT operations
   - Solution: Add system memory or reduce the number of QAT instances

### AES-NI problems

**Problem:** AES-NI is not detected or not being used.

**Solution:**

1. Make sure the CPU supports AES-NI:
   ```bash
   grep -o 'aes' /proc/cpuinfo | uniq
   ```

2. Check that AES-NI is enabled in the BIOS:
   - Some BIOSes have an option to disable AES-NI
   - It is usually found in the "Security" or "Advanced CPU Configuration" section

3. On virtual machines, make sure the AES flag is passed through to the guest:
   - VMware: Enable "Expose hardware assisted virtualization"
   - KVM/QEMU: Use CPU type "host-passthrough" or add the "+aes" flag
   - Hyper-V: Use a CPU with AES-NI support and enable nested virtualization

4. For Yandex Cloud, choose a VM type with a modern processor:
   - Recommended: Intel Ice Lake, Intel Cascade Lake
   - AES-NI is available on all VM types except legacy ones

## Best practices

### Choosing algorithms for maximum performance

**With Intel QAT:**

1. **Priority 1: AES-128-GCM**
   - Best performance on QAT
   - Good security for most use cases
   - Low latency

2. **Priority 2: AES-256-GCM**
   - A balance of performance and security
   - Recommended for critical data
   - Marginally slower than AES-128-GCM

3. **Avoid:**
   - ChaCha20-Poly1305 (not supported by QAT)
   - 3DES (legacy, slow)
   - AES-CBC with a separate HMAC (slower than GCM)

**Without QAT (AES-NI only):**

1. **Priority 1: ChaCha20-Poly1305**
   - Best software performance
   - Good security
   - Does not require AES-NI

2. **Priority 2: AES-256-GCM**
   - Good acceleration through AES-NI
   - Broad compatibility

3. **Priority 3: AES-256-CBC + SHA256**
   - A conservative choice for maximum compatibility

### Configuration for different scenarios

**High throughput (bulk transfer):**

```bash
configure
# Large packet size, minimal overhead
set vpn ipsec esp-group HIGH-THROUGHPUT lifetime 3600
set vpn ipsec esp-group HIGH-THROUGHPUT mode tunnel
set vpn ipsec esp-group HIGH-THROUGHPUT pfs disable
set vpn ipsec esp-group HIGH-THROUGHPUT proposal 1 encryption aes128gcm128
commit
save
```

**Low latency (real-time traffic):**

```bash
configure
# Fast algorithms, frequent key rotation for security
set vpn ipsec esp-group LOW-LATENCY lifetime 1800
set vpn ipsec esp-group LOW-LATENCY mode tunnel
set vpn ipsec esp-group LOW-LATENCY pfs dh-group14
set vpn ipsec esp-group LOW-LATENCY proposal 1 encryption aes128gcm64
set vpn ipsec ike-group LOW-LATENCY-IKE lifetime 14400
commit
save
```

**Maximum security:**

```bash
configure
# Strong algorithms, frequent key rotation
set vpn ipsec esp-group MAX-SECURITY lifetime 1800
set vpn ipsec esp-group MAX-SECURITY mode tunnel
set vpn ipsec esp-group MAX-SECURITY pfs dh-group16
set vpn ipsec esp-group MAX-SECURITY proposal 1 encryption aes256gcm128
set vpn ipsec ike-group MAX-SECURITY-IKE lifetime 7200
set vpn ipsec ike-group MAX-SECURITY-IKE proposal 1 dh-group 16
set vpn ipsec ike-group MAX-SECURITY-IKE proposal 1 encryption aes256gcm128
commit
save
```

### Monitoring and maintenance

**Regular QAT monitoring:**

1. Create a script for a daily status check:

```bash
configure
set system task-scheduler task qat-daily-check executable path /config/scripts/qat-check.sh
set system task-scheduler task qat-daily-check interval 1d
commit
save
```

2. Create the check script itself:

```bash
cat > /config/scripts/qat-check.sh << 'EOF'
#!/bin/bash

LOG_FILE="/var/log/qat-health.log"
DATE=$(date "+%Y-%m-%d %H:%M:%S")

echo "[$DATE] QAT Health Check" >> $LOG_FILE

# Check the status
STATUS=$(show system acceleration qat status | grep "Active Devices")
echo "  $STATUS" >> $LOG_FILE

# Check for errors
ERRORS=$(show system acceleration qat device qat_dev0 flows | grep "Errors:")
echo "  $ERROR S" >> $LOG_FILE

# Check performance
OPS=$(show system acceleration qat device qat_dev0 flows | grep "Operations/sec")
echo "  $OPS" >> $LOG_FILE

# Alert if problems are detected
if echo "$ERRORS" | grep -v "Errors: 0" > /dev/null; then
    echo "  WARNING: QAT errors detected!" >> $LOG_FILE
    # Optional: send a notification
fi

echo "" >> $LOG_FILE
EOF

chmod +x /config/scripts/qat-check.sh
```

**Proactive performance monitoring:**

```bash
# Using SNMP for monitoring (if configured)
configure
set service snmp community public authorization ro
set service snmp community public network 10.0.0.0/8
commit
save
```

Monitoring through external systems (Zabbix, Prometheus):
- CPU Usage (OID: .1.3.6.1.4.1.2021.11)
- Network throughput
- IPsec tunnel status (custom scripts)

### Backup and recovery

**Saving a configuration with QAT:**

```bash
# Back up the current configuration
save /config/backup/config-with-qat-$(date +%Y%m%d).boot

# Copy to a remote server
scp /config/backup/config-with-qat-$(date +%Y%m%d).boot user@backup-server:/backups/vyos/
```

**Recovering after a QAT failure:**

If the QAT hardware fails:

```bash
configure
# Disable QAT to keep running on software cryptography
delete system acceleration qat
commit
save

# Restart IPsec
restart vpn ipsec
```

Performance will drop, but connectivity will be preserved.

### Updating firmware and drivers

**Updating the QAT firmware:**

1. Check the current version:
   ```bash
   show system acceleration qat status | grep Firmware
   ```

2. Download the new firmware from the vendor's site

3. Install it (the procedure is platform-dependent):
   ```bash
   # Example for the Intel Atom C3000 Series
   sudo qat_fwupdate -d qat_dev0 -f /path/to/new_firmware.bin
   ```

4. Reboot the system:
   ```bash
   reboot now
   ```

**Upgrading VyOS while keeping QAT:**

```bash
# Save the configuration before upgrading
save

# Add the new VyOS image
add system image <URL_to_new_image>

# After the reboot, check QAT
show system acceleration qat status

# If QAT is not working, reinstall the driver
# (usually automatic when the new kernel boots)
```

### Security

**Security recommendations when using QAT:**

1. **Update the firmware regularly**
   - Vendors release security updates
   - Review the release notes for CVEs

2. **Monitor for anomalies**
   - A sharp drop in performance may indicate an attack
   - An unusually high QAT error rate

3. **Logging**
   ```bash
   configure
   set system syslog global facility all level info
   set system syslog global facility security level warning
   commit
   save
   ```

4. **Restrict access to QAT statistics**
   - Statistics can reveal information about the traffic
   - Restrict access to show commands via RBAC (if in use)

## Additional resources

### Official documentation

- **VyOS Documentation**: https://docs.vyos.io/
- **Intel QAT Documentation**: https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html
- **Intel QAT Software**: https://github.com/intel/QAT_Engine
- **strongSwan QAT Plugin**: https://wiki.strongswan.org/projects/strongswan/wiki/QAT

### Hardware support

- **Intel Atom C3000 Series**: https://ark.intel.com/content/www/us/en/ark/products/series/95217/intel-atom-processor-c3000-series.html
- **Intel Xeon Scalable Processors**: https://www.intel.com/content/www/us/en/products/docs/processors/xeon/xeon-scalable-processors.html

### Community and support

- **VyOS Community Forum**: https://forum.vyos.io/
- **VyOS Slack**: https://slack.vyos.io/
- **Yandex Cloud Support**: https://cloud.yandex.ru/docs/support/
- **VK Cloud Support**: https://mcs.mail.ru/help/

### Testing tools

- **iperf3**: Network throughput testing
  ```bash
  sudo apt install iperf3  # On the test machines
  ```

- **OpenSSL**: Benchmarking cryptographic operations
  ```bash
  openssl speed -evp aes-256-gcm
  ```

- **ping with a large packet size**: Latency test
  ```bash
  ping -s 1400 -c 100 10.0.2.10
  ```

## Conclusion

Hardware acceleration of cryptographic operations in VyOS delivers a significant performance boost for VPN and IPsec connections. Intel QAT provides the largest performance gain (up to 3x), while AES-NI offers solid acceleration in virtualized environments and on processors without dedicated cryptographic modules.

### Key takeaways

1. **Intel QAT** - the best choice for high-load VPN gateways on bare-metal servers
2. **AES-NI** - the optimal solution for virtualized environments (Yandex Cloud, VK Cloud VMs)
3. **AES-GCM algorithms** deliver the best performance with hardware acceleration
4. **Regular monitoring** of QAT is necessary to ensure stable operation
5. **Fallback software cryptography** keeps the system operational if QAT fails

### Recommended configuration

**For bare-metal with QAT (VK Cloud, dedicated):**
- Enable `set system acceleration qat`
- Use AES-256-GCM with SHA256
- Set up QAT monitoring
- Update the firmware regularly

**For virtualized environments (Yandex Cloud):**
- Use AES-256-GCM (AES-NI is used automatically)
- Choose VMs with modern processors
- Consider ChaCha20-Poly1305 for better software performance

**For hybrid scenarios:**
- Configure both algorithms in priority order
- Provide a fallback to software cryptography
- Use identical algorithms on both ends of the tunnel

By following the recommendations in this guide, you can make the most of hardware acceleration in VyOS and ensure high performance for your network infrastructure.