RSA Keys

RSA Keys - RSA Authentication for VPN

Overview of RSA authentication for VPN

RSA authentication is a cryptographic method for verifying the identity of IPsec VPN tunnels in VyOS. Unlike Pre-Shared Key (PSK) authentication, RSA uses asymmetric cryptography with a pair of public and private keys, providing greater security and flexibility when configuring VPN connections.

Key advantages of RSA authentication

Security:

The private key is never transmitted over the network
Eliminates the risk of compromising a single shared key (PSK)
Keys cannot be recovered by brute-force attack
Support for cryptographically strong keys of 2048-4096 bits

Scalability:

Simplified key management in large networks
No configuration changes required when adding new nodes (Hub-and-Spoke)
Centralized management of public keys
Easy key rotation without changing the entire infrastructure

Flexibility:

Support for dynamic IP addresses (Dynamic IP)
Ability to use domain names for identification
Compatibility with various topologies (Site-to-Site, Hub-and-Spoke, Full-Mesh)
Integration with PKI infrastructure

How RSA authentication works

RSA authentication in IPsec VPN is based on the following principles:

Key pair generation: Each VPN node generates a unique pair of keys (public and private)
Public key exchange: Public keys are securely exchanged between nodes
IKE Phase 1: When the tunnel is established, the peers mutually authenticate using RSA signatures
Identity verification: Each side verifies the identity of the remote node using its public key
Tunnel establishment: After successful authentication, a secure IPsec tunnel is established

Architectural use-case scenarios

Site-to-Site VPN:

Connecting two offices or data centers
Static or dynamic IP addresses
Mutual authentication of both ends of the tunnel

Hub-and-Spoke VPN:

Central office (Hub) with multiple branch offices (Spokes)
The Hub has a static IP, while Spokes may have dynamic IPs
Centralized key management on the Hub

Full-Mesh VPN:

Multiple nodes, each connected to every other
Requires public key exchange between all nodes
Optimal traffic routing without transit nodes

Generating an RSA key pair

Basic key generation

VyOS uses a built-in PKI (Public Key Infrastructure) subsystem for key management. An RSA key pair is generated with the following command:

generate pki key-pair install <key-pair-name>

Generation parameters:

<key-pair-name> - a unique name for the key pair used to identify it in the configuration
By default, a 2048-bit RSA key is generated
The system will prompt you to set a passphrase to encrypt the private key (optional)

Key generation example:

vyos@router1:~$ generate pki key-pair install router1-key
Enter private key passphrase:
Retype private key passphrase:
Private key stored as 'router1-key'

Generating keys with advanced parameters

Choosing the key length:

VyOS supports various RSA key lengths. To generate a key of a specific length, use the following command:

generate pki key-pair type rsa length <bit-length> install <key-pair-name>

Recommended key lengths:

2048 bits: Baseline security level, fast processing
3072 bits: Enhanced security for corporate networks
4096 bits: Maximum security for critical systems

Example of generating a 4096-bit key:

vyos@router1:~$ generate pki key-pair type rsa length 4096 install router1-key-4096
Enter private key passphrase:
Retype private key passphrase:
Private key stored as 'router1-key-4096'

Generating a key without encrypting the private key

For automated systems or test environments, you can generate a key without a passphrase:

generate pki key-pair install <key-pair-name> no-password

Important: Unencrypted private keys pose a security risk. Use this option only in isolated test environments.

Viewing generated keys

Viewing the public key:

show pki key-pair <key-pair-name> public

Example output:

vyos@router1:~$ show pki key-pair router1-key public
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw1qHLMXLMbz0Z0X+YoXr
+vRkZJmBYRf3UqL6QgPPExRsG5gN7j8FqXqVLxFZZQ9PVnQ4/BgL1JGsZWx2OcDr
... (remainder of the key) ...
-----END PUBLIC KEY-----

Viewing the private key:

show pki key-pair <key-pair-name> private

Warning: The private key must be kept secret. Never share the private key with third parties.

Saving keys to files

To exchange public keys or create backups, you can export keys to files:

show pki key-pair router1-key public | save /config/router1-public.key
show pki key-pair router1-key private | save /config/router1-private.key

Exchanging public keys between routers

The key exchange process

To establish an RSA-authenticated VPN tunnel, you must exchange public keys between the routers:

Export the public key from Router1:

vyos@router1:~$ show pki key-pair router1-key public
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw1qHLMXLMbz0Z0X+YoXr
+vRkZJmBYRf3UqL6QgPPExRsG5gN7j8FqXqVLxFZZQ9PVnQ4/BgL1JGsZWx2OcDr
K4Hd8XYzLmH5cN3vQ4FGBfQWx2yL8oPxRnE9K5vFmZqL8rN4TpWx6Q8vL2nR5F7K
... (full public key) ...
-----END PUBLIC KEY-----

Import the public key on Router2:

Copy the entire public key text and import it on the second router:

vyos@router2# set pki key-pair router1-remote public key '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw1qHLMXLMbz0Z0X+YoXr
... (the entire public key on a single line or line by line) ...
-----END PUBLIC KEY-----'

Repeat the process for Router2:

Likewise, export the Router2 public key and import it on Router1.

Alternative method: Import from a file

On Router1 (export):

vyos@router1:~$ show pki key-pair router1-key public | save /config/router1-public.key

Transferring the file:

Copy the file /config/router1-public.key to Router2 by any secure method (SCP, USB, secure channel).

On Router2 (import):

vyos@router2# loadkey pki key-pair router1-remote public /tmp/router1-public.key

Or manually through the configuration:

vyos@router2# set pki key-pair router1-remote
vyos@router2# loadkey pki key-pair router1-remote public /tmp/router1-public.key
vyos@router2# commit

Verifying imported keys

Viewing all key pairs:

show pki key-pair

Output:

Key Pair Name    Type    Comment
---------------  ------  ---------
router1-key      rsa     Local key
router2-remote   rsa     Remote peer key

Viewing a specific public key:

show pki key-pair router2-remote public

Managing public and private keys

How keys are stored in VyOS

Keys in VyOS are stored in the configuration under the pki node:

pki {
    key-pair router1-key {
        private {
            key "LS0tLS1CRUdJTiBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0K..."
        }
        public {
            key "LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5C..."
        }
    }
    key-pair router2-remote {
        public {
            key "LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5C..."
        }
    }
}

Important:

Local key pairs contain both the public and private parts
Remote keys contain only the public part
Keys are stored in Base64 encoding

Adding a remote node’s public key

Via the command line:

configure
set pki key-pair <remote-key-name> public key '<public key>'
commit
save

Example:

vyos@router1# set pki key-pair yc-datacenter-remote public key '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2pL5Hx7vN4Zm...
-----END PUBLIC KEY-----'
vyos@router1# commit
vyos@router1# save

Deleting a key pair

Deleting the entire key pair:

configure
delete pki key-pair <key-pair-name>
commit
save

Deleting only the public key:

configure
delete pki key-pair <key-pair-name> public
commit
save

Warning: Before deleting, make sure the key is not being used by active VPN tunnels.

Key rotation

Regular key rotation improves security. The rotation process:

Generate a new key pair:

generate pki key-pair install router1-key-new

Exchange the new public keys:

Export and import the new public keys on the remote nodes.

Update the VPN configuration:

configure
set vpn ipsec authentication pki local-key router1-key-new
set vpn ipsec authentication pki rsa remote-key router2-key-new
commit
save

Delete the old keys:

After verifying that everything works, delete the old keys:

configure
delete pki key-pair router1-key
commit
save

Backing up keys

Exporting the configuration with keys:

show configuration commands | grep pki | save /config/pki-backup.txt

Full backup:

save /config/config.boot.backup

Restore:

load /config/config.boot.backup
commit

Configuring an IPsec VPN with RSA authentication

Basic configuration structure

An IPsec VPN with RSA authentication in VyOS is configured in the following main sections:

IKE Group - IKE Phase 1 parameters (authentication and key exchange)
ESP Group - IPsec Phase 2 parameters (data encryption)
VPN Interface - definition of the VPN interface and tunnel parameters
Authentication - RSA authentication settings
Connection - connection parameters and operating modes

Creating an IKE Group

The IKE Group defines the parameters for IKE Phase 1 (ISAKMP):

configure

set vpn ipsec ike-group IKE-RSA proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-RSA proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-RSA proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-RSA lifetime '28800'
set vpn ipsec ike-group IKE-RSA dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-RSA dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-RSA dead-peer-detection timeout '120'

commit

Parameters:

dh-group: Diffie-Hellman group (2, 5, 14, 15, 16, 19, 20)
- Group 14: 2048-bit MODP (recommended minimum)
- Group 19: 256-bit ECP (modern standard)
- Group 20: 384-bit ECP (enhanced security)
encryption: Encryption algorithm (aes128, aes192, aes256, aes128gcm16, aes256gcm16)
hash: Hashing algorithm (sha1, sha256, sha384, sha512)
lifetime: IKE SA lifetime in seconds (default 28800 = 8 hours)
dead-peer-detection: Detection of an unreachable node

Creating an ESP Group

The ESP Group defines the parameters for IPsec Phase 2:

set vpn ipsec esp-group ESP-RSA proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-RSA proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-RSA lifetime '3600'
set vpn ipsec esp-group ESP-RSA pfs 'dh-group14'

commit

Parameters:

encryption: Data encryption algorithm
hash: HMAC algorithm for data integrity
lifetime: IPsec SA lifetime (default 3600 = 1 hour)
pfs (Perfect Forward Secrecy): Generation of new keys on each SA rekey

Configuring a Site-to-Site VPN with RSA

Scenario: Two offices with static IP addresses

Topology:

Router1 (Office A): 203.0.113.10, LAN 10.10.1.0/24
Router2 (Office B): 198.51.100.20, LAN 10.10.2.0/24

Router1 configuration:

configure

# IKE and ESP groups (as described above)
set vpn ipsec ike-group IKE-RSA proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-RSA proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-RSA proposal 1 hash 'sha256'

set vpn ipsec esp-group ESP-RSA proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-RSA proposal 1 hash 'sha256'

# Site-to-site connection setup
set vpn ipsec site-to-site peer office-b authentication mode 'rsa'
set vpn ipsec site-to-site peer office-b authentication rsa local-key 'router1-key'
set vpn ipsec site-to-site peer office-b authentication rsa remote-key 'router2-remote'
set vpn ipsec site-to-site peer office-b authentication local-id '203.0.113.10'
set vpn ipsec site-to-site peer office-b authentication remote-id '198.51.100.20'

set vpn ipsec site-to-site peer office-b connection-type 'initiate'
set vpn ipsec site-to-site peer office-b ike-group 'IKE-RSA'
set vpn ipsec site-to-site peer office-b local-address '203.0.113.10'
set vpn ipsec site-to-site peer office-b remote-address '198.51.100.20'

set vpn ipsec site-to-site peer office-b tunnel 1 esp-group 'ESP-RSA'
set vpn ipsec site-to-site peer office-b tunnel 1 local prefix '10.10.1.0/24'
set vpn ipsec site-to-site peer office-b tunnel 1 remote prefix '10.10.2.0/24'

commit
save

Router2 configuration:

configure

# IKE and ESP groups (identical to Router1)
set vpn ipsec ike-group IKE-RSA proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-RSA proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-RSA proposal 1 hash 'sha256'

set vpn ipsec esp-group ESP-RSA proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-RSA proposal 1 hash 'sha256'

# Site-to-site connection setup
set vpn ipsec site-to-site peer office-a authentication mode 'rsa'
set vpn ipsec site-to-site peer office-a authentication rsa local-key 'router2-key'
set vpn ipsec site-to-site peer office-a authentication rsa remote-key 'router1-remote'
set vpn ipsec site-to-site peer office-a authentication local-id '198.51.100.20'
set vpn ipsec site-to-site peer office-a authentication remote-id '203.0.113.10'

set vpn ipsec site-to-site peer office-a connection-type 'respond'
set vpn ipsec site-to-site peer office-a ike-group 'IKE-RSA'
set vpn ipsec site-to-site peer office-a local-address '198.51.100.20'
set vpn ipsec site-to-site peer office-a remote-address '203.0.113.10'

set vpn ipsec site-to-site peer office-a tunnel 1 esp-group 'ESP-RSA'
set vpn ipsec site-to-site peer office-a tunnel 1 local prefix '10.10.2.0/24'
set vpn ipsec site-to-site peer office-a tunnel 1 remote prefix '10.10.1.0/24'

commit
save

Key parameters:

authentication mode 'rsa' - use RSA authentication
local-key - name of the local key pair
remote-key - name of the remote node’s public key
local-id and remote-id - identifiers for IKE (IP addresses or FQDNs)
connection-type - ‘initiate’ (initiator) or ‘respond’ (responder)

Support for dynamic IP addresses

Configuration with local-address “any”

VyOS supports dynamic IP addresses for VPN tunnels, which is especially useful for connections from offices that receive dynamic IPs from their provider.

Scenario: Router1 has a dynamic IP, Router2 has a static IP

Router1 configuration (dynamic IP):

configure

set vpn ipsec ike-group IKE-DYNAMIC proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-DYNAMIC proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-DYNAMIC proposal 1 hash 'sha256'

set vpn ipsec esp-group ESP-DYNAMIC proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-DYNAMIC proposal 1 hash 'sha256'

set vpn ipsec site-to-site peer central-office authentication mode 'rsa'
set vpn ipsec site-to-site peer central-office authentication rsa local-key 'branch-key'
set vpn ipsec site-to-site peer central-office authentication rsa remote-key 'central-remote'
set vpn ipsec site-to-site peer central-office authentication local-id 'branch-office@company.local'
set vpn ipsec site-to-site peer central-office authentication remote-id '203.0.113.100'

set vpn ipsec site-to-site peer central-office connection-type 'initiate'
set vpn ipsec site-to-site peer central-office ike-group 'IKE-DYNAMIC'
set vpn ipsec site-to-site peer central-office local-address 'any'
set vpn ipsec site-to-site peer central-office remote-address '203.0.113.100'

set vpn ipsec site-to-site peer central-office tunnel 1 esp-group 'ESP-DYNAMIC'
set vpn ipsec site-to-site peer central-office tunnel 1 local prefix '10.20.1.0/24'
set vpn ipsec site-to-site peer central-office tunnel 1 remote prefix '10.10.0.0/16'

commit
save

Router2 configuration (static IP - central office):

configure

set vpn ipsec ike-group IKE-DYNAMIC proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-DYNAMIC proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-DYNAMIC proposal 1 hash 'sha256'

set vpn ipsec esp-group ESP-DYNAMIC proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-DYNAMIC proposal 1 hash 'sha256'

set vpn ipsec site-to-site peer branch-office authentication mode 'rsa'
set vpn ipsec site-to-site peer branch-office authentication rsa local-key 'central-key'
set vpn ipsec site-to-site peer branch-office authentication rsa remote-key 'branch-remote'
set vpn ipsec site-to-site peer branch-office authentication local-id '203.0.113.100'
set vpn ipsec site-to-site peer branch-office authentication remote-id 'branch-office@company.local'

set vpn ipsec site-to-site peer branch-office connection-type 'respond'
set vpn ipsec site-to-site peer branch-office ike-group 'IKE-DYNAMIC'
set vpn ipsec site-to-site peer branch-office local-address '203.0.113.100'
set vpn ipsec site-to-site peer branch-office remote-address 'any'

set vpn ipsec site-to-site peer branch-office tunnel 1 esp-group 'ESP-DYNAMIC'
set vpn ipsec site-to-site peer branch-office tunnel 1 local prefix '10.10.0.0/16'
set vpn ipsec site-to-site peer branch-office tunnel 1 remote prefix '10.20.1.0/24'

commit
save

Key characteristics:

local-address 'any' - use any available IP address
remote-address 'any' - accept connections from any IP address
local-id and remote-id use an FQDN or email-like identifier for dynamic IPs
connection-type 'initiate' on the side with the dynamic IP
connection-type 'respond' on the side with the static IP

Using an FQDN as the remote-id

For dynamic IP addresses, it is recommended to use an FQDN or email-like identifier:

set vpn ipsec site-to-site peer remote-office authentication local-id 'vyos.headquarters.example.com'
set vpn ipsec site-to-site peer remote-office authentication remote-id 'vyos.branch.example.com'

Advantages of an FQDN:

Identification is independent of the IP address
Support for DynDNS with dynamic IPs
Simplified management in large networks
Human-readable identifiers in logs

Scenario with both nodes on dynamic IPs

If both nodes have dynamic IP addresses, you need a third node with a static IP to initiate the connection, or use DynDNS:

# On both nodes
set vpn ipsec site-to-site peer remote-site local-address 'any'
set vpn ipsec site-to-site peer remote-site remote-address '<dyndns-hostname>'
set vpn ipsec site-to-site peer remote-site authentication local-id '<local-fqdn>'
set vpn ipsec site-to-site peer remote-site authentication remote-id '<remote-fqdn>'

Configuring remote-id and local-id

Purpose of the IKE identifiers

IKE identifiers (local-id and remote-id) are used to:

Identify nodes when establishing an IPsec tunnel
Match public keys to remote nodes
Validate connections from the correct sources
Support dynamic IPs through immutable identifiers

Identifier types

IP address:

set vpn ipsec site-to-site peer remote-site authentication local-id '203.0.113.10'
set vpn ipsec site-to-site peer remote-site authentication remote-id '198.51.100.20'

Used for static IP addresses; simple identification.

FQDN (Fully Qualified Domain Name):

set vpn ipsec site-to-site peer remote-site authentication local-id 'router1.example.com'
set vpn ipsec site-to-site peer remote-site authentication remote-id 'router2.example.com'

Recommended for dynamic IPs, with DNS support.

Email-like identifier:

set vpn ipsec site-to-site peer remote-site authentication local-id 'office-a@company.local'
set vpn ipsec site-to-site peer remote-site authentication remote-id 'office-b@company.local'

Convenient for management in large networks; does not require DNS.

Distinguished Name (DN):

set vpn ipsec site-to-site peer remote-site authentication local-id 'C=RU, O=Company, CN=router1'
set vpn ipsec site-to-site peer remote-site authentication remote-id 'C=RU, O=Company, CN=router2'

Used when integrating with a PKI/CA infrastructure.

Rules for matching identifiers

Important:

The local-id on Router1 must match the remote-id on Router2
The remote-id on Router1 must match the local-id on Router2

Example of a correct configuration:

# Router1
set vpn ipsec site-to-site peer peer2 authentication local-id 'router1.company.com'
set vpn ipsec site-to-site peer peer2 authentication remote-id 'router2.company.com'

# Router2
set vpn ipsec site-to-site peer peer1 authentication local-id 'router2.company.com'
set vpn ipsec site-to-site peer peer1 authentication remote-id 'router1.company.com'

Identifiers in a Hub-and-Spoke topology

Hub (central node):

# A separate peer with a unique remote-id for each Spoke
set vpn ipsec site-to-site peer spoke1 authentication local-id 'hub.company.com'
set vpn ipsec site-to-site peer spoke1 authentication remote-id 'spoke1.company.com'

set vpn ipsec site-to-site peer spoke2 authentication local-id 'hub.company.com'
set vpn ipsec site-to-site peer spoke2 authentication remote-id 'spoke2.company.com'

Spoke (branch office):

set vpn ipsec site-to-site peer hub authentication local-id 'spoke1.company.com'
set vpn ipsec site-to-site peer hub authentication remote-id 'hub.company.com'

Integration with ESP and IKE groups

Choosing cryptographic algorithms

Recommended combinations for various scenarios:

Maximum security (government organizations, financial sector):

# IKE Group
set vpn ipsec ike-group IKE-HIGH-SEC proposal 1 dh-group '20'
set vpn ipsec ike-group IKE-HIGH-SEC proposal 1 encryption 'aes256gcm16'
set vpn ipsec ike-group IKE-HIGH-SEC proposal 1 hash 'sha512'
set vpn ipsec ike-group IKE-HIGH-SEC lifetime '14400'

# ESP Group
set vpn ipsec esp-group ESP-HIGH-SEC proposal 1 encryption 'aes256gcm16'
set vpn ipsec esp-group ESP-HIGH-SEC proposal 1 hash 'sha512'
set vpn ipsec esp-group ESP-HIGH-SEC lifetime '3600'
set vpn ipsec esp-group ESP-HIGH-SEC pfs 'dh-group20'

Balance of security and performance (corporate networks):

# IKE Group
set vpn ipsec ike-group IKE-BALANCED proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-BALANCED proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-BALANCED proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-BALANCED lifetime '28800'

# ESP Group
set vpn ipsec esp-group ESP-BALANCED proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-BALANCED proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-BALANCED lifetime '3600'
set vpn ipsec esp-group ESP-BALANCED pfs 'dh-group14'

Maximum performance (test environments, high throughput):

# IKE Group
set vpn ipsec ike-group IKE-PERFORMANCE proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-PERFORMANCE proposal 1 encryption 'aes128gcm16'
set vpn ipsec ike-group IKE-PERFORMANCE proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-PERFORMANCE lifetime '28800'

# ESP Group
set vpn ipsec esp-group ESP-PERFORMANCE proposal 1 encryption 'aes128gcm16'
set vpn ipsec esp-group ESP-PERFORMANCE proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-PERFORMANCE lifetime '3600'
set vpn ipsec esp-group ESP-PERFORMANCE pfs 'dh-group14'

Multiple proposals for compatibility

For compatibility with various VPN concentrators, you can configure multiple proposals:

set vpn ipsec ike-group IKE-MULTI proposal 1 dh-group '20'
set vpn ipsec ike-group IKE-MULTI proposal 1 encryption 'aes256gcm16'
set vpn ipsec ike-group IKE-MULTI proposal 1 hash 'sha512'

set vpn ipsec ike-group IKE-MULTI proposal 2 dh-group '14'
set vpn ipsec ike-group IKE-MULTI proposal 2 encryption 'aes256'
set vpn ipsec ike-group IKE-MULTI proposal 2 hash 'sha256'

set vpn ipsec ike-group IKE-MULTI proposal 3 dh-group '14'
set vpn ipsec ike-group IKE-MULTI proposal 3 encryption 'aes128'
set vpn ipsec ike-group IKE-MULTI proposal 3 hash 'sha256'

VyOS will try to negotiate the parameters in priority order (proposal 1, then 2, then 3).

Dead Peer Detection (DPD)

DPD provides automatic tunnel recovery when connectivity is lost:

set vpn ipsec ike-group IKE-GROUP dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection timeout '120'

Parameters:

action: ‘restart’ (recreate the tunnel), ‘clear’ (delete the SA), ‘hold’ (leave unchanged)
interval: Interval for sending DPD messages (seconds)
timeout: Timeout for waiting for a response (seconds)

Recommendations:

For stable links: interval=60, timeout=300
For unstable links: interval=30, timeout=120
For critical connections: interval=10, timeout=30

Configuring lifetime

IKE lifetime (ISAKMP SA):

set vpn ipsec ike-group IKE-GROUP lifetime '28800'  # 8 hours

A longer lifetime reduces CPU load during rekeying but increases the window of vulnerability.

ESP lifetime (IPsec SA):

set vpn ipsec esp-group ESP-GROUP lifetime '3600'  # 1 hour

A shorter lifetime improves security through more frequent key changes.

Recommended values:

IKE: 28800 (8 hours) - 86400 (24 hours)
ESP: 3600 (1 hour) - 14400 (4 hours)

Example: Site-to-Site VPN between Yandex Cloud and on-premises

Solution architecture

Scenario:

Yandex Cloud VPC: VyOS router in the cloud with public IP 51.250.10.50, internal network 10.128.0.0/16
On-Premises: VyOS router in the office with public IP 203.0.113.25, internal network 192.168.10.0/24
Requirements: Secure connection between the cloud and the office, RSA authentication, support for a dynamic IP on the office side

Network topology

┌─────────────────────────────────────────────────────────────┐
│ Yandex Cloud VPC (ru-central1-a)                            │
│                                                              │
│  ┌──────────────────────────────────┐                       │
│  │ VyOS Router (yc-router)          │                       │
│  │ Public IP: 51.250.10.50          │                       │
│  │ Private IP: 10.128.0.10          │                       │
│  │ eth0: 51.250.10.50 (WAN)         │                       │
│  │ eth1: 10.128.0.10 (LAN)          │                       │
│  └──────────────┬───────────────────┘                       │
│                 │                                            │
│  ┌──────────────┴───────────────────┐                       │
│  │ Subnet: 10.128.0.0/24            │                       │
│  │ VMs: 10.128.0.11 - 10.128.0.254  │                       │
│  └──────────────────────────────────┘                       │
│                                                              │
└──────────────────────────┬───────────────────────────────────┘
                           │
                           │ Internet
                           │ IPsec VPN Tunnel (RSA Auth)
                           │
┌──────────────────────────┴───────────────────────────────────┐
│ On-Premises Office                                           │
│                                                              │
│  ┌──────────────────────────────────┐                       │
│  │ VyOS Router (office-router)      │                       │
│  │ Public IP: 203.0.113.25 (dynamic)│                       │
│  │ Private IP: 192.168.10.1         │                       │
│  │ eth0: 203.0.113.25 (WAN)         │                       │
│  │ eth1: 192.168.10.1 (LAN)         │                       │
│  └──────────────┬───────────────────┘                       │
│                 │                                            │
│  ┌──────────────┴───────────────────┐                       │
│  │ LAN: 192.168.10.0/24             │                       │
│  │ Hosts: 192.168.10.10 - .254      │                       │
│  └──────────────────────────────────┘                       │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Key generation

On the Yandex Cloud VyOS Router:

vyos@yc-router:~$ configure
vyos@yc-router# generate pki key-pair install yc-cloud-key
Enter private key passphrase: ********
Retype private key passphrase: ********
vyos@yc-router# commit
vyos@yc-router# save

On the On-Premises VyOS Router:

vyos@office-router:~$ configure
vyos@office-router# generate pki key-pair install office-onprem-key
Enter private key passphrase: ********
Retype private key passphrase: ********
vyos@office-router# commit
vyos@office-router# save

Exchanging public keys

Exporting the Yandex Cloud public key:

vyos@yc-router# run show pki key-pair yc-cloud-key public
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1K8Hx5vQ2Lm7pN9wZ...
(copy the entire key)
-----END PUBLIC KEY-----

Importing it on the On-Premises Router:

vyos@office-router# set pki key-pair yc-remote-key public key '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1K8Hx5vQ2Lm7pN9wZ...
-----END PUBLIC KEY-----'
vyos@office-router# commit

Similarly for the office public key:

vyos@office-router# run show pki key-pair office-onprem-key public
# Copy the key and import it on yc-router as office-remote-key

Yandex Cloud Router configuration

configure

# IKE Group
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-YC-OFFICE lifetime '28800'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection timeout '120'

# ESP Group
set vpn ipsec esp-group ESP-YC-OFFICE proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-YC-OFFICE proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-YC-OFFICE lifetime '3600'
set vpn ipsec esp-group ESP-YC-OFFICE pfs 'dh-group14'

# Site-to-Site Peer Configuration
set vpn ipsec site-to-site peer office-branch authentication mode 'rsa'
set vpn ipsec site-to-site peer office-branch authentication rsa local-key 'yc-cloud-key'
set vpn ipsec site-to-site peer office-branch authentication rsa remote-key 'office-remote-key'
set vpn ipsec site-to-site peer office-branch authentication local-id 'yc-router@yandex-cloud.local'
set vpn ipsec site-to-site peer office-branch authentication remote-id 'office-router@company.local'

set vpn ipsec site-to-site peer office-branch connection-type 'respond'
set vpn ipsec site-to-site peer office-branch ike-group 'IKE-YC-OFFICE'
set vpn ipsec site-to-site peer office-branch local-address '51.250.10.50'
set vpn ipsec site-to-site peer office-branch remote-address 'any'

# Tunnel Configuration
set vpn ipsec site-to-site peer office-branch tunnel 1 esp-group 'ESP-YC-OFFICE'
set vpn ipsec site-to-site peer office-branch tunnel 1 local prefix '10.128.0.0/16'
set vpn ipsec site-to-site peer office-branch tunnel 1 remote prefix '192.168.10.0/24'

commit
save

On-Premises Router configuration

configure

# IKE Group (identical to the YC Router)
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-YC-OFFICE proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-YC-OFFICE lifetime '28800'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-YC-OFFICE dead-peer-detection timeout '120'

# ESP Group (identical to the YC Router)
set vpn ipsec esp-group ESP-YC-OFFICE proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-YC-OFFICE proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-YC-OFFICE lifetime '3600'
set vpn ipsec esp-group ESP-YC-OFFICE pfs 'dh-group14'

# Site-to-Site Peer Configuration
set vpn ipsec site-to-site peer yandex-cloud authentication mode 'rsa'
set vpn ipsec site-to-site peer yandex-cloud authentication rsa local-key 'office-onprem-key'
set vpn ipsec site-to-site peer yandex-cloud authentication rsa remote-key 'yc-remote-key'
set vpn ipsec site-to-site peer yandex-cloud authentication local-id 'office-router@company.local'
set vpn ipsec site-to-site peer yandex-cloud authentication remote-id 'yc-router@yandex-cloud.local'

set vpn ipsec site-to-site peer yandex-cloud connection-type 'initiate'
set vpn ipsec site-to-site peer yandex-cloud ike-group 'IKE-YC-OFFICE'
set vpn ipsec site-to-site peer yandex-cloud local-address 'any'
set vpn ipsec site-to-site peer yandex-cloud remote-address '51.250.10.50'

# Tunnel Configuration
set vpn ipsec site-to-site peer yandex-cloud tunnel 1 esp-group 'ESP-YC-OFFICE'
set vpn ipsec site-to-site peer yandex-cloud tunnel 1 local prefix '192.168.10.0/24'
set vpn ipsec site-to-site peer yandex-cloud tunnel 1 remote prefix '10.128.0.0/16'

commit
save

Configuring routing

On the Yandex Cloud Router:

configure

# Static route for the office network over the VPN
set protocols static route 192.168.10.0/24 interface tunnel0

# Or via next-hop (if VTI is used)
# set protocols static route 192.168.10.0/24 next-hop <vti-interface-ip>

commit
save

On the On-Premises Router:

configure

# Static route for the cloud network over the VPN
set protocols static route 10.128.0.0/16 interface tunnel0

commit
save

Configuring NAT (optional)

If NAT is required so that VPN clients can access the internet through Yandex Cloud:

On the Yandex Cloud Router:

configure

# Exclude VPN traffic from NAT
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address '10.128.0.0/16'
set nat source rule 100 destination address '192.168.10.0/24'
set nat source rule 100 exclude

# NAT for internet traffic
set nat source rule 200 outbound-interface 'eth0'
set nat source rule 200 source address '10.128.0.0/16'
set nat source rule 200 translation address 'masquerade'

commit
save

Example: Hub-and-Spoke VPN with RSA authentication (VK Cloud)

Solution architecture

Scenario:

Hub (VK Cloud): Central VyOS router with public IP 89.208.220.50, network 10.0.0.0/16
Spoke 1 (Moscow branch): VyOS router, dynamic IP, network 10.10.1.0/24
Spoke 2 (Saint Petersburg branch): VyOS router, dynamic IP, network 10.10.2.0/24
Spoke 3 (Yekaterinburg branch): VyOS router, static IP 198.51.100.100, network 10.10.3.0/24

Network topology

                          ┌─────────────────────────────┐
                          │ VK Cloud (Moscow Region)    │
                          │                             │
                          │  ┌──────────────────────┐   │
                          │  │ Hub VyOS Router      │   │
                          │  │ 89.208.220.50        │   │
                          │  │ 10.0.0.1 (LAN)       │   │
                          │  └──────────┬───────────┘   │
                          │             │               │
                          │  ┌──────────┴───────────┐   │
                          │  │ VK Cloud Network     │   │
                          │  │ 10.0.0.0/16          │   │
                          │  └──────────────────────┘   │
                          │                             │
                          └──────────────┬──────────────┘
                                         │
                 ┌───────────────────────┼───────────────────────┐
                 │                       │                       │
                 │                       │                       │
     ┌───────────▼───────────┐ ┌────────▼────────────┐ ┌────────▼────────────┐
     │ Spoke 1: Moscow       │ │ Spoke 2: SPb        │ │ Spoke 3: Ekb        │
     │ Dynamic IP            │ │ Dynamic IP          │ │ Static IP           │
     │ 10.10.1.0/24          │ │ 10.10.2.0/24        │ │ 198.51.100.100      │
     │                       │ │                     │ │ 10.10.3.0/24        │
     └───────────────────────┘ └─────────────────────┘ └─────────────────────┘

Generating keys on all nodes

Hub (VK Cloud):

vyos@hub-router:~$ configure
vyos@hub-router# generate pki key-pair install hub-vkcloud-key
vyos@hub-router# commit
vyos@hub-router# save

Spoke 1 (Moscow):

vyos@spoke1-moscow:~$ configure
vyos@spoke1-moscow# generate pki key-pair install spoke1-moscow-key
vyos@spoke1-moscow# commit
vyos@spoke1-moscow# save

Spoke 2 (Saint Petersburg):

vyos@spoke2-spb:~$ configure
vyos@spoke2-spb# generate pki key-pair install spoke2-spb-key
vyos@spoke2-spb# commit
vyos@spoke2-spb# save

Spoke 3 (Yekaterinburg):

vyos@spoke3-ekb:~$ configure
vyos@spoke3-ekb# generate pki key-pair install spoke3-ekb-key
vyos@spoke3-ekb# commit
vyos@spoke3-ekb# save

Exchanging public keys

Importing the Spoke public keys on the Hub:

vyos@hub-router# set pki key-pair spoke1-remote-key public key '<spoke1-public-key>'
vyos@hub-router# set pki key-pair spoke2-remote-key public key '<spoke2-public-key>'
vyos@hub-router# set pki key-pair spoke3-remote-key public key '<spoke3-public-key>'
vyos@hub-router# commit

Importing the Hub public key on each Spoke:

# On Spoke 1
vyos@spoke1-moscow# set pki key-pair hub-remote-key public key '<hub-public-key>'

# On Spoke 2
vyos@spoke2-spb# set pki key-pair hub-remote-key public key '<hub-public-key>'

# On Spoke 3
vyos@spoke3-ekb# set pki key-pair hub-remote-key public key '<hub-public-key>'

Hub Router configuration (VK Cloud)

configure

# Shared IKE and ESP groups for all Spokes
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-HUB lifetime '28800'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '120'

set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-HUB lifetime '3600'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group14'

# Spoke 1 Configuration (Dynamic IP)
set vpn ipsec site-to-site peer spoke1 authentication mode 'rsa'
set vpn ipsec site-to-site peer spoke1 authentication rsa local-key 'hub-vkcloud-key'
set vpn ipsec site-to-site peer spoke1 authentication rsa remote-key 'spoke1-remote-key'
set vpn ipsec site-to-site peer spoke1 authentication local-id 'hub@vkcloud.local'
set vpn ipsec site-to-site peer spoke1 authentication remote-id 'spoke1-moscow@company.local'

set vpn ipsec site-to-site peer spoke1 connection-type 'respond'
set vpn ipsec site-to-site peer spoke1 ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer spoke1 local-address '89.208.220.50'
set vpn ipsec site-to-site peer spoke1 remote-address 'any'

set vpn ipsec site-to-site peer spoke1 tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer spoke1 tunnel 1 local prefix '10.0.0.0/16'
set vpn ipsec site-to-site peer spoke1 tunnel 1 remote prefix '10.10.1.0/24'

# Spoke 2 Configuration (Dynamic IP)
set vpn ipsec site-to-site peer spoke2 authentication mode 'rsa'
set vpn ipsec site-to-site peer spoke2 authentication rsa local-key 'hub-vkcloud-key'
set vpn ipsec site-to-site peer spoke2 authentication rsa remote-key 'spoke2-remote-key'
set vpn ipsec site-to-site peer spoke2 authentication local-id 'hub@vkcloud.local'
set vpn ipsec site-to-site peer spoke2 authentication remote-id 'spoke2-spb@company.local'

set vpn ipsec site-to-site peer spoke2 connection-type 'respond'
set vpn ipsec site-to-site peer spoke2 ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer spoke2 local-address '89.208.220.50'
set vpn ipsec site-to-site peer spoke2 remote-address 'any'

set vpn ipsec site-to-site peer spoke2 tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer spoke2 tunnel 1 local prefix '10.0.0.0/16'
set vpn ipsec site-to-site peer spoke2 tunnel 1 remote prefix '10.10.2.0/24'

# Spoke 3 Configuration (Static IP)
set vpn ipsec site-to-site peer spoke3 authentication mode 'rsa'
set vpn ipsec site-to-site peer spoke3 authentication rsa local-key 'hub-vkcloud-key'
set vpn ipsec site-to-site peer spoke3 authentication rsa remote-key 'spoke3-remote-key'
set vpn ipsec site-to-site peer spoke3 authentication local-id 'hub@vkcloud.local'
set vpn ipsec site-to-site peer spoke3 authentication remote-id 'spoke3-ekb@company.local'

set vpn ipsec site-to-site peer spoke3 connection-type 'initiate'
set vpn ipsec site-to-site peer spoke3 ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer spoke3 local-address '89.208.220.50'
set vpn ipsec site-to-site peer spoke3 remote-address '198.51.100.100'

set vpn ipsec site-to-site peer spoke3 tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer spoke3 tunnel 1 local prefix '10.0.0.0/16'
set vpn ipsec site-to-site peer spoke3 tunnel 1 remote prefix '10.10.3.0/24'

commit
save

Spoke Router configuration

Spoke 1 (Moscow - Dynamic IP):

configure

set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-HUB lifetime '28800'
set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '120'

set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha256'
set vpn ipsec esp-group ESP-HUB lifetime '3600'
set vpn ipsec esp-group ESP-HUB pfs 'dh-group14'

set vpn ipsec site-to-site peer hub authentication mode 'rsa'
set vpn ipsec site-to-site peer hub authentication rsa local-key 'spoke1-moscow-key'
set vpn ipsec site-to-site peer hub authentication rsa remote-key 'hub-remote-key'
set vpn ipsec site-to-site peer hub authentication local-id 'spoke1-moscow@company.local'
set vpn ipsec site-to-site peer hub authentication remote-id 'hub@vkcloud.local'

set vpn ipsec site-to-site peer hub connection-type 'initiate'
set vpn ipsec site-to-site peer hub ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer hub local-address 'any'
set vpn ipsec site-to-site peer hub remote-address '89.208.220.50'

set vpn ipsec site-to-site peer hub tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer hub tunnel 1 local prefix '10.10.1.0/24'
set vpn ipsec site-to-site peer hub tunnel 1 remote prefix '10.0.0.0/16'

commit
save

Spoke 2 (Saint Petersburg - Dynamic IP):

configure

# IKE and ESP groups identical to Spoke 1

set vpn ipsec site-to-site peer hub authentication mode 'rsa'
set vpn ipsec site-to-site peer hub authentication rsa local-key 'spoke2-spb-key'
set vpn ipsec site-to-site peer hub authentication rsa remote-key 'hub-remote-key'
set vpn ipsec site-to-site peer hub authentication local-id 'spoke2-spb@company.local'
set vpn ipsec site-to-site peer hub authentication remote-id 'hub@vkcloud.local'

set vpn ipsec site-to-site peer hub connection-type 'initiate'
set vpn ipsec site-to-site peer hub ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer hub local-address 'any'
set vpn ipsec site-to-site peer hub remote-address '89.208.220.50'

set vpn ipsec site-to-site peer hub tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer hub tunnel 1 local prefix '10.10.2.0/24'
set vpn ipsec site-to-site peer hub tunnel 1 remote prefix '10.0.0.0/16'

commit
save

Spoke 3 (Yekaterinburg - Static IP):

configure

# IKE and ESP groups identical to the other Spokes

set vpn ipsec site-to-site peer hub authentication mode 'rsa'
set vpn ipsec site-to-site peer hub authentication rsa local-key 'spoke3-ekb-key'
set vpn ipsec site-to-site peer hub authentication rsa remote-key 'hub-remote-key'
set vpn ipsec site-to-site peer hub authentication local-id 'spoke3-ekb@company.local'
set vpn ipsec site-to-site peer hub authentication remote-id 'hub@vkcloud.local'

set vpn ipsec site-to-site peer hub connection-type 'respond'
set vpn ipsec site-to-site peer hub ike-group 'IKE-HUB'
set vpn ipsec site-to-site peer hub local-address '198.51.100.100'
set vpn ipsec site-to-site peer hub remote-address '89.208.220.50'

set vpn ipsec site-to-site peer hub tunnel 1 esp-group 'ESP-HUB'
set vpn ipsec site-to-site peer hub tunnel 1 local prefix '10.10.3.0/24'
set vpn ipsec site-to-site peer hub tunnel 1 remote prefix '10.0.0.0/16'

commit
save

Spoke-to-Spoke routing through the Hub

To provide connectivity between Spoke nodes, you must configure routing through the Hub:

On the Hub Router:

configure

# Enable IP forwarding (usually enabled by default)
set system ip forwarding

# Static routes for each Spoke
set protocols static route 10.10.1.0/24 interface tunnel0
set protocols static route 10.10.2.0/24 interface tunnel1
set protocols static route 10.10.3.0/24 interface tunnel2

# Or use dynamic routing (BGP/OSPF)
# OSPF example:
set protocols ospf area 0 network '10.0.0.0/16'
set protocols ospf area 0 network '10.10.1.0/24'
set protocols ospf area 0 network '10.10.2.0/24'
set protocols ospf area 0 network '10.10.3.0/24'

commit
save

On each Spoke Router:

configure

# Default route through the Hub for the other Spoke networks
set protocols static route 10.10.0.0/16 interface tunnel0

# Or specific routes
set protocols static route 10.10.2.0/24 interface tunnel0
set protocols static route 10.10.3.0/24 interface tunnel0

commit
save

Configuration examples for various scenarios

Scenario 1: Site-to-Site with two tunnels (redundancy)

To improve fault tolerance, you can configure two tunnels between the same nodes:

configure

# Primary Tunnel (main link)
set vpn ipsec site-to-site peer remote-site tunnel 1 esp-group 'ESP-PRIMARY'
set vpn ipsec site-to-site peer remote-site tunnel 1 local prefix '10.10.1.0/24'
set vpn ipsec site-to-site peer remote-site tunnel 1 remote prefix '10.10.2.0/24'
set vpn ipsec site-to-site peer remote-site tunnel 1 priority '10'

# Backup Tunnel (backup link)
set vpn ipsec site-to-site peer remote-site-backup authentication mode 'rsa'
set vpn ipsec site-to-site peer remote-site-backup authentication rsa local-key 'local-key'
set vpn ipsec site-to-site peer remote-site-backup authentication rsa remote-key 'remote-key-backup'
set vpn ipsec site-to-site peer remote-site-backup remote-address '<backup-ip>'
set vpn ipsec site-to-site peer remote-site-backup tunnel 1 esp-group 'ESP-BACKUP'
set vpn ipsec site-to-site peer remote-site-backup tunnel 1 local prefix '10.10.1.0/24'
set vpn ipsec site-to-site peer remote-site-backup tunnel 1 remote prefix '10.10.2.0/24'
set vpn ipsec site-to-site peer remote-site-backup tunnel 1 priority '20'

commit
save

Scenario 2: VPN with traffic separation (Split Tunneling)

Configuring multiple tunnels for different subnets:

configure

# Tunnel 1: for the server subnet
set vpn ipsec site-to-site peer remote-site tunnel 1 esp-group 'ESP-SERVERS'
set vpn ipsec site-to-site peer remote-site tunnel 1 local prefix '10.10.1.0/24'
set vpn ipsec site-to-site peer remote-site tunnel 1 remote prefix '10.20.10.0/24'

# Tunnel 2: for workstations
set vpn ipsec site-to-site peer remote-site tunnel 2 esp-group 'ESP-WORKSTATIONS'
set vpn ipsec site-to-site peer remote-site tunnel 2 local prefix '10.10.2.0/24'
set vpn ipsec site-to-site peer remote-site tunnel 2 remote prefix '10.20.20.0/24'

# Tunnel 3: for management systems (with enhanced security)
set vpn ipsec site-to-site peer remote-site tunnel 3 esp-group 'ESP-MGMT'
set vpn ipsec site-to-site peer remote-site tunnel 3 local prefix '10.10.100.0/24'
set vpn ipsec site-to-site peer remote-site tunnel 3 remote prefix '10.20.100.0/24'

commit
save

Scenario 3: VPN with QoS for traffic prioritization

configure

# Defining traffic classes
set traffic-policy shaper VPN-SHAPER bandwidth '100mbit'
set traffic-policy shaper VPN-SHAPER class 10 bandwidth '40%'
set traffic-policy shaper VPN-SHAPER class 10 match VOICE ip dscp 'ef'
set traffic-policy shaper VPN-SHAPER class 20 bandwidth '30%'
set traffic-policy shaper VPN-SHAPER class 20 match VIDEO ip dscp 'af41'
set traffic-policy shaper VPN-SHAPER class 30 bandwidth '30%'
set traffic-policy shaper VPN-SHAPER default bandwidth '20%'

# Applying the policy to the VPN interface
set interfaces tunnel tun0 traffic-policy out 'VPN-SHAPER'

commit
save

Scenario 4: Integration with VRRP for high availability

Two VyOS routers in HA mode with a shared virtual IP for the VPN:

Router 1 (Master):

configure

# VRRP Configuration
set high-availability vrrp group VPN-HA vrid '10'
set high-availability vrrp group VPN-HA interface 'eth0'
set high-availability vrrp group VPN-HA virtual-address '203.0.113.100/24'
set high-availability vrrp group VPN-HA priority '200'

# VPN Configuration using VRRP virtual IP
set vpn ipsec site-to-site peer remote-site local-address '203.0.113.100'

commit
save

Router 2 (Backup):

configure

# VRRP Configuration
set high-availability vrrp group VPN-HA vrid '10'
set high-availability vrrp group VPN-HA interface 'eth0'
set high-availability vrrp group VPN-HA virtual-address '203.0.113.100/24'
set high-availability vrrp group VPN-HA priority '100'

# Identical VPN Configuration
set vpn ipsec site-to-site peer remote-site local-address '203.0.113.100'

commit
save

Verification and diagnostic commands

Checking IPsec status

Overall IPsec status:

show vpn ipsec sa

Example output:

Connection                State    Uptime    Bytes In/Out
------------------------  -------  --------  --------------
office-branch             up       00:15:23  125K/98K
  Tunnel 1                up       00:15:20  125K/98K

Detailed connection information:

show vpn ipsec sa detail

The output includes:

IKE version and algorithms
Tunnel states
SA establishment and rekey times
Packet and byte counters
Remote and Local ID

Checking IKE SA (Security Associations)

show vpn ike sa

Example output:

Peer ID / IP                            Local ID / IP
--------------------------------------  ----------------------------
office-router@company.local             yc-router@yandex-cloud.local
198.51.100.20                           51.250.10.50

    State  Encrypt  Hash    D-H Group  NAT-T  A-Time  L-Time
    -----  -------  ------  ---------  -----  ------  ------
    up     aes256   sha256  14         no     2341    28800

Viewing the VPN configuration

Full VPN configuration:

show configuration commands | grep vpn

A specific peer:

show configuration commands | grep "vpn ipsec site-to-site peer office-branch"

Checking keys

List of all key pairs:

show pki key-pair

Output:

Key Pair Name         Type    Comment
--------------------  ------  ---------
yc-cloud-key          rsa     Local key
office-remote-key     rsa     Remote peer

Viewing the public key:

show pki key-pair yc-cloud-key public

Monitoring traffic

Tunnel statistics:

show vpn ipsec sa statistics

Viewing counters:

show interfaces tunnel

Debugging and logging

Enabling detailed IPsec logging:

configure
set vpn ipsec logging log-level '2'
commit

Logging levels:

0: Critical errors
1: Errors
2: Warnings and important events (recommended)
3: Informational messages
4: Debug information (for diagnostics only)

Viewing IPsec logs:

show log vpn ipsec

Real-time monitoring:

monitor log vpn ipsec

System logs:

show log | match ipsec
show log | match charon

Testing connectivity

Ping through the VPN tunnel:

ping 10.128.0.11 source-address 192.168.10.1 count 5

Traceroute through the tunnel:

traceroute 10.128.0.11 source-address 192.168.10.1

Checking routing:

show ip route
show ip route 10.128.0.0/16

Troubleshooting

Authentication errors

Problem: The tunnel does not establish; authentication errors appear in the logs

Symptoms:

IKE authentication credentials are unacceptable
received NO_PROPOSAL_CHOSEN error notify

Solutions:

Verify that the keys match:

# On Router1
show pki key-pair remote-router-key public

# Compare with the local key on Router2
show pki key-pair local-router-key public

The Router1 public key must exactly match the key imported on Router2.

Check local-id and remote-id:

# On Router1
show configuration commands | grep "authentication.*-id"

# On Router2
show configuration commands | grep "authentication.*-id"

Make sure that the local-id on Router1 = the remote-id on Router2, and vice versa.

Check the authentication mode:

show configuration commands | grep "authentication mode"

mode 'rsa' must be set on both sides.

Key Mismatch

Problem: The keys do not match or are corrupted

Diagnostics:

# Export and compare the keys
show pki key-pair remote-key public | save /tmp/remote-key.txt

# Compare with the original key from the remote node

Solution:

Delete the corrupted key:

configure
delete pki key-pair remote-key
commit

Import the key again:

set pki key-pair remote-key public key '<correct-public-key>'
commit
save

Restart IPsec:

restart vpn

Proposal issues (encryption algorithms)

Problem: NO_PROPOSAL_CHOSEN error

Symptoms:

no matching proposal found, sending NO_PROPOSAL_CHOSEN

Solution:

Make sure the IKE and ESP groups are identical on both sides:

# Router1
show configuration commands | grep "ike-group IKE-GROUP"
show configuration commands | grep "esp-group ESP-GROUP"

# Router2 - the configuration must be identical

Add multiple proposals for compatibility:

configure
set vpn ipsec ike-group IKE-GROUP proposal 2 dh-group '14'
set vpn ipsec ike-group IKE-GROUP proposal 2 encryption 'aes128'
set vpn ipsec ike-group IKE-GROUP proposal 2 hash 'sha256'
commit

The tunnel establishes, but traffic does not pass

Problem: The IPsec SA is active, but ping does not work

Diagnostics:

Check the tunnel prefixes:

show configuration commands | grep "tunnel.*prefix"

Make sure the local and remote prefixes are configured correctly and mirrored on both sides.

Check firewall rules:

show firewall

Make sure that traffic from the VPN networks is allowed.

Check NAT:

show nat source rules

VPN traffic must be excluded from NAT:

configure
set nat source rule 10 outbound-interface 'eth0'
set nat source rule 10 source address '10.10.1.0/24'
set nat source rule 10 destination address '10.10.2.0/24'
set nat source rule 10 exclude
commit

Check routing:

show ip route

There must be routes to the remote VPN networks through the tunnel interface.

Dead Peer Detection is not working

Problem: The tunnel does not recover after a disconnection

Solution:

configure
set vpn ipsec ike-group IKE-GROUP dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection timeout '120'
commit
save

For more aggressive DPD:

set vpn ipsec ike-group IKE-GROUP dead-peer-detection interval '10'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection timeout '30'

Issues with dynamic IPs

Problem: The tunnel does not establish after an IP address change

Solution:

On the side with the dynamic IP:

configure
set vpn ipsec site-to-site peer remote-site local-address 'any'
set vpn ipsec site-to-site peer remote-site connection-type 'initiate'
commit

On the side with the static IP:

configure
set vpn ipsec site-to-site peer remote-site remote-address 'any'
set vpn ipsec site-to-site peer remote-site connection-type 'respond'
commit

Force reconnection:

reset vpn ipsec-peer <peer-name>

Logs for diagnostics

Enabling detailed logging:

configure
set vpn ipsec logging log-level '3'
set vpn ipsec logging log-modes 'ike'
set vpn ipsec logging log-modes 'esp'
commit

Viewing detailed logs:

show log vpn ipsec | tail 100
monitor log vpn ipsec

charon system logs (IKE daemon):

show log | match charon

Full reset and reconnection

Resetting a specific peer:

reset vpn ipsec-peer office-branch

Full IPsec restart:

restart vpn

Verification after restart:

show vpn ipsec sa
show vpn ike sa

Best practices and recommendations

Key management

1. Key length:

At least 2048 bits for corporate networks
3072 bits for enhanced security
4096 bits for critical government systems

# Recommended generation
generate pki key-pair type rsa length 3072 install <key-name>

2. Regular key rotation:

Rotate keys every 12-24 months
Rotate immediately if compromise is suspected
Document the rotation process

Rotation process:

# 1. Generate new keys
generate pki key-pair install new-key

# 2. Exchange public keys
# 3. Update the VPN configuration
configure
set vpn ipsec site-to-site peer remote authentication rsa local-key 'new-key'
commit

# 4. After verification, delete the old keys
delete pki key-pair old-key
commit
save

3. Protecting private keys:

Always use passphrase protection for private keys
Restrict access to the VyOS configuration
Regularly back up keys to secure storage
Never transmit private keys over unsecured channels

4. Centralized management:

For large networks, use a key management system:

# Script for automated key exchange
#!/bin/bash
ROUTERS="router1 router2 router3"
for router in $ROUTERS; do
    ssh vyos@$router "show pki key-pair local-key public" > ${router}-public.key
done

Choosing cryptographic parameters

1. Diffie-Hellman groups:

Group 14 (2048-bit MODP): Minimum recommended
Group 19 (256-bit ECP): Optimal balance
Group 20 (384-bit ECP): Maximum security

# Recommended configuration
set vpn ipsec ike-group IKE-SECURE proposal 1 dh-group '19'

2. Encryption algorithms:

AES-256: Standard for corporate networks
AES-256-GCM: Modern AEAD mode with authentication
AES-128-GCM: For high-performance systems

# Modern secure configuration
set vpn ipsec ike-group IKE-MODERN proposal 1 encryption 'aes256gcm16'
set vpn ipsec esp-group ESP-MODERN proposal 1 encryption 'aes256gcm16'

3. Hashing functions:

SHA-256: Standard for most use cases
SHA-384/512: For systems with heightened requirements

4. Perfect Forward Secrecy (PFS):

Always enable PFS to protect against compromise of long-term keys:

set vpn ipsec esp-group ESP-GROUP pfs 'dh-group19'

Configuring lifetime

Recommendations for SA lifetime:

IKE Lifetime:

set vpn ipsec ike-group IKE-GROUP lifetime '28800'  # 8 hours

Do not set the lifetime too short (< 1 hour) - it increases CPU load
Optimal range: 8-24 hours
For critical systems: 4-8 hours

ESP Lifetime:

set vpn ipsec esp-group ESP-GROUP lifetime '3600'  # 1 hour

Optimal range: 1-4 hours
For heavily loaded links: 30-60 minutes
Take traffic volume and performance into account

Monitoring and logging

1. Configuring logging levels:

configure
# For production - level 2 (warnings)
set vpn ipsec logging log-level '2'

# For debugging - level 3-4
set vpn ipsec logging log-level '3'
set vpn ipsec logging log-modes 'ike'
set vpn ipsec logging log-modes 'esp'
set vpn ipsec logging log-modes 'cfg'

commit
save

2. Automated monitoring:

Create a script to check the VPN status:

#!/bin/bash
# /config/scripts/vpn-monitor.sh

PEERS="office-branch datacenter-vpn"
LOG_FILE="/var/log/vpn-monitor.log"

for peer in $PEERS; do
    STATUS=$(vtysh -c "show vpn ipsec sa | grep $peer | grep -c 'up'")
    if [ $STATUS -eq 0 ]; then
        echo "$(date): WARNING - VPN peer $peer is DOWN" >> $LOG_FILE
        # Attempt recovery
        vtysh -c "reset vpn ipsec-peer $peer"
    fi
done

Running it via cron:

configure
set system task-scheduler task vpn-monitor interval '5m'
set system task-scheduler task vpn-monitor executable path '/config/scripts/vpn-monitor.sh'
commit

3. SNMP monitoring:

configure
set service snmp community public authorization 'ro'
set service snmp community public network '10.0.0.0/8'
commit

4. Syslog integration:

configure
set system syslog host 10.0.1.100 facility all level 'info'
set system syslog host 10.0.1.100 facility security level 'warning'
commit

Performance and optimization

1. Configuring MTU for the VPN:

configure
# Reduce MTU to avoid fragmentation
set interfaces ethernet eth0 mtu '1500'
set interfaces tunnel tun0 mtu '1400'

# Enable MSS clamping
set policy route MSS-CLAMP rule 10 protocol 'tcp'
set policy route MSS-CLAMP rule 10 tcp flags 'SYN'
set policy route MSS-CLAMP rule 10 set tcp-mss '1360'

commit

2. Hardware acceleration (if supported):

# Check AES-NI support
show system cpu features

3. Optimization for high-performance links:

configure
# Use GCM mode for hardware acceleration
set vpn ipsec ike-group IKE-PERF proposal 1 encryption 'aes128gcm16'
set vpn ipsec esp-group ESP-PERF proposal 1 encryption 'aes128gcm16'

# Increase buffers (if necessary)
set system sysctl parameter net.core.rmem_max value '134217728'
set system sysctl parameter net.core.wmem_max value '134217728'

commit

Security and hardening

1. Firewall for VPN traffic:

configure
# Allow only IKE and ESP
set firewall name WAN_LOCAL rule 100 action 'accept'
set firewall name WAN_LOCAL rule 100 protocol 'udp'
set firewall name WAN_LOCAL rule 100 destination port '500'
set firewall name WAN_LOCAL rule 100 description 'Allow IKE'

set firewall name WAN_LOCAL rule 110 action 'accept'
set firewall name WAN_LOCAL rule 110 protocol 'udp'
set firewall name WAN_LOCAL rule 110 destination port '4500'
set firewall name WAN_LOCAL rule 110 description 'Allow NAT-T'

set firewall name WAN_LOCAL rule 120 action 'accept'
set firewall name WAN_LOCAL rule 120 protocol 'esp'
set firewall name WAN_LOCAL rule 120 description 'Allow ESP'

commit

2. Restricting management access:

configure
set service ssh access-control allow from '10.0.0.0/8'
set service ssh access-control allow from '192.168.0.0/16'
set service ssh access-control deny from '0.0.0.0/0'

set service https access-control allow from '10.0.0.0/8'
set service https access-control deny from '0.0.0.0/0'

commit

3. Rate limiting for DoS protection:

configure
set firewall name WAN_LOCAL rule 90 action 'drop'
set firewall name WAN_LOCAL rule 90 protocol 'udp'
set firewall name WAN_LOCAL rule 90 destination port '500,4500'
set firewall name WAN_LOCAL rule 90 recent count '10'
set firewall name WAN_LOCAL rule 90 recent time '60'
set firewall name WAN_LOCAL rule 90 state new 'enable'

commit

Documenting the configuration

1. Descriptions in the configuration:

configure
set vpn ipsec site-to-site peer office-branch description 'VPN to Moscow Office'
set vpn ipsec ike-group IKE-CORPORATE description 'Corporate VPN IKE parameters'
set vpn ipsec esp-group ESP-CORPORATE description 'Corporate VPN ESP parameters'
commit

2. Maintaining a changelog:

# Adding a comment to the configuration
configure
commit comment "Added VPN tunnel to new branch office in SPb"

3. Backup:

# Automatic configuration backup
configure
set system task-scheduler task backup-config crontab-spec '0 2 * * *'
set system task-scheduler task backup-config executable path '/config/scripts/backup.sh'
commit

Backup script:

#!/bin/bash
# /config/scripts/backup.sh
DATE=$(date +%Y%m%d-%H%M%S)
cp /config/config.boot /config/backups/config.boot.$DATE
# Keep the last 30 copies
ls -t /config/backups/config.boot.* | tail -n +31 | xargs rm -f

Testing and validation

1. Regular testing:

Check connectivity over the VPN weekly
Test failover mechanisms monthly
Verify recovery after a power outage

2. Load testing:

# iperf3 VPN throughput testing
iperf3 -c 10.10.2.1 -t 60 -P 4

3. Security verification:

# Validate the cryptographic parameters
show vpn ipsec sa detail | grep -E "encr|hash|DH"

Security recommendations

Protecting private keys

1. Passphrase protection:

Always generate keys with passphrase protection:

generate pki key-pair install secure-key
# Enter a strong passphrase when prompted

2. Restricting system access:

configure
# Two-factor authentication via RADIUS/TACACS+
set system login radius server 10.0.1.50 key 'radius-secret'
set system login user admin authentication encrypted-password '<hash>'
set system login user admin authentication radius

# Disable root login
set system login user root authentication encrypted-password '!'

commit

3. Access auditing:

configure
set system syslog global facility auth level 'info'
set system syslog global facility authpriv level 'info'
commit

Protection against attacks

1. Anti-replay protection:

IPsec automatically enables anti-replay protection. Verification:

show vpn ipsec sa detail | grep -i replay

2. DPD for attack detection:

configure
set vpn ipsec ike-group IKE-GROUP dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-GROUP dead-peer-detection interval '30'
commit

3. Rate limiting:

configure
set firewall name WAN_LOCAL rule 85 action 'drop'
set firewall name WAN_LOCAL rule 85 protocol 'udp'
set firewall name WAN_LOCAL rule 85 destination port '500'
set firewall name WAN_LOCAL rule 85 recent count '20'
set firewall name WAN_LOCAL rule 85 recent time '60'
commit

Standards compliance

1. GOST compliance (for Russia):

VyOS supports the standard algorithms. GOST cryptography requires additional integration.

2. PCI DSS compliance:

Use of AES-256 or higher
Regular key rotation (at least annually)
Logging of all security events
Restriction of administrative access

3. HIPAA compliance:

Encryption of all data in transit (AES-256)
Node authentication (RSA 2048+ bits)
Access auditing and logging
Regular security assessments

Disaster recovery planning

1. Documenting the configuration:

VPN topology diagrams
A list of all public keys and their locations
Recovery procedures

2. Backups:

# Automatic backup of the configuration and keys
configure
set system task-scheduler task daily-backup crontab-spec '0 3 * * *'
set system task-scheduler task daily-backup executable path '/config/scripts/full-backup.sh'
commit

3. Recovery testing:

Regularly verify the recovery process:

Restoring the configuration from a backup
Importing keys onto a new system
Verifying VPN functionality after recovery

Conclusion

RSA authentication for IPsec VPN in VyOS provides a reliable and scalable method for protecting VPN tunnels. Its key advantages include:

Security: Asymmetric cryptography eliminates the risks associated with shared keys
Scalability: Simplified key management in large networks
Flexibility: Support for dynamic IP addresses and various topologies
Compatibility: Standard protocols ensure compatibility with various VPN solutions

Key recommendations:

Use keys of at least 2048 bits (3072 bits recommended)
Rotate keys regularly (every 12-24 months)
Protect private keys with passphrases
Use modern cryptographic algorithms (AES-256, SHA-256, DH group 14+)
Enable PFS (Perfect Forward Secrecy)
Configure Dead Peer Detection for automatic recovery
Monitor the status of VPN tunnels
Regularly update VyOS to receive security fixes
Document the configuration and procedures
Perform regular testing and validation

With proper configuration and management, RSA authentication provides reliable protection for VPN infrastructure of any scale - from simple Site-to-Site connections to complex Hub-and-Spoke and Full-Mesh topologies.

Reviewed by OpenNIX Team · Last updated on 29 July 2026

OpenConnect VPN Server - SSL VPN for Remote Access PPTP Server - Legacy VPN (Deprecated)