IPsec VPN Troubleshooting in pfSense - Diagnostics Guide

IPsec is a protocol with strict requirements for parameter matching on both sides of a tunnel. A mismatch in even a single parameter (encryption algorithm, DH group, subnet mask) causes the connection to fail, and error messages do not always point to the root cause. A systematic approach to diagnostics reduces troubleshooting time from hours to minutes.

This guide describes the methodology for diagnosing IPsec VPN issues in pfSense: from enabling debug logs through analyzing specific error messages. The material applies to both site-to-site and mobile client configurations.

Enabling Debug Logs

By default, pfSense logs only basic IPsec events. For detailed diagnostics, the logging level must be increased.

Configuring the Logging Level

Navigate to VPN > IPsec > Advanced Settings and configure the logging parameters:

Click Save. Changing the logging level does not disrupt existing IPsec tunnels.

Warning:

The Diag level generates a significant volume of logs. After completing diagnostics, it is recommended to revert levels to Control to avoid filling disk space.

Viewing Logs

IPsec logs are available under Status > System Logs > IPsec. Use the search field at the top of the page to filter entries.

To view logs in real time via the command line:

# Real-time IPsec log monitoring
clog -f /var/log/ipsec.log

# Filter for specific peer IP
clog -f /var/log/ipsec.log | grep "203.0.113.10"

# Filter for errors only
clog -f /var/log/ipsec.log | grep -i "error\|failed\|no proposal"

Key Log Indicators

The following entries indicate successful or failed negotiation:

Common Phase 1 Errors

Phase 1 (IKE SA) handles mutual authentication and security parameter negotiation. Errors at this stage completely prevent tunnel establishment.

Encryption Parameter Mismatch

When using AES-GCM in Phase 1, note that some implementations (Cisco ASA, older MikroTik versions) do not support GCM for IKE. In such cases, use AES-CBC with a separate hash algorithm.

Authentication Errors

Warning:

When copying a PSK via the clipboard, invisible characters (spaces, newline characters) are often appended. It is recommended to type the key manually or use a random string generator.

Identifier Errors

Unreachable Remote Peer

DPD Errors

Common Phase 2 Errors

Phase 2 (Child SA / IPsec SA) is negotiated within the secure Phase 1 channel. Phase 2 errors occur when Phase 1 is established - the Phase 1 status shows Established, but no Child SA is created.

Subnet Mismatch (Traffic Selectors)

Warning:

Subnet mismatch is the most common cause of Phase 2 failure. The subnet mask must match exactly: 10.1.0.0/24 on one side and 10.1.0.0/24 on the other (not 10.1.0.0/23 or 10.1.0.0/32).

Phase 2 Encryption Parameter Mismatch

PFS Mismatch

Tunnel Established but No Traffic

If Phase 1 and Phase 2 are established (status Established), but user traffic does not pass, the problem lies outside IPsec negotiation.

NAT-T Issues

NAT Traversal (NAT-T) is required when a NAT device (router, ISP equipment) sits between two IPsec tunnel endpoints. NAT-T encapsulates ESP packets in UDP port 4500, allowing them to pass through NAT.

When NAT-T Is Required

• One or both IPsec endpoints are behind NAT (have a private IP address on the WAN interface)
• A router performing address translation sits between the endpoints
• The ISP uses Carrier-Grade NAT (CG-NAT)

Common NAT-T Problems

Verifying NAT-T

To verify NAT-T operation, perform a packet capture on the WAN interface:

# Capture IKE and NAT-T traffic
tcpdump -ni em0 udp port 500 or udp port 4500

# Expected output with NAT-T active:
# 10.0.0.1.4500 > 203.0.113.10.4500: UDP-encap: ESP(spi=0x12345678,seq=0x1)

If traffic is visible on port 4500 with the UDP-encap: ESP notation, NAT-T is functioning correctly. If traffic uses the ESP protocol (protocol 50) in the presence of NAT, the tunnel will be unstable.

Diagnostic Commands

pfSense provides a set of commands for IPsec diagnostics via Diagnostics > Command Prompt or an SSH connection.

IPsec SA Status

# Complete SA status with negotiated parameters
ipsec statusall

# Brief status (established tunnels only)
ipsec status

# View IP addresses assigned to mobile clients
ipsec leases

The ipsec statusall command displays:

• Negotiated encryption and hash algorithms
• DH and PFS groups
• Time until SA expiration
• Traffic Selectors (subnets)
• Byte counts
• Current IKE SA and Child SA status

Security Policy Database (SPD)

# View installed security policies
setkey -DP

# View installed security associations
setkey -D

The setkey -DP command shows which packets should be processed by IPsec (routed into the tunnel). If a policy is missing for the required subnet pair, traffic will not be encrypted.

Firewall Rules

# View all active firewall rules
pfctl -sr

# View rules for IPsec interface (enc0)
pfctl -sr | grep enc0

# View blocked packets in real time
tcpdump -ni pflog0

Packet Capture

# Capture IKE negotiation on WAN
tcpdump -ni em0 udp port 500 or udp port 4500

# Capture ESP traffic (without NAT-T)
tcpdump -ni em0 proto 50

# Capture decrypted traffic on IPsec interface
tcpdump -ni enc0

# Capture with detailed output and write to file
tcpdump -ni em0 -vvv -s 0 -w /tmp/ipsec-capture.pcap udp port 500 or udp port 4500

Capturing on the enc0 interface shows decrypted traffic inside the tunnel. This allows you to determine whether packets reach pfSense through the tunnel and in what form.

Viewing Routes

# View routing table
netstat -rn

# Check if route for remote subnet exists
netstat -rn | grep "10.2.0.0"

# View gateway groups status
pfctl -vvsA

Managing the IPsec Service

# Restart IPsec service (drops all tunnels)
ipsec restart

# Reload configuration without dropping tunnels
ipsec update

# Terminate specific IKE SA
ipsec down <connection_name>

# Re-initiate specific connection
ipsec up <connection_name>

Warning:

The ipsec restart command drops all active tunnels. In production environments, it is preferable to use ipsec update to apply configuration changes without disrupting existing connections.

Rekeying Issues

IPsec tunnels have a limited lifetime. When this time expires, rekeying occurs - new keying material is negotiated. Errors during rekeying cause periodic tunnel disruptions.

Diagnostic Checklist

When IPsec tunnel issues arise, it is recommended to work through the following checklist sequentially.

1. Network Connectivity

• Remote peer is reachable: ping <remote_public_ip>
• Route to remote peer exists: traceroute <remote_public_ip>
• UDP 500 is not blocked: nc -uzv <remote_public_ip> 500
• UDP 4500 is not blocked: nc -uzv <remote_public_ip> 4500

2. Phase 1 Parameters

• IKE version matches (IKEv1 or IKEv2)
• Encryption algorithm matches
• Hash Algorithm matches
• DH Group matches
• Lifetime matches (difference is tolerated; the lower value is used)
• Authentication method matches (PSK or certificates)
• PSK is identical on both sides (when using PSK)
• Identifiers (My/Peer) are configured correctly

3. Phase 2 Parameters

• Subnets (Traffic Selectors) match in mirror
• Subnet masks are correct (/24 vs /32)
• Phase 2 Encryption Algorithm matches
• Phase 2 Hash Algorithm matches
• PFS Group matches or is disabled on both sides
• Protocol is ESP (not AH)

4. Certificates (When Used)

• CA certificate imported on both sides
• Server certificate has not expired (check the date)
• SAN in the server certificate contains the server IP or FQDN
• Certificate chain of trust is complete
• For Apple devices: server certificate Lifetime is no more than 825 days

5. Firewall Rules

• UDP 500 and 4500 are permitted on WAN
• Rules on the IPsec tab allow traffic between subnets
• No conflicting floating rules
• NAT Traversal is enabled (Auto or Force) when NAT is present

6. Routing and SA

• ipsec statusall shows established SAs
• setkey -DP contains policies for the required subnets
• netstat -rn contains a route to the remote subnet (for VTI)
• Byte counters in Status > IPsec increment when traffic passes

Related Sections

• IPsec Site-to-Site VPN in pfSense - configuring site-to-site tunnels, IPsec terminology, and Phase 1/Phase 2 parameters
• IPsec IKEv2 for Mobile Clients in pfSense - configuring IKEv2 for remote access, certificates, and client device setup
• pfSense Firewall Rules - principles of creating and managing firewall rules for VPN traffic
• NAT in pfSense - NAT configuration, including the interaction between NAT and IPsec