# Conntrack Sync

> Conntrack Sync on VyOS - stateful failover, VRRP integration and connection state synchronization for high availability in the cloud

Source: https://opennix.org/en/docs/vyos/services/vyos-conntrack-sync/


Conntrack Sync (connection tracking synchronization) is a feature of the Netfilter framework that synchronizes connection tracking states between network devices. It provides stateful failover and high availability for firewall and NAT.

## Overview

Connection tracking follows every network connection through the router at the protocol level. This makes it possible to:

- Support stateful firewall rules
- Provide state-aware NAT
- Synchronize active connections between HA nodes
- Minimize connection disruption during failover

Conntrack Sync synchronizes these states across multiple VyOS routers, allowing the backup node to immediately take over active connections without dropping them.

### Supported protocols

Conntrack Sync can synchronize states for the following protocols:

- TCP
- UDP
- ICMP
- SCTP
- DCCP
- IPv6-ICMP

### Architecture

```
[Primary VyOS]                    [Backup VyOS]
     |                                  |
     |    Conntrack Sync Protocol       |
     |<-------------------------------->|
     |    (Multicast or Unicast)        |
     |                                  |
[Conntrack Table]              [Conntrack Table]
   - Active connections           - Synced connections
   - NAT mappings                 - NAT mappings
   - Firewall states              - Firewall states
```

During failover the backup node holds a full table of active connections and can continue processing without interrupting client sessions.

## Basic configuration

### Minimal setup

Minimal configuration for conntrack sync:

```
set service conntrack-sync interface eth1
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
```

### Configuration structure

```
service conntrack-sync
  ├── interface <name>              # Interface for synchronization
  ├── mcast-group <address>         # Multicast group (default: 225.0.0.50)
  ├── failover-mechanism
  │   └── vrrp
  │       └── sync-group <name>     # VRRP sync-group
  ├── accept-protocol <protocol>    # Restrict synchronized protocols
  ├── ignore-address <ip>           # Ignore IP addresses
  ├── sync-queue-size <size>        # Synchronization queue size
  ├── expect-sync <protocol>        # Synchronize expect entries
  ├── event-listen-queue-size <size>
  └── startup-resync                # Request a full resync at startup
```

## Configuration parameters

### Interface

The interface used to transmit synchronization data:

```
set service conntrack-sync interface eth1
```

Recommendations:
- Use a dedicated interface for synchronization (not a public one)
- Typically this is an interface on the private network between HA nodes
- The interface must be up and have an IP address

### Multicast Group

The multicast group used to transmit synchronization data:

```
set service conntrack-sync mcast-group 225.0.0.50
```

Default value: `225.0.0.50`

Multicast requires:
- An L2 segment between nodes (no multicast routing)
- The firewall to permit multicast traffic
- The interface to support multicast

### Unicast (alternative to Multicast)

For networks without multicast support, use unicast:

```
set service conntrack-sync expect-sync all
set service conntrack-sync interface eth1
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
```

Unicast uses a peer-to-peer connection instead of a multicast group.

### Failover Mechanism - VRRP Integration

Integration with VRRP for automatic state management:

```
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
```

With this, conntrack-sync will:
- Be active only on the VRRP MASTER node
- Switch over automatically during a VRRP failover
- Synchronize states in the background

Requires a configured VRRP sync-group. Example VRRP configuration:

```
set high-availability vrrp group WAN vrid 10
set high-availability vrrp group WAN interface eth0
set high-availability vrrp group WAN virtual-address 203.0.113.1/24
set high-availability vrrp group WAN priority 200

set high-availability vrrp sync-group SYNC1 member WAN
```

### Accept Protocol

Restricting the protocols that are synchronized:

```
set service conntrack-sync accept-protocol tcp
set service conntrack-sync accept-protocol udp
set service conntrack-sync accept-protocol icmp
```

Available protocols:
- `tcp` - TCP connections
- `udp` - UDP sessions
- `icmp` - ICMP requests/replies
- `sctp` - SCTP connections
- `dccp` - DCCP connections
- `ipv6-icmp` - ICMPv6

By default all protocols are synchronized.

Recommendations:
- To reduce overhead, synchronize only the critical protocols
- TCP is usually the most important for stateful failover
- UDP can generate many entries (DNS, NTP)

### Ignore Address

Excluding specific IP addresses from synchronization:

```
set service conntrack-sync ignore-address 192.168.1.1
set service conntrack-sync ignore-address 203.0.113.0/24
```

Use it to:
- Exclude service addresses (monitoring, management)
- Exclude local services
- Reduce synchronization overhead

### Expect Sync

Synchronizing expect entries for complex protocols:

```
set service conntrack-sync expect-sync ftp
set service conntrack-sync expect-sync h323
set service conntrack-sync expect-sync nfs
set service conntrack-sync expect-sync sip
set service conntrack-sync expect-sync sqlnet
```

Available protocols:
- `ftp` - FTP (active mode)
- `h323` - H.323 VoIP
- `nfs` - Network File System
- `sip` - SIP VoIP
- `sqlnet` - Oracle SQLNet
- `all` - all protocols

Expect entries are required for protocols that open additional connections (for example, the FTP data channel).

### Queue Sizes

Configuring queue sizes:

**Sync Queue Size:**
```
set service conntrack-sync sync-queue-size 8192
```
Queue for outbound synchronization messages. Default: 1024.

**Event Listen Queue Size:**
```
set service conntrack-sync event-listen-queue-size 16384
```
Queue for incoming conntrack events. Default: 8192.

Increase these when there is a large number of connections or when synchronization messages are being lost.

### Startup Resync

Full connection table synchronization at startup:

```
set service conntrack-sync startup-resync
```

When enabled:
- The new node requests the full connection table from the master
- It increases startup time but guarantees a complete synchronization
- Useful with frequent reboots or upgrades

## Complete High Availability configuration

### Scenario: Active-Standby Firewall/NAT

**Topology:**
```
                    Internet
                       |
                [Virtual IP: 203.0.113.1]
                       |
         +-------------+-------------+
         |                           |
[vyos-primary]              [vyos-backup]
eth0: 203.0.113.10          eth0: 203.0.113.11
eth1: 192.168.1.1           eth1: 192.168.1.2
eth2: 10.0.0.1 (sync)       eth2: 10.0.0.2 (sync)
         |                           |
         +-------------+-------------+
                       |
                  Internal LAN
```

**vyos-primary configuration:**

```
# Base interfaces
set interfaces ethernet eth0 address 203.0.113.10/24
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth1 address 192.168.1.1/24
set interfaces ethernet eth1 description 'LAN'
set interfaces ethernet eth2 address 10.0.0.1/24
set interfaces ethernet eth2 description 'HA-SYNC'

# VRRP for the WAN interface
set high-availability vrrp group WAN vrid 10
set high-availability vrrp group WAN interface eth0
set high-availability vrrp group WAN virtual-address 203.0.113.1/24
set high-availability vrrp group WAN priority 200
set high-availability vrrp group WAN preempt true
set high-availability vrrp group WAN authentication type 'plaintext-password'
set high-availability vrrp group WAN authentication password 'SecurePassword123'

# VRRP for the LAN interface
set high-availability vrrp group LAN vrid 20
set high-availability vrrp group LAN interface eth1
set high-availability vrrp group LAN virtual-address 192.168.1.254/24
set high-availability vrrp group LAN priority 200
set high-availability vrrp group LAN preempt true
set high-availability vrrp group LAN authentication type 'plaintext-password'
set high-availability vrrp group LAN authentication password 'SecurePassword123'

# VRRP Sync Group
set high-availability vrrp sync-group SYNC1 member WAN
set high-availability vrrp sync-group SYNC1 member LAN

# Conntrack Sync
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync accept-protocol tcp
set service conntrack-sync accept-protocol udp
set service conntrack-sync accept-protocol icmp
set service conntrack-sync ignore-address 10.0.0.0/24
set service conntrack-sync expect-sync ftp
set service conntrack-sync expect-sync sip

# NAT for internet egress
set nat source rule 100 source address 192.168.1.0/24
set nat source rule 100 outbound-interface name eth0
set nat source rule 100 translation address masquerade

# Firewall - WAN_LOCAL
set firewall ipv4 name WAN_LOCAL default-action drop
set firewall ipv4 name WAN_LOCAL rule 10 action accept
set firewall ipv4 name WAN_LOCAL rule 10 state established
set firewall ipv4 name WAN_LOCAL rule 10 state related
set firewall ipv4 name WAN_LOCAL rule 20 action drop
set firewall ipv4 name WAN_LOCAL rule 20 state invalid
set firewall ipv4 name WAN_LOCAL rule 30 action accept
set firewall ipv4 name WAN_LOCAL rule 30 protocol icmp

# Applying the firewall on WAN
set firewall interface eth0 local name WAN_LOCAL
```

**vyos-backup configuration:**

```
# Base interfaces
set interfaces ethernet eth0 address 203.0.113.11/24
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth1 address 192.168.1.2/24
set interfaces ethernet eth1 description 'LAN'
set interfaces ethernet eth2 address 10.0.0.2/24
set interfaces ethernet eth2 description 'HA-SYNC'

# VRRP for the WAN interface
set high-availability vrrp group WAN vrid 10
set high-availability vrrp group WAN interface eth0
set high-availability vrrp group WAN virtual-address 203.0.113.1/24
set high-availability vrrp group WAN priority 100
set high-availability vrrp group WAN preempt false
set high-availability vrrp group WAN authentication type 'plaintext-password'
set high-availability vrrp group WAN authentication password 'SecurePassword123'

# VRRP for the LAN interface
set high-availability vrrp group LAN vrid 20
set high-availability vrrp group LAN interface eth1
set high-availability vrrp group LAN virtual-address 192.168.1.254/24
set high-availability vrrp group LAN priority 100
set high-availability vrrp group LAN preempt false
set high-availability vrrp group LAN authentication type 'plaintext-password'
set high-availability vrrp group LAN authentication password 'SecurePassword123'

# VRRP Sync Group
set high-availability vrrp sync-group SYNC1 member WAN
set high-availability vrrp sync-group SYNC1 member LAN

# Conntrack Sync (identical to primary)
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync accept-protocol tcp
set service conntrack-sync accept-protocol udp
set service conntrack-sync accept-protocol icmp
set service conntrack-sync ignore-address 10.0.0.0/24
set service conntrack-sync expect-sync ftp
set service conntrack-sync expect-sync sip

# NAT (identical to primary)
set nat source rule 100 source address 192.168.1.0/24
set nat source rule 100 outbound-interface name eth0
set nat source rule 100 translation address masquerade

# Firewall (identical to primary)
set firewall ipv4 name WAN_LOCAL default-action drop
set firewall ipv4 name WAN_LOCAL rule 10 action accept
set firewall ipv4 name WAN_LOCAL rule 10 state established
set firewall ipv4 name WAN_LOCAL rule 10 state related
set firewall ipv4 name WAN_LOCAL rule 20 action drop
set firewall ipv4 name WAN_LOCAL rule 20 state invalid
set firewall ipv4 name WAN_LOCAL rule 30 action accept
set firewall ipv4 name WAN_LOCAL rule 30 protocol icmp

set firewall interface eth0 local name WAN_LOCAL
```

**Key points:**
- Primary has priority 200, backup has 100
- Primary uses preempt, backup does not (this prevents flapping)
- Both nodes have identical NAT and firewall configuration
- Synchronization happens over the dedicated interface eth2
- The VRRP sync-group guarantees state consistency

## Operational commands

### Viewing the connection table

**IPv4 conntrack table:**
```
show conntrack table ipv4
```

Example output:
```
ipv4     tcp      ESTABLISHED src=192.168.1.100 dst=203.0.113.50 sport=45678 dport=443
ipv4     tcp      TIME_WAIT src=192.168.1.101 dst=203.0.113.51 sport=52341 dport=80
ipv4     udp      src=192.168.1.102 dst=8.8.8.8 sport=53124 dport=53
```

**IPv6 conntrack table:**
```
show conntrack table ipv6
```

**Filtering by protocol:**
```
show conntrack table ipv4 protocol tcp
show conntrack table ipv4 protocol udp
```

**Filtering by address:**
```
show conntrack table ipv4 source 192.168.1.100
show conntrack table ipv4 destination 203.0.113.50
```

### Viewing Conntrack Sync state

**Synchronization status:**
```
show conntrack-sync status
```

Example output:
```
sync-interface        : eth2
failover-mechanism    : vrrp [sync-group SYNC1]
last-rx               : 0.123s ago
last-tx               : 0.098s ago
```

**Internal cache (local connections):**
```
show conntrack-sync cache internal
```

Shows the connections tracked locally for synchronization.

**External cache (received connections):**
```
show conntrack-sync cache external
```

Shows the connections received from the peer node.

**Synchronization statistics:**
```
show conntrack-sync statistics
```

Example output:
```
cache stats:
  current entries: 1523
  peak entries:    2891
  insertions:      45234
  deletions:       43711

message traffic:
  received:
    packets:       123456
    bytes:         45678901
    errors:        0
  sent:
    packets:       123401
    bytes:         45654321
    errors:        0
```

### Managing Conntrack

**Deleting all connections:**
```
delete conntrack table ipv4
```

**Deleting connections by protocol:**
```
delete conntrack table ipv4 protocol tcp
```

**Deleting connections by address:**
```
delete conntrack table ipv4 source 192.168.1.100
```

### Restarting the service

```
restart conntrack-sync
```

## Configuration examples

### Minimal configuration (without VRRP)

```
set service conntrack-sync interface eth1
set service conntrack-sync mcast-group 225.0.0.50
```

Note: Without VRRP, both nodes actively synchronize connections, which can lead to duplication.

### TCP-only synchronization (optimization)

```
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync accept-protocol tcp
```

### Synchronization for VoIP (SIP/H.323)

```
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync accept-protocol tcp
set service conntrack-sync accept-protocol udp
set service conntrack-sync expect-sync sip
set service conntrack-sync expect-sync h323
```

### High-load configuration

```
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync accept-protocol tcp
set service conntrack-sync accept-protocol udp
set service conntrack-sync sync-queue-size 16384
set service conntrack-sync event-listen-queue-size 32768
set service conntrack-sync startup-resync
```

### Excluding service addresses

```
set service conntrack-sync interface eth2
set service conntrack-sync mcast-group 225.0.0.50
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1
set service conntrack-sync ignore-address 10.0.0.0/24
set service conntrack-sync ignore-address 192.168.255.0/24
set service conntrack-sync ignore-address 169.254.0.0/16
```

## Troubleshooting

### Checking synchronization status

1. Verify that conntrack-sync is running:
```
show conntrack-sync status
```

Expected:
- sync-interface should be up
- last-rx and last-tx should show recent activity

2. Check the VRRP state:
```
show vrrp
```

Make sure the MASTER node has an active conntrack-sync.

3. Check the statistics:
```
show conntrack-sync statistics
```

Pay attention to:
- `errors` should be 0 or minimal
- `received packets` should be increasing on the BACKUP node
- `sent packets` should be increasing on the MASTER node

### Connections are not synchronizing

**Problem:** The backup node does not see connections from the master.

**Checks:**

1. Verify multicast connectivity:
```
# On the master
set system ipv4 multicast-routing mroute 0.0.0.0/0 interface eth2

# Test with tcpdump
sudo tcpdump -i eth2 host 225.0.0.50
```

2. Check the firewall on the sync interface:
```
# Allow multicast on eth2
set firewall interface eth2 local ipv4 name ALLOW_SYNC
set firewall ipv4 name ALLOW_SYNC rule 10 action accept
set firewall ipv4 name ALLOW_SYNC rule 10 destination address 225.0.0.50
set firewall ipv4 name ALLOW_SYNC rule 10 protocol udp
```

3. Verify the interfaces are in the same L2 segment:
```
# On both nodes
ping -I eth2 <peer-ip>
```

4. Check the logs:
```
show log | match conntrack
```

### High CPU usage from conntrack-sync

**Causes:**
- Too many connections being synchronized
- accept-protocol is misconfigured
- ignore-address is not used for service addresses

**Solution:**

1. Restrict the synchronized protocols:
```
set service conntrack-sync accept-protocol tcp
```

2. Exclude service addresses:
```
set service conntrack-sync ignore-address 10.0.0.0/8
set service conntrack-sync ignore-address 192.168.0.0/16
```

3. Increase the queue sizes:
```
set service conntrack-sync sync-queue-size 16384
set service conntrack-sync event-listen-queue-size 32768
```

4. Check the number of connections:
```
show conntrack table ipv4 | count
```

### Failover drops connections

**Problem:** Connections are dropped during a VRRP switchover.

**Causes:**
- Conntrack-sync did not manage to synchronize the connections in time
- Firewall state is not synchronized
- NAT mappings are not synchronized

**Solution:**

1. Make sure conntrack-sync is tied to VRRP:
```
show service conntrack-sync
```

2. Enable startup-resync:
```
set service conntrack-sync startup-resync
```

3. Check the VRRP transition delay:
```
set high-availability vrrp group WAN transition-script mode-force
```

4. Make sure the firewall rules are identical:
```
# On both nodes
show firewall
```

5. Check the NAT rules:
```
# On both nodes
show nat source rules
```

### Conntrack table is full

**Problem:** `nf_conntrack: table full, dropping packet`

**Check:**
```
# Current usage
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
```

**Solution:**

1. Increase the table size:
```
set system conntrack table-size 262144
```

2. Reduce the conntrack timeouts:
```
set system conntrack timeout tcp close-wait 10
set system conntrack timeout tcp time-wait 10
```

3. Configure ignore-address to reduce the number of entries:
```
set service conntrack-sync ignore-address <high-traffic-subnet>
```

4. Reboot after changing table-size:
```
reboot now
```

### Diagnostics with tcpdump

Monitoring conntrack-sync traffic:

```
# Multicast traffic
sudo tcpdump -i eth2 -n host 225.0.0.50 -vv

# All UDP on the sync interface
sudo tcpdump -i eth2 -n udp -vv

# Writing to a file
sudo tcpdump -i eth2 -n host 225.0.0.50 -w /tmp/conntrack-sync.pcap
```

## Best practices

### Design

1. **Dedicated synchronization interface** - use a separate interface for conntrack-sync, do not mix it with production traffic

2. **Use a private network** - the sync interface should be on a private network that is not reachable from outside

3. **L2 adjacency** - for multicast, the nodes must be in the same L2 segment (no routing)

4. **Sufficient bandwidth** - for high-load systems, use at least 1Gbit/s for sync

### Optimization

1. **Restrict protocols** - synchronize only the critical protocols (usually TCP)
```
set service conntrack-sync accept-protocol tcp
```

2. **Exclude service addresses** - do not synchronize monitoring, management, or heartbeat
```
set service conntrack-sync ignore-address 10.0.0.0/8
```

3. **Tune queue sizes** - increase queue sizes for high loads
```
set service conntrack-sync sync-queue-size 16384
set service conntrack-sync event-listen-queue-size 32768
```

4. **Use expect-sync selectively** - enable only the protocols you need (FTP, SIP)

### Security

1. **Protect the sync interface** - configure the firewall to allow only conntrack-sync traffic
```
set firewall interface eth2 local ipv4 name SYNC_LOCAL
set firewall ipv4 name SYNC_LOCAL default-action drop
set firewall ipv4 name SYNC_LOCAL rule 10 action accept
set firewall ipv4 name SYNC_LOCAL rule 10 destination address 225.0.0.50
set firewall ipv4 name SYNC_LOCAL rule 10 protocol udp
```

2. **Isolate the VLAN** - use a separate VLAN for HA/sync traffic

3. **Monitoring** - set up alerts for synchronization errors

### Monitoring

1. **Check the statistics regularly:**
```
show conntrack-sync statistics
```

2. **Track errors:**
```
show log | match conntrack
```

3. **Monitor the table size:**
```
show conntrack table ipv4 | count
```

4. **Check the VRRP state:**
```
show vrrp
```

### Failover testing

1. **Preparation:**
   - Establish active connections (for example, SSH, HTTP)
   - Start a ping monitor

2. **Trigger a failover:**
```
# On the master node
set high-availability vrrp group WAN disable
commit
```

3. **Verification:**
   - Connections should survive without a drop
   - Ping should show no significant loss
   - The backup should become MASTER

4. **Recovery:**
```
delete high-availability vrrp group WAN disable
commit
```

### Maintenance

1. **Backup** - save the configuration regularly:
```
save /config/backup/config.$(date +%Y%m%d).boot
```

2. **Configuration synchronization** - both nodes must have identical:
   - Firewall rules
   - NAT rules
   - Conntrack-sync parameters
   - VRRP configuration

3. **Planned maintenance:**
   - Move the node to backup before maintenance
   - Verify synchronization after upgrades
   - Test failover after changes

## Cloud platform specifics

### Yandex Cloud

**Limitations:**
- Multicast is usually not supported
- Use unicast or a VPN between the nodes

**Recommended architecture:**
```
set service conntrack-sync interface eth1
set service conntrack-sync failover-mechanism vrrp sync-group SYNC1

# Use an internal network for synchronization
# eth1 must be in an internal subnet
```

**Network Load Balancer:**
- Configure the NLB to distribute traffic
- Use health checks for automatic failover
- Conntrack-sync will provide a seamless transition

### VK Cloud

Same as Yandex Cloud:
- Use private networks for sync
- Configure security groups to allow sync traffic
- Test failover in a staging environment before production

### General cloud recommendations

1. **Use internal networks** - for sync traffic
2. **Configure security groups** - allow only the necessary traffic
3. **Monitoring** - use cloud metrics to track state
4. **Automation** - use Terraform/Ansible for deployment
5. **Backup** - regularly save snapshots and configurations

## Next steps

- [High Availability (VRRP)](/docs/vyos/ha/) - configuring fault tolerance
- [Firewall](/docs/vyos/firewall/) - stateful firewall rules
- [NAT](/docs/vyos/nat/) - Network Address Translation
- [Traffic Policy](/docs/vyos/qos/) - traffic management

## Additional resources

- [VyOS Documentation - Conntrack](https://docs.vyos.io/en/latest/configuration/service/conntrack-sync.html)
- [Netfilter Conntrack Documentation](https://www.netfilter.org/documentation/)
- [VRRP RFC 5798](https://tools.ietf.org/html/rfc5798)