# Failover - Automatic Route Switching

> Failover routing on VyOS - policy-based route switching, health checks (ICMP, ARP, TCP) and nexthop monitoring in the cloud

Source: https://opennix.org/en/docs/vyos/routing/vyos-failover/


Failover routing in VyOS provides automatic route switching based on availability monitoring of target hosts or services, allowing you to build fault-tolerant network solutions without using dynamic routing protocols.

## Overview

Failover routing is an automatic routing mechanism that adds or removes static routes from the kernel routing table based on the results of availability checks (health checks) against target addresses.

**How it works**:
- Routes are configured manually by the administrator
- A route is installed into the routing table only if the health-check target is reachable
- If the target becomes unreachable, the route is automatically removed
- When availability is restored, the route is reinstated

**Differences from other mechanisms**:

| Mechanism | Automation | Detection speed | Complexity | Overhead |
|----------|---------------|----------------------|-----------|----------|
| Failover routes | Health checks | Medium (seconds) | Low | Minimal |
| BFD | Monitoring protocol | Very high (ms) | Medium | Low |
| Dynamic protocols | Full | High | High | High |
| Static routes + distance | None | None | Low | None |

**Advantages**:
- Simple setup without dynamic protocols
- Automatic switching on failures
- Flexible health check mechanisms
- Support for various check types
- Minimal resource consumption
- Predictable behavior

**Disadvantages**:
- Slower than BFD
- Requires configuring health check targets
- Not suitable for complex topologies
- Limited scalability

## Core Concepts

### Failover Route

A failover route consists of:
- **Subnet** - the destination network
- **Next-hop** - the next hop (gateway)
- **Target** - the address to check for availability
- **Check type** - the check type (ICMP, ARP, TCP)
- **Check parameters** - the check parameters

### Health Check

The availability check mechanism:
- Periodic checks of target addresses
- Various check types (ping, ARP, TCP port)
- Configurable intervals and timeouts
- Check policies (all-available, any-available)

### Route Installation

Route installation logic:
1. The health check runs at the configured interval
2. If the target is reachable, the route is installed
3. If the target is unreachable, the route is removed
4. When restored, the route is reinstated

## Basic Configuration

### Simple failover route

Minimal configuration with an ICMP health check:

```bash
configure

# Failover route to the 203.0.113.0/24 network
set protocols failover route 203.0.113.0/24 next-hop 192.0.2.1
set protocols failover route 203.0.113.0/24 next-hop 192.0.2.1 check target '192.0.2.1'
set protocols failover route 203.0.113.0/24 next-hop 192.0.2.1 check type 'icmp'
set protocols failover route 203.0.113.0/24 next-hop 192.0.2.1 interface 'eth0'

commit
save
```

This route:
- Is installed if 192.0.2.1 responds to ICMP ping
- Is removed if 192.0.2.1 becomes unreachable
- Uses the eth0 interface as the egress

### Command syntax

```bash
set protocols failover route <subnet> next-hop <ip-address>
set protocols failover route <subnet> next-hop <ip-address> check target '<target-ip>'
set protocols failover route <subnet> next-hop <ip-address> check type '<type>'
set protocols failover route <subnet> next-hop <ip-address> interface '<interface>'
set protocols failover route <subnet> next-hop <ip-address> metric <1-65535>
set protocols failover route <subnet> next-hop <ip-address> check timeout <1-300>
```

**Parameters**:
- `<subnet>` - destination network in IP/prefix format
- `<ip-address>` - next-hop address
- `<target-ip>` - address for the health check
- `<type>` - check type: icmp, arp, tcp
- `<interface>` - outgoing interface
- `<metric>` - route metric (default 1)
- `<timeout>` - check timeout in seconds (default 10)

## Health Check Types

### ICMP (Ping)

The ICMP check (default) is the most common type.

```bash
set protocols failover route 10.0.0.0/8 next-hop 192.0.2.1 check target '192.0.2.1'
set protocols failover route 10.0.0.0/8 next-hop 192.0.2.1 check type 'icmp'
set protocols failover route 10.0.0.0/8 next-hop 192.0.2.1 check timeout 5
```

**ICMP parameters**:
- 2 ICMP Echo Request packets are sent
- Response timeout is 1 second per packet
- The health check succeeds if at least one response is received
- The overall timeout is configured separately

**When to use**:
- Checking gateway availability
- Checking connectivity to a remote host
- General link monitoring

**Limitations**:
- May be blocked by a firewall
- Does not verify service health
- May produce false positives under ICMP rate limiting

### ARP

The ARP check is for local networks (Layer 2).

```bash
set protocols failover route 172.16.0.0/12 next-hop 192.168.1.1 check target '192.168.1.1'
set protocols failover route 172.16.0.0/12 next-hop 192.168.1.1 check type 'arp'
set protocols failover route 172.16.0.0/12 next-hop 192.168.1.1 interface 'eth1'
```

**ARP parameters**:
- 2 ARP Request packets are sent
- Response timeout is 1 second per packet
- The health check succeeds if at least one ARP Reply is received
- Works only within the local subnet

**When to use**:
- Checking a local gateway
- Monitoring devices on the same L2 network
- When ICMP is blocked

**Limitations**:
- Local subnet only
- Does not work across routers
- Checks only Layer 2 availability

### TCP Port

The TCP check verifies the availability of a specific service.

```bash
set protocols failover route 198.51.100.0/24 next-hop 203.0.113.1 check target '203.0.113.10'
set protocols failover route 198.51.100.0/24 next-hop 203.0.113.1 check type 'tcp'
set protocols failover route 198.51.100.0/24 next-hop 203.0.113.1 check port 443
set protocols failover route 198.51.100.0/24 next-hop 203.0.113.1 interface 'eth0'
```

**TCP parameters**:
- Verifies the ability to establish a TCP connection
- Requires specifying the port via `check port <1-65535>`
- The health check succeeds if the port is open (the TCP handshake completes)
- The connection is closed immediately after the check

**When to use**:
- Checking web server health (80, 443)
- Monitoring specific services
- When both network and service availability matter

**Limitations**:
- Requires an open TCP port
- Slower than ICMP/ARP
- May be blocked by a firewall

**Example with an HTTPS server**:

```bash
# The route to the servers is active only if the web server is running
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check target '10.100.1.100'
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check type 'tcp'
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check port 443
```

## Multiple Target Addresses

Support for checking several target addresses with different policies.

### Multiple targets with the any-available policy

Default: the route is active if at least one target is reachable.

```bash
set protocols failover route 10.20.0.0/16 next-hop 192.0.2.1 check target '8.8.8.8'
set protocols failover route 10.20.0.0/16 next-hop 192.0.2.1 check target '8.8.4.4'
set protocols failover route 10.20.0.0/16 next-hop 192.0.2.1 check policy 'any-available'
```

**any-available logic**:
- The route is installed if any of the targets is reachable
- The route is removed only if all targets are unreachable
- Useful for checking overall link connectivity

### Multiple targets with the all-available policy

The route is active only if all targets are reachable.

```bash
set protocols failover route 10.30.0.0/16 next-hop 192.0.2.5 check target '10.30.1.1'
set protocols failover route 10.30.0.0/16 next-hop 192.0.2.5 check target '10.30.2.1'
set protocols failover route 10.30.0.0/16 next-hop 192.0.2.5 check target '10.30.3.1'
set protocols failover route 10.30.0.0/16 next-hop 192.0.2.5 check policy 'all-available'
```

**all-available logic**:
- The route is installed only if all targets are reachable
- The route is removed if at least one target is unreachable
- Useful for critical paths that require full availability

**Choosing a policy**:
- `any-available` - for general connectivity checks (default)
- `all-available` - to guarantee full infrastructure availability

## Failover Scenarios

### Dual ISP Failover

Automatic switching between two internet links.

**Topology**:
```
                 Internet
                    |
          +---------+---------+
          |                   |
      ISP1 (Primary)      ISP2 (Backup)
    203.0.113.1/30       198.51.100.1/30
          |                   |
        eth0                eth1
          |                   |
          +------- VyOS -------+
                    |
             LAN (192.168.1.0/24)
```

**Configuration**:

```bash
configure

# ISP interfaces
set interfaces ethernet eth0 address '203.0.113.2/30'
set interfaces ethernet eth0 description 'ISP1-Primary'

set interfaces ethernet eth1 address '198.51.100.2/30'
set interfaces ethernet eth1 description 'ISP2-Backup'

# Primary ISP - failover route (metric 10)
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '1.1.1.1'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check policy 'any-available'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check timeout 10
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 metric 10

# Backup ISP - failover route (metric 20)
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check target '1.1.1.1'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check policy 'any-available'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check timeout 10
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 interface 'eth1'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 metric 20

# NAT for both ISPs
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.1.0/24'
set nat source rule 100 translation address 'masquerade'

set nat source rule 200 outbound-interface name 'eth1'
set nat source rule 200 source address '192.168.1.0/24'
set nat source rule 200 translation address 'masquerade'

commit
save
```

**How it works**:
1. The Primary ISP (metric 10) is installed if the DNS servers are reachable
2. The Backup ISP (metric 20) is installed in parallel
3. The kernel selects the route with the lower metric (Primary)
4. If the Primary fails, its route is removed and traffic flows through the Backup
5. When the Primary is restored, its route is reinstated and traffic switches back

### Active-Backup Gateway

Failover between two gateways in the same network.

**Topology**:
```
    Gateway1 (Primary)     Gateway2 (Backup)
      192.168.1.1            192.168.1.2
          |                       |
          +----------+------------+
                     |
                   VyOS
                  eth0: 192.168.1.10
                     |
            LAN: 192.168.10.0/24
```

**Configuration**:

```bash
configure

# Primary gateway
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.1
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.1 check target '192.168.1.1'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.1 check type 'arp'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.1 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.1 metric 10

# Backup gateway
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.2
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.2 check target '192.168.1.2'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.2 check type 'arp'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.2 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 192.168.1.2 metric 20

commit
save
```

**Using the ARP check**:
- Fast checking of local gateways
- Not affected by ICMP blocking
- Checks only L2 availability

### Multi-site VPN Failover

Automatic switching between VPN tunnels.

**Configuration**:

```bash
configure

# Primary VPN tunnel (via ISP1)
set protocols failover route 10.20.0.0/16 next-hop 10.100.1.1
set protocols failover route 10.20.0.0/16 next-hop 10.100.1.1 check target '10.20.1.1'
set protocols failover route 10.20.0.0/16 next-hop 10.100.1.1 check type 'icmp'
set protocols failover route 10.20.0.0/16 next-hop 10.100.1.1 interface 'tun0'
set protocols failover route 10.20.0.0/16 next-hop 10.100.1.1 metric 10

# Backup VPN tunnel (via ISP2)
set protocols failover route 10.20.0.0/16 next-hop 10.100.2.1
set protocols failover route 10.20.0.0/16 next-hop 10.100.2.1 check target '10.20.1.1'
set protocols failover route 10.20.0.0/16 next-hop 10.100.2.1 check type 'icmp'
set protocols failover route 10.20.0.0/16 next-hop 10.100.2.1 interface 'tun1'
set protocols failover route 10.20.0.0/16 next-hop 10.100.2.1 metric 20

commit
save
```

## Examples for Cloud Providers

### Yandex Cloud: Dual ISP with NAT Failover

Configuration for Yandex Cloud with two public IP addresses.

**Architecture**:
```
Yandex Cloud VPC: 10.128.0.0/24

VyOS Instance:
- eth0 (external): 10.128.0.10
  - Public IP1: 51.250.10.20 (Primary)
  - Public IP2: 51.250.10.21 (Backup)
- eth1 (internal): 192.168.1.1/24
```

**Full configuration**:

```bash
configure

# Interfaces
set interfaces ethernet eth0 address '10.128.0.10/24'
set interfaces ethernet eth0 description 'Yandex-Cloud-External'

set interfaces ethernet eth1 address '192.168.1.1/24'
set interfaces ethernet eth1 description 'Internal-Network'

# Default route via the Yandex Cloud gateway
set protocols static route 0.0.0.0/0 next-hop 10.128.0.1

# Failover routes to check public connectivity
# Check availability via the Primary IP
set protocols failover route 8.8.8.8/32 next-hop 10.128.0.1
set protocols failover route 8.8.8.8/32 next-hop 10.128.0.1 check target '8.8.8.8'
set protocols failover route 8.8.8.8/32 next-hop 10.128.0.1 check type 'icmp'
set protocols failover route 8.8.8.8/32 next-hop 10.128.0.1 check timeout 5
set protocols failover route 8.8.8.8/32 next-hop 10.128.0.1 interface 'eth0'

# Source NAT via the Primary IP (higher priority)
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.1.0/24'
set nat source rule 100 translation address '51.250.10.20'
set nat source rule 100 description 'NAT-Primary-IP'

# Source NAT via the Backup IP (lower priority, used when the Primary fails)
set nat source rule 200 outbound-interface name 'eth0'
set nat source rule 200 source address '192.168.1.0/24'
set nat source rule 200 translation address '51.250.10.21'
set nat source rule 200 description 'NAT-Backup-IP'

# Destination NAT for inbound connections
set nat destination rule 10 destination address '51.250.10.20'
set nat destination rule 10 destination port '443'
set nat destination rule 10 inbound-interface name 'eth0'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 translation address '192.168.1.100'
set nat destination rule 10 translation port '443'

commit
save
```

**Verification**:

```bash
# Failover route status
show ip route failover

# Check NAT
show nat source statistics

# Monitor switchovers
monitor log tail
```

### VK Cloud: Multi-AZ Gateway Failover

Failover between gateways in different VK Cloud availability zones.

**Architecture**:
```
VK Cloud VPC: 10.0.0.0/16

Availability Zone 1:
- Gateway1: 10.0.1.1 (Primary)
- VyOS eth0: 10.0.1.10

Availability Zone 2:
- Gateway2: 10.0.2.1 (Backup)
- VyOS eth1: 10.0.2.10
```

**Configuration**:

```bash
configure

# Interfaces in different AZs
set interfaces ethernet eth0 address '10.0.1.10/24'
set interfaces ethernet eth0 description 'VK-Cloud-AZ1'

set interfaces ethernet eth1 address '10.0.2.10/24'
set interfaces ethernet eth1 description 'VK-Cloud-AZ2'

# Failover default route via AZ1 (Primary)
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 check target '10.0.1.1'
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 check policy 'all-available'
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 10.0.1.1 metric 10

# Failover default route via AZ2 (Backup)
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 check target '10.0.2.1'
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 check policy 'all-available'
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 interface 'eth1'
set protocols failover route 0.0.0.0/0 next-hop 10.0.2.1 metric 20

# Policy routing for symmetric traffic
set policy route PBR rule 10 source address '10.0.1.0/24'
set policy route PBR rule 10 set table '100'

set policy route PBR rule 20 source address '10.0.2.0/24'
set policy route PBR rule 20 set table '200'

# Routing tables for the different AZs
set protocols static table 100 route 0.0.0.0/0 next-hop 10.0.1.1
set protocols static table 200 route 0.0.0.0/0 next-hop 10.0.2.1

commit
save
```

**Verifying Multi-AZ failover**:

```bash
# Check routes in the main table
show ip route

# Check policy routing
show policy route statistics

# Check availability of both AZs
ping 10.0.1.1 source-address 10.0.1.10
ping 10.0.2.1 source-address 10.0.2.10
```

### Combined scenario: VPN + ISP Failover

Failover between a VPN link and a direct ISP route.

**Configuration**:

```bash
configure

# VPN tunnel (Primary) - via WireGuard
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1 check target '10.50.1.1'
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1 check type 'icmp'
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1 check timeout 5
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1 interface 'wg0'
set protocols failover route 10.50.0.0/16 next-hop 10.200.0.1 metric 10

# Direct ISP route (Backup) - via the internet
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50 check target '10.50.1.1'
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50 check type 'tcp'
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50 check port 443
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50 interface 'eth0'
set protocols failover route 10.50.0.0/16 next-hop 203.0.113.50 metric 50

commit
save
```

**Logic**:
- Primary: traffic through the secure VPN tunnel
- Backup: if the VPN fails, traffic through the unencrypted internet
- The TCP check on the backup confirms the availability of the end service

## Advanced Configuration

### Custom Timeout and Interval

Configuring timeouts for various scenarios.

**Fast switching**:

```bash
# Aggressive checks for critical links
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check target '10.100.1.1'
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check type 'icmp'
set protocols failover route 10.100.0.0/16 next-hop 192.0.2.10 check timeout 3
```

Fast detection (3 seconds), but more overhead.

**Slow switching**:

```bash
# Conservative checks for stability
set protocols failover route 10.200.0.0/16 next-hop 192.0.2.20
set protocols failover route 10.200.0.0/16 next-hop 192.0.2.20 check target '10.200.1.1'
set protocols failover route 10.200.0.0/16 next-hop 192.0.2.20 check type 'icmp'
set protocols failover route 10.200.0.0/16 next-hop 192.0.2.20 check timeout 30
```

Slow detection (30 seconds), avoids false positives.

### Metrics for prioritization

Using metrics for fine-grained control.

```bash
# Tier 1: Fastest link (fiber)
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 metric 10

# Tier 2: Medium link (cable)
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 metric 20

# Tier 3: Slow link (DSL backup)
set protocols failover route 0.0.0.0/0 next-hop 192.0.2.1
set protocols failover route 0.0.0.0/0 next-hop 192.0.2.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 192.0.2.1 metric 30

# Tier 4: Last resort (4G/LTE)
set protocols failover route 0.0.0.0/0 next-hop 10.64.0.1
set protocols failover route 0.0.0.0/0 next-hop 10.64.0.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 10.64.0.1 metric 100
```

Cascading failover with a clear priority hierarchy.

### Integration with Firewall

Failover that accounts for firewall zones.

```bash
configure

# Failover through different firewall zones
set firewall zone WAN1 interface 'eth0'
set firewall zone WAN2 interface 'eth1'

# Failover routes
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 metric 10

set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 interface 'eth1'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 metric 20

# Firewall rules are applied automatically on switchover
set firewall zone WAN1 from LAN firewall name LAN-to-WAN
set firewall zone WAN2 from LAN firewall name LAN-to-WAN

commit
save
```

## Monitoring and Verification

### Viewing failover routes

**All failover routes**:

```bash
show protocols failover
```

Output:
```
Failover routes:
  route 0.0.0.0/0 next-hop 203.0.113.1
    check target: 8.8.8.8
    check type: icmp
    check timeout: 10
    interface: eth0
    metric: 10
    status: active

  route 0.0.0.0/0 next-hop 198.51.100.1
    check target: 8.8.8.8
    check type: icmp
    check timeout: 10
    interface: eth1
    metric: 20
    status: standby
```

**Active routes in the table**:

```bash
show ip route
```

Only failover routes that have passed the health check are shown.

**Details of a specific route**:

```bash
show ip route 0.0.0.0/0
```

Output:
```
Routing entry for 0.0.0.0/0
  Known via "failover", distance 10, metric 10, best
  Last update 00:05:23 ago
  * 203.0.113.1, via eth0
```

### Health Check Monitoring

**Check status**:

```bash
show protocols failover route 0.0.0.0/0 next-hop 203.0.113.1
```

Output:
```
Route: 0.0.0.0/0
Next-hop: 203.0.113.1
Check targets:
  8.8.8.8 - UP (last check: 2s ago, rtt: 15ms)
  1.1.1.1 - UP (last check: 2s ago, rtt: 12ms)
Check policy: any-available
Check type: icmp
Check timeout: 10s
Route status: ACTIVE
Uptime: 2h 15m 42s
```

**Failover event logs**:

```bash
show log | match failover
```

Output:
```
Oct 15 10:23:45 vyos-gw failover[1234]: Route 0.0.0.0/0 via 203.0.113.1: target 8.8.8.8 DOWN
Oct 15 10:23:45 vyos-gw failover[1234]: Route 0.0.0.0/0 via 203.0.113.1: REMOVED from table
Oct 15 10:25:12 vyos-gw failover[1234]: Route 0.0.0.0/0 via 203.0.113.1: target 8.8.8.8 UP
Oct 15 10:25:12 vyos-gw failover[1234]: Route 0.0.0.0/0 via 203.0.113.1: INSTALLED to table
```

### Real-time monitoring

**Continuous route monitoring**:

```bash
monitor protocols failover
```

**Log monitoring**:

```bash
monitor log | match failover
```

**Ping monitoring of a target**:

```bash
monitor ping 8.8.8.8
```

### Switchover statistics

**Number of failover events**:

```bash
show protocols failover statistics
```

Output:
```
Failover Statistics:
  Total routes configured: 2
  Active routes: 1
  Standby routes: 1

  Route 0.0.0.0/0 via 203.0.113.1:
    Uptime: 2h 15m 42s
    Total checks: 1521
    Successful checks: 1521
    Failed checks: 0
    Failover count: 0

  Route 0.0.0.0/0 via 198.51.100.1:
    Uptime: 0s (standby)
    Total checks: 1521
    Successful checks: 1521
    Failed checks: 0
    Failover count: 0
```

## Troubleshooting

### Failover route does not install

**Problem**: The route does not appear in the routing table.

**Diagnostics**:

```bash
# Check the configuration
show configuration commands | match failover

# Check target availability
ping <target-ip> source-address <nexthop-interface-ip>

# Check the interface
show interfaces <interface>

# Check the firewall
show firewall
```

**Common causes**:
- The target is unreachable
- The interface is down
- The firewall blocks health check packets
- Incorrect target IP
- Routing loop

**Solution**:

```bash
# Temporarily increase the timeout
set protocols failover route <subnet> next-hop <ip> check timeout 30

# Change the check type
set protocols failover route <subnet> next-hop <ip> check type arp

# Try a different target
set protocols failover route <subnet> next-hop <ip> check target '<alternative-ip>'

commit
```

### Frequent switching (flapping)

**Problem**: The route is constantly added and removed.

**Diagnostics**:

```bash
# Monitor the logs
show log | match failover | tail -50

# Check latency to the target
monitor ping <target-ip>

# Check for packet loss
ping <target-ip> count 100
```

**Common causes**:
- An unstable link
- Too aggressive a timeout
- The target is overloaded
- ICMP rate limiting
- Network issues

**Solution - increasing the timeout**:

```bash
# Increase the timeout
set protocols failover route <subnet> next-hop <ip> check timeout 30

# Add hysteresis (use all-available with multiple targets)
set protocols failover route <subnet> next-hop <ip> check target '<target1>'
set protocols failover route <subnet> next-hop <ip> check target '<target2>'
set protocols failover route <subnet> next-hop <ip> check target '<target3>'
set protocols failover route <subnet> next-hop <ip> check policy 'all-available'

commit
```

### Slow switching

**Problem**: Failover happens too slowly.

**Diagnostics**:

```bash
# Check the current timeout
show protocols failover route <subnet> next-hop <ip>
```

**Solution**:

```bash
# Reduce the timeout
set protocols failover route <subnet> next-hop <ip> check timeout 5

# Use a faster check type (ARP instead of ICMP)
set protocols failover route <subnet> next-hop <ip> check type arp

commit
```

**Alternative - BFD**:

For very fast switching (milliseconds), use BFD instead of failover:

```bash
set protocols bfd peer <nexthop-ip> interval transmit 300
set protocols bfd peer <nexthop-ip> interval receive 300
set protocols bfd peer <nexthop-ip> interval multiplier 3

set protocols static route <subnet> next-hop <ip> bfd

commit
```

### Health check does not work

**Problem**: The health check always shows DOWN.

**ICMP diagnostics**:

```bash
# Check ICMP with a source interface
ping <target> source-address <interface-ip>

# Tcpdump for analysis
monitor traffic interface <interface> filter icmp
```

**ARP diagnostics**:

```bash
# Check the ARP table
show arp

# Clear the ARP cache
reset arp interface <interface> address <target>
```

**TCP diagnostics**:

```bash
# Check the TCP port
telnet <target> <port>

# Check with curl
run curl -v --connect-timeout 5 http://<target>:<port>
```

**Common causes**:
- A firewall on the target blocks ICMP
- The target does not respond to ARP
- The TCP port is closed or filtered
- Source routing problems
- Asymmetric routing

**Solution**:

```bash
# Change the check type
set protocols failover route <subnet> next-hop <ip> check type tcp
set protocols failover route <subnet> next-hop <ip> check port 443

# Or use a different target
set protocols failover route <subnet> next-hop <ip> check target '<new-target>'

commit
```

### Asymmetric routing

**Problem**: Inbound and outbound traffic take different paths.

**Diagnostics**:

```bash
# Traceroute in both directions
traceroute <destination>

# On the remote host
traceroute <local-address>
```

**Solution - Policy Based Routing**:

```bash
# Routing based on source address
set policy route PBR rule 10 source address '<subnet>'
set policy route PBR rule 10 set table 100

set protocols static table 100 route 0.0.0.0/0 next-hop <specific-gateway>

# Apply to the interface
set interfaces ethernet <interface> policy route PBR

commit
```

## Best Practices

### Choosing Health Check parameters

**Timeout**:
- **3-5 seconds** - for critical links that require fast failover
- **10 seconds** - standard value (default)
- **20-30 seconds** - for unstable links, avoiding flapping

**Check type**:
- **ICMP** - for general connectivity checks (standard choice)
- **ARP** - for local gateways on the same L2 network
- **TCP** - for checking a specific service

**Target selection**:
- Use stable, always-available targets (8.8.8.8, 1.1.1.1)
- For critical routes, use multiple targets with all-available
- For general routes, use any-available

### Metrics and priorities

**Rule**: The difference in metrics should be large enough to establish a clear priority.

```bash
# Good: clear hierarchy
metric 10, 20, 30, 50, 100

# Bad: values too close together
metric 10, 11, 12, 13
```

**Recommended scheme**:
- Primary: 10
- Secondary: 20
- Tertiary: 30
- Emergency backup: 100

### Avoiding Flapping

**Use hysteresis**:

```bash
# Multiple targets with all-available
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '1.1.1.1'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check policy 'all-available'
```

The route is removed only if all targets are unreachable.

**Adequate timeout**:

```bash
# Not too aggressive
set protocols failover route <subnet> next-hop <ip> check timeout 15
```

### Documentation

**Use description**:

```bash
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 description 'Primary-ISP-Fiber-1Gbps'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 description 'Backup-ISP-Cable-100Mbps'
```

### Monitoring and Alerting

**Syslog setup**:

```bash
set system syslog global facility protocols level info
set system syslog host 192.168.1.100 facility protocols level warning
```

**Logging failover events**:

```bash
# All events are recorded in syslog
show log | match failover
```

### Testing Failover

**Regular testing**:

```bash
# Simulate a failure - temporarily disable the interface
set interfaces ethernet eth0 disable
commit

# Check the switchover
show ip route

# Restore
delete interfaces ethernet eth0 disable
commit
```

**Testing with controlled traffic**:

```bash
# Continuous ping during the test
monitor ping 8.8.8.8
```

### Combining with other technologies

**Failover + VRRP**:
- VRRP for local gateway redundancy
- Failover for upstream routes

**Failover + NAT**:
- Make sure NAT rules are applied to both interfaces
- Use masquerade for automatic adaptation

**Failover + VPN**:
- The health check should verify the end host behind the VPN
- Use a TCP check for critical VPN services

## Comparison with Alternatives

### Failover vs Static Routes + Distance

**Static routes + distance**:
```bash
set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 distance 10
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 20
```

**Problems**:
- No automatic failover
- The backup route activates only if the Primary nexthop is unreachable (not pingable)
- Slow failure detection

**Failover routes**:
```bash
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.1 check target '8.8.8.8' metric 10
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.1 check target '8.8.8.8' metric 20
```

**Advantages**:
- Active monitoring
- Fast detection
- Verification of real connectivity

### Failover vs BFD

**BFD**:
```bash
set protocols bfd peer 203.0.113.1 interval transmit 300
set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
```

**BFD advantages**:
- Very fast detection (milliseconds)
- A monitoring protocol
- Bidirectional checking

**Failover advantages**:
- Simpler to configure
- Does not require BFD support on the nexthop
- Flexible check types (ICMP, ARP, TCP)
- Can verify end-to-end connectivity

**When to use BFD**:
- Critical applications requiring sub-second failover
- The nexthop supports BFD
- Bidirectional checking is needed

**When to use Failover**:
- The nexthop does not support BFD
- You need to check a specific service (TCP)
- Simplicity matters more than speed

### Failover vs dynamic protocols (OSPF/BGP)

**OSPF/BGP**:
- Automatic routing
- Fast convergence
- Scalability

**Failover routes**:
- Manual configuration
- Controlled behavior
- Simplicity

**When to use Failover**:
- Simple topologies (2-3 routes)
- Stub networks
- The ISP does not support BGP
- Full control is needed

**When to use dynamic protocols**:
- Complex topologies
- Multiple paths
- Fast adaptation is required
- Enterprise networks

## Performance and Limitations

### Performance

**Health check overhead**:
- ICMP: minimal (2 packets every N seconds)
- ARP: minimal (2 packets every N seconds)
- TCP: medium (a full 3-way handshake)

**CPU impact**:
- Negligible for 10-20 failover routes
- Increases with the number of routes and the check frequency

**Recommendations**:
- Up to 50 failover routes on standard hardware
- Timeout ≥ 5 seconds for minimal overhead

### Limitations

**Technical**:
- No support for IPv6 failover routes (IPv4 only)
- One check type per route
- Limited check types (ICMP, ARP, TCP)

**Operational**:
- Switchover speed: seconds (depends on the timeout)
- No stateful failover for TCP connections
- May cause asymmetric routing

## Integration with Monitoring Systems

### SNMP Monitoring

```bash
configure

# Enable SNMP
set service snmp community public authorization ro
set service snmp community public network 192.168.1.0/24
set service snmp listen-address 192.168.1.1

commit
```

**Monitoring via SNMP**:
- IP-MIB::ipRouteTable - the routing table
- IF-MIB::ifOperStatus - interface status

### Syslog Integration

```bash
# Send failover events to a syslog server
set system syslog host 192.168.1.100 facility protocols level info
set system syslog host 192.168.1.100 port 514
set system syslog host 192.168.1.100 protocol udp
```

### Custom Scripts

Integration with an event handler for automatic actions:

```bash
set system event-handler event failover filter pattern 'failover.*REMOVED'
set system event-handler event failover script path '/config/scripts/failover-alert.sh'
```

**Example script** `/config/scripts/failover-alert.sh`:

```bash
#!/bin/bash
# Send a notification on failover
MESSAGE="Failover event detected on $(hostname) at $(date)"
echo "$MESSAGE" | mail -s "VyOS Failover Alert" admin@example.com
```

## Full Configuration Examples

### Enterprise Branch Office

Full configuration for a branch office with dual ISP and VPN.

```bash
configure

# Interfaces
set interfaces ethernet eth0 address '203.0.113.10/30'
set interfaces ethernet eth0 description 'ISP1-Primary'

set interfaces ethernet eth1 address '198.51.100.10/30'
set interfaces ethernet eth1 description 'ISP2-Backup'

set interfaces ethernet eth2 address '192.168.100.1/24'
set interfaces ethernet eth2 description 'LAN'

# Failover default routes
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 check target '1.1.1.1'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 check policy 'any-available'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 interface 'eth0'
set protocols failover route 0.0.0.0/0 next-hop 203.0.113.9 metric 10

set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 check target '8.8.8.8'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 check target '1.1.1.1'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 check policy 'any-available'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 check type 'icmp'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 interface 'eth1'
set protocols failover route 0.0.0.0/0 next-hop 198.51.100.9 metric 20

# NAT
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.100.0/24'
set nat source rule 100 translation address 'masquerade'

set nat source rule 200 outbound-interface name 'eth1'
set nat source rule 200 source address '192.168.100.0/24'
set nat source rule 200 translation address 'masquerade'

# Firewall
set firewall zone LAN interface 'eth2'
set firewall zone WAN1 interface 'eth0'
set firewall zone WAN2 interface 'eth1'

set firewall zone WAN1 from LAN firewall name LAN-to-WAN
set firewall zone WAN2 from LAN firewall name LAN-to-WAN

set firewall name LAN-to-WAN default-action accept
set firewall name LAN-to-WAN enable-default-log

# Syslog
set system syslog global facility protocols level info
set system syslog host 192.168.100.50 facility all level warning

commit
save
```

## Next Steps

- [Static Routes](/docs/vyos/routing/vyos-static/) - basic static routing
- [OSPF](/docs/vyos/routing/vyos-ospf/) - dynamic routing in the enterprise
- [BGP](/docs/vyos/routing/vyos-bgp/) - for connecting to an ISP
- [BFD](/docs/vyos/routing/vyos-bfd/) - fast failure detection
- [VPN](/docs/vyos/vpn/) - for secure tunnels
- [High Availability](/docs/vyos/ha/) - VRRP for gateway redundancy