# OSPF Unnumbered with ECMP

> OSPF unnumbered interfaces with Equal-Cost Multipath routing for datacenter spine-leaf topologies and cloud deployments in Yandex Cloud and VK Cloud

Source: https://opennix.org/en/docs/vyos/examples/ospf-unnumbered-ecmp/


OSPF unnumbered lets you use interfaces without assigning an IP subnet, simplifying the configuration of point-to-point links. ECMP (Equal-Cost Multipath) provides load balancing across several paths that share the same metric.

## Use Case

### Applicability

- **Datacenter Spine-Leaf**: L3 underlay with minimal addressing
- **Cloud Networking**: Multi-AZ topologies in Yandex Cloud / VK Cloud
- **Campus Networks**: L3 core with ECMP between buildings
- **Service Provider**: Carrier network with load balancing

### Benefits

1. **Simplified Address Planning**: No need for a /30 or /31 on every link
2. **ECMP Load Balancing**: Automatic traffic distribution
3. **Fast Convergence**: Sub-second failover with the right timers
4. **Scalability**: Easy to add new links
5. **Resource Efficiency**: Saves IP addresses

## Network Topology

### Basic topology (2 routers, 2 links)

```
                    ┌─────────────────┐
                    │   Router A      │
                    │   Area 0.0.0.0  │
                    │   RID: 192.168.0.1
                    │                 │
                    │ lo: 192.168.0.1 │
                    │ eth1: /32       │
                    │ eth2: /32       │
                    └────┬───────┬────┘
                         │       │
                    eth1 │       │ eth2
                   /32 IP│       │/32 IP
                    eth1 │       │ eth2
                    ┌────┴───────┴────┐
                    │   Router B      │
                    │   Area 0.0.0.0  │
                    │   RID: 192.168.0.2
                    │                 │
                    │ lo: 192.168.0.2 │
                    │ eth1: /32       │
                    │ eth2: /32       │
                    └─────────────────┘
```

### Datacenter Spine-Leaf with ECMP

```
     ┌─────────┐           ┌─────────┐
     │ Spine 1 │           │ Spine 2 │
     │Area 0   │═══════════│Area 0   │
     └─┬─┬─┬─┬─┘           └─┬─┬─┬─┬─┘
       │ │ │ │               │ │ │ │
    ┌──┘ │ │ └──┐         ┌──┘ │ │ └──┐
    │    │ │    │         │    │ │    │
 ┌──┴┐ ┌─┴┐ ┌──┴┐     ┌──┴┐ ┌─┴┐ ┌──┴┐
 │L1 │ │L2│ │L3 │ ... │L4 │ │L5│ │L6 │
 │A0 │ │A0│ │A0 │     │A0 │ │A0│ │A0 │
 └───┘ └──┘ └───┘     └───┘ └──┘ └───┘

ECMP: Each Leaf has 2+ paths to other Leaf switches through the Spines
```

## Requirements

### Minimum requirements

- VyOS 1.4 (Sagitta) or newer
- OSPF support (FRRouting)
- A unique Router ID for each router
- Loopback interfaces

### Network parameters (example)

| Parameter | Router A | Router B |
|----------|----------|----------|
| Router ID | 192.168.0.1 | 192.168.0.2 |
| Loopback | 192.168.0.1/32 | 192.168.0.2/32 |
| eth1 | 192.168.0.1/32 | 192.168.0.2/32 |
| eth2 | 192.168.0.1/32 | 192.168.0.2/32 |
| OSPF Area | 0.0.0.0 | 0.0.0.0 |

## Router A Configuration

### Interfaces

```bash
configure

# Loopback for the Router ID
set interfaces loopback lo address '192.168.0.1/32'
set interfaces loopback lo description 'Router ID + OSPF announce'

# eth1 - Unnumbered (uses the loopback address)
set interfaces ethernet eth1 address '192.168.0.1/32'
set interfaces ethernet eth1 description 'OSPF link to Router B eth1'
set interfaces ethernet eth1 ip ospf authentication md5 key-id 1 md5-key 'SecureOSPFPassword123'
set interfaces ethernet eth1 ip ospf network 'point-to-point'

# eth2 - Unnumbered (same IP as the loopback)
set interfaces ethernet eth2 address '192.168.0.1/32'
set interfaces ethernet eth2 description 'OSPF link to Router B eth2'
set interfaces ethernet eth2 ip ospf authentication md5 key-id 1 md5-key 'SecureOSPFPassword123'
set interfaces ethernet eth2 ip ospf network 'point-to-point'

commit
```

### OSPF Configuration

```bash
configure

# OSPF core parameters
set protocols ospf parameters router-id '192.168.0.1'
set protocols ospf parameters abr-type 'cisco'

# Area 0 (backbone)
set protocols ospf area 0.0.0.0 network '192.168.0.1/32'

# OSPF log adjacency changes
set protocols ospf log-adjacency-changes detail

# Timers for fast convergence
set protocols ospf timers throttle spf delay '50'
set protocols ospf timers throttle spf initial-holdtime '100'
set protocols ospf timers throttle spf max-holdtime '1000'

commit
save
```

### Explanation of the key parameters

- **`/32` on the interface**: Unnumbered - the interface uses the loopback address
- **`network point-to-point`**: OSPF skips DR/BDR election, so convergence is faster
- **`authentication md5`**: Secures the OSPF adjacency
- **`throttle spf`**: Fast SPF recalculation on changes (50 ms delay)

## Router B Configuration

```bash
configure

# Loopback
set interfaces loopback lo address '192.168.0.2/32'
set interfaces loopback lo description 'Router ID + OSPF announce'

# Interfaces
set interfaces ethernet eth1 address '192.168.0.2/32'
set interfaces ethernet eth1 description 'OSPF link to Router A eth1'
set interfaces ethernet eth1 ip ospf authentication md5 key-id 1 md5-key 'SecureOSPFPassword123'
set interfaces ethernet eth1 ip ospf network 'point-to-point'

set interfaces ethernet eth2 address '192.168.0.2/32'
set interfaces ethernet eth2 description 'OSPF link to Router A eth2'
set interfaces ethernet eth2 ip ospf authentication md5 key-id 1 md5-key 'SecureOSPFPassword123'
set interfaces ethernet eth2 ip ospf network 'point-to-point'

# OSPF
set protocols ospf parameters router-id '192.168.0.2'
set protocols ospf parameters abr-type 'cisco'
set protocols ospf area 0.0.0.0 network '192.168.0.2/32'
set protocols ospf log-adjacency-changes detail
set protocols ospf timers throttle spf delay '50'
set protocols ospf timers throttle spf initial-holdtime '100'
set protocols ospf timers throttle spf max-holdtime '1000'

commit
save
```

## Yandex Cloud Integration

### Scenario: Multi-AZ Spine-Leaf in Yandex Cloud

```
┌─────────────────────────────────────────────┐
│         Yandex Cloud VPC                    │
│                                             │
│  ┌──────────────┐       ┌──────────────┐   │
│  │  Zone A      │       │  Zone B      │   │
│  │              │       │              │   │
│  │ ┌─────────┐  │       │ ┌─────────┐  │   │
│  │ │ Spine-A │  │═══════│═│ Spine-B │  │   │
│  │ │ VyOS    │◄─┼───────┼─┤ VyOS    │  │   │
│  │ └────┬────┘  │       │ └────┬────┘  │   │
│  │      ║       │       │      ║       │   │
│  │  ┌───╩───┐   │       │  ┌───╩───┐   │   │
│  │  │ Leaf  │   │       │  │ Leaf  │   │   │
│  │  │ VMs   │   │       │  │ VMs   │   │   │
│  │  └───────┘   │       │  └───────┘   │   │
│  └──────────────┘       └──────────────┘   │
└─────────────────────────────────────────────┘
```

### Spine Configuration (Yandex Cloud)

```bash
configure

# Management interface (DHCP from Yandex Cloud)
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description 'Yandex Cloud VPC management'

# Loopback
set interfaces loopback lo address '10.255.0.1/32'
set interfaces loopback lo description 'Spine-A Router ID'

# Unnumbered interfaces to the Leaf routers
set interfaces ethernet eth1 address '10.255.0.1/32'
set interfaces ethernet eth1 description 'Link to Leaf-1'
set interfaces ethernet eth1 ip ospf network 'point-to-point'
set interfaces ethernet eth1 ip ospf authentication md5 key-id 1 md5-key 'YandexCloudOSPF2025'

set interfaces ethernet eth2 address '10.255.0.1/32'
set interfaces ethernet eth2 description 'Link to Leaf-2'
set interfaces ethernet eth2 ip ospf network 'point-to-point'
set interfaces ethernet eth2 ip ospf authentication md5 key-id 1 md5-key 'YandexCloudOSPF2025'

set interfaces ethernet eth3 address '10.255.0.1/32'
set interfaces ethernet eth3 description 'Link to Leaf-3'
set interfaces ethernet eth3 ip ospf network 'point-to-point'
set interfaces ethernet eth3 ip ospf authentication md5 key-id 1 md5-key 'YandexCloudOSPF2025'

# OSPF
set protocols ospf parameters router-id '10.255.0.1'
set protocols ospf parameters abr-type 'cisco'
set protocols ospf area 0.0.0.0 network '10.255.0.1/32'
set protocols ospf log-adjacency-changes detail

# Fast convergence timers
set protocols ospf timers throttle spf delay '50'
set protocols ospf timers throttle spf initial-holdtime '100'
set protocols ospf timers throttle spf max-holdtime '1000'

# ECMP configuration (FRR)
# Configured through vtysh below

commit
save

# FRR ECMP configuration
vtysh
conf t
router ospf
maximum-paths 16
exit
exit
exit
```

### Leaf Configuration (Yandex Cloud)

```bash
configure

# Loopback
set interfaces loopback lo address '10.255.1.1/32'
set interfaces loopback lo description 'Leaf-1 Router ID'

# Unnumbered interfaces to the Spine routers (ECMP)
set interfaces ethernet eth1 address '10.255.1.1/32'
set interfaces ethernet eth1 description 'Link to Spine-A'
set interfaces ethernet eth1 ip ospf network 'point-to-point'
set interfaces ethernet eth1 ip ospf authentication md5 key-id 1 md5-key 'YandexCloudOSPF2025'

set interfaces ethernet eth2 address '10.255.1.1/32'
set interfaces ethernet eth2 description 'Link to Spine-B'
set interfaces ethernet eth2 ip ospf network 'point-to-point'
set interfaces ethernet eth2 ip ospf authentication md5 key-id 1 md5-key 'YandexCloudOSPF2025'

# LAN interface for VM connectivity
set interfaces ethernet eth3 address '10.10.1.1/24'
set interfaces ethernet eth3 description 'VMs subnet'

# OSPF
set protocols ospf parameters router-id '10.255.1.1'
set protocols ospf parameters abr-type 'cisco'
set protocols ospf area 0.0.0.0 network '10.255.1.1/32'
set protocols ospf area 0.0.0.0 network '10.10.1.0/24'
set protocols ospf log-adjacency-changes detail
set protocols ospf timers throttle spf delay '50'
set protocols ospf timers throttle spf initial-holdtime '100'
set protocols ospf timers throttle spf max-holdtime '1000'

commit
save

# ECMP
vtysh
conf t
router ospf
maximum-paths 16
exit
exit
exit
```

### Monitoring through Yandex Monitoring

```bash
# Export OSPF neighbor metrics
show ip ospf neighbor | grep -c "Full" > /tmp/ospf_neighbors_full.txt

# Yandex Monitoring custom metrics (via the agent)
# /etc/yandex/unified_agent/config.yml
```

## VK Cloud Integration

### Configuration for VK Cloud Networking

```bash
configure

# VK Cloud management (OpenStack Neutron)
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description 'VK Cloud management'

# The rest of the configuration is the same as for Yandex Cloud

# VK Cloud specifics:
# 1. Security Groups - allow OSPF (IP protocol 89)
# 2. Floating IP - management only, not for OSPF
# 3. MTU 1450 for overlay networks

set interfaces ethernet eth1 mtu '1450'
set interfaces ethernet eth2 mtu '1450'

commit
save
```

### Security Groups (VK Cloud)

In the VK Cloud Console, create a Security Group:
- **Rule 1**: OSPF (IP Protocol 89) from VPC CIDR
- **Rule 2**: ICMP from VPC CIDR
- **Rule 3**: SSH (TCP 22) from trusted IPs

## Verifying the Configuration

### Checking OSPF neighbors

```bash
# OSPF neighbors
show ip ospf neighbor

# Expected output:
# Neighbor ID     Pri State           Up Time         Dead Time Address         Interface
# 192.168.0.2       1 Full/Point-to-Point  00:05:30        00:00:35  192.168.0.2     eth1:192.168.0.1
# 192.168.0.2       1 Full/Point-to-Point  00:05:25        00:00:38  192.168.0.2     eth2:192.168.0.1

# State must be "Full"
```

### Checking the OSPF database

```bash
# OSPF LSDB
show ip ospf database

# Router LSAs
show ip ospf database router

# Details of a specific LSA
show ip ospf database router 192.168.0.1
```

### Checking ECMP routes

```bash
# Routing table
show ip route ospf

# Expected output with ECMP:
# O   192.168.0.2/32 [110/10] via 192.168.0.2, eth1, weight 1, 00:05:30
#                            via 192.168.0.2, eth2, weight 1, 00:05:25

# Two paths to the same destination = ECMP is working
```

### Checking OSPF interfaces

```bash
# OSPF interfaces
show ip ospf interface

# Details of a specific interface
show ip ospf interface eth1

# Important parameters:
# - Network type: POINTOPOINT
# - Timer intervals: Hello 10s, Dead 40s
# - State: Point-to-Point
```

### Connectivity and ECMP test

```bash
# Ping the neighbor's loopback
ping 192.168.0.2 count 10

# Traceroute
traceroute 192.168.0.2

# Check load balancing (iperf3)
# On Router B:
iperf3 -s

# On Router A (several parallel streams for ECMP):
iperf3 -c 192.168.0.2 -P 4 -t 30

# Monitor interface utilization
watch -n 1 'show interfaces ethernet eth1 ; show interfaces ethernet eth2'
```

## Troubleshooting

### Problem 1: OSPF neighbors do not come up

**Symptoms**:
```bash
show ip ospf neighbor
# Empty or State = "Init" / "2-Way"
```

**Causes and solutions**:

1. **MD5 authentication mismatch**:
   ```bash
   # Check the configuration
   show configuration commands | grep "ospf authentication"

   # Make sure the key-id and md5-key match on both ends
   ```

2. **Network type mismatch**:
   ```bash
   # Both sides must be point-to-point
   show ip ospf interface eth1 | grep "Network Type"

   # Fix it
   set interfaces ethernet eth1 ip ospf network 'point-to-point'
   commit
   ```

3. **Firewall blocks OSPF (protocol 89)**:
   ```bash
   # Allow OSPF
   set firewall ipv4-name WAN_LOCAL rule 100 action 'accept'
   set firewall ipv4-name WAN_LOCAL rule 100 protocol 'ospf'
   commit
   ```

### Problem 2: ECMP does not work

**Symptoms**:
```bash
show ip route 192.168.0.2/32
# Shows only one path, although there should be two
```

**Solution**:
```bash
# Check the FRR ECMP configuration
vtysh
show running-config | grep maximum-paths

# If it is not set:
conf t
router ospf
maximum-paths 16
exit
exit
write memory
exit

# Also check kernel ECMP support
sysctl net.ipv4.fib_multipath_hash_policy
# Should be 1 (layer 3+4 hash)

# Set it if needed
sysctl -w net.ipv4.fib_multipath_hash_policy=1

# Make it permanent
echo "net.ipv4.fib_multipath_hash_policy=1" >> /etc/sysctl.conf
```

### Problem 3: Slow convergence

**Symptoms**:
When a link goes down, switchover takes > 10 seconds

**Solution**:

1. **Reduce the OSPF timers**:
   ```bash
   configure
   set interfaces ethernet eth1 ip ospf dead-interval '12'
   set interfaces ethernet eth1 ip ospf hello-interval '3'
   set interfaces ethernet eth2 ip ospf dead-interval '12'
   set interfaces ethernet eth2 ip ospf hello-interval '3'
   commit
   ```

2. **Enable BFD**:
   ```bash
   # BFD on the interfaces
   set interfaces ethernet eth1 ip enable-bfd
   set interfaces ethernet eth2 ip enable-bfd

   # BFD for OSPF
   vtysh
   conf t
   router ospf
   bfd
   exit
   interface eth1
   ip ospf bfd
   exit
   interface eth2
   ip ospf bfd
   exit
   exit
   write memory
   exit
   ```

   **Result**: Sub-second failover (< 1 second)

### Problem 4: Routing loops

**Symptoms**:
Traceroute shows loops and packets circle around

**Causes**:
- Incorrect OSPF area configuration
- Redistribution loops

**Solution**:
```bash
# Check the area assignment
show ip ospf interface

# All unnumbered interfaces must be in the same area
# Check redistribution
show configuration commands | grep redistribute

# Use a route-map to filter
set policy route-map OSPF-REDIST rule 10 action 'deny'
set policy route-map OSPF-REDIST rule 10 match interface 'eth0'
set policy route-map OSPF-REDIST rule 20 action 'permit'

set protocols ospf redistribute connected route-map 'OSPF-REDIST'
commit
```

## Best Practices

### 1. Router ID Planning

- Use loopback addresses as the Router ID
- Assign them from a single /16 block (for example, 10.255.0.0/16)
- Spine: 10.255.0.x, Leaf: 10.255.1.x, 10.255.2.x, ...

### 2. OSPF Timers

**Production (default)**:
- Hello: 10 seconds
- Dead: 40 seconds
- Convergence: ~40 seconds

**Fast Convergence**:
```bash
set interfaces ethernet ethX ip ospf dead-interval '12'
set interfaces ethernet ethX ip ospf hello-interval '3'
```
- Convergence: ~12 seconds

**BFD (best)**:
- Convergence: < 1 second

### 3. ECMP Configuration

```bash
# FRR
vtysh
conf t
router ospf
maximum-paths 16
exit
exit
write memory

# Kernel
sysctl -w net.ipv4.fib_multipath_hash_policy=1
sysctl -w net.ipv4.fib_multipath_use_neigh=1
```

### 4. Security

```bash
# MD5 authentication on all OSPF interfaces
set interfaces ethernet ethX ip ospf authentication md5 key-id 1 md5-key 'StrongPassword'

# Passive interfaces (do not send OSPF Hello)
set protocols ospf passive-interface 'eth0'
set protocols ospf passive-interface 'eth3'
```

### 5. Logging

```bash
# OSPF adjacency changes
set protocols ospf log-adjacency-changes detail

# Syslog
set system syslog global facility protocols level 'info'

# Monitoring
watch -n 5 'show ip ospf neighbor'
```

## Advanced Configuration

### Stub Area to save resources

```bash
# Leaf routers in a stub area (fewer LSAs, less CPU/memory)
set protocols ospf area 0.0.0.1 area-type stub
set protocols ospf area 0.0.0.1 network '10.255.1.0/24'
```

### OSPF Cost Tuning

```bash
# Change the cost to prioritize paths
set interfaces ethernet eth1 ip ospf cost '10'
set interfaces ethernet eth2 ip ospf cost '20'

# eth1 becomes the preferred path
```

### BFD for sub-second failover

```bash
# Detailed BFD configuration
set protocols bfd peer 192.168.0.2 multihop local-address '192.168.0.1'
set protocols bfd peer 192.168.0.2 interval transmit '300'
set protocols bfd peer 192.168.0.2 interval receive '300'
set protocols bfd peer 192.168.0.2 interval multiplier '3'

# 300ms * 3 = 900ms maximum failure detection time
```

## References

- [VyOS OSPF Configuration](https://docs.vyos.io/en/latest/configuration/protocols/ospf.html)
- [RFC 2328 - OSPF Version 2](https://datatracker.ietf.org/doc/html/rfc2328)
- [RFC 3137 - OSPF Stub Router Advertisement](https://datatracker.ietf.org/doc/html/rfc3137)
- [FRRouting OSPF Documentation](http://docs.frrouting.org/en/latest/ospfd.html)
- [Yandex Cloud VPC](https://cloud.yandex.ru/docs/vpc/)
- [VK Cloud Networks](https://mcs.mail.ru/docs/networks)