Documentation
VyOS Documentation
Routing in VyOS
OpenFabric - Data Center Routing Protocol

OpenFabric - Data Center Routing Protocol

Overview

OpenFabric is a link-state routing protocol designed specifically for modern data center (DC) architectures. The protocol is derived from IS-IS and optimized for Spine-Leaf (Clos) topologies with a large number of ECMP paths.

Key Features

  • IS-IS foundation: Builds on the proven IS-IS protocol technology with data center optimizations
  • Flat topology: No L1/L2 levels - all routers reside at the same level
  • Dual-stack: Native support for IPv4 and IPv6 simultaneously
  • ECMP: Efficient support for multiple equal-cost paths
  • Scalability: Optimized for large Spine-Leaf topologies
  • Fast convergence: Minimal reaction time to topology changes

OpenFabric Architecture

OpenFabric is designed to operate in the Clos (Spine-Leaf) topology, which has become the de facto standard for modern data centers:

┌─────────────────────────────────────────────────────────┐
│                    Leaf Layer (Tier 2)                   │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐   │
│  │ Leaf 1  │  │ Leaf 2  │  │ Leaf 3  │  │ Leaf 4  │   │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘   │
└───────┼────────────┼────────────┼────────────┼─────────┘
        │            │            │            │
        └────┬───────┴─────┬──────┴─────┬──────┘
             │             │            │
┌────────────┼─────────────┼────────────┼────────────────┐
│            │             │            │                 │
│  ┌─────────▼──┐  ┌───────▼────┐  ┌───▼───────┐       │
│  │  Spine 1   │  │  Spine 2   │  │  Spine 3  │       │
│  └────────────┘  └────────────┘  └───────────┘       │
│                  Spine Layer (Tier 1)                  │
└────────────────────────────────────────────────────────┘

Differences from Traditional IS-IS

CharacteristicIS-ISOpenFabric
Hierarchy levelsL1, L2, L1/L2Flat topology
Target use caseService provider networksData centers
TopologyArbitrarySpine-Leaf (Clos)
OptimizationGeneral purposeFor ECMP and DC
Flood optimizationStandardEnhanced for fabric

Basic Configuration

Network Entity Title (NET)

The NET is a unique identifier for a router within an OpenFabric domain. Configuring the NET is mandatory.

NET format:

49.AREA.SYSTEM-ID.SELECTOR

Components:

  • AFI (Authority and Format Identifier): 49 - private addressing
  • Area Identifier: The area identifier (typically 0001)
  • System Identifier: A unique system identifier (48 bits, usually 12 hexadecimal digits)
  • NET Selector: Always 00

NET examples:

# NET derived from IP address 192.168.100.2
49.0001.1921.6810.0002.00

# NET derived from MAC address 00:50:56:00:10:02
49.0001.0050.5600.1002.00

# Arbitrary NET
49.0001.0000.0000.0001.00

NET configuration:

set protocols openfabric net 49.0001.1921.6810.0002.00

Creating an OpenFabric Domain

A domain is a logical grouping of routers and interfaces within OpenFabric.

# Create a domain named DATACENTER
set protocols openfabric domain DATACENTER

# Configure the NET for the domain
set protocols openfabric net 49.0001.1921.6810.0001.00

Enabling Interfaces

Interfaces must be explicitly enabled within the OpenFabric domain, specifying the address families.

# Enable interface eth1 for IPv4
set protocols openfabric domain DATACENTER interface eth1 address-family ipv4

# Enable interface eth2 for IPv4 and IPv6
set protocols openfabric domain DATACENTER interface eth2 address-family ipv4
set protocols openfabric domain DATACENTER interface eth2 address-family ipv6

# Enable the loopback interface
set protocols openfabric domain DATACENTER interface lo address-family ipv4

Minimal Working Configuration

# Spine router configuration
set protocols openfabric net 49.0001.1921.6810.0001.00
set protocols openfabric domain DC interface eth1 address-family ipv4
set protocols openfabric domain DC interface eth2 address-family ipv4
set protocols openfabric domain DC interface eth3 address-family ipv4
set protocols openfabric domain DC interface eth4 address-family ipv4
set protocols openfabric domain DC interface lo address-family ipv4
commit
save

Fabric Tier Configuration

The Fabric Tier is a level in the Spine-Leaf hierarchy that defines the role of a router.

Fabric Levels

  • Tier 0: Reserved for future use
  • Tier 1: Spine layer (aggregation)
  • Tier 2: Leaf layer (server access)

Tier Configuration

# Spine router configuration (Tier 1)
set protocols openfabric domain DATACENTER fabric-tier 1

# Leaf router configuration (Tier 2)
set protocols openfabric domain DATACENTER fabric-tier 2

Important: Correct Tier configuration is critical for the flooding mechanism to work optimally and to prevent loops.

Interface Parameters

Hello Interval and Multiplier

Hello packets are used to discover neighbors and maintain adjacencies.

# Set the hello interval (default: 3 seconds)
set protocols openfabric domain DATACENTER interface eth1 hello-interval 5

# Set the hello multiplier (default: 3)
# Dead interval = hello-interval × hello-multiplier
set protocols openfabric domain DATACENTER interface eth1 hello-multiplier 3

Calculating the Dead interval:

Dead Interval = Hello Interval × Hello Multiplier
Example: 5 seconds × 3 = 15 seconds

Interface Metric

The metric affects path selection. A lower metric is preferred.

# Set the interface metric (default: 10)
set protocols openfabric domain DATACENTER interface eth1 metric 100

# Different metrics for different interfaces
set protocols openfabric domain DATACENTER interface eth1 metric 10
set protocols openfabric domain DATACENTER interface eth2 metric 20

Metric range: 1 - 16,777,215

Passive Interface

A passive interface advertises networks but does not form adjacencies.

# Useful for loopback interfaces and interfaces facing servers
set protocols openfabric domain DATACENTER interface lo passive

# Server-facing interface without OpenFabric
set protocols openfabric domain DATACENTER interface eth5 passive
set protocols openfabric domain DATACENTER interface eth5 address-family ipv4

CSNP and PSNP Intervals

CSNP (Complete Sequence Number PDU) and PSNP (Partial Sequence Number PDU) are used to synchronize the link-state database.

# Set the CSNP interval (default: 10 seconds)
set protocols openfabric domain DATACENTER interface eth1 csnp-interval 15

# Set the PSNP interval (default: 2 seconds)
set protocols openfabric domain DATACENTER interface eth1 psnp-interval 3

LSP (Link State PDU) Parameters

LSP Generation

Control the rate of new LSP generation to optimize load.

# Minimum interval between LSP generations (milliseconds)
set protocols openfabric domain DATACENTER lsp-gen-interval 1

# Maximum interval between LSP generations (seconds)
set protocols openfabric domain DATACENTER lsp-mtu 1497

LSP Refresh Interval

LSPs are refreshed periodically to prevent them from expiring.

# LSP refresh interval (default: 900 seconds)
set protocols openfabric domain DATACENTER lsp-refresh-interval 600

# Maximum LSP lifetime (default: 1200 seconds)
set protocols openfabric domain DATACENTER max-lsp-lifetime 1800

Recommendation: max-lsp-lifetime must be greater than lsp-refresh-interval.

Overload Bit

The overload bit signals to neighbors that a router is overloaded and should not be used for transit traffic.

# Set the overload bit
set protocols openfabric domain DATACENTER overload

# Set the overload bit for a specific period (seconds)
set protocols openfabric domain DATACENTER overload on-startup 120

Use cases:

  • Planned maintenance
  • Graceful restart
  • Temporary performance issues

Purge Originator

Identify the source of an LSP purge in the network.

# Enable purge originator identification
set protocols openfabric domain DATACENTER purge-originator

Authentication

Domain-Level Authentication

# Plaintext authentication
set protocols openfabric domain DATACENTER domain-password plaintext-password MySecretPassword

# MD5 authentication
set protocols openfabric domain DATACENTER domain-password md5 MySecretPassword

Interface-Level Authentication

# Plaintext interface authentication
set protocols openfabric domain DATACENTER interface eth1 password plaintext-password InterfacePassword

# MD5 interface authentication
set protocols openfabric domain DATACENTER interface eth1 password md5 InterfacePassword

Recommendation: Use MD5 authentication for stronger security.

Route Redistribution

Redistributing Connected Networks

# Redistribute all connected interfaces
set protocols openfabric domain DATACENTER redistribute ipv4 connected

# Redistribute with route-map filtering
set protocols openfabric domain DATACENTER redistribute ipv4 connected route-map CONNECTED-TO-FABRIC

Redistributing Static Routes

# Redistribute static routes
set protocols openfabric domain DATACENTER redistribute ipv4 static

# Redistribute with a metric
set protocols openfabric domain DATACENTER redistribute ipv4 static metric 50

Redistributing from Other Protocols

# Redistribute from OSPF
set protocols openfabric domain DATACENTER redistribute ipv4 ospf

# Redistribute from BGP
set protocols openfabric domain DATACENTER redistribute ipv4 bgp

# Redistribute from Kernel
set protocols openfabric domain DATACENTER redistribute ipv4 kernel

Route-map for Filtering

# Create a route-map for filtering
set policy route-map CONNECTED-TO-FABRIC rule 10 action permit
set policy route-map CONNECTED-TO-FABRIC rule 10 match interface eth5
set policy route-map CONNECTED-TO-FABRIC rule 10 set metric 20

set policy route-map CONNECTED-TO-FABRIC rule 20 action deny

# Apply the route-map
set protocols openfabric domain DATACENTER redistribute ipv4 connected route-map CONNECTED-TO-FABRIC

IPv6 Configuration

OpenFabric natively supports IPv6 alongside IPv4.

# Enable IPv6 on interfaces
set protocols openfabric domain DATACENTER interface eth1 address-family ipv6
set protocols openfabric domain DATACENTER interface eth2 address-family ipv6
set protocols openfabric domain DATACENTER interface lo address-family ipv6

# Redistribute IPv6 routes
set protocols openfabric domain DATACENTER redistribute ipv6 connected
set protocols openfabric domain DATACENTER redistribute ipv6 static

# Dual-stack configuration
set protocols openfabric domain DATACENTER interface eth1 address-family ipv4
set protocols openfabric domain DATACENTER interface eth1 address-family ipv6

Flooding Optimization

OpenFabric includes flooding optimization mechanisms to reduce redundant traffic in Spine-Leaf topologies.

How It Works

In traditional IS-IS, each router sends LSPs to all of its neighbors. In a Spine-Leaf topology, this creates redundancy:

  • A Leaf sends LSPs to all Spines
  • Each Spine sends the LSPs back to all Leaves
  • Result: the same LSPs are transmitted multiple times

OpenFabric with correctly configured Tiers optimizes this process:

  • A Leaf (Tier 2) sends LSPs to only one Spine (Tier 1)
  • The Spine distributes the LSPs to the other Spines and to all Leaves
  • A significant reduction in control-plane traffic

Configuration for Optimization

# Spine configuration
set protocols openfabric domain DATACENTER fabric-tier 1

# Leaf configuration
set protocols openfabric domain DATACENTER fabric-tier 2

Important: All routers at the same physical layer must have the same Fabric Tier.

Configuration Examples

Example 1: Yandex Cloud - OpenFabric for a Clos Data Center

Architecture: 3 Spine routers and 4 Leaf routers in Yandex Cloud.

Topology

                   Yandex Cloud VPC
┌────────────────────────────────────────────────────────┐
│                                                          │
│  Spine Layer (AS 64512)                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Spine-1    │  │   Spine-2    │  │   Spine-3    │ │
│  │ 10.0.1.1/32  │  │ 10.0.1.2/32  │  │ 10.0.1.3/32  │ │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘ │
│         │                  │                  │          │
│         └─────────┬────────┴────────┬─────────┘         │
│                   │                 │                    │
│  Leaf Layer       │                 │                    │
│  ┌────────────────▼──┐  ┌──────────▼────────┐          │
│  │     Leaf-1        │  │      Leaf-2       │          │
│  │   10.0.2.1/32     │  │   10.0.2.2/32     │          │
│  └───────────────────┘  └───────────────────┘          │
│  ┌───────────────────┐  ┌───────────────────┐          │
│  │     Leaf-3        │  │      Leaf-4       │          │
│  │   10.0.2.3/32     │  │   10.0.2.4/32     │          │
│  └───────────────────┘  └───────────────────┘          │
│                                                          │
│  Subnet for Spine-Leaf links: 10.0.100.0/24            │
│  Subnet for Server links: 10.0.200.0/22                │
└────────────────────────────────────────────────────────┘

Spine-1 Configuration

# Interfaces
set interfaces ethernet eth0 address 10.0.100.1/31
set interfaces ethernet eth1 address 10.0.100.9/31
set interfaces ethernet eth2 address 10.0.100.17/31
set interfaces ethernet eth3 address 10.0.100.25/31
set interfaces loopback lo address 10.0.1.1/32

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0100.0000.1001.00
set protocols openfabric domain YANDEX fabric-tier 1

# Enable interfaces in OpenFabric
set protocols openfabric domain YANDEX interface eth0 address-family ipv4
set protocols openfabric domain YANDEX interface eth1 address-family ipv4
set protocols openfabric domain YANDEX interface eth2 address-family ipv4
set protocols openfabric domain YANDEX interface eth3 address-family ipv4
set protocols openfabric domain YANDEX interface lo address-family ipv4
set protocols openfabric domain YANDEX interface lo passive

# Timer optimization for the data center
set protocols openfabric domain YANDEX interface eth0 hello-interval 3
set protocols openfabric domain YANDEX interface eth0 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth1 hello-interval 3
set protocols openfabric domain YANDEX interface eth1 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth2 hello-interval 3
set protocols openfabric domain YANDEX interface eth2 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth3 hello-interval 3
set protocols openfabric domain YANDEX interface eth3 hello-multiplier 3

# Metrics (identical for ECMP)
set protocols openfabric domain YANDEX interface eth0 metric 10
set protocols openfabric domain YANDEX interface eth1 metric 10
set protocols openfabric domain YANDEX interface eth2 metric 10
set protocols openfabric domain YANDEX interface eth3 metric 10

# LSP parameters
set protocols openfabric domain YANDEX lsp-gen-interval 1
set protocols openfabric domain YANDEX lsp-refresh-interval 600
set protocols openfabric domain YANDEX max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain YANDEX domain-password md5 YandexCloudSecret2024

commit
save

Spine-2 Configuration

# Interfaces
set interfaces ethernet eth0 address 10.0.100.3/31
set interfaces ethernet eth1 address 10.0.100.11/31
set interfaces ethernet eth2 address 10.0.100.19/31
set interfaces ethernet eth3 address 10.0.100.27/31
set interfaces loopback lo address 10.0.1.2/32

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0100.0000.1002.00
set protocols openfabric domain YANDEX fabric-tier 1

# Enable interfaces
set protocols openfabric domain YANDEX interface eth0 address-family ipv4
set protocols openfabric domain YANDEX interface eth1 address-family ipv4
set protocols openfabric domain YANDEX interface eth2 address-family ipv4
set protocols openfabric domain YANDEX interface eth3 address-family ipv4
set protocols openfabric domain YANDEX interface lo address-family ipv4
set protocols openfabric domain YANDEX interface lo passive

# Timers and metrics (identical to Spine-1)
set protocols openfabric domain YANDEX interface eth0 hello-interval 3
set protocols openfabric domain YANDEX interface eth0 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth0 metric 10
set protocols openfabric domain YANDEX interface eth1 hello-interval 3
set protocols openfabric domain YANDEX interface eth1 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth1 metric 10
set protocols openfabric domain YANDEX interface eth2 hello-interval 3
set protocols openfabric domain YANDEX interface eth2 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth2 metric 10
set protocols openfabric domain YANDEX interface eth3 hello-interval 3
set protocols openfabric domain YANDEX interface eth3 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth3 metric 10

# LSP parameters
set protocols openfabric domain YANDEX lsp-gen-interval 1
set protocols openfabric domain YANDEX lsp-refresh-interval 600
set protocols openfabric domain YANDEX max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain YANDEX domain-password md5 YandexCloudSecret2024

commit
save

Spine-3 Configuration

# Interfaces
set interfaces ethernet eth0 address 10.0.100.5/31
set interfaces ethernet eth1 address 10.0.100.13/31
set interfaces ethernet eth2 address 10.0.100.21/31
set interfaces ethernet eth3 address 10.0.100.29/31
set interfaces loopback lo address 10.0.1.3/32

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0100.0000.1003.00
set protocols openfabric domain YANDEX fabric-tier 1

# Enable interfaces
set protocols openfabric domain YANDEX interface eth0 address-family ipv4
set protocols openfabric domain YANDEX interface eth1 address-family ipv4
set protocols openfabric domain YANDEX interface eth2 address-family ipv4
set protocols openfabric domain YANDEX interface eth3 address-family ipv4
set protocols openfabric domain YANDEX interface lo address-family ipv4
set protocols openfabric domain YANDEX interface lo passive

# Timers and metrics
set protocols openfabric domain YANDEX interface eth0 hello-interval 3
set protocols openfabric domain YANDEX interface eth0 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth0 metric 10
set protocols openfabric domain YANDEX interface eth1 hello-interval 3
set protocols openfabric domain YANDEX interface eth1 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth1 metric 10
set protocols openfabric domain YANDEX interface eth2 hello-interval 3
set protocols openfabric domain YANDEX interface eth2 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth2 metric 10
set protocols openfabric domain YANDEX interface eth3 hello-interval 3
set protocols openfabric domain YANDEX interface eth3 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth3 metric 10

# LSP parameters
set protocols openfabric domain YANDEX lsp-gen-interval 1
set protocols openfabric domain YANDEX lsp-refresh-interval 600
set protocols openfabric domain YANDEX max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain YANDEX domain-password md5 YandexCloudSecret2024

commit
save

Leaf-1 Configuration

# Interfaces to Spine
set interfaces ethernet eth0 address 10.0.100.0/31
set interfaces ethernet eth1 address 10.0.100.2/31
set interfaces ethernet eth2 address 10.0.100.4/31

# Interface to servers
set interfaces ethernet eth3 address 10.0.200.1/24

# Loopback
set interfaces loopback lo address 10.0.2.1/32

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0100.0000.2001.00
set protocols openfabric domain YANDEX fabric-tier 2

# Enable uplink interfaces
set protocols openfabric domain YANDEX interface eth0 address-family ipv4
set protocols openfabric domain YANDEX interface eth1 address-family ipv4
set protocols openfabric domain YANDEX interface eth2 address-family ipv4
set protocols openfabric domain YANDEX interface lo address-family ipv4
set protocols openfabric domain YANDEX interface lo passive

# Timers and metrics for uplinks
set protocols openfabric domain YANDEX interface eth0 hello-interval 3
set protocols openfabric domain YANDEX interface eth0 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth0 metric 10
set protocols openfabric domain YANDEX interface eth1 hello-interval 3
set protocols openfabric domain YANDEX interface eth1 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth1 metric 10
set protocols openfabric domain YANDEX interface eth2 hello-interval 3
set protocols openfabric domain YANDEX interface eth2 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth2 metric 10

# Redistribute the connected server network
set protocols openfabric domain YANDEX redistribute ipv4 connected route-map SERVER-NETWORKS

# Route-map for filtering
set policy route-map SERVER-NETWORKS rule 10 action permit
set policy route-map SERVER-NETWORKS rule 10 match interface eth3

# LSP parameters
set protocols openfabric domain YANDEX lsp-gen-interval 1
set protocols openfabric domain YANDEX lsp-refresh-interval 600
set protocols openfabric domain YANDEX max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain YANDEX domain-password md5 YandexCloudSecret2024

commit
save

Leaf-2 Configuration

# Interfaces to Spine
set interfaces ethernet eth0 address 10.0.100.8/31
set interfaces ethernet eth1 address 10.0.100.10/31
set interfaces ethernet eth2 address 10.0.100.12/31

# Interface to servers
set interfaces ethernet eth3 address 10.0.201.1/24

# Loopback
set interfaces loopback lo address 10.0.2.2/32

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0100.0000.2002.00
set protocols openfabric domain YANDEX fabric-tier 2

# Enable interfaces (same as Leaf-1)
set protocols openfabric domain YANDEX interface eth0 address-family ipv4
set protocols openfabric domain YANDEX interface eth1 address-family ipv4
set protocols openfabric domain YANDEX interface eth2 address-family ipv4
set protocols openfabric domain YANDEX interface lo address-family ipv4
set protocols openfabric domain YANDEX interface lo passive

# Timers and metrics
set protocols openfabric domain YANDEX interface eth0 hello-interval 3
set protocols openfabric domain YANDEX interface eth0 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth0 metric 10
set protocols openfabric domain YANDEX interface eth1 hello-interval 3
set protocols openfabric domain YANDEX interface eth1 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth1 metric 10
set protocols openfabric domain YANDEX interface eth2 hello-interval 3
set protocols openfabric domain YANDEX interface eth2 hello-multiplier 3
set protocols openfabric domain YANDEX interface eth2 metric 10

# Redistribute server networks
set protocols openfabric domain YANDEX redistribute ipv4 connected route-map SERVER-NETWORKS
set policy route-map SERVER-NETWORKS rule 10 action permit
set policy route-map SERVER-NETWORKS rule 10 match interface eth3

# LSP parameters
set protocols openfabric domain YANDEX lsp-gen-interval 1
set protocols openfabric domain YANDEX lsp-refresh-interval 600
set protocols openfabric domain YANDEX max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain YANDEX domain-password md5 YandexCloudSecret2024

commit
save

Leaf-3 and Leaf-4 Configuration

The configuration is similar to Leaf-1 and Leaf-2 with the corresponding changes:

  • Leaf-3: NET 49.0001.0100.0000.2003.00, Loopback 10.0.2.3/32, Server subnet 10.0.202.0/24
  • Leaf-4: NET 49.0001.0100.0000.2004.00, Loopback 10.0.2.4/32, Server subnet 10.0.203.0/24

Example 2: VK Cloud - Spine-Leaf Fabric with OpenFabric

Architecture: 2 Spine and 2 Leaf for VK Cloud with IPv4 and IPv6 support.

Topology

                   VK Cloud VPC
┌─────────────────────────────────────────────────────┐
│                                                       │
│  Spine Layer                                         │
│  ┌─────────────────┐       ┌─────────────────┐     │
│  │    Spine-A      │       │    Spine-B      │     │
│  │  10.10.0.1/32   │       │  10.10.0.2/32   │     │
│  │  2001:db8::1/128│       │  2001:db8::2/128│     │
│  └────────┬────────┘       └────────┬────────┘     │
│           │                          │               │
│           └──────────┬───────────────┘              │
│                      │                               │
│  Leaf Layer          │                               │
│  ┌──────────────────▼───┐   ┌──────────────────┐   │
│  │      Leaf-A          │   │      Leaf-B      │   │
│  │   10.10.1.1/32       │   │   10.10.1.2/32   │   │
│  │   2001:db8::11/128   │   │   2001:db8::12/128│  │
│  └──────────────────────┘   └──────────────────┘   │
│                                                      │
│  P2P Links: 10.10.100.0/24, 2001:db8:100::/64      │
│  Server Networks: 10.10.200.0/22, 2001:db8:200::/56│
└─────────────────────────────────────────────────────┘

Spine-A Configuration (VK Cloud)

# Interfaces
set interfaces ethernet eth0 address 10.10.100.1/31
set interfaces ethernet eth0 address 2001:db8:100::1/127
set interfaces ethernet eth1 address 10.10.100.3/31
set interfaces ethernet eth1 address 2001:db8:100::3/127
set interfaces loopback lo address 10.10.0.1/32
set interfaces loopback lo address 2001:db8::1/128

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0101.0000.0001.00
set protocols openfabric domain VKCLOUD fabric-tier 1

# Enable interfaces with dual-stack
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv6
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv6
set protocols openfabric domain VKCLOUD interface lo address-family ipv4
set protocols openfabric domain VKCLOUD interface lo address-family ipv6
set protocols openfabric domain VKCLOUD interface lo passive

# Optimization for low latency
set protocols openfabric domain VKCLOUD interface eth0 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth0 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth0 metric 10
set protocols openfabric domain VKCLOUD interface eth1 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth1 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth1 metric 10

# LSP parameters
set protocols openfabric domain VKCLOUD lsp-gen-interval 1
set protocols openfabric domain VKCLOUD lsp-refresh-interval 900
set protocols openfabric domain VKCLOUD max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain VKCLOUD domain-password md5 VKCloudFabric2024

commit
save

Spine-B Configuration (VK Cloud)

# Interfaces
set interfaces ethernet eth0 address 10.10.100.5/31
set interfaces ethernet eth0 address 2001:db8:100::5/127
set interfaces ethernet eth1 address 10.10.100.7/31
set interfaces ethernet eth1 address 2001:db8:100::7/127
set interfaces loopback lo address 10.10.0.2/32
set interfaces loopback lo address 2001:db8::2/128

# OpenFabric configuration (same as Spine-A)
set protocols openfabric net 49.0001.0101.0000.0002.00
set protocols openfabric domain VKCLOUD fabric-tier 1

# Dual-stack interfaces
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv6
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv6
set protocols openfabric domain VKCLOUD interface lo address-family ipv4
set protocols openfabric domain VKCLOUD interface lo address-family ipv6
set protocols openfabric domain VKCLOUD interface lo passive

# Timers
set protocols openfabric domain VKCLOUD interface eth0 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth0 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth0 metric 10
set protocols openfabric domain VKCLOUD interface eth1 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth1 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth1 metric 10

# LSP parameters
set protocols openfabric domain VKCLOUD lsp-gen-interval 1
set protocols openfabric domain VKCLOUD lsp-refresh-interval 900
set protocols openfabric domain VKCLOUD max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain VKCLOUD domain-password md5 VKCloudFabric2024

commit
save

Leaf-A Configuration (VK Cloud)

# Uplink interfaces
set interfaces ethernet eth0 address 10.10.100.0/31
set interfaces ethernet eth0 address 2001:db8:100::0/127
set interfaces ethernet eth1 address 10.10.100.4/31
set interfaces ethernet eth1 address 2001:db8:100::4/127

# Server-facing interface
set interfaces ethernet eth2 address 10.10.200.1/24
set interfaces ethernet eth2 address 2001:db8:200:1::1/64

# Loopback
set interfaces loopback lo address 10.10.1.1/32
set interfaces loopback lo address 2001:db8::11/128

# OpenFabric basic configuration
set protocols openfabric net 49.0001.0101.0000.1001.00
set protocols openfabric domain VKCLOUD fabric-tier 2

# Uplink interfaces in OpenFabric
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv6
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv6
set protocols openfabric domain VKCLOUD interface lo address-family ipv4
set protocols openfabric domain VKCLOUD interface lo address-family ipv6
set protocols openfabric domain VKCLOUD interface lo passive

# Timers and metrics
set protocols openfabric domain VKCLOUD interface eth0 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth0 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth0 metric 10
set protocols openfabric domain VKCLOUD interface eth1 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth1 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth1 metric 10

# Redistribute server networks (IPv4 and IPv6)
set protocols openfabric domain VKCLOUD redistribute ipv4 connected route-map SERVERS
set protocols openfabric domain VKCLOUD redistribute ipv6 connected route-map SERVERS

# Route-map for server interfaces
set policy route-map SERVERS rule 10 action permit
set policy route-map SERVERS rule 10 match interface eth2

# LSP parameters
set protocols openfabric domain VKCLOUD lsp-gen-interval 1
set protocols openfabric domain VKCLOUD lsp-refresh-interval 900
set protocols openfabric domain VKCLOUD max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain VKCLOUD domain-password md5 VKCloudFabric2024

commit
save

Leaf-B Configuration (VK Cloud)

# Uplink interfaces
set interfaces ethernet eth0 address 10.10.100.2/31
set interfaces ethernet eth0 address 2001:db8:100::2/127
set interfaces ethernet eth1 address 10.10.100.6/31
set interfaces ethernet eth1 address 2001:db8:100::6/127

# Server-facing interface
set interfaces ethernet eth2 address 10.10.201.1/24
set interfaces ethernet eth2 address 2001:db8:200:2::1/64

# Loopback
set interfaces loopback lo address 10.10.1.2/32
set interfaces loopback lo address 2001:db8::12/128

# OpenFabric configuration (same as Leaf-A)
set protocols openfabric net 49.0001.0101.0000.1002.00
set protocols openfabric domain VKCLOUD fabric-tier 2

# Dual-stack uplinks
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth0 address-family ipv6
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv4
set protocols openfabric domain VKCLOUD interface eth1 address-family ipv6
set protocols openfabric domain VKCLOUD interface lo address-family ipv4
set protocols openfabric domain VKCLOUD interface lo address-family ipv6
set protocols openfabric domain VKCLOUD interface lo passive

# Timers
set protocols openfabric domain VKCLOUD interface eth0 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth0 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth0 metric 10
set protocols openfabric domain VKCLOUD interface eth1 hello-interval 2
set protocols openfabric domain VKCLOUD interface eth1 hello-multiplier 3
set protocols openfabric domain VKCLOUD interface eth1 metric 10

# Redistribute server networks
set protocols openfabric domain VKCLOUD redistribute ipv4 connected route-map SERVERS
set protocols openfabric domain VKCLOUD redistribute ipv6 connected route-map SERVERS
set policy route-map SERVERS rule 10 action permit
set policy route-map SERVERS rule 10 match interface eth2

# LSP parameters
set protocols openfabric domain VKCLOUD lsp-gen-interval 1
set protocols openfabric domain VKCLOUD lsp-refresh-interval 900
set protocols openfabric domain VKCLOUD max-lsp-lifetime 1200

# Authentication
set protocols openfabric domain VKCLOUD domain-password md5 VKCloudFabric2024

commit
save

Verification and Monitoring

Checking OpenFabric Neighbors

# Show all OpenFabric neighbors
show openfabric neighbor

# Show neighbors for a specific domain
show openfabric domain DATACENTER neighbor

# Detailed information about a specific neighbor
show openfabric neighbor detail

Example output:

Area DATACENTER:
 System Id           Interface   L  State        Holdtime SNPA
 0100.0000.1002      eth0        2  Up           27       2020.2020.2020
 0100.0000.1003      eth1        2  Up           25       2020.2020.2021

Checking the Topology Database

# Show the LSP database
show openfabric database

# Detailed database
show openfabric database detail

# Database for a specific domain
show openfabric domain DATACENTER database

Checking Routes

# Show routes learned via OpenFabric
show ip route openfabric

# Show OpenFabric IPv6 routes
show ipv6 route openfabric

# Show the entire routing table
show ip route

Example output:

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

F>* 10.0.2.1/32 [115/20] via 10.0.100.0, eth0, weight 1, 00:15:23
  *                      via 10.0.100.2, eth1, weight 1, 00:15:23
  *                      via 10.0.100.4, eth2, weight 1, 00:15:23
F>* 10.0.200.0/24 [115/20] via 10.0.100.0, eth0, weight 1, 00:15:23

Checking Interfaces

# Show OpenFabric interface information
show openfabric interface

# Detailed interface information
show openfabric interface eth0

# Show the status of domain interfaces
show openfabric domain DATACENTER interface

Checking the Topology

# Show SPF statistics
show openfabric spf-delay-ietf

# Show hostname mapping (if configured)
show openfabric hostname

# Show summary information
show openfabric summary

Monitoring the Fabric Tier

# Check the router's fabric tier
show openfabric domain DATACENTER summary | grep Tier

# Check tier information in the database
show openfabric database detail | grep -A 5 "Fabric Tier"

Troubleshooting

Problem: Neighbors Do Not Come Up

Diagnostics:

# Check the interface configuration
show configuration commands | grep openfabric

# Check the interface state
show interfaces

# Check authentication
show openfabric database detail | grep Authentication

# Debug adjacency
monitor log

Common causes:

  1. Authentication mismatch
  2. Incorrect interface configuration
  3. MTU mismatch
  4. Physical link problems

Solution:

# Check the MTU
show interfaces ethernet eth0

# Temporarily disable authentication for testing
delete protocols openfabric domain DATACENTER domain-password
commit

# Check hello packets
sudo tcpdump -i eth0 -vv proto 124

Problem: Routes Do Not Appear in the Table

Diagnostics:

# Check the LSP database
show openfabric database

# Check redistribution
show configuration commands | grep redistribute

# Check the route-map
show route-map

# Check the routing table
show ip route openfabric

Common causes:

  1. Incorrect redistribution
  2. A route-map is blocking routes
  3. NET configuration problems
  4. The interface is not enabled in passive mode

Solution:

# Verify that connected networks are being redistributed
set protocols openfabric domain DATACENTER redistribute ipv4 connected

# Verify passive mode on the loopback
set protocols openfabric domain DATACENTER interface lo passive

# Check the route-map
show policy route-map SERVERS

Problem: ECMP Paths Are Not Used

Diagnostics:

# Check interface metrics
show openfabric interface

# Check the FIB
show ip route 10.0.2.1

# Check the kernel routing table
ip route show

Common causes:

  1. Different metrics on the interfaces
  2. Incorrect Fabric Tier configuration
  3. The kernel does not support multipath

Solution:

# Make sure the metrics are identical
set protocols openfabric domain DATACENTER interface eth0 metric 10
set protocols openfabric domain DATACENTER interface eth1 metric 10
set protocols openfabric domain DATACENTER interface eth2 metric 10

# Check the Fabric Tier
set protocols openfabric domain DATACENTER fabric-tier 2

Problem: High CPU Load Due to LSP Flooding

Diagnostics:

# Monitor CPU
top

# Check the LSP generation rate
show openfabric database detail | grep "Generated"

# Check the Fabric Tier configuration
show configuration commands | grep fabric-tier

Solution:

# Increase the LSP generation interval
set protocols openfabric domain DATACENTER lsp-gen-interval 5

# Increase the refresh interval
set protocols openfabric domain DATACENTER lsp-refresh-interval 900

# Configure the Fabric Tier correctly
# Spine
set protocols openfabric domain DATACENTER fabric-tier 1
# Leaf
set protocols openfabric domain DATACENTER fabric-tier 2

Problem: Frequent Neighbor Flapping

Diagnostics:

# Monitor logs
monitor log | match openfabric

# Check hello parameters
show openfabric interface eth0

# Check link quality
ping 10.0.100.1 size 1400 count 100

Solution:

# Increase the hello interval and multiplier
set protocols openfabric domain DATACENTER interface eth0 hello-interval 5
set protocols openfabric domain DATACENTER interface eth0 hello-multiplier 5

# Reduce the MTU if there is fragmentation
set interfaces ethernet eth0 mtu 1400

Debug Commands

# Enable debug for OpenFabric
# Warning: generates a lot of logs
sudo vtysh -c "debug openfabric events"
sudo vtysh -c "debug openfabric adj-packets"
sudo vtysh -c "debug openfabric lsp-gen"
sudo vtysh -c "debug openfabric update-packets"

# View FRR logs
tail -f /var/log/frr/frr.log

# Disable debug
sudo vtysh -c "no debug openfabric events"
sudo vtysh -c "no debug openfabric adj-packets"

Best Practices

Network Design

  1. Fabric Tier configuration

    • Always configure the Fabric Tier to optimize flooding
    • Spine: Tier 1, Leaf: Tier 2
    • All routers at the same physical layer must have the same Tier

  2. NET addressing

    • Use a consistent scheme for the System ID
    • Recommended: the last 6 bytes of the loopback IP address
    • Document your NET addressing scheme

  3. Metrics

    • Use identical metrics on all uplinks for ECMP
    • Change the metric only for traffic engineering

  4. Loopback interfaces

    • Always include the loopback in OpenFabric
    • Always use passive mode for the loopback
    • Use /32 (IPv4) or /128 (IPv6) for the loopback

Timers and Optimization

  1. Hello parameters for the data center

    • Hello interval: 2-3 seconds
    • Hello multiplier: 3
    • Dead interval: 6-9 seconds (fast failure detection)

  2. LSP parameters

    • lsp-gen-interval: 1-2 milliseconds (fast reaction)
    • lsp-refresh-interval: 600-900 seconds
    • max-lsp-lifetime: 1200-1800 seconds

  3. CSNP/PSNP

    • Use the default values
    • Change them only for specific problems

Security

  1. Authentication

    • Always use MD5 authentication
    • Use strong passwords
    • Rotate passwords periodically
    • You can use different passwords at the domain and interface levels

  2. Restricting the protocol

    • Enable OpenFabric only on the necessary interfaces
    • Use passive mode for non-fabric interfaces

Scalability

  1. Fabric size

    • Recommended: up to 512 Leaves per 3-4 Spines
    • For larger deployments, consider a Multi-POD architecture

  2. Number of routes

    • OpenFabric is efficient up to tens of thousands of routes
    • For very large tables, consider summarization

  3. ECMP paths

    • Typical: 4-8 ECMP paths from Leaf to Spine
    • Can be scaled to 16-32 paths

Operational Procedures

  1. Router maintenance

    # Set the overload bit before maintenance
set protocols openfabric domain DATACENTER overload
commit

# Perform maintenance

# Clear the overload bit
delete protocols openfabric domain DATACENTER overload
commit

  2. Graceful restart

    # Overload for a specific period
set protocols openfabric domain DATACENTER overload on-startup 300
commit

  3. Monitoring

    • Monitor the number of neighbors
    • Monitor the number of routes
    • Set up alerts for neighbor flapping
    • Monitor CPU utilization on Spines

BGP Integration

To connect the fabric to external networks, use BGP on the Leaf or Spine:

# Border Leaf configuration
set protocols bgp system-as 65000
set protocols bgp neighbor 192.0.2.1 remote-as 65001
set protocols bgp neighbor 192.0.2.1 address-family ipv4-unicast

# Redistribute OpenFabric into BGP
set protocols bgp address-family ipv4-unicast redistribute openfabric

# Redistribute BGP into OpenFabric
set protocols openfabric domain DATACENTER redistribute ipv4 bgp route-map BGP-TO-FABRIC

Dual-stack Recommendations

  1. Enable IPv4 and IPv6 on all interfaces simultaneously
  2. Use identical metrics for IPv4 and IPv6
  3. Apply the same route-map to both address families
  4. Test failover for both protocols

Conclusion

OpenFabric is an efficient routing protocol for modern data centers, optimized for the Spine-Leaf architecture. Proper configuration of the Fabric Tier, metrics, and timers ensures fast convergence, optimal use of ECMP paths, and minimal control-plane traffic overhead.

Key points for a successful deployment:

  • Proper Fabric Tier configuration (Spine = Tier 1, Leaf = Tier 2)
  • Identical metrics on all uplinks for ECMP
  • Aggressive timers for fast failure detection
  • Use of MD5 authentication
  • Regular monitoring of neighbors and routes

OpenFabric in VyOS provides enterprise-grade functionality for building fault-tolerant, scalable data center networks in the Yandex Cloud and VK Cloud cloud environments.

Reviewed by OpenNix LLC · Last updated on
Segment Routing in VyOSPIM - Protocol Independent Multicast (IPv4)