Documentation
VyOS Documentation
VyOS Interfaces
VXLAN Interfaces in VyOS

VXLAN Interfaces in VyOS

VXLAN (Virtual Extensible LAN) is a network virtualization technology for building overlay networks on top of an existing IP infrastructure.

Overview

VXLAN solves the scaling problems of traditional VLAN networks:

  • Encapsulates Layer 2 Ethernet frames into Layer 4 UDP datagrams
  • Uses a 24-bit VNI (VXLAN Network Identifier) instead of a 12-bit VLAN ID
  • Supports up to 16 million logical networks (compared to 4096 VLANs)
  • Operates over an existing IP network (underlay)
  • Lets you stretch L2 segments across L3 boundaries

Key components:

  • VNI (VXLAN Network Identifier) - the virtual network identifier (24 bits)
  • VTEP (VXLAN Tunnel Endpoint) - the encapsulation/decapsulation point
  • Underlay network - the physical IP network that carries the encapsulated traffic
  • Overlay network - the virtual L2 network on top of the underlay

UDP port:

  • VyOS default: 8472
  • IANA standard: 4789

Use cases:

  • Network virtualization in cloud providers
  • Multi-tenant environments
  • Data center interconnect (DCI)
  • Linking geographically distributed L2 domains
  • Kubernetes/Docker overlay networks

Basic Configuration

Creating a VXLAN Interface

set interfaces vxlan vxlan100 vni 100
commit

Minimal configuration - only the VNI (network identifier).

Assigning an IP Address

set interfaces vxlan vxlan100 address 10.0.100.1/24
commit

Interface Description

set interfaces vxlan vxlan100 description 'VXLAN Overlay Network 100'
commit

MTU

It is recommended to reduce the MTU because of the encapsulation overhead:

set interfaces vxlan vxlan100 mtu 1450
commit

VXLAN adds 50 bytes of overhead:

  • 20 bytes outer IP header
  • 8 bytes UDP header
  • 8 bytes VXLAN header
  • 14 bytes outer Ethernet header

For the standard MTU of 1500:

  • VXLAN MTU = 1500 - 50 = 1450

For jumbo frames (9000):

  • VXLAN MTU = 9000 - 50 = 8950

VXLAN Operating Modes

1. Multicast VXLAN

Uses multicast groups for automatic VTEP discovery and for flooding BUM traffic (Broadcast, Unknown unicast, Multicast).

Configuration:

set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 mtu 1450
set interfaces vxlan vxlan100 group 239.0.0.100
commit

Parameters:

  • vni - the VXLAN network identifier
  • source-interface - the interface for underlay traffic
  • group - the multicast group for this VNI
  • mtu - the maximum packet size

How it works:

  1. BUM traffic is sent to the multicast group
  2. All VTEPs in the group receive the traffic
  3. VTEPs learn MAC addresses and build the FDB (Forwarding Database)
  4. Unicast traffic goes directly between VTEPs

Requirements:

  • The underlay network must support multicast (PIM)
  • Not all cloud providers support multicast

2. Unicast VXLAN (Point-to-Point)

Explicitly specifies the remote VTEP by IP address. Does not require multicast.

Configuration:

set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 remote 203.0.113.10
set interfaces vxlan vxlan100 mtu 1450
commit

Parameters:

  • remote - the IP address of the remote VTEP

How it works:

  • All traffic is sent to the specified remote IP
  • Suitable for point-to-point connections
  • Does not scale for a large number of VTEPs

Advantages:

  • Works in networks without multicast (most clouds)
  • Simple configuration for a small number of nodes
  • Deterministic behavior

Disadvantages:

  • Does not support automatic VTEP discovery
  • Requires manual configuration for each VTEP pair
  • Scales poorly

3. Unicast VXLAN with Multiple Remote Endpoints

To connect to multiple VTEPs, use an FDB (Forwarding Database) or a bridge with FDB entries.

set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0

set interfaces bridge br100
set interfaces bridge br100 member interface vxlan100

# Static FDB entries (MAC to VTEP mapping)
# Syntax depends on the VyOS version
# Alternatively, use BGP EVPN for automatic distribution
commit

4. VXLAN via source-address

Instead of source-interface, you can specify the source IP directly:

set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 198.51.100.5
set interfaces vxlan vxlan100 remote 203.0.113.10
commit

Useful when you have multiple interfaces or a loopback address.

VLAN-to-VNI Mapping (Single VXLAN Device)

A Single VXLAN Device (SVD) lets you use one VXLAN interface for multiple VNIs by mapping VLANs to VNIs.

SVD Configuration

# VXLAN interface with a base VNI
set interfaces vxlan vxlan0 vni 10000
set interfaces vxlan vxlan0 source-interface eth0

# Bridge for VLAN management
set interfaces bridge br0
set interfaces bridge br0 member interface vxlan0
set interfaces bridge br0 enable-vlan

# VLAN-to-VNI mapping
set interfaces bridge br0 member interface vxlan0 vlan-to-vni 10 vni 10010
set interfaces bridge br0 member interface vxlan0 vlan-to-vni 20 vni 10020
set interfaces bridge br0 member interface vxlan0 vlan-to-vni 30 vni 10030

# Physical interface for local VLANs
set interfaces ethernet eth1 vif 10
set interfaces ethernet eth1 vif 20
set interfaces ethernet eth1 vif 30

set interfaces bridge br0 member interface eth1.10 allowed-vlan 10
set interfaces bridge br0 member interface eth1.20 allowed-vlan 20
set interfaces bridge br0 member interface eth1.30 allowed-vlan 30

commit

How it works:

  • VLAN 10 is locally mapped to VNI 10010
  • VLAN 20 is locally mapped to VNI 10020
  • VLAN 30 is locally mapped to VNI 10030
  • A single VXLAN interface serves all VNIs
  • The bridge performs VLAN-aware switching

Advantages:

  • Scales to thousands of VNIs on a single interface
  • Simplified configuration
  • Lower system resource usage

Configuration Examples

Point-to-Point VXLAN (Yandex Cloud)

Connecting two VyOS instances in different Yandex Cloud subnets:

Node 1 (192.0.2.10):

# Underlay configuration
set interfaces ethernet eth0 address 192.0.2.10/24

# VXLAN tunnel
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 192.0.2.10
set interfaces vxlan vxlan100 remote 192.0.2.20
set interfaces vxlan vxlan100 mtu 1450

# Overlay address
set interfaces vxlan vxlan100 address 10.100.0.1/24
set interfaces vxlan vxlan100 description 'VXLAN to Node2'

commit

Node 2 (192.0.2.20):

# Underlay configuration
set interfaces ethernet eth0 address 192.0.2.20/24

# VXLAN tunnel
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 192.0.2.20
set interfaces vxlan vxlan100 remote 192.0.2.10
set interfaces vxlan vxlan100 mtu 1450

# Overlay address
set interfaces vxlan vxlan100 address 10.100.0.2/24
set interfaces vxlan vxlan100 description 'VXLAN to Node1'

commit

Now the nodes can communicate over the 10.100.0.0/24 overlay network.

VXLAN Bridge for L2 Stretching (VK Cloud)

Stretching an L2 domain between two data centers:

Site A (198.51.100.10):

# VXLAN interface
set interfaces vxlan vxlan200 vni 200
set interfaces vxlan vxlan200 source-address 198.51.100.10
set interfaces vxlan vxlan200 remote 203.0.113.10
set interfaces vxlan vxlan200 mtu 1450

# Bridge with a local interface
set interfaces bridge br200
set interfaces bridge br200 member interface vxlan200
set interfaces bridge br200 member interface eth1
set interfaces bridge br200 description 'Stretched L2 VLAN 200'

# IP for inter-site routing (optional)
set interfaces bridge br200 address 10.200.0.1/24

commit

Site B (203.0.113.10):

# VXLAN interface
set interfaces vxlan vxlan200 vni 200
set interfaces vxlan vxlan200 source-address 203.0.113.10
set interfaces vxlan vxlan200 remote 198.51.100.10
set interfaces vxlan vxlan200 mtu 1450

# Bridge with a local interface
set interfaces bridge br200
set interfaces bridge br200 member interface vxlan200
set interfaces bridge br200 member interface eth1
set interfaces bridge br200 description 'Stretched L2 VLAN 200'

# IP for inter-site routing (optional)
set interfaces bridge br200 address 10.200.0.2/24

commit

Devices on eth1 at both sites are in the same L2 domain.

Multi-tenant VXLAN (cloud provider)

Isolating traffic from different customers using separate VNIs:

# Customer A - VNI 1000
set interfaces vxlan vxlan1000 vni 1000
set interfaces vxlan vxlan1000 source-interface eth0
set interfaces vxlan vxlan1000 remote 192.0.2.20

set interfaces bridge brA
set interfaces bridge brA member interface vxlan1000
set interfaces bridge brA member interface eth1.100
set interfaces bridge brA address 10.10.0.1/24
set interfaces bridge brA description 'Tenant A'

# Customer B - VNI 2000
set interfaces vxlan vxlan2000 vni 2000
set interfaces vxlan vxlan2000 source-interface eth0
set interfaces vxlan vxlan2000 remote 192.0.2.20

set interfaces bridge brB
set interfaces bridge brB member interface vxlan2000
set interfaces bridge brB member interface eth1.200
set interfaces bridge brB address 10.20.0.1/24
set interfaces bridge brB description 'Tenant B'

# Customer C - VNI 3000
set interfaces vxlan vxlan3000 vni 3000
set interfaces vxlan vxlan3000 source-interface eth0
set interfaces vxlan vxlan3000 remote 192.0.2.20

set interfaces bridge brC
set interfaces bridge brC member interface vxlan3000
set interfaces bridge brC member interface eth1.300
set interfaces bridge brC address 10.30.0.1/24
set interfaces bridge brC description 'Tenant C'

commit

Each customer is fully isolated at the L2 level through a separate VNI.

VXLAN Multicast (on-premises)

For on-premises data centers with multicast support:

# VXLAN with multicast
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 group 239.1.1.100
set interfaces vxlan vxlan100 mtu 1450

# Bridge for local L2
set interfaces bridge br100
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 member interface eth1
set interfaces bridge br100 member interface eth2

commit

Automatic VTEP discovery through the multicast group 239.1.1.100.

VXLAN over loopback (recommended for production)

Using a loopback address for stability:

# Loopback interface
set interfaces loopback lo address 198.51.100.1/32

# VXLAN with source-address on the loopback
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 198.51.100.1
set interfaces vxlan vxlan100 remote 203.0.113.1
set interfaces vxlan vxlan100 mtu 1450

# Route to the remote loopback
set protocols static route 203.0.113.1/32 next-hop 192.0.2.1

commit

A loopback address does not depend on the state of physical interfaces.

VXLAN with IPv6 underlay

VXLAN supports IPv6 for the underlay network:

# IPv6 underlay
set interfaces ethernet eth0 address 2001:db8:1::10/64

# VXLAN over IPv6
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 2001:db8:1::10
set interfaces vxlan vxlan100 remote 2001:db8:2::10
set interfaces vxlan vxlan100 mtu 1450

# IPv4 overlay
set interfaces vxlan vxlan100 address 10.100.0.1/24

commit

VXLAN with firewall

Protecting VXLAN traffic with a firewall:

# VXLAN interface
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 remote 192.0.2.20
set interfaces vxlan vxlan100 address 10.100.0.1/24

# Firewall for overlay traffic
set firewall ipv4 input filter rule 100 action accept
set firewall ipv4 input filter rule 100 source address 10.100.0.0/24
set firewall ipv4 input filter rule 100 description 'Allow VXLAN overlay'

# Firewall for underlay VXLAN (UDP 8472)
set firewall ipv4 input filter rule 200 action accept
set firewall ipv4 input filter rule 200 protocol udp
set firewall ipv4 input filter rule 200 destination port 8472
set firewall ipv4 input filter rule 200 source address 192.0.2.0/24
set firewall ipv4 input filter rule 200 description 'Allow VXLAN encapsulation'

commit

VXLAN Full-Mesh (3 nodes)

Building a fully meshed topology for 3 nodes:

Option 1: Separate VXLAN per pair (not recommended): Too many interfaces for a large number of nodes.

Option 2: Multicast (if supported):

# Identical configuration on all nodes
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 group 239.0.0.100
set interfaces vxlan vxlan100 mtu 1450

set interfaces bridge br100
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 address 10.100.0.X/24

commit

Option 3: BGP EVPN (best practice for production): Use BGP EVPN for automatic distribution of MAC/IP routes and to build a VXLAN fabric.

Integration with Cloud Providers

Yandex Cloud specifics

Limitations:

  • Multicast is not supported
  • Use unicast VXLAN only
  • MTU: 1450 is recommended (Yandex Cloud MTU = 1500)
  • Security groups: allow UDP 8472 between nodes

Security Group configuration:

# In the Yandex Cloud console, create a rule:
Protocol: UDP
Port: 8472
Source: CIDR block with the VTEP node addresses

Example for an HA cluster:

# Node 1 (primary)
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 remote 10.128.0.20
set interfaces vxlan vxlan100 address 172.16.0.1/24

# VRRP for HA
set high-availability vrrp group vxlan100 vrid 100
set high-availability vrrp group vxlan100 interface vxlan100
set high-availability vrrp group vxlan100 address 172.16.0.254/24
set high-availability vrrp group vxlan100 priority 200

commit

VK Cloud specifics

Limitations:

  • Multicast is not supported
  • MTU: 1450 is recommended
  • Firewall: allow UDP 8472

Firewall configuration: In the VK Cloud panel:

Inbound traffic rule:
- Protocol: UDP
- Port: 8472
- Source: VTEP IP addresses

Overlay for Kubernetes:

# VXLAN for the Kubernetes CNI
set interfaces vxlan vxlan10 vni 10
set interfaces vxlan vxlan10 source-interface eth0
set interfaces vxlan vxlan10 remote 10.0.0.20

set interfaces bridge cni0
set interfaces bridge cni0 member interface vxlan10
set interfaces bridge cni0 address 10.244.0.1/24

# IP forwarding for pod routing
# VyOS enables forwarding automatically

commit

General recommendations for clouds

  1. Use unicast VXLAN - multicast is usually not supported
  2. MTU 1450 - to prevent fragmentation
  3. Security groups/Firewall - allow UDP 8472 between VTEPs
  4. Source-address on a loopback - for stability during failover
  5. Monitor the underlay - check the reachability of remote VTEPs
  6. BGP EVPN - for automation in large deployments

UDP Port Configuration

Changing the default port (8472) to the IANA one (4789):

set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 port 4789
set interfaces vxlan vxlan100 source-interface eth0
set interfaces vxlan vxlan100 remote 192.0.2.20
commit

When to change the port:

  • Integration with hardware from other vendors (Cisco, Arista use 4789)
  • Corporate standards require the IANA port
  • Firewall rules are already configured for 4789

Important: The port must match on all VTEPs within the same VNI.

MAC Address

Setting a static MAC address for the VXLAN interface:

set interfaces vxlan vxlan100 mac 00:50:56:00:00:01
commit

May be required for:

  • Integration with legacy systems
  • MAC-based licensing
  • Debugging

ARP and ND

ARP Cache Timeout

set interfaces vxlan vxlan100 ip arp-cache-timeout 3600
commit

Disable ARP

set interfaces vxlan vxlan100 ip disable-arp-filter
commit

IPv6 Neighbor Discovery

set interfaces vxlan vxlan100 ipv6 dup-addr-detect-transmits 1
commit

Operational Commands

Viewing VXLAN Interfaces

show interfaces vxlan

Example output:

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
vxlan100         10.100.0.1/24                     u/u  VXLAN Overlay 100
vxlan200         10.200.0.1/24                     u/u  VXLAN Overlay 200

Detailed VXLAN Information

show interfaces vxlan vxlan100

Example output:

vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 00:50:56:00:12:34 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.1/24 brd 10.100.0.255 scope global vxlan100
    vxlan id 100 remote 192.0.2.20 dev eth0 srcport 0 0 dstport 8472

VXLAN Statistics

show interfaces vxlan vxlan100 statistics

Example output:

RX:  bytes    packets  errors  dropped  overrun  mcast
     1048576  1024     0       0        0        0
TX:  bytes    packets  errors  dropped  carrier  collsns
     2097152  2048     0       0        0        0

FDB (Forwarding Database)

Viewing the MAC addresses learned in VXLAN:

show bridge vxlan vxlan100 fdb

Example output:

port    mac addr           flags
vxlan100  00:0c:29:12:34:56  dst 192.0.2.20
vxlan100  00:0c:29:ab:cd:ef  dst 192.0.2.30

Checking Connectivity

Ping over the overlay network:

ping 10.100.0.2 source-address 10.100.0.1

Traceroute:

traceroute 10.100.0.2 source-address 10.100.0.1

Monitoring Encapsulated Traffic

monitor interfaces vxlan vxlan100 traffic

Capturing VXLAN packets:

monitor traffic interface eth0 filter 'udp port 8472'

Troubleshooting

VXLAN Interface Does Not Come Up

Check the source-interface:

show interfaces ethernet eth0

Make sure the source interface is in the UP state.

Check the configuration:

show configuration interfaces vxlan vxlan100

No Connectivity over VXLAN

1. Check underlay connectivity:

ping 192.0.2.20 source-address 192.0.2.10

If the underlay does not work, VXLAN will not work either.

2. Check the firewall/security groups:

# On the remote node
sudo tcpdump -i eth0 udp port 8472

If packets do not arrive, the problem is in the firewall.

3. Check the VNI: The VNI must match on both ends of the tunnel.

4. Check the MTU:

ping 10.100.0.2 size 1400 do-not-fragment

If packets do not pass, reduce the MTU.

FDB Is Not Learning (MACs Not Visible)

Check the bridge configuration:

show bridge brX fdb

Make sure the VXLAN interface is in the bridge:

show bridge brX

Check the aging time:

show configuration interfaces bridge brX

Broadcast Storm in VXLAN

Possible causes:

  • A loop in the overlay topology
  • Incorrect STP configuration
  • Duplicate MAC addresses

Solution:

# Enable STP on the bridge
set interfaces bridge br100 stp
commit

# Check STP
show bridge br100 spanning-tree

High Packet Fragmentation

Reduce the VXLAN MTU:

set interfaces vxlan vxlan100 mtu 1400
commit

Or increase the underlay MTU (if possible):

set interfaces ethernet eth0 mtu 1600
commit

Remote VTEP Unreachable

Check the route:

show ip route 192.0.2.20

Check ARP:

show arp interface eth0

Traceroute to the remote VTEP:

traceroute 192.0.2.20

VXLAN Works Only in One Direction

Check for a symmetric configuration on both ends:

  • The VNI must match
  • The UDP port must match
  • The remote IP must point to the correct peer

Poor VXLAN Performance

1. Check the CPU load:

show system cpu

VXLAN encapsulation consumes CPU.

2. Enable offloading (if supported):

show interfaces ethernet eth0 offload

3. Use jumbo frames:

# Underlay
set interfaces ethernet eth0 mtu 9000

# VXLAN
set interfaces vxlan vxlan100 mtu 8950

commit

4. Check the underlay bandwidth: There may be a bottleneck in the physical network.

MTU Calculation

Standard MTU (1500)

VXLAN MTU = Underlay MTU - VXLAN Overhead
VXLAN MTU = 1500 - 50 = 1450

Overhead components:

  • Outer Ethernet: 14 bytes
  • Outer IP: 20 bytes (IPv4) or 40 bytes (IPv6)
  • UDP: 8 bytes
  • VXLAN: 8 bytes
  • Total: 50 bytes (IPv4), 70 bytes (IPv6)

Jumbo Frames (9000)

VXLAN MTU = 9000 - 50 = 8950

With a VLAN Tag

VXLAN MTU = 1500 - 50 - 4 = 1446

A VLAN tag adds 4 bytes.

Automatic MTU Detection

# Path MTU Discovery is enabled by default
show interfaces vxlan vxlan100

Performance

Optimization Recommendations

1. Hardware offloading: Use a NIC with VXLAN offload support.

2. Jumbo frames: Increase the MTU to 9000 on the underlay and 8950 on the overlay.

3. Source on a loopback: Use loopback addresses for VTEPs - more stable during failover.

4. Multipath routing: Use ECMP to load-balance VXLAN traffic.

5. CPU pinning: For critical workloads, configure CPU affinity.

Performance Testing

Throughput:

# On the server
iperf3 -s -B 10.100.0.1

# On the client
iperf3 -c 10.100.0.1 -t 60 -P 4

Latency:

ping 10.100.0.2 -i 0.2 -c 1000

Packet loss:

mtr 10.100.0.2 -c 100

Security

VXLAN Encryption

By default, VXLAN does not encrypt traffic. To encrypt it:

Option 1: IPsec over VXLAN:

# Create an IPsec tunnel between the VTEPs
set vpn ipsec site-to-site peer 192.0.2.20 ...

# VXLAN over the IPsec tunnel
set interfaces vxlan vxlan100 source-interface ipsec0

Option 2: WireGuard underlay:

# WireGuard tunnel
set interfaces wireguard wg0 ...

# VXLAN over WireGuard
set interfaces vxlan vxlan100 source-interface wg0

Firewall for VXLAN

Protecting the underlay:

# Allow VXLAN only from known VTEPs
set firewall ipv4 input filter rule 100 action accept
set firewall ipv4 input filter rule 100 protocol udp
set firewall ipv4 input filter rule 100 destination port 8472
set firewall ipv4 input filter rule 100 source address 192.0.2.0/24

set firewall ipv4 input filter rule 110 action drop
set firewall ipv4 input filter rule 110 protocol udp
set firewall ipv4 input filter rule 110 destination port 8472

commit

Protecting the overlay:

# Firewall for overlay traffic
set firewall ipv4 forward filter rule 200 action accept
set firewall ipv4 forward filter rule 200 inbound-interface name vxlan100
set firewall ipv4 forward filter rule 200 state established
set firewall ipv4 forward filter rule 200 state related

commit

Rate Limiting

Protection against flooding:

set firewall ipv4 input filter rule 300 action accept
set firewall ipv4 input filter rule 300 protocol udp
set firewall ipv4 input filter rule 300 destination port 8472
set firewall ipv4 input filter rule 300 limit rate 1000/second
commit

Monitoring

Key Metrics

1. Interface statistics:

show interfaces vxlan vxlan100 statistics

Track:

  • RX/TX errors
  • Dropped packets
  • Overruns

2. FDB size:

show bridge brX fdb | count

3. CPU usage:

show system cpu

VXLAN encapsulation consumes CPU.

4. Underlay health:

ping 192.0.2.20 source-address 192.0.2.10

Logging

Enabling debug logs for VXLAN:

# In VyOS 1.4
set system syslog global facility local7 level debug

# In VyOS 1.5
# See the logging documentation

Viewing the logs:

show log | match vxlan

Integration

With BGP EVPN

BGP EVPN automates the distribution of MAC/IP routes across a VXLAN fabric:

# Loopback for the VTEP
set interfaces loopback lo address 192.0.2.1/32

# VXLAN
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 192.0.2.1

# Bridge
set interfaces bridge br100
set interfaces bridge br100 member interface vxlan100

# BGP for the underlay
set protocols bgp system-as 65000
set protocols bgp neighbor 10.0.0.1 remote-as 65000

# BGP EVPN
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp address-family l2vpn-evpn advertise-default-gw
set protocols bgp address-family l2vpn-evpn neighbor 10.0.0.1 activate

commit

With OSPF (underlay)

OSPF for routing between VTEPs:

# Loopback
set interfaces loopback lo address 192.0.2.1/32

# OSPF
set protocols ospf area 0 network 192.0.2.1/32
set protocols ospf area 0 network 10.0.0.0/24

# VXLAN with source on the loopback
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 192.0.2.1
set interfaces vxlan vxlan100 remote 192.0.2.2

commit

OSPF advertises the loopback addresses, providing underlay connectivity.

With VRF

Isolating the VXLAN overlay in a separate VRF:

# VRF for the tenant
set vrf name TENANT table 100

# VXLAN in the VRF
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 vrf TENANT
set interfaces vxlan vxlan100 address 10.100.0.1/24

commit

With QoS

Prioritizing VXLAN traffic:

# QoS for the VXLAN underlay
set qos policy shaper VXLAN-QOS bandwidth 1gbit
set qos policy shaper VXLAN-QOS class 10 match vxlan ip protocol udp
set qos policy shaper VXLAN-QOS class 10 match vxlan ip destination port 8472
set qos policy shaper VXLAN-QOS class 10 bandwidth 800mbit

set interfaces ethernet eth0 qos egress VXLAN-QOS

commit

Technology Comparison

VXLAN vs VLAN

CharacteristicVXLANVLAN
Identifier24 bits (16M)12 bits (4096)
ScalabilityVery highLimited
Works across L3YesNo
Overhead50 bytes4 bytes
ComplexityHighLow
Use caseData center, cloudCampus, enterprise

VXLAN vs GRE

CharacteristicVXLANGRE
LayerL2 over L3L3 over L3
MulticastSupportedNo
NAT traversalYes (UDP)Limited
IdentifierVNI (24 bits)Key (32 bits)
ScalabilityHighMedium

VXLAN vs MPLS L2VPN

CharacteristicVXLANMPLS L2VPN
ProtocolUDPMPLS
ComplexityMediumHigh
CostLowHigh
Control planeBGP EVPN/MulticastLDP/BGP
Use caseCloud/DCService provider

Best Practices

  1. MTU planning - always subtract 50 bytes from the underlay MTU
  2. Source on a loopback - for stability during failover
  3. Unicast for clouds - multicast is usually not supported
  4. Underlay firewall - allow UDP 8472 only from known VTEPs
  5. FDB monitoring - track the size of the forwarding database
  6. BGP EVPN for production - automation in large deployments
  7. Jumbo frames - if the underlay supports them
  8. VNI plan - document VNI assignments
  9. Overlay security - use a firewall on overlay interfaces
  10. Testing - test failover scenarios

Limitations

  • VXLAN adds 50 bytes of overhead (IPv4) or 70 bytes (IPv6)
  • Encapsulation consumes CPU (hardware offload is recommended)
  • Multicast VXLAN does not work in most cloud providers
  • Careful MTU planning is required
  • FDB tables can grow large in big networks
  • Debugging is harder than with ordinary L2/L3 networks

Examples for Specific Scenarios

Kubernetes Overlay

# VXLAN for the Kubernetes pod network
set interfaces vxlan vxlan-pods vni 1000
set interfaces vxlan vxlan-pods source-interface eth0
set interfaces vxlan vxlan-pods remote 10.0.0.20
set interfaces vxlan vxlan-pods mtu 1450

set interfaces bridge kube-bridge
set interfaces bridge kube-bridge member interface vxlan-pods
set interfaces bridge kube-bridge address 10.244.0.1/24

commit

Multi-DC DCI (Data Center Interconnect)

# DC1 VTEP
set interfaces loopback lo address 198.51.100.1/32
set interfaces vxlan vxlan-dci vni 5000
set interfaces vxlan vxlan-dci source-address 198.51.100.1
set interfaces vxlan vxlan-dci remote 203.0.113.1
set interfaces vxlan vxlan-dci mtu 1450

# QoS for DCI
set qos policy shaper DCI bandwidth 10gbit

commit

NFV Service Chaining

# VNF1 -> VXLAN -> VNF2 -> VXLAN -> VNF3

# VXLAN segments
set interfaces vxlan vxlan-vnf1-vnf2 vni 2001
set interfaces vxlan vxlan-vnf1-vnf2 source-interface eth0
set interfaces vxlan vxlan-vnf1-vnf2 remote 192.0.2.20

set interfaces vxlan vxlan-vnf2-vnf3 vni 2002
set interfaces vxlan vxlan-vnf2-vnf3 source-interface eth0
set interfaces vxlan vxlan-vnf2-vnf3 remote 192.0.2.30

# Bridges for service chaining
set interfaces bridge br-vnf1
set interfaces bridge br-vnf1 member interface vxlan-vnf1-vnf2
set interfaces bridge br-vnf1 member interface eth1

set interfaces bridge br-vnf2
set interfaces bridge br-vnf2 member interface vxlan-vnf2-vnf3
set interfaces bridge br-vnf2 member interface eth2

commit

Next Steps

  • Bridge interfaces - L2 bridging and VLAN-aware mode
  • GRE tunnels - an alternative tunneling technology
  • VRF - routing table isolation for multi-tenancy
  • BGP - the control plane for a VXLAN fabric
  • Firewall - protecting the VXLAN overlay and underlay
  • QoS - prioritizing VXLAN traffic

References

Reviewed by OpenNix LLC · Last updated on
Pseudo-Ethernet (MACVLAN) Interfaces in VyOSDummy Interfaces in VyOS