rfc9469.original   rfc9469.txt 
NVO3 Workgroup J. Rabadan, Ed. Internet Engineering Task Force (IETF) J. Rabadan, Ed.
Internet-Draft M. Bocci Request for Comments: 9469 M. Bocci
Intended status: Informational Nokia Category: Informational Nokia
Expires: 30 October 2023 S. Boutros ISSN: 2070-1721 S. Boutros
Ciena Ciena
A. Sajassi A. Sajassi
Cisco Cisco
28 April 2023 September 2023
Applicability of EVPN to NVO3 Networks Applicability of Ethernet Virtual Private Network (EVPN) to Network
draft-ietf-nvo3-evpn-applicability-06 Virtualization over Layer 3 (NVO3) Networks
Abstract Abstract
Ethernet Virtual Private Network (EVPN) provides a unified control- An Ethernet Virtual Private Network (EVPN) provides a unified control
plane that solves the Network Virtualization Edge (NVE) auto- plane that solves the issues of Network Virtualization Edge (NVE)
discovery, tenant MAC/IP dissemination and advanced features required auto-discovery, tenant Media Access Control (MAC) / IP dissemination,
by Network Virtualization Over Layer-3 (NVO3) networks. EVPN is a and advanced features in a scablable way as required by Network
scalable solution for NVO3 networks and keeps the independence of the Virtualization over Layer 3 (NVO3) networks. EVPN is a scalable
underlay IP Fabric, i.e. there is no need to enable PIM in the solution for NVO3 networks and keeps the independence of the underlay
underlay network and maintain multicast states for tenant Broadcast IP Fabric, i.e., there is no need to enable Protocol Independent
Domains. This document describes the use of EVPN for NVO3 networks, Multicast (PIM) in the underlay network and maintain multicast states
discusses its applicability to basic Layer-2 and Layer-3 connectivity for tenant Broadcast Domains. This document describes the use of
requirements, as well as advanced features such as MAC-mobility, MAC EVPN for NVO3 networks and discusses its applicability to basic Layer
Protection and Loop Protection, multi-homing, Data Center 2 and Layer 3 connectivity requirements and to advanced features such
Interconnect (DCI) and much more. No new EVPN procedures are as MAC Mobility, MAC Protection and Loop Protection, multihoming,
introduced. Data Center Interconnect (DCI), and much more. No new EVPN
procedures are introduced.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on 30 October 2023. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9469.
Copyright Notice Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. Code Components carefully, as they describe your rights and restrictions with respect
extracted from this document must include Revised BSD License text as to this document. Code Components extracted from this document must
described in Section 4.e of the Trust Legal Provisions and are include Revised BSD License text as described in Section 4.e of the
provided without warranty as described in the Revised BSD License. Trust Legal Provisions and are provided without warranty as described
in the Revised BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction
2. EVPN and NVO3 Terminology . . . . . . . . . . . . . . . . . . 3 2. EVPN and NVO3 Terminology
3. Why is EVPN Needed in NVO3 Networks? . . . . . . . . . . . . 7 3. Why is EVPN Needed in NVO3 Networks?
4. Applicability of EVPN to NVO3 Networks . . . . . . . . . . . 9 4. Applicability of EVPN to NVO3 Networks
4.1. EVPN Route Types Used in NVO3 Networks . . . . . . . . . 9 4.1. EVPN Route Types Used in NVO3 Networks
4.2. EVPN Basic Applicability for Layer-2 Services . . . . . . 11 4.2. EVPN Basic Applicability for Layer 2 Services
4.2.1. Auto-Discovery and Auto-Provisioning . . . . . . . . 12 4.2.1. Auto-Discovery and Auto-Provisioning
4.2.2. Remote NVE Auto-Discovery . . . . . . . . . . . . . . 13 4.2.2. Remote NVE Auto-Discovery
4.2.3. Distribution of Tenant MAC and IP Information . . . . 14 4.2.3. Distribution of Tenant MAC and IP Information
4.3. EVPN Basic Applicability for Layer-3 Services . . . . . . 15 4.3. EVPN Basic Applicability for Layer 3 Services
4.4. EVPN as Control Plane for NVO3 Encapsulations and 4.4. EVPN as Control Plane for NVO3 Encapsulations and Geneve
GENEVE . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.5. EVPN OAM and Application to NVO3
4.5. EVPN OAM and Application to NVO3 . . . . . . . . . . . . 18 4.6. EVPN as the Control Plane for NVO3 Security
4.6. EVPN as the Control Plane for NVO3 Security . . . . . . . 18 4.7. Advanced EVPN Features for NVO3 Networks
4.7. Advanced EVPN Features for NVO3 Networks . . . . . . . . 18 4.7.1. Virtual Machine (VM) Mobility
4.7.1. Virtual Machine (VM) Mobility . . . . . . . . . . . . 18 4.7.2. MAC Protection, Duplication Detection, and Loop
4.7.2. MAC Protection, Duplication Detection and Loop Protection
Protection . . . . . . . . . . . . . . . . . . . . . 19 4.7.3. Reduction/Optimization of BUM Traffic in Layer 2
4.7.3. Reduction/Optimization of BUM Traffic in Layer-2 Services
Services . . . . . . . . . . . . . . . . . . . . . . 19 4.7.4. Ingress Replication (IR) Optimization for BUM Traffic
4.7.4. Ingress Replication (IR) Optimization for BUM 4.7.5. EVPN Multihoming
Traffic . . . . . . . . . . . . . . . . . . . . . . . 20 4.7.6. EVPN Recursive Resolution for Inter-subnet Unicast
4.7.5. EVPN Multi-Homing . . . . . . . . . . . . . . . . . . 21 Forwarding
4.7.6. EVPN Recursive Resolution for Inter-Subnet Unicast 4.7.7. EVPN Optimized Inter-subnet Multicast Forwarding
Forwarding . . . . . . . . . . . . . . . . . . . . . 22 4.7.8. Data Center Interconnect (DCI)
4.7.7. EVPN Optimized Inter-Subnet Multicast Forwarding . . 23 5. Security Considerations
4.7.8. Data Center Interconnect (DCI) . . . . . . . . . . . 24 6. IANA Considerations
5. Security Considerations . . . . . . . . . . . . . . . . . . . 24 7. References
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 7.1. Normative References
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 24 7.2. Informative References
7.1. Normative References . . . . . . . . . . . . . . . . . . 24 Acknowledgments
7.2. Informative References . . . . . . . . . . . . . . . . . 25 Authors' Addresses
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 29
Appendix B. Contributors . . . . . . . . . . . . . . . . . . . . 29
Appendix C. Authors' Addresses . . . . . . . . . . . . . . . . . 29
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 29
1. Introduction 1. Introduction
In Network Virtualization Over Layer-3 (NVO3) networks, Network In Network Virtualization over Layer 3 (NVO3) networks, Network
Virtualization Edge devices (NVEs) sit at the edge of the underlay Virtualization Edge (NVE) devices sit at the edge of the underlay
network and provide Layer-2 and Layer-3 connectivity among Tenant network and provide Layer 2 and Layer 3 connectivity among Tenant
Systems (TSes) of the same tenant. The NVEs need to build and Systems (TSes) of the same tenant. The NVEs need to build and
maintain mapping tables so that they can deliver encapsulated packets maintain mapping tables so they can deliver encapsulated packets to
to their intended destination NVE(s). While there are different their intended destination NVE(s). While there are different options
options to create and disseminate the mapping table entries, NVEs may to create and disseminate the mapping table entries, NVEs may
exchange that information directly among themselves via a control- exchange that information directly among themselves via a control
plane protocol, such as Ethernet Virtual Private Network (EVPN). plane protocol, such as Ethernet Virtual Private Network (EVPN).
EVPN provides an efficient, flexible and unified control-plane option EVPN provides an efficient, flexible, and unified control plane
that can be used for Layer-2 and Layer-3 Virtual Network (VN) service option that can be used for Layer 2 and Layer 3 Virtual Network (VN)
connectivity. This document does not introduce any new procedures in service connectivity. This document does not introduce any new
EVPN. procedures in EVPN.
In this document, we assume that the EVPN control-plane module In this document, we assume that the EVPN control plane module
resides in the NVEs. The NVEs can be virtual switches in resides in the NVEs. The NVEs can be virtual switches in
hypervisors, Top Of Rack (TOR) switches or Leaf switches or Data hypervisors, Top-of-Rack (ToR) switches or Leaf switches, or Data
Center Gateways. As described in [RFC7365], Network Virtualization Center Gateways. As described in [RFC7365], Network Virtualization
Authorities (NVAs) may be used to provide the forwarding information Authorities (NVAs) may be used to provide the forwarding information
to the NVEs, and in that case, EVPN could be used to disseminate the to the NVEs, and in that case, EVPN could be used to disseminate the
information across multiple federated NVAs. The applicability of information across multiple federated NVAs. The applicability of
EVPN would then be similar to the one described in this document. EVPN would then be similar to the one described in this document.
However, for simplicity, the description assumes control-plane However, for simplicity, the description assumes control plane
communication among NVE(s). communication among NVE(s).
2. EVPN and NVO3 Terminology 2. EVPN and NVO3 Terminology
This document uses the terminology of [RFC7365], in addition to the This document uses the terminology of [RFC7365] in addition to the
terms that follow. terms that follow.
* AC: Attachment Circuit or logical interface associated to a given AC: Attachment Circuit or logical interface associated with a given
BT. To determine the AC on which a packet arrived, the NVE will BT. To determine the AC on which a packet arrived, the NVE will
examine the physical/logical port and/or VLAN tags (where the VLAN examine the physical/logical port and/or VLAN tags (where the VLAN
tags can be individual c-tags, s-tags or ranges of both). tags can be individual c-tags, s-tags, or ranges of both).
* ARP and ND: Address Resolution Protocol (IPv4) and Neighbor ARP and NDP: Address Resolution Protocol (IPv4) and Neighbor
Discovery protocol (IPv6). Discovery Protocol (IPv6), respectively.
* BD: or Broadcast Domain, it corresponds to a tenant IP subnet. If BD: Broadcast Domain that corresponds to a tenant IP subnet. If no
no suppression techniques are used, a BUM frame that is injected suppression techniques are used, a BUM frame that is injected in a
in a Broadcast Domain will reach all the NVEs that are attached to Broadcast Domain will reach all the NVEs that are attached to that
that Broadcast Domain. An EVI may contain one or multiple Broadcast Domain. An EVI may contain one or multiple Broadcast
Broadcast Domains depending on the service model [RFC7432]. This Domains depending on the service model [RFC7432]. This document
document will use the term Broadcast Domain to refer to a tenant will use the term Broadcast Domain to refer to a tenant subnet.
subnet.
* BT: a Bridge Table, as defined in [RFC7432]. A BT is the BT: Bridge Table, as defined in [RFC7432]. A BT is the
instantiation of a Broadcast Domain in an NVE. When there is a instantiation of a Broadcast Domain in an NVE. When there is a
single Broadcast Domain on a given EVI, the MAC-VRF is equivalent single Broadcast Domain on a given EVI, the MAC-VRF is equivalent
to the BT on that NVE. Although a Broadcast Domain spans multiple to the BT on that NVE. Although a Broadcast Domain spans multiple
NVEs, and a BT is really the instantiation of a Broadcast Domain NVEs and a BT is really the instantiation of a Broadcast Domain in
in an NVE, this document uses BT and Broadcast Domain an NVE, this document uses BT and Broadcast Domain
interchangeably. interchangeably.
* BUM: Broadcast, Unknown unicast and Multicast frames. BUM: Broadcast, Unknown Unicast, and Multicast frames
* Clos: a multistage network topology described in [CLOS1953], where Clos: A multistage network topology described in [CLOS1953], where
all the edge switches (or Leafs) are connected to all the core all the edge switches (or Leafs) are connected to all the core
switches (or Spines). Typically used in Data Centers. switches (or Spines). Typically used in Data Centers.
* DF and NDF: they refer to Designated Forwarder and Non-Designated DF and NDF: Designated Forwarder and Non-Designated Forwarder,
Forwarder, which are the roles that a given PE can have in a given respectively. These are the roles that a given PE can have in a
ES. given ES.
* ECMP: Equal Cost Multi-Path. ECMP: Equal-Cost Multipath
* ES: Ethernet Segment. When a Tenant System (TS) is connected to ES: Ethernet Segment. When a Tenant System (TS) is connected to one
one or more NVEs via a set of Ethernet links, then that set of or more NVEs via a set of Ethernet links, that set of links is
links is referred to as an 'Ethernet segment'. Each ES is referred to as an "Ethernet Segment". Each ES is represented by a
represented by a unique Ethernet Segment Identifier (ESI) in the unique Ethernet Segment Identifier (ESI) in the NVO3 network, and
NVO3 network and the ESI is used in EVPN routes that are specific the ESI is used in EVPN routes that are specific to that ES.
to that ES.
* Ethernet Tag: Used to represent a Broadcast Domain that is Ethernet Tag: Used to represent a Broadcast Domain that is
configured on a given ES for the purpose of Designated Forwarder configured on a given ES for the purpose of Designated Forwarder
election. Note that any of the following may be used to represent election. Note that any of the following may be used to represent
a Broadcast Domain: VIDs (including Q-in-Q tags), configured IDs, a Broadcast Domain: VIDs (including Q-in-Q tags), configured IDs,
VNIs (Virtual Extensible Local Area Network (VXLAN) Network VNIs, normalized VIDs, Service Instance Identifiers (I-SIDs),
Identifiers), normalized VIDs, I-SIDs (Service Instance etc., as long as the representation of the Broadcast Domains is
Identifiers), etc., as long as the representation of the Broadcast configured consistently across the multihomed PEs attached to that
Domains is configured consistently across the multihomed PEs ES.
attached to that ES.
* EVI: or EVPN Instance. It is a Layer-2 Virtual Network that uses EVI or EVPN Instance: A Layer 2 Virtual Network that uses an EVPN
an EVPN control-plane to exchange reachability information among control plane to exchange reachability information among the
the member NVEs. It corresponds to a set of MAC-VRFs of the same member NVEs. It corresponds to a set of MAC-VRFs of the same
tenant. See MAC-VRF in this section. tenant. See MAC-VRF in this section.
* EVPN: Ethernet Virtual Private Networks, as described in EVPN: Ethernet Virtual Private Network, as described in [RFC7432].
[RFC7432].
* EVPN VLAN-aware bundle service model: similar to the VLAN-bundle EVPN VLAN-Aware Bundle Service Interface: Similar to the VLAN-bundle
model but each individual VLAN value is mapped to a different interface but each individual VLAN value is mapped to a different
Broadcast Domain. In this model there are multiple Broadcast Broadcast Domain. In this interface, there are multiple Broadcast
Domains per EVI for a given tenant. Each Broadcast Domain is Domains per EVI for a given tenant. Each Broadcast Domain is
identified by an "Ethernet Tag", that is a control-plane value identified by an "Ethernet Tag", which is a control plane value
that identifies the routes for the Broadcast Domain within the that identifies the routes for the Broadcast Domain within the
EVI. EVI.
* EVPN VLAN-based service model: one of the three service models EVPN VLAN-Based Service Interface: One of the three service
defined in [RFC7432]. It is characterized as a Broadcast Domain interfaces defined in [RFC7432]. It is characterized as a
that uses a single VLAN per physical access port to attach tenant Broadcast Domain that uses a single VLAN per physical access port
traffic to the Broadcast Domain. In this service model, there is to attach tenant traffic to the Broadcast Domain. In this service
only one Broadcast Domain per EVI. interface, there is only one Broadcast Domain per EVI.
* EVPN VLAN-bundle service model: similar to VLAN-based but uses a EVPN VLAN-Bundle Service Interface: Similar to the VLAN-based
bundle of VLANs per physical port to attach tenant traffic to the interface but uses a bundle of VLANs per physical port to attach
Broadcast Domain. As in VLAN-based, in this model there is a tenant traffic to the Broadcast Domain. Like the VLAN-based
single Broadcast Domain per EVI. interface, there is only one Broadcast Domain per EVI.
* GENEVE: Generic Network Virtualization Encapsulation, an NVO3 Geneve: Generic Network Virtualization Encapsulation. An NVO3
encapsulation defined in [RFC8926]. encapsulation defined in [RFC8926].
* IP-VRF: an IP Virtual Routing and Forwarding table, as defined in IP-VRF: IP Virtual Routing and Forwarding table, as defined in
[RFC4364]. It stores IP Prefixes that are part of the tenant's IP [RFC4364]. It stores IP Prefixes that are part of the tenant's IP
space, and are distributed among NVEs of the same tenant by EVPN. space and are distributed among NVEs of the same tenant by EVPN.
Route Distinguisher (RD) and Route Target(s) (RTs) are required A Route Distinguisher (RD) and one or more Route Targets (RTs) are
properties of an IP-VRF. An IP-VRF is instantiated in an NVE for required properties of an IP-VRF. An IP-VRF is instantiated in an
a given tenant, if the NVE is attached to multiple subnets of the NVE for a given tenant if the NVE is attached to multiple subnets
tenant and local inter-subnet-forwarding is required across those of the tenant and local inter-subnet forwarding is required across
subnets. those subnets.
* IRB: Integrated Routing and Bridging interface. It refers to the IRB: Integrated Routing and Bridging. It refers to the logical
logical interface that connects a Broadcast Domain instance (or a interface that connects a Broadcast Domain instance (or a BT) to
BT) to an IP- VRF and allows to forward packets with destination an IP-VRF and forwards packets with a destination in a different
in a different subnet. subnet.
* MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in MAC-VRF: A MAC Virtual Routing and Forwarding table, as defined in
[RFC7432]. The instantiation of an EVI (EVPN Instance) in an NVE. [RFC7432]. The instantiation of an EVI (EVPN Instance) in an NVE.
Route Distinguisher (RD) and Route Target(s) (RTs) are required A Route Distinguisher (RD) and one or more RTs are required
properties of a MAC-VRF and they are normally different from the properties of a MAC-VRF, and they are normally different from the
ones defined in the associated IP-VRF (if the MAC-VRF has an IRB ones defined in the associated IP-VRF (if the MAC-VRF has an IRB
interface). interface).
* NVE: Network Virtualization Edge device, a network entity that NVE: Network Virtualization Edge. A network entity that sits at the
sits at the edge of an underlay network and implements Layer-2 edge of an underlay network and implements Layer 2 and/or Layer 3
and/or Layer-3 network virtualization functions. The network- network virtualization functions. The network-facing side of the
facing side of the NVE uses the underlying Layer-3 network to NVE uses the underlying Layer 3 network to tunnel tenant frames to
tunnel tenant frames to and from other NVEs. The tenant-facing and from other NVEs. The tenant-facing side of the NVE sends and
side of the NVE sends and receives Ethernet frames to and from receives Ethernet frames to and from individual Tenant Systems.
individual Tenant Systems. In this document, an NVE could be In this document, an NVE could be implemented as a virtual switch
implemented as a virtual switch within a hypervisor, a switch or a within a hypervisor, a switch, or a router, and it runs EVPN in
router, and runs EVPN in the control-plane. the control plane.
* NVO3 tunnels: Network Virtualization Over Layer-3 tunnels. In NVO3 tunnels: Network Virtualization over Layer 3 tunnels. In this
this document, NVO3 tunnels refer to a way to encapsulate tenant document, NVO3 tunnels refer to a way to encapsulate tenant frames
frames or packets into IP packets whose IP Source Addresses (SA) or packets into IP packets, whose IP Source Addresses (SAs) or
or Destination Addresses (DA) belong to the underlay IP address Destination Addresses (DAs) belong to the underlay IP address
space, and identify NVEs connected to the same underlay network. space, and identify NVEs connected to the same underlay network.
Examples of NVO3 tunnel encapsulations are VXLAN [RFC7348], GENEVE Examples of NVO3 tunnel encapsulations are VXLAN [RFC7348], Geneve
[RFC8926] or MPLSoUDP [RFC7510]. [RFC8926], or MPLSoUDP [RFC7510].
* PE: Provider Edge router. PE: Provider Edge
* PMSI: Provider Multicast Service Interface. PMSI: Provider Multicast Service Interface
* PTA: Provider Multicast Service Interface Tunnel Attribute. PTA: PMSI Tunnel Attribute
* RT and RD: Route Target and Route Distinguisher. RT and RD: Route Target and Route Distinguisher, respectively.
* RT-1, RT-2, RT-3, etc.: they refer to Route Type followed by the RT-1, RT-2, RT-3, etc.: These refer to the Route Types followed by
type number as defined in the IANA registry for EVPN route types. the type numbers as defined in the "EVPN Route Types" IANA
registry (see <https://www.iana.org/assignments/evpn/>).
* SA and DA: Source Address and Destination Address. They are used SA and DA: Source Address and Destination Address, respectively.
along with MAC or IP, e.g. IP SA or MAC DA. They are used along with MAC or IP, e.g., IP SA or MAC DA.
* SBD: Supplementary Broadcast Domain. Defined in [RFC9136], it is SBD: Supplementary Broadcast Domain, as defined in [RFC9136]. It is
a Broadcast Domain that does not have any Attachment Circuits, a Broadcast Domain that does not have any Attachment Circuits,
only IRB interfaces, and provides connectivity among all the IP- only has IRB interfaces, and provides connectivity among all the
VRFs of a tenant in the Interface-ful IP-VRF-to-IP-VRF models. IP-VRFs of a tenant in the Interface-ful IP-VRF-to-IP-VRF models.
* TS: Tenant System. A physical or virtual system that can play the TS: Tenant System. A physical or virtual system that can play the
role of a host or a forwarding element such as a router, switch, role of a host or a forwarding element, such as a router, switch,
firewall, etc. It belongs to a single tenant and connects to one firewall, etc. It belongs to a single tenant and connects to one
or more Broadcast Domains of that tenant. or more Broadcast Domains of that tenant.
* VIDs: Virtual Local Area Network Identifiers. VID: Virtual Local Area Network Identifier
* VNI: Virtual Network Identifier. Irrespective of the NVO3 VNI: Virtual Network Identifier. Irrespective of the NVO3
encapsulation, the tunnel header always includes a VNI that is encapsulation, the tunnel header always includes a VNI that is
added at the ingress NVE (based on the mapping table lookup) and added at the ingress NVE (based on the mapping table lookup) and
identifies the BT at the egress NVE. This VNI is called VNI in identifies the BT at the egress NVE. This VNI is called VNI in
VXLAN or GENEVE, VSID in nvGRE or Label in MPLSoGRE or MPLSoUDP. VXLAN or Geneve, Virtual Subnet ID (VSID) in nvGRE, or Label in
This document will refer to VNI as a generic Virtual Network MPLSoGRE or MPLSoUDP. This document refers to VNI as a generic
Identifier for any NVO3 encapsulation. VNI for any NVO3 encapsulation.
* VXLAN: Virtual eXtensible Local Area Network, an NVO3 VXLAN: Virtual eXtensible Local Area Network. An NVO3 encapsulation
encapsulation defined in [RFC7348]. defined in [RFC7348].
3. Why is EVPN Needed in NVO3 Networks? 3. Why is EVPN Needed in NVO3 Networks?
Data Centers have adopted NVO3 architectures mostly due to the issues Data Centers have adopted NVO3 architectures mostly due to the issues
discussed in [RFC7364]. The architecture of a Data Center is discussed in [RFC7364]. The architecture of a Data Center is
nowadays based on a Clos design, where every Leaf is connected to a nowadays based on a Clos design, where every Leaf is connected to a
layer of Spines, and there is a number of Equal Cost Multi-Paths layer of Spines and there is a number of ECMPs between any two Leaf
between any two leaf nodes. All the links between Leaf and Spine nodes. All the links between Leaf and Spine nodes are routed links,
nodes are routed links, forming what we also know as an underlay IP forming what we also know as an underlay IP Fabric. The underlay IP
Fabric. The underlay IP Fabric does not have issues with loops or Fabric does not have issues with loops or flooding (like old Spanning
flooding (like old Spanning Tree Data Center designs did), Tree Data Center designs did), convergence is fast, and ECMP
convergence is fast and Equal Cost Multi-Path generally distributes generally distributes utilization well across all the links.
utilization well across all the links.
On this architecture, and as discussed by [RFC7364], multi-tenant On this architecture, and as discussed by [RFC7364], multi-tenant
intra-subnet and inter-subnet connectivity services are provided by intra-subnet and inter-subnet connectivity services are provided by
NVO3 tunnels. VXLAN [RFC7348] or GENEVE [RFC8926] are two examples NVO3 tunnels. VXLAN [RFC7348] and Geneve [RFC8926] are two examples
of such NVO3 tunnels. of such NVO3 tunnels.
Why is a control-plane protocol along with NVO3 tunnels helpful? Why is a control plane protocol along with NVO3 tunnels helpful?
There are three main reasons: There are three main reasons:
a. Auto-discovery of the remote NVEs that are attached to the same a. Auto-discovery of the remote NVEs that are attached to the same
VPN instance (Layer-2 and/or Layer-3) as the ingress NVE is. VPN instance (Layer 2 and/or Layer 3) as the ingress NVE is.
b. Dissemination of the MAC/IP host information so that mapping b. Dissemination of the MAC/IP host information so that mapping
tables can be populated on the remote NVEs. tables can be populated on the remote NVEs.
c. Advanced features such as MAC Mobility, MAC Protection, BUM and c. Advanced features such as MAC Mobility, MAC Protection, BUM and
ARP/ND traffic reduction/suppression, Multi-homing, Prefix ARP/ND traffic reduction/suppression, multihoming, functionality
Independent Convergence (PIC) like functionality similar to Prefix Independent Convergence (PIC) [RTGWG-BGP-PIC],
[I-D.ietf-rtgwg-bgp-pic], Fast Convergence, etc. fast convergence, etc.
A possible approach to achieve points (a) and (b) above for "Flood and learn" is a possible approach to achieve points (a) and
multipoint Ethernet services, is "flood and learn". "Flood and (b) above for multipoint Ethernet services. "Flood and learn" refers
learn" refers to not using a specific control-plane on the NVEs, but to "flooding" BUM traffic from the ingress NVE to all the egress NVEs
rather "flood" BUM traffic from the ingress NVE to all the egress attached to the same Broadcast Domain instead of using a specific
NVEs attached to the same Broadcast Domain. The egress NVEs may then control plane on the NVEs. The egress NVEs may then use data path
use data path source MAC address "learning" on the frames received source MAC address "learning" on the frames received over the NVO3
over the NVO3 tunnels. When the destination host replies and the tunnels. When the destination host replies and the frames arrive at
frames arrive at the NVE that initially flooded BUM frames, the NVE the NVE that initially flooded BUM frames, the NVE will also "learn"
will also "learn" the source MAC address of the frame encapsulated on the source MAC address of the frame encapsulated on the NVO3 tunnel.
the NVO3 tunnel. This approach has the following drawbacks: This approach has the following drawbacks:
* In order to flood a given BUM frame, the ingress NVE must know the * In order to flood a given BUM frame, the ingress NVE must know the
IP addresses of the remote NVEs attached to the same Broadcast IP addresses of the remote NVEs attached to the same Broadcast
Domain. This may be done as follows: Domain. This may be done as follows:
- The remote tunnel IP addresses can be statically provisioned on - The remote tunnel IP addresses can be statically provisioned on
the ingress NVE. If the ingress NVE receives a BUM frame for the ingress NVE. If the ingress NVE receives a BUM frame for
the Broadcast Domain on an ingress Attachment Circuit, it will the Broadcast Domain on an ingress Attachment Circuit, it will
do ingress replication and will send the frame to all the do ingress replication and will send the frame to all the
configured egress NVE destination IP addresses in the Broadcast configured egress NVE destination IP addresses in the Broadcast
Domain. Domain.
- All the NVEs attached to the same Broadcast Domain can - All the NVEs attached to the same Broadcast Domain can
subscribe to an underlay IP Multicast Group that is dedicated subscribe to an underlay IP multicast group that is dedicated
to that Broadcast Domain. When an ingress NVE receives a BUM to that Broadcast Domain. When an ingress NVE receives a BUM
frame on an ingress Attachment Circuit, it will send a single frame on an ingress Attachment Circuit, it will send a single
copy of the frame encapsulated into an NVO3 tunnel, using the copy of the frame encapsulated into an NVO3 tunnel using the
multicast address as destination IP address of the tunnel. multicast address as the destination IP address of the tunnel.
This solution requires Protocol Independent Multicast (PIM) in This solution requires PIM in the underlay network and the
the underlay network and the association of individual association of individual Broadcast Domains to underlay IP
Broadcast Domains to underlay IP multicast groups. multicast groups.
* "Flood and learn" solves the issues of auto-discovery and learning * "Flood and learn" solves the issues of auto-discovery and the
of the MAC to VNI/tunnel IP mapping on the NVEs for a given learning of the MAC to VNI/tunnel IP mapping on the NVEs for a
Broadcast Domain. However, it does not provide a solution for given Broadcast Domain. However, it does not provide a solution
advanced features and it does not scale well (mostly due to the for advanced features, and it does not scale well (mostly due to
need for constant flooding and the underlay PIM states that must the need for constant flooding and the underlay PIM states that
be maintained). must be maintained).
EVPN provides a unified control-plane that solves the NVE auto- EVPN provides a unified control plane that solves the issues of NVE
discovery, tenant MAC/IP dissemination and advanced features in a auto-discovery, tenant MAC/IP dissemination, and advanced features in
scalable way and keeping the independence of the underlay IP Fabric, a scalable way and keeps the independence of the underlay IP Fabric;
i.e., there is no need to enable PIM in the underlay network and i.e., there is no need to enable PIM in the underlay network and
maintain multicast states for tenant Broadcast Domains. maintain multicast states for tenant Broadcast Domains.
Section 4 describes how EVPN can be used to meet the control-plane Section 4 describes how EVPN can be used to meet the control plane
requirements in an NVO3 network. requirements in an NVO3 network.
4. Applicability of EVPN to NVO3 Networks 4. Applicability of EVPN to NVO3 Networks
This section discusses the applicability of EVPN to NVO3 networks. This section discusses the applicability of EVPN to NVO3 networks.
The intent is not to provide a comprehensive explanation of the The intent is not to provide a comprehensive explanation of the
protocol itself but give an introduction and point at the protocol itself, but to give an introduction and point at the
corresponding reference document, so that the reader can easily find corresponding reference document so the reader can easily find more
more details if needed. details if needed.
4.1. EVPN Route Types Used in NVO3 Networks 4.1. EVPN Route Types Used in NVO3 Networks
EVPN supports multiple Route Types and each type has a different EVPN supports multiple Route Types, and each type has a different
function. For convenience, Table 1 shows a summary of all the function. For convenience, Table 1 shows a summary of all the
existing EVPN route types and its usage. In this document we may existing EVPN Route Types and their usages. In this document, we may
refer to these route types as RT-x routes, where x is the type number refer to these route types as RT-x routes, where x is the type number
included in the first column of Table 1. included in the first column of Table 1.
+======+================+=======================================+ +======+================+=======================================+
| Type | Description | Usage | | Type | Description | Usage |
+======+================+=======================================+ +======+================+=======================================+
| 1 | Ethernet Auto- | Multi-homing: used for MAC mass- | | 1 | Ethernet Auto- | Multihoming: Used for MAC mass- |
| | Discovery | withdraw when advertised per Ethernet | | | Discovery | withdraw when advertised per Ethernet |
| | | Segment, and used for aliasing/backup | | | | Segment and for aliasing/backup |
| | | functions when advertised per EVI | | | | functions when advertised per EVI. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 2 | MAC/IP | Host MAC/IP dissemination, supports | | 2 | MAC/IP | Host MAC/IP dissemination; supports |
| | Advertisement | MAC mobility and protection | | | Advertisement | MAC Mobility and protection. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 3 | Inclusive | NVE discovery and BUM flooding tree | | 3 | Inclusive | NVE discovery and BUM flooding tree |
| | Multicast | setup | | | Multicast | setup. |
| | Ethernet Tag | | | | Ethernet Tag | |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 4 | Ethernet | Multi-homing: ES auto-discovery and | | 4 | Ethernet | Multihoming: ES auto-discovery and DF |
| | Segment | DF Election | | | Segment | election. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 5 | IP Prefix | IP Prefix dissemination | | 5 | IP Prefix | IP Prefix dissemination. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 6 | Selective | Indicate interest for a multicast S,G | | 6 | Selective | Indicate interest for a multicast S,G |
| | Multicast | or *,G | | | Multicast | or *,G. |
| | Ethernet Tag | | | | Ethernet Tag | |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 7 | Multicast Join | Multi-homing: S,G or *,G state synch | | 7 | Multicast Join | Multihoming: S,G or *,G state synch. |
| | Synch | | | | Synch | |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 8 | Multicast | Multi-homing: S,G or *,G leave synch | | 8 | Multicast | Multihoming: S,G or *,G leave synch. |
| | Leave Synch | | | | Leave Synch | |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 9 | Per-Region | BUM tree creation across regions | | 9 | Per-Region | BUM tree creation across regions. |
| | I-PMSI A-D | | | | I-PMSI A-D | |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 10 | S-PMSI A-D | Multicast tree for S,G or *,G states | | 10 | S-PMSI A-D | Multicast tree for S,G or *,G states. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
| 11 | Leaf A-D | Used for responses to explicit | | 11 | Leaf A-D | Used for responses to explicit |
| | | tracking | | | | tracking. |
+------+----------------+---------------------------------------+ +------+----------------+---------------------------------------+
Table 1: EVPN route types Table 1: EVPN Route Types
4.2. EVPN Basic Applicability for Layer-2 Services 4.2. EVPN Basic Applicability for Layer 2 Services
Although the applicability of EVPN to NVO3 networks spans multiple Although the applicability of EVPN to NVO3 networks spans multiple
documents, EVPN's baseline specification is [RFC7432]. [RFC7432] documents, EVPN's baseline specification is [RFC7432]. [RFC7432]
allows multipoint layer-2 VPNs to be operated as [RFC4364] IP-VPNs, allows multipoint Layer 2 VPNs to be operated as IP VPNs [RFC4364],
where MACs and the information to set up flooding trees are where MACs and the information to set up flooding trees are
distributed by MP-BGP [RFC4760]. Based on [RFC7432], [RFC8365] distributed by Multiprotocol BGP (MP-BGP) [RFC4760]. Based on
describes how to use EVPN to deliver Layer-2 services specifically in [RFC7432], [RFC8365] describes how to use EVPN to deliver Layer 2
NVO3 Networks. services specifically in NVO3 networks.
Figure 1 represents a Layer-2 service deployed with an EVPN Broadcast Figure 1 represents a Layer 2 service deployed with an EVPN Broadcast
Domain in an NVO3 network. Domain in an NVO3 network.
+--TS2---+ +--TS2---+
* | Single-Active * | Single-Active
* | ESI-1 * | ESI-1
+----+ +----+ +----+ +----+
|BD1 | |BD1 | |BD1 | |BD1 |
+-------------| |--| |-----------+ +-------------| |--| |-----------+
| +----+ +----+ | | +----+ +----+ |
| NVE2 NVE3 NVE4 | NVE2 NVE3 NVE4
| EVPN NVO3 Network +----+ | EVPN NVO3 Network +----+
NVE1(IP-A) | BD1|-----+ NVE1(IP-A) | BD1|-----+
+-------------+ RT-2 | | | +-------------+ RT-2 | | |
| | +-------+ +----+ | | | +-------+ +----+ |
| +----+ | |MAC1 | NVE5 TS3 | +----+ | |MAC1 | NVE5 TS3
TS1--------|BD1 | | |IP1 | +----+ | TS1--------|BD1 | | |IP1 | +----+ |
MAC1 | +----+ | |Label L|---> | BD1|-----+ MAC1 | +----+ | |Label L|---> | BD1|-----+
IP1 | | |NH IP-A| | | All-Active IP1 | | |NH IP-A| | | All-Active
| Hypervisor | +-------+ +----+ ESI-2 | Hypervisor | +-------+ +----+ ESI-2
+-------------+ | +-------------+ |
+--------------------------------------+ +--------------------------------------+
Figure 1: EVPN for L2 in an NVO3 Network - example Figure 1: EVPN for L2 in an NVO3 Network - Example
In a simple NVO3 network, such as the example of Figure 1, these are In a simple NVO3 network, such as the example of Figure 1, these are
the basic constructs that EVPN uses for Layer-2 services (or Layer-2 the basic constructs that EVPN uses for Layer 2 services (or Layer 2
Virtual Networks): Virtual Networks):
* BD1 is an EVPN Broadcast Domain for a given tenant and TS1, TS2 * BD1 is an EVPN Broadcast Domain for a given tenant and TS1, TS2,
and TS3 are connected to it. The five represented NVEs are and TS3 are connected to it. The five represented NVEs are
attached to BD1 and are connected to the same underlay IP network. attached to BD1 and are connected to the same underlay IP network.
That is, each NVE learns the remote NVEs' loopback addresses via That is, each NVE learns the remote NVEs' loopback addresses via
underlay routing protocol. underlay routing protocol.
* NVE1 is deployed as a virtual switch in a Hypervisor with IP-A as * NVE1 is deployed as a virtual switch in a hypervisor with IP-A as
underlay loopback IP address. The rest of the NVEs in Figure 1 underlay loopback IP address. The rest of the NVEs in Figure 1
are physical switches and TS2/TS3 are multi-homed to them. TS1 is are physical switches and TS2/TS3 are multihomed to them. TS1 is
a virtual machine, identified by MAC1 and IP1. TS2 and TS3 are a virtual machine, identified by MAC1 and IP1. TS2 and TS3 are
physically dual-connected to NVEs, hence they are normally not physically dual-connected to NVEs; hence, they are normally not
considered virtual machines. considered virtual machines.
* The terms Single-Active and All-Active in Figure 1 refer to the * The terms Single-Active and All-Active in Figure 1 refer to the
mode in which the TS2 and TS3 are multi-homed to the NVEs in BD1. mode in which the TS2 and TS3 are multihomed to the NVEs in BD1.
In All-Active mode, all the multi-homing links are active and can In All-Active mode, all the multihoming links are active and can
send or receive traffic. In Single-Active mode, only one link (of send or receive traffic. In Single-Active mode, only one link (of
the set of links connected to the NVEs) is active. the set of links connected to the NVEs) is active.
4.2.1. Auto-Discovery and Auto-Provisioning 4.2.1. Auto-Discovery and Auto-Provisioning
Auto-discovery is one of the basic capabilities of EVPN. The Auto-discovery is one of the basic capabilities of EVPN. The
provisioning of EVPN components in NVEs is significantly automated, provisioning of EVPN components in NVEs is significantly automated,
simplifying the deployment of services and minimizing manual simplifying the deployment of services and minimizing manual
operations that are prone to human error. operations that are prone to human error.
These are some of the Auto-Discovery and Auto-Provisioning These are some of the auto-discovery and auto-provisioning
capabilities available in EVPN: capabilities available in EVPN:
* Automation on Ethernet Segments (ES): an Ethernet Segment is * Automation on Ethernet Segments (ESes): An Ethernet Segment is
defined as a group of NVEs that are attached to the same Tenant defined as a group of NVEs that are attached to the same Tenant
System or network. An Ethernet Segment is identified by an System or network. An Ethernet Segment is identified by an
Ethernet Segment Identifier (ESI) in the control plane, but Ethernet Segment Identifier (ESI) in the control plane, but
neither the ESI nor the NVEs that share the same Ethernet Segment neither the ESI nor the NVEs that share the same Ethernet Segment
are required to be manually provisioned in the local NVE: are required to be manually provisioned in the local NVE.
- If the multi-homed Tenant System or network are running - If the multihomed Tenant System or network is running
protocols such as LACP (Link Aggregation Control Protocol) protocols, such as the Link Aggregation Control Protocol (LACP)
[IEEE.802.1AX_2014], MSTP (Multiple-instance Spanning Tree [IEEE.802.1AX_2014], the Multiple Spanning Tree Protocol
Protocol), G.8032, etc. and all the NVEs in the Ethernet (MSTP), G.8032, etc., and all the NVEs in the Ethernet Segment
Segment can listen to the protocol PDUs to uniquely identify can listen to the protocol PDUs to uniquely identify the
the multi-homed Tenant System/network, then the ESI can be multihomed Tenant System/network, then the ESI can be "auto-
"auto-sensed" or "auto-provisioned" following the guidelines in sensed" or "auto-provisioned" following the guidelines in
[RFC7432] section 5. The ESI can also be auto-derived out of Section 5 of [RFC7432]. The ESI can also be auto-derived out
other parameters that are common to all NVEs attached to the of other parameters that are common to all NVEs attached to the
same Ethernet Segment. same Ethernet Segment.
- As described in [RFC7432], EVPN can also auto-derive the BGP - As described in [RFC7432], EVPN can also auto-derive the BGP
parameters required to advertise the presence of a local parameters required to advertise the presence of a local
Ethernet Segment in the control plane (RT and RD). Local Ethernet Segment in the control plane (RT and RD). Local
Ethernet Segments are advertised using Ethernet Segment routes Ethernet Segments are advertised using Ethernet Segment routes,
and the ESI-import Route-Target used by Ethernet Segment routes and the ESI-import Route Target used by Ethernet Segment routes
can be auto-derived based on the procedures of [RFC7432], can be auto-derived based on the procedures of Section 7.6 of
section 7.6. [RFC7432].
- By listening to other Ethernet Segment routes that match the - By listening to other Ethernet Segment routes that match the
local ESI and import Route Target, an NVE can also auto- local ESI and import Route Target, an NVE can also auto-
discover the other NVEs participating in the multi-homing for discover the other NVEs participating in the multihoming for
the Ethernet Segment. the Ethernet Segment.
- Once the NVE has auto-discovered all the NVEs attached to the - Once the NVE has auto-discovered all the NVEs attached to the
same Ethernet Segment, the NVE can automatically perform the same Ethernet Segment, the NVE can automatically perform the
Designated Forwarder Election algorithm (which determines the Designated Forwarder election algorithm (which determines the
NVE that will forward traffic to the multi-homed Tenant System/ NVE that will forward traffic to the multihomed Tenant System/
network). EVPN guarantees that all the NVEs in the Ethernet network). EVPN guarantees that all the NVEs in the Ethernet
Segment have a consistent Designated Forwarder Election. Segment have a consistent Designated Forwarder election.
* Auto-provisioning of services: when deploying a Layer-2 Service * Auto-provisioning of services: When deploying a Layer 2 service
for a tenant in an NVO3 network, all the NVEs attached to the same for a tenant in an NVO3 network, all the NVEs attached to the same
subnet must be configured with a MAC-VRF and the Broadcast Domain subnet must be configured with a MAC-VRF and the Broadcast Domain
for the subnet, as well as certain parameters for them. Note for the subnet, as well as certain parameters for them. Note that
that, if the EVPN service model is VLAN-based or VLAN-bundle, if the EVPN service interfaces are VLAN-based or VLAN-bundle,
implementations do not normally have a specific provisioning for implementations do not normally have a specific provisioning for
the Broadcast Domain (since it is in that case the same construct the Broadcast Domain since, in this case, it is the same construct
as the MAC-VRF). EVPN allows auto-deriving as many MAC-VRF as the MAC-VRF. EVPN allows auto-deriving as many MAC-VRF
parameters as possible. As an example, the MAC-VRF's Route Target parameters as possible. As an example, the MAC-VRF's Route Target
and Route Distinguisher for the EVPN routes may be auto-derived. and Route Distinguisher for the EVPN routes may be auto-derived.
Section 5.1.2.1 in [RFC8365] specifies how to auto-derive a MAC- Section 5.1.2.1 of [RFC8365] specifies how to auto-derive a MAC-
VRF's Route Target as long as VLAN-based service model is VRF's Route Target as long as a VLAN-based service interface is
implemented. [RFC7432] specifies how to auto-derive the Route implemented. [RFC7432] specifies how to auto-derive the Route
Distinguisher. Distinguisher.
4.2.2. Remote NVE Auto-Discovery 4.2.2. Remote NVE Auto-Discovery
Auto-discovery via MP-BGP [RFC4760] is used to discover the remote Auto-discovery via MP-BGP [RFC4760] is used to discover the remote
NVEs attached to a given Broadcast Domain, the NVEs participating in NVEs attached to a given Broadcast Domain, the NVEs participating in
a given redundancy group, the tunnel encapsulation types supported by a given redundancy group, the tunnel encapsulation types supported by
an NVE, etc. an NVE, etc.
In particular, when a new MAC-VRF and Broadcast Domain are enabled, In particular, when a new MAC-VRF and Broadcast Domain are enabled,
the NVE will advertise a new Inclusive Multicast Ethernet Tag route. the NVE will advertise a new Inclusive Multicast Ethernet Tag route.
Besides other fields, the Inclusive Multicast Ethernet Tag route will Besides other fields, the Inclusive Multicast Ethernet Tag route will
encode the IP address of the advertising NVE, the Ethernet Tag (which encode the IP address of the advertising NVE, the Ethernet Tag (which
is zero in case of VLAN-based and VLAN-bundle models) and also a PMSI is zero in the case of VLAN-based and VLAN-bundle interfaces), and a
Tunnel Attribute (PTA) that indicates the information about the PMSI Tunnel Attribute (PTA) that indicates the information about the
intended way to deliver BUM traffic for the Broadcast Domain. intended way to deliver BUM traffic for the Broadcast Domain.
In the example of Figure 1, when BD1 is enabled, NVE1 will send an When BD1 is enabled in the example of Figure 1, NVE1 will send an
Inclusive Multicast Ethernet Tag route including its own IP address, Inclusive Multicast Ethernet Tag route including its own IP address,
Ethernet-Tag for BD1 and the PMSI Tunnel Attribute to the remote an Ethernet-Tag for BD1, and the PMSI Tunnel Attribute to the remote
NVEs. Assuming Ingress Replication (IR) is used, the Inclusive NVEs. Assuming Ingress Replication (IR) is used, the Inclusive
Multicast Ethernet Tag route will include an identification for Multicast Ethernet Tag route will include an identification for
Ingress Replication in the PMSI Tunnel Attribute and the Virtual Ingress Replication in the PMSI Tunnel Attribute and the VNI that the
Network Identifier that the other NVEs in the Broadcast Domain must other NVEs in the Broadcast Domain must use to send BUM traffic to
use to send BUM traffic to the advertising NVE. The other NVEs in the advertising NVE. The other NVEs in the Broadcast Domain will
the Broadcast Domain will import the Inclusive Multicast Ethernet Tag import the Inclusive Multicast Ethernet Tag route and will add NVE1's
route and will add NVE1's IP address to the flooding list for BD1. IP address to the flooding list for BD1. Note that the Inclusive
Note that the Inclusive Multicast Ethernet Tag route is also sent Multicast Ethernet Tag route is also sent with a BGP encapsulation
with a BGP encapsulation attribute [RFC9012] that indicates what NVO3 attribute [RFC9012] that indicates what NVO3 encapsulation the remote
encapsulation the remote NVEs should use when sending BUM traffic to NVEs should use when sending BUM traffic to NVE1.
NVE1.
Refer to [RFC7432] for more information about the Inclusive Multicast Refer to [RFC7432] for more information about the Inclusive Multicast
Ethernet Tag route and forwarding of BUM traffic, and to [RFC8365] Ethernet Tag route and forwarding of BUM traffic. See [RFC8365] for
for its considerations on NVO3 networks. its considerations on NVO3 networks.
4.2.3. Distribution of Tenant MAC and IP Information 4.2.3. Distribution of Tenant MAC and IP Information
Tenant MAC/IP information is advertised to remote NVEs using MAC/IP Tenant MAC/IP information is advertised to remote NVEs using MAC/IP
Advertisement routes. Following the example of Figure 1: Advertisement routes. Following the example of Figure 1:
* In a given EVPN Broadcast Domain, Tenant Systems' MAC addresses * In a given EVPN Broadcast Domain, the MAC addresses of TSes are
are first learned at the NVE they are attached to, via data path first learned at the NVE they are attached to via data path or
or management plane learning. In Figure 1 we assume NVE1 learns management plane learning. In Figure 1, we assume NVE1 learns
MAC1/IP1 in the management plane (for instance, via Cloud MAC1/IP1 in the management plane (for instance, via Cloud
Management System) since the NVE is a virtual switch. NVE2, NVE3, Management System) since the NVE is a virtual switch. NVE2, NVE3,
NVE4 and NVE5 are TOR/Leaf switches and they normally learn MAC NVE4, and NVE5 are ToR/Leaf switches, and they normally learn MAC
addresses via data path. addresses via data path.
* Once NVE1's BD1 learns MAC1/IP1, NVE1 advertises that information * Once NVE1's BD1 learns MAC1/IP1, NVE1 advertises that information
along with a Virtual Network Identifier and Next Hop IP-A in an along with a VNI and Next-Hop IP-A in a MAC/IP Advertisement
MAC/IP Advertisement route. The EVPN routes are advertised using route. The EVPN routes are advertised using the Route
the Route Distinguisher/Route Targets of the MAC-VRF where the Distinguisher / Route Targets of the MAC-VRF where the Broadcast
Broadcast Domain belongs. Similarly, all the NVEs in BD1 learn Domain belongs. Similarly, all the NVEs in BD1 learn local MAC/IP
local MAC/IP addresses and advertise them in MAC/IP Advertisement addresses and advertise them in MAC/IP Advertisement routes.
routes.
* The remote NVEs can then add MAC1 to their mapping table for BD1 * The remote NVEs can then add MAC1 to their mapping table for BD1
(BT). For instance, when TS3 sends frames to NVE4 with (BT). For instance, when TS3 sends frames to NVE4 with the
destination MAC address = MAC1, NVE4 does a MAC lookup on the destination MAC address = MAC1, NVE4 does a MAC lookup on the
Bridge Table that yields IP-A and Label L. NVE4 can then Bridge Table that yields IP-A and Label L. NVE4 can then
encapsulate the frame into an NVO3 tunnel with IP-A as the tunnel encapsulate the frame into an NVO3 tunnel with IP-A as the tunnel
destination IP address and L as the Virtual Network Identifier. destination IP address and L as the VNI. Note that the MAC/IP
Note that the MAC/IP Advertisement route may also contain the Advertisement route may also contain the host's IP address (as
host's IP address (as in the example of Figure 1). While the MAC shown in the example of Figure 1). While the MAC of the received
of the received MAC/IP Advertisement route is installed in the MAC/IP Advertisement route is installed in the Bridge Table, the
Bridge Table, the IP address may be installed in the Proxy-ARP/ND IP address may be installed in the Proxy ARP/ND table (if enabled)
table (if enabled) or in the ARP/IP-VRF tables if the Broadcast or in the ARP/IP-VRF tables if the Broadcast Domain has an IRB.
Domain has an IRB. See Section 4.7.3 to see more information See Section 4.7.3 for more information about Proxy ARP/ND and
about Proxy-ARP/ND and Section 4.3. for more details about IRB and Section 4.3 for more details about IRB and Layer 3 services.
Layer-3 services.
Refer to [RFC7432] and [RFC8365] for more information about the MAC/ Refer to [RFC7432] and [RFC8365] for more information about the MAC/
IP Advertisement route and forwarding of known unicast traffic. IP Advertisement route and the forwarding of known unicast traffic.
4.3. EVPN Basic Applicability for Layer-3 Services 4.3. EVPN Basic Applicability for Layer 3 Services
[RFC9136] and [RFC9135] are the reference documents that describe how [RFC9136] and [RFC9135] are the reference documents that describe how
EVPN can be used for Layer-3 services. Inter Subnet Forwarding in EVPN can be used for Layer 3 services. Inter-subnet forwarding in
EVPN networks is implemented via IRB interfaces between Broadcast EVPN networks is implemented via IRB interfaces between Broadcast
Domains and IP-VRFs. An EVPN Broadcast Domain corresponds to an IP Domains and IP-VRFs. An EVPN Broadcast Domain corresponds to an IP
subnet. When IP packets generated in a Broadcast Domain are destined subnet. When IP packets generated in a Broadcast Domain are destined
to a different subnet (different Broadcast Domain) of the same to a different subnet (different Broadcast Domain) of the same
tenant, the packets are sent to the IRB attached to the local tenant, the packets are sent to the IRB attached to the local
Broadcast Domain in the source NVE. As discussed in [RFC9135], Broadcast Domain in the source NVE. As discussed in [RFC9135],
depending on how the IP packets are forwarded between the ingress NVE depending on how the IP packets are forwarded between the ingress NVE
and the egress NVE, there are two forwarding models: Asymmetric and and the egress NVE, there are two forwarding models: Asymmetric and
Symmetric model. Symmetric.
The Asymmetric model is illustrated in the example of Figure 2 and it The Asymmetric model is illustrated in the example of Figure 2, and
requires the configuration of all the Broadcast Domains of the tenant it requires the configuration of all the Broadcast Domains of the
in all the NVEs attached to the same tenant. In that way, there is tenant in all the NVEs attached to the same tenant. That way, there
no need to advertise IP Prefixes between NVEs since all the NVEs are is no need to advertise IP Prefixes between NVEs since all the NVEs
attached to all the subnets. It is called Asymmetric because the are attached to all the subnets. It is called "Asymmetric" because
ingress and egress NVEs do not perform the same number of lookups in the ingress and egress NVEs do not perform the same number of lookups
the data plane. In Figure 2, if TS1 and TS2 are in different in the data plane. In Figure 2, if TS1 and TS2 are in different
subnets, and TS1 sends IP packets to TS2, the following lookups are subnets and TS1 sends IP packets to TS2, the following lookups are
required in the data path: a MAC lookup (on BD1's table), an IP required in the data path: a MAC lookup at BD1's table, an IP lookup
lookup (on the IP-VRF) and a MAC lookup (on BD2's table) at the at the IP-VRF, a MAC lookup at BD2's table at the ingress NVE1, and
ingress NVE1 and then only a MAC lookup at the egress NVE. The two only a MAC lookup at the egress NVE. The two IP-VRFs in Figure 2 are
IP-VRFs in Figure 2 are not connected by tunnels and all the not connected by tunnels, and all the connectivity between the NVEs
connectivity between the NVEs is done based on tunnels between the is done based on tunnels between the Broadcast Domains.
Broadcast Domains.
+-------------------------------------+ +-------------------------------------+
| EVPN NVO3 | | EVPN NVO3 |
| | | |
NVE1 NVE2 NVE1 NVE2
+--------------------+ +--------------------+ +--------------------+ +--------------------+
| +---+IRB +------+ | | +------+IRB +---+ | | +---+IRB +------+ | | +------+IRB +---+ |
TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD1| | TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD1| |
| +---+ | | | | | | +---+ | | +---+ | | | | | | +---+ |
| +---+ | | | | | | +---+ | | +---+ | | | | | | +---+ |
| |BD2|----| | | | | |----|BD2|----TS2 | |BD2|----| | | | | |----|BD2|----TS2
| +---+IRB +------+ | | +------+IRB +---+ | | +---+IRB +------+ | | +------+IRB +---+ |
+--------------------+ +--------------------+ +--------------------+ +--------------------+
| | | |
+-------------------------------------+ +-------------------------------------+
Figure 2: EVPN for L3 in an NVO3 Network - Asymmetric model Figure 2: EVPN for L3 in an NVO3 Network - Asymmetric Model
In the Symmetric model, depicted in Figure 3, the same number of data In the Symmetric model, depicted in Figure 3, the same number of data
path lookups is needed at the ingress and egress NVEs. For example, path lookups is needed at the ingress and egress NVEs. For example,
if TS1 sends IP packets to TS3, the following data path lookups are if TS1 sends IP packets to TS3, the following data path lookups are
required: a MAC lookup at NVE1's BD1 table, an IP lookup at NVE1's required: a MAC lookup at NVE1's BD1 table, an IP lookup at NVE1's
IP-VRF and then IP lookup and MAC lookup at NVE2's IP-VRF and BD3 IP-VRF, and an IP lookup and MAC lookup at NVE2's IP-VRF and BD3,
respectively. In the Symmetric model, the Inter Subnet connectivity respectively. In the Symmetric model, the inter-subnet connectivity
between NVEs is done based on tunnels between the IP-VRFs. between NVEs is done based on tunnels between the IP-VRFs.
+-------------------------------------+ +-------------------------------------+
| EVPN NVO3 | | EVPN NVO3 |
| | | |
NVE1 NVE2 NVE1 NVE2
+--------------------+ +--------------------+ +--------------------+ +--------------------+
| +---+IRB +------+ | | +------+IRB +---+ | | +---+IRB +------+ | | +------+IRB +---+ |
TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD3|-----TS3 TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD3|-----TS3
| +---+ | | | | | | +---+ | | +---+ | | | | | | +---+ |
| +---+IRB | | | | +------+ | | +---+IRB | | | | +------+ |
TS2-----|BD2|----| | | +--------------------+ TS2-----|BD2|----| | | +--------------------+
| +---+ +------+ | | | +---+ +------+ | |
+--------------------+ | +--------------------+ |
| | | |
+-------------------------------------+ +-------------------------------------+
Figure 3: EVPN for L3 in an NVO3 Network - Symmetric model Figure 3: EVPN for L3 in an NVO3 Network - Symmetric Model
The Symmetric model scales better than the Asymmetric model because The Symmetric model scales better than the Asymmetric model because
it does not require the NVEs to be attached to all the tenant's it does not require the NVEs to be attached to all the tenant's
subnets. However, it requires the use of NVO3 tunnels on the IP-VRFs subnets. However, it requires the use of NVO3 tunnels on the IP-VRFs
and the exchange of IP Prefixes between the NVEs in the control and the exchange of IP Prefixes between the NVEs in the control
plane. EVPN uses MAC/IP Advertisement routes for the exchange of plane. EVPN uses MAC/IP Advertisement routes for the exchange of
host IP routes and IP Prefixes routes for the exchange of prefixes of host IP routes and IP Prefix routes for the exchange of prefixes of
any length (including host routes too). As an example, in Figure 3, any length, including host routes. As an example, in Figure 3, NVE2
NVE2 needs to advertise TS3's host route and/or TS3's subnet, so that needs to advertise TS3's host route and/or TS3's subnet so that the
the IP lookup on NVE1's IP-VRF succeeds. IP lookup on NVE1's IP-VRF succeeds.
[RFC9135] specifies the use of MAC/IP Advertisement routes for the [RFC9135] specifies the use of MAC/IP Advertisement routes for the
advertisement of host routes. Section 4.4.1 in [RFC9136] specifies advertisement of host routes. Section 4.4.1 of [RFC9136] specifies
the use of IP Prefix routes for the advertisement of IP Prefixes in the use of IP Prefix routes for the advertisement of IP Prefixes in
an "Interface-less IP-VRF-to-IP-VRF Model". The Symmetric model for an "Interface-less IP-VRF-to-IP-VRF Model". The Symmetric model for
host routes can be implemented following either approach: host routes can be implemented following either approach:
a. [RFC9135] uses MAC/IP Advertisement routes to convey the a. [RFC9135] uses MAC/IP Advertisement routes to convey the
information to populate Layer-2, ARP/ND and Layer-3 Forwarding information to populate Layer 2, ARP/ND, and Layer 3 Forwarding
Information Base tables in the remote NVE. For instance, in Information Base tables in the remote NVE. For instance, in
Figure 3, NVE2 would advertise a MAC/IP Advertisement route with Figure 3, NVE2 would advertise a MAC/IP Advertisement route with
TS3's IP and MAC addresses, and including two labels/Virtual TS3's IP and MAC addresses and include two labels / VNIs: a
Network Identifiers: a label-3/VNI-3 that identifies BD3 for MAC label-3/VNI-3 that identifies BD3 for MAC lookup (that would be
lookup (that would be used for Layer-2 traffic in case NVE1 was used for Layer 2 traffic in case NVE1 was attached to BD3 too)
attached to BD3 too) and a label-1/VNI-1 that identifies the IP- and a label-1/VNI-1 that identifies the IP-VRF for IP lookup
VRF for IP lookup (and will be used for Layer-3 traffic). NVE1 (that would be used for Layer 3 traffic). NVE1 imports the MAC/
imports the MAC/IP Advertisement route and installs TS3's IP in IP Advertisement route and installs TS3's IP in the IP-VRF route
the IP-VRF route table with label-1/VNI-1. Traffic from e.g., table with label-1/VNI-1. Traffic, e.g., from TS2 to TS3, would
TS2 to TS3, will be encapsulated with label-1/VNI-1 and forwarded be encapsulated with label-1/VNI-1 and forwarded to NVE2.
to NVE2.
b. [RFC9136] uses MAC/IP Advertisement routes to convey the b. [RFC9136] uses MAC/IP Advertisement routes to convey the
information to populate the Layer-2 Forwarding Information Base information to populate the Layer 2 Forwarding Information Base,
and ARP/ND tables, and IP Prefix routes to populate the IP-VRF ARP/ND tables, and IP Prefix routes to populate the IP-VRF Layer
Layer-3 Forwarding Information Base table. For instance, in 3 Forwarding Information Base table. For instance, in Figure 3,
Figure 3, NVE2 would advertise a MAC/IP Advertisement route NVE2 would advertise a MAC/IP Advertisement route including TS3's
including TS3's MAC and IP addresses with a single label-3/VNI-3. MAC and IP addresses with a single label-3/VNI-3. In this
In this example, this MAC/IP Advertisement route wouldn't be example, this MAC/IP Advertisement route wouldn't be imported by
imported by NVE1 because NVE1 is not attached to BD3. In NVE1 because NVE1 is not attached to BD3. In addition, NVE2
addition, NVE2 would advertise an IP Prefix route with TS3's IP would advertise an IP Prefix route with TS3's IP address and
address and label-1/VNI-1. This IP Prefix route would be label-1/VNI-1. This IP Prefix route would be imported by NVE1's
imported by NVE1's IP-VRF and the host route installed in the IP-VRF and the host route installed in the Layer 3 Forwarding
Layer-3 Forwarding Information Base associated to label-1/VNI-1. Information Base associated with label-1/VNI-1. Traffic from TS2
Traffic from TS2 to TS3 would be encapsulated with label-1/VNI-1. to TS3 would be encapsulated with label-1/VNI-1.
4.4. EVPN as Control Plane for NVO3 Encapsulations and GENEVE 4.4. EVPN as Control Plane for NVO3 Encapsulations and Geneve
[RFC8365] describes how to use EVPN for NVO3 encapsulations, such us [RFC8365] describes how to use EVPN for NVO3 encapsulations, such us
VXLAN, nvGRE or MPLSoGRE. The procedures can be easily applicable to VXLAN, nvGRE, or MPLSoGRE. The procedures can be easily applicable
any other NVO3 encapsulation, in particular GENEVE. to any other NVO3 encapsulation, particularly Geneve.
The Generic Network Virtualization Encapsulation [RFC8926] is the Geneve [RFC8926] is the proposed standard encapsulation specified in
proposed standard encapsulation specified in the IETF Network the IETF Network Virtualization Overlays Working Group. The EVPN
Virtualization Overlays Working Group. The EVPN control plane can control plane can signal the Geneve encapsulation type in the BGP
signal the GENEVE encapsulation type in the BGP Tunnel Encapsulation Tunnel Encapsulation Extended Community (see [RFC9012]).
Extended Community (see [RFC9012]).
GENEVE requires a control plane [I-D.ietf-nvo3-encap] to: Geneve requires a control plane [NVO3-ENCAP] to:
1. Negotiate a subset of GENEVE option TLVs that can be carried on a * Negotiate a subset of Geneve option TLVs that can be carried on a
GENEVE tunnel Geneve tunnel,
2. Enforce an order for GENEVE option TLVs and * Enforce an order for Geneve option TLVs, and
3. Limit the total number of options that could be carried on a * Limit the total number of options that could be carried on a
GENEVE tunnel. Geneve tunnel.
The EVPN control plane can easily extend the BGP Tunnel Encapsulation The EVPN control plane can easily extend the BGP Tunnel Encapsulation
Attribute sub-TLV [RFC9012] to specify the GENEVE tunnel options that attribute sub-TLV [RFC9012] to specify the Geneve tunnel options that
can be received or transmitted over a GENEVE tunnels by a given NVE. can be received or transmitted over a Geneve tunnel by a given NVE.
[I-D.ietf-bess-evpn-geneve] describes the EVPN control plane [BESS-EVPN-GENEVE] describes the EVPN control plane extensions to
extensions to support GENEVE. support Geneve.
4.5. EVPN OAM and Application to NVO3 4.5. EVPN OAM and Application to NVO3
EVPN OAM (as in [I-D.ietf-bess-evpn-lsp-ping]) defines mechanisms to EVPN Operations, Administration, and Maintenance (OAM), as described
detect data plane failures in an EVPN deployment over an MPLS in [BESS-EVPN-LSP-PING], defines mechanisms to detect data plane
network. These mechanisms detect failures related to P2P and P2MP failures in an EVPN deployment over an MPLS network. These
connectivity, for multi-tenant unicast and multicast Layer-2 traffic, mechanisms detect failures related to point-to-point (P2P) and Point-
between multi-tenant access nodes connected to EVPN PE(s), and in a to-Multipoint (P2MP) connectivity, for multi-tenant unicast and
single-homed, single-active or all-active redundancy model. multicast Layer 2 traffic, between multi-tenant access nodes
connected to EVPN PE(s), and in a single-homed, Single-Active, or
All-Active redundancy model.
In general, EVPN OAM mechanisms defined for EVPN deployed in MPLS In general, EVPN OAM mechanisms defined for EVPN deployed in MPLS
networks are equally applicable for EVPN in NVO3 networks. networks are equally applicable for EVPN in NVO3 networks.
4.6. EVPN as the Control Plane for NVO3 Security 4.6. EVPN as the Control Plane for NVO3 Security
EVPN can be used to signal the security protection capabilities of a EVPN can be used to signal the security protection capabilities of a
sender NVE, as well as what portion of an NVO3 packet (taking a sender NVE, as well as what portion of an NVO3 packet (taking a
GENEVE packet as an example) can be protected by the sender NVE, to Geneve packet as an example) can be protected by the sender NVE, to
ensure the privacy and integrity of tenant traffic carried over the ensure the privacy and integrity of tenant traffic carried over the
NVO3 tunnels [I-D.sajassi-bess-secure-evpn]. NVO3 tunnels [BESS-SECURE-EVPN].
4.7. Advanced EVPN Features for NVO3 Networks 4.7. Advanced EVPN Features for NVO3 Networks
This section describes how EVPN can be used to deliver advanced This section describes how EVPN can be used to deliver advanced
capabilities in NVO3 networks. capabilities in NVO3 networks.
4.7.1. Virtual Machine (VM) Mobility 4.7.1. Virtual Machine (VM) Mobility
[RFC7432] replaces the classic Ethernet Flood-and-Learn behavior [RFC7432] replaces the classic Ethernet "flood and learn" behavior
among NVEs with BGP-based MAC learning, which in return provides more among NVEs with BGP-based MAC learning. In return, this provides
control over the location of MAC addresses in the Broadcast Domain more control over the location of MAC addresses in the Broadcast
and consequently advanced features, such as MAC Mobility. If we Domain and consequently advanced features, such as MAC Mobility. If
assume that VM Mobility means the VM's MAC and IP addresses move with we assume that Virtual Machine (VM) Mobility means the VM's MAC and
the VM, EVPN's MAC Mobility is the required procedure that IP addresses move with the VM, EVPN's MAC Mobility is the required
facilitates VM Mobility. According to [RFC7432] section 15, when a procedure that facilitates VM Mobility. According to Section 15 of
MAC is advertised for the first time in a Broadcast Domain, all the [RFC7432], when a MAC is advertised for the first time in a Broadcast
NVEs attached to the Broadcast Domain will store Sequence Number zero Domain, all the NVEs attached to the Broadcast Domain will store
for that MAC. When the MAC "moves" within the same Broadcast Domain Sequence Number zero for that MAC. When the MAC "moves" to a remote
but to a remote NVE, the NVE that just learned locally the MAC, NVE within the same Broadcast Domain, the NVE that just learned the
increases the Sequence Number in the MAC/IP Advertisement route's MAC MAC locally increases the Sequence Number in the MAC/IP Advertisement
Mobility extended community to indicate that it owns the MAC now. route's MAC Mobility extended community to indicate that it owns the
That makes all the NVE in the Broadcast Domain change their tables MAC now. That makes all the NVEs in the Broadcast Domain change
immediately with no need to wait for any aging timer. EVPN their tables immediately with no need to wait for any aging timer.
guarantees a fast MAC Mobility without flooding or black-holes in the EVPN guarantees a fast MAC Mobility without flooding or packet drops
Broadcast Domain. in the Broadcast Domain.
4.7.2. MAC Protection, Duplication Detection and Loop Protection 4.7.2. MAC Protection, Duplication Detection, and Loop Protection
The advertisement of MACs in the control plane, allows advanced The advertisement of MACs in the control plane allows advanced
features such as MAC protection, Duplication Detection and Loop features such as MAC Protection, Duplication Detection, and Loop
Protection. Protection.
[RFC7432] MAC Protection refers to EVPN's ability to indicate - in a In a MAC/IP Advertisement route, MAC Protection refers to EVPN's
MAC/IP Advertisement route - that a MAC must be protected by the NVE ability to indicate that a MAC must be protected by the NVE receiving
receiving the route. The Protection is indicated in the "Sticky bit" the route [RFC7432]. The Protection is indicated in the "Sticky bit"
of the MAC Mobility extended community sent along the MAC/IP of the MAC Mobility extended community sent along the MAC/IP
Advertisement route for a MAC. NVEs' Attachment Circuits that are Advertisement route for a MAC. NVEs' Attachment Circuits that are
connected to subject-to-be-protected servers or VMs, may set the connected to subject-to-be-protected servers or VMs may set the
Sticky bit on the MAC/IP Advertisement routes sent for the MACs Sticky bit on the MAC/IP Advertisement routes sent for the MACs
associated to the Attachment Circuits. Also, statically configured associated with the Attachment Circuits. Also, statically configured
MAC addresses should be advertised as Protected MAC addresses, since MAC addresses should be advertised as Protected MAC addresses since
they are not subject to MAC Mobility procedures. they are not subject to MAC Mobility procedures.
[RFC7432] MAC Duplication Detection refers to EVPN's ability to MAC Duplication Detection refers to EVPN's ability to detect
detect duplicate MAC addresses. A "MAC move" is a relearn event that duplicate MAC addresses [RFC7432]. A "MAC move" is a relearn event
happens at an access Attachment Circuit or through a MAC/IP that happens at an access Attachment Circuit or through a MAC/IP
Advertisement route with a Sequence Number that is higher than the Advertisement route with a Sequence Number that is higher than the
stored one for the MAC. When a MAC moves a number of times N within stored one for the MAC. When a MAC moves a number of times (N)
an M-second window between two NVEs, the MAC is declared as Duplicate within an M-second window between two NVEs, the MAC is declared as a
and the detecting NVE does not re-advertise the MAC anymore. duplicate and the detecting NVE does not re-advertise the MAC
anymore.
[RFC7432] provides MAC Duplication Detection, and with an extension [RFC7432] provides MAC Duplication Detection, and with an extension,
it can protect the Broadcast Domain against loops created by backdoor it can protect the Broadcast Domain against loops created by backdoor
links between NVEs. The same principle (based on the Sequence links between NVEs. The same principle (based on the Sequence
Number) may be extended to protect the Broadcast Domain against Number) may be extended to protect the Broadcast Domain against
loops. When a MAC is detected as duplicate, the NVE may install it loops. When a MAC is detected as a duplicate, the NVE may install it
as a drop-MAC and discard received frames with source MAC address or as a drop-MAC and discard received frames with source MAC address or
destination MAC address matching that duplicate MAC. The MAC the destination MAC address matching that duplicate MAC. The MAC
Duplication extension to support Loop Protection is described in Duplication extension to support Loop Protection is described in
[I-D.ietf-bess-rfc7432bis], section 15.3. Section 15.3 of [BESS-RFC7432BIS].
4.7.3. Reduction/Optimization of BUM Traffic in Layer-2 Services 4.7.3. Reduction/Optimization of BUM Traffic in Layer 2 Services
In Broadcast Domains with a significant amount of flooding due to In Broadcast Domains with a significant amount of flooding due to
Unknown unicast and Broadcast frames, EVPN may help reduce and Unknown Unicast and broadcast frames, EVPN may help reduce and
sometimes even suppress the flooding. sometimes even suppress the flooding.
In Broadcast Domains where most of the Broadcast traffic is caused by In Broadcast Domains where most of the broadcast traffic is caused by
ARP (Address Resolution Protocol) and ND (Neighbor Discovery) the Address Resolution Protocol (ARP) and the Neighbor Discovery
protocols on the Tenant Systems, EVPN's Proxy-ARP and Proxy-ND Protocol (NDP) on the Tenant Systems, EVPN's Proxy ARP and Proxy ND
capabilities may reduce the flooding drastically. The use of Proxy- capabilities may reduce the flooding drastically. The use of Proxy
ARP/ND is specified in [RFC9161]. ARP/ND is specified in [RFC9161].
Proxy-ARP/ND procedures along with the assumption that Tenant Systems Proxy ARP/ND procedures, along with the assumption that Tenant
always issue a GARP (Gratuitous ARP) or an unsolicited Neighbor Systems always issue a Gratuitous ARP (GARP) or an unsolicited
Advertisement message when they come up in the Broadcast Domain, may Neighbor Advertisement message when they come up in the Broadcast
drastically reduce the unknown unicast flooding in the Broadcast Domain, may drastically reduce the Unknown Unicast flooding in the
Domain. Broadcast Domain.
The flooding caused by Tenant Systems' IGMP/MLD or PIM messages in The flooding caused by Tenant Systems' IGMP / Multicast Listener
the Broadcast Domain may also be suppressed by the use of IGMP/MLD Discovery (MLD) or PIM messages in the Broadcast Domain may also be
and PIM Proxy functions, as specified in [RFC9251] and suppressed by the use of IGMP/MLD and PIM Proxy functions, as
[I-D.skr-bess-evpn-pim-proxy]. These two documents also specify how specified in [RFC9251] and [BESS-EVPN-PIM-PROXY]. These two
to forward IP multicast traffic efficiently within the same Broadcast documents also specify how to forward IP multicast traffic
Domain, translate soft state IGMP/MLD/PIM messages into hard state efficiently within the same Broadcast Domain, translate soft state
BGP routes and provide fast-convergence redundancy for IP Multicast IGMP/MLD/PIM messages into hard state BGP routes, and provide fast
on multi-homed Ethernet Segments (ESes). convergence redundancy for IP multicast on multihomed ESes.
4.7.4. Ingress Replication (IR) Optimization for BUM Traffic 4.7.4. Ingress Replication (IR) Optimization for BUM Traffic
When an NVE attached to a given Broadcast Domain needs to send BUM When an NVE attached to a given Broadcast Domain needs to send BUM
traffic for the Broadcast Domain to the remote NVEs attached to the traffic for the Broadcast Domain to the remote NVEs attached to the
same Broadcast Domain, Ingress Replication is a very common option in same Broadcast Domain, Ingress Replication is a very common option in
NVO3 networks, since it is completely independent of the multicast NVO3 networks since it is completely independent of the multicast
capabilities of the underlay network. Also, if the optimization capabilities of the underlay network. Also, if the optimization
procedures to reduce/suppress the flooding in the Broadcast Domain procedures to reduce/suppress the flooding in the Broadcast Domain
are enabled (Section 4.7.3), in spite of creating multiple copies of are enabled (Section 4.7.3) in spite of creating multiple copies of
the same frame at the ingress NVE, Ingress Replication may be good the same frame at the ingress NVE, Ingress Replication may be good
enough. However, in Broadcast Domains where Multicast (or Broadcast) enough. However, in Broadcast Domains where Multicast (or Broadcast)
traffic is significant, Ingress Replication may be very inefficient traffic is significant, Ingress Replication may be very inefficient
and cause performance issues on virtual-switch-based NVEs. and cause performance issues on virtual switch-based NVEs.
[I-D.ietf-bess-evpn-optimized-ir] specifies the use of AR (Assisted [BESS-EVPN-OPTIMIZED-IR] specifies the use of Assisted Replication
Replication) NVO3 tunnels in EVPN Broadcast Domains. AR retains the (AR) NVO3 tunnels in EVPN Broadcast Domains. AR retains the
independence of the underlay network while providing a way to forward independence of the underlay network while providing a way to forward
Broadcast and Multicast traffic efficiently. AR uses AR-REPLICATORs Broadcast and multicast traffic efficiently. AR uses AR-REPLICATORs
that can replicate the Broadcast/Multicast traffic on behalf of the that can replicate the broadcast/multicast traffic on behalf of the
AR-LEAF NVEs. The AR-LEAF NVEs are typically virtual-switches or AR-LEAF NVEs. The AR-LEAF NVEs are typically virtual switches or
NVEs with limited replication capabilities. AR can work in a single- NVEs with limited replication capabilities. AR can work in a single-
stage replication mode (Non-Selective Mode) or in a dual-stage stage replication mode (Non-Selective Mode) or in a dual-stage
replication mode (Selective Mode). Both modes are detailed in replication mode (Selective Mode). Both modes are detailed in
[I-D.ietf-bess-evpn-optimized-ir]. [BESS-EVPN-OPTIMIZED-IR].
In addition, [I-D.ietf-bess-evpn-optimized-ir] also describes a In addition, [BESS-EVPN-OPTIMIZED-IR] describes a procedure to avoid
procedure to avoid sending Broadcast, Multicast or Unknown unicast to sending BUM to certain NVEs that do not need that type of traffic.
certain NVEs that do not need that type of traffic. This is done by This is done by enabling Pruned Flood Lists (PFLs) on a given
enabling PFL (Pruned Flood Lists) on a given Broadcast Domain. For Broadcast Domain. For instance, a virtual switch NVE that learns all
instance, a virtual-switch NVE that learns all its local MAC its local MAC addresses for a Broadcast Domain via a Cloud Management
addresses for a Broadcast Domain via Cloud Management System, does System does not need to receive the Broadcast Domain's Unknown
not need to receive the Broadcast Domain's Unknown unicast traffic. Unicast traffic. PFLs help optimize the BUM flooding in the
Pruned Flood Lists help optimize the BUM flooding in the Broadcast Broadcast Domain.
Domain.
4.7.5. EVPN Multi-Homing 4.7.5. EVPN Multihoming
Another fundamental concept in EVPN is multi-homing. A given Tenant Another fundamental concept in EVPN is multihoming. A given Tenant
System can be multi-homed to two or more NVEs for a given Broadcast System can be multihomed to two or more NVEs for a given Broadcast
Domain, and the set of links connected to the same Tenant System is Domain, and the set of links connected to the same Tenant System is
defined as Ethernet Segment (ES). EVPN supports single-active and defined as an ES. EVPN supports Single-Active and All-Active
all-active multi-homing. In single-active multi-homing only one link multihoming. In Single-Active multihoming, only one link in the
in the Ethernet Segment is active. In all-active multi-homing all Ethernet Segment is active. In All-Active multihoming, all the links
the links in the Ethernet Segment are active for unicast traffic. in the Ethernet Segment are active for unicast traffic. Both modes
Both modes support load-balancing: support load-balancing:
* Single-active multi-homing means per-service load-balancing to/ * Single-Active multihoming means per-service load-balancing to/from
from the Tenant System. For example, in Figure 1, for BD1, only the Tenant System. For example, in Figure 1 for BD1, only one of
one of the NVEs can forward traffic from/to TS2. For a different the NVEs can forward traffic from/to TS2. For a different
Broadcast Domain, the other NVE may forward traffic. Broadcast Domain, the other NVE may forward traffic.
* All-active multi-homing means per-flow load-balanding for unicast * All-active multihoming means per-flow load-balancing for unicast
frames to/from the Tenant System. That is, in Figure 1 and for frames to/from the Tenant System. That is, in Figure 1 and for
BD1, both NVE4 and NVE5 can forward known unicast traffic to/from BD1, both NVE4 and NVE5 can forward known unicast traffic to/from
TS3. For BUM traffic only one of the two NVEs can forward traffic TS3. For BUM traffic, only one of the two NVEs can forward
to TS3, and both can forward traffic from TS3. traffic to TS3, and both can forward traffic from TS3.
There are two key aspects in the EVPN multi-homing procedures: There are two key aspects in the EVPN multihoming procedures:
* DF (Designated Forwarder) election: the Designated Forwarder is Designated Forwarder (DF) election:
the NVE that forwards the traffic to the Ethernet Segment in The Designated Forwarder is the NVE that forwards the traffic to
single-active mode. In case of all-active, the Designated the Ethernet Segment in Single-Active mode. In the case of All-
Forwarder is the NVE that forwards the BUM traffic to the Ethernet Active mode, the Designated Forwarder is the NVE that forwards the
Segment. BUM traffic to the Ethernet Segment.
* Split-horizon function: prevents the Tenant System from receiving Split-horizon function:
echoed BUM frames that the Tenant System itself sent to the Prevents the Tenant System from receiving echoed BUM frames that
Ethernet Segment. This is especially relevant in all-active the Tenant System itself sent to the Ethernet Segment. This is
Ethernet Segments, where the Tenant System may forward BUM frames especially relevant in All-Active ESes where the TS may forward
to a non-Designated Forwarder NVE that can flood the BUM frames BUM frames to a Non-Designated Forwarder NVE that can flood the
back to the Designated Forwarder NVE and then the Tenant System. BUM frames back to the Designated Forwarder NVE and then back to
As an example, in Figure 1, assuming NVE4 is the Designated the TS. As an example, assuming NVE4 is the Designated Forwarder
Forwarder for ESI-2 in BD1, BUM frames sent from TS3 to NVE5 will for ESI-2 in BD1, Figure 1 shows that BUM frames sent from TS3 to
be received at NVE4 and, since NVE4 is the Designated Forwarder NVE5 will be received at NVE4. NVE4 will forward them back to TS3
for BD1, it will forward them back to TS3. Split-horizon allows since NVE4 is the Designated Forwarder for BD1. Split-horizon
NVE4 (and any multi-homed NVE for that matter) to identify if an allows NVE4 (and any multihomed NVE for that matter) to identify
EVPN BUM frame is coming from the same Ethernet Segment or if an EVPN BUM frame is coming from the same Ethernet Segment or a
different, and if the frame belongs to the same ESI-2, NVE4 will different one. If the frame belongs to the same ESI-2, NVE4 will
not forward the BUM frame to TS3, in spite of being the Designated not forward the BUM frame to TS3 in spite of being the Designated
Forwarder. Forwarder.
While [RFC7432] describes the default algorithm for the Designated While [RFC7432] describes the default algorithm for the Designated
Forwarder Election, [RFC8584] and [I-D.ietf-bess-evpn-pref-df] Forwarder election, [RFC8584] and [BESS-EVPN-PREF-DF] specify other
specify other algorithms and procedures that optimize the Designated algorithms and procedures that optimize the Designated Forwarder
Forwarder Election. election.
The Split-horizon function is specified in [RFC7432] and it is The split-horizon function is specified in [RFC7432], and it is
carried out by using a special ESI-label that it identifies in the carried out by using a special ESI-label that it identifies in the
data path, all the BUM frames being originated from a given NVE and data path with all the BUM frames originating from a given NVE and
Ethernet Segment. Since the ESI-label is an MPLS label, it cannot be Ethernet Segment. Since the ESI-label is an MPLS label, it cannot be
used in all the non-MPLS NVO3 encapsulations, therefore [RFC8365] used in all the non-MPLS NVO3 encapsulations. Therefore, [RFC8365]
defines a modified Split-horizon procedure that is based on the defines a modified split-horizon procedure that is based on the
source IP address of the NVO3 tunnel, and it is known as "Local- source IP address of the NVO3 tunnel; it is known as "Local-Bias".
Bias". It is worth noting that Local-Bias only works for all-active It is worth noting that Local-Bias only works for All-Active
multi-homing, and not for single-active multi-homing. multihoming, and not for Single-Active multihoming.
4.7.6. EVPN Recursive Resolution for Inter-Subnet Unicast Forwarding 4.7.6. EVPN Recursive Resolution for Inter-subnet Unicast Forwarding
Section 4.3 describes how EVPN can be used for Inter Subnet Section 4.3 describes how EVPN can be used for inter-subnet
Forwarding among subnets of the same tenant. MAC/IP Advertisement forwarding among subnets of the same tenant. MAC/IP Advertisement
routes and IP Prefix routes allow the advertisement of host routes routes and IP Prefix routes allow the advertisement of host routes
and IP Prefixes (IP Prefix route) of any length. The procedures and IP Prefixes (IP Prefix route) of any length. The procedures
outlined by Section 4.3 are similar to the ones in [RFC4364], only outlined by Section 4.3 are similar to the ones in [RFC4364], but
for NVO3 tunnels. However, [RFC9136] also defines advanced Inter they are only for NVO3 tunnels. However, [RFC9136] also defines
Subnet Forwarding procedures that allow the resolution of IP Prefix advanced inter-subnet forwarding procedures that allow the resolution
routes to not only BGP next-hops but also "overlay indexes" that can of IP Prefix routes not only to BGP next hops but also to "overlay
be a MAC, a Gateway IP (GW-IP) or an ESI, all of them in the tenant indexes" that can be a MAC, a Gateway IP (GW-IP), or an ESI, all of
space. them in the tenant space.
Figure 4 illustrates an example that uses Recursive Resolution to a Figure 4 illustrates an example that uses Recursive Resolution to a
GW-IP as per [RFC9136] section 4.4.2. In this example, IP-VRFs in GW-IP as per Section 4.4.2 of [RFC9136]. In this example, IP-VRFs in
NVE1 and NVE2 are connected by an SBD (Supplementary Broadcast NVE1 and NVE2 are connected by a Supplementary Broadcast Domain
Domain). An SBD is a Broadcast Domain that connects all the IP-VRFs (SBD). An SBD is a Broadcast Domain that connects all the IP-VRFs of
of the same tenant, via IRB, and has no Attachment Circuits. NVE1 the same tenant via IRB and has no Attachment Circuits. NVE1
advertises the host route TS2-IP/L (IP address and Prefix Length of advertises the host route TS2-IP/L (IP address and Prefix Length of
TS2) in an IP Prefix route with overlay index GW-IP=IP1. Also, IP1 TS2) in an IP Prefix route with overlay index GW-IP=IP1. Also, IP1
is advertised in an MAC/IP Advertisement route associated to M1, is advertised in a MAC/IP Advertisement route associated with M1,
VNI-S and BGP next-hop NVE1. Upon importing the two routes, NVE2 VNI-S, and BGP next-hop NVE1. Upon importing the two routes, NVE2
installs TS2-IP/L in the IP-VRF with a next-hop that is the GW-IP installs TS2-IP/L in the IP-VRF with a next hop that is the GW-IP
IP1. NVE2 also installs M1 in the Supplementary Broadcast Domain, IP1. NVE2 also installs M1 in the Supplementary Broadcast Domain,
with VNI-S and NVE1 as next-hop. If TS3 sends a packet with IP with VNI-S and NVE1 as next hop. If TS3 sends a packet with IP
DA=TS2, NVE2 will perform a Recursive Resolution of the IP Prefix DA=TS2, NVE2 will perform a Recursive Resolution of the IP Prefix
route prefix information to the forwarding information of the route prefix information to the forwarding information of the
correlated MAC/IP Advertisement route. The IP Prefix route's correlated MAC/IP Advertisement route. The IP Prefix route's
Recursive Resolution has several advantages such as better Recursive Resolution has several advantages, such as better
convergence in scaled networks (since multiple IP Prefix routes can convergence in scaled networks (since multiple IP Prefix routes can
be invalidated with a single withdrawal of the overlay index route) be invalidated with a single withdrawal of the overlay index route)
or the ability to advertise multiple IP Prefix routes from an overlay or the ability to advertise multiple IP Prefix routes from an overlay
index that can move or change dynamically. [RFC9136] describes a few index that can move or change dynamically. [RFC9136] describes a few
use-cases. use cases.
+-------------------------------------+ +-------------------------------------+
| EVPN NVO3 | | EVPN NVO3 |
| + | +
NVE1 NVE2 NVE1 NVE2
+--------------------+ +--------------------+ +--------------------+ +--------------------+
| +---+IRB +------+ | | +------+IRB +---+ | | +---+IRB +------+ | | +------+IRB +---+ |
TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD3|-----TS3 TS1-----|BD1|----|IP-VRF| | | |IP-VRF|----|BD3|-----TS3
| +---+ | |-(SBD)------(SBD)-| | +---+ | | +---+ | |-(SBD)------(SBD)-| | +---+ |
| +---+IRB | |IRB(IP1/M1) IRB+------+ | | +---+IRB | |IRB(IP1/M1) IRB+------+ |
TS2-----|BD2|----| | | +-----------+--------+ TS2-----|BD2|----| | | +-----------+--------+
| +---+ +------+ | | | +---+ +------+ | |
+--------------------+ | +--------------------+ |
| RT-2(M1,IP1,VNI-S,NVE1)--> | | RT-2(M1,IP1,VNI-S,NVE1)--> |
| RT-5(TS2-IP/L,GW-IP=IP1)--> | | RT-5(TS2-IP/L,GW-IP=IP1)--> |
+-------------------------------------+ +-------------------------------------+
Figure 4: EVPN for L3 - Recursive Resolution example Figure 4: EVPN for L3 - Recursive Resolution Example
4.7.7. EVPN Optimized Inter-Subnet Multicast Forwarding 4.7.7. EVPN Optimized Inter-subnet Multicast Forwarding
The concept of the Supplementary Broadcast Domain described in The concept of the Supplementary Broadcast Domain described in
Section 4.7.6 is also used in [I-D.ietf-bess-evpn-irb-mcast] for the Section 4.7.6 is also used in [BESS-EVPN-IRB-MCAST] for the
procedures related to Inter Subnet Multicast Forwarding across procedures related to inter-subnet multicast forwarding across
Broadcast Domains of the same tenant. For instance, Broadcast Domains of the same tenant. For instance,
[I-D.ietf-bess-evpn-irb-mcast] allows the efficient forwarding of IP [BESS-EVPN-IRB-MCAST] allows the efficient forwarding of IP multicast
multicast traffic from any Broadcast Domain to any other Broadcast traffic from any Broadcast Domain to any other Broadcast Domain (or
Domain (or even to the same Broadcast Domain where the Source even to the same Broadcast Domain where the source resides). The
resides). The [I-D.ietf-bess-evpn-irb-mcast] procedures are [BESS-EVPN-IRB-MCAST] procedures are supported along with EVPN
supported along with EVPN multi-homing, and for any tree allowed on multihoming and for any tree allowed on NVO3 networks, including IR
NVO3 networks, including IR or AR. [I-D.ietf-bess-evpn-irb-mcast] or AR. [BESS-EVPN-IRB-MCAST] also describes the interoperability
also describes the interoperability between EVPN and other multicast between EVPN and other multicast technologies such as Multicast VPN
technologies such as MVPN (Multicast VPN) and PIM for inter-subnet (MVPN) and PIM for inter-subnet multicast.
multicast.
[I-D.ietf-bess-evpn-mvpn-seamless-interop] describes another [BESS-EVPN-MVPN-SEAMLESS-INTEROP] describes another potential
potential solution to support EVPN to MVPN interoperability. solution to support EVPN to MVPN interoperability.
4.7.8. Data Center Interconnect (DCI) 4.7.8. Data Center Interconnect (DCI)
Tenant Layer-2 and Layer-3 services deployed on NVO3 networks must Tenant Layer 2 and Layer 3 services deployed on NVO3 networks must
often be extended to remote NVO3 networks that are connected via non- often be extended to remote NVO3 networks that are connected via non-
NOV3 Wide Area Networks (mostly MPLS based Wide Area Networks). NOV3 Wide Area Networks (WANs) (mostly MPLS-based WANs). [RFC9014]
[RFC9014] defines some architectural models that can be used to defines some architectural models that can be used to interconnect
interconnect NVO3 networks via MPLS Wide Area Networks. NVO3 networks via MPLS WANs.
When NVO3 networks are connected by MPLS Wide Area Networks, When NVO3 networks are connected by MPLS WANs, [RFC9014] specifies
[RFC9014] specifies how EVPN can be used end-to-end, in spite of how EVPN can be used end to end in spite of using a different
using a different encapsulation in the Wide Area Network. [RFC9014] encapsulation in the WAN. [RFC9014] also supports the use of NVO3 or
also supports the use of NVO3 or Segment Routing (encoding 32-bit or Segment Routing (encoding 32-bit or 128-bit Segment Identifiers into
128-bit Segment Identifiers into labels or IPv6 addresses labels or IPv6 addresses, respectively) transport tunnels in the WAN.
respectively) transport tunnels in the Wide Area Network.
Even if EVPN can also be used in the Wide Area Network for Layer-2 Even if EVPN can also be used in the WAN for Layer 2 and Layer 3
and Layer-3 services, there may be a need to provide a Gateway services, there may be a need to provide a Gateway function between
function between EVPN for NVO3 encapsulations and IPVPN for MPLS EVPN for NVO3 encapsulations and IP VPN for MPLS tunnels if the
tunnels, if the operator uses IPVPN in the Wide Area Network. operator uses IP VPN in the WAN. [BESS-EVPN-IPVPN-INTERWORKING]
[I-D.ietf-bess-evpn-ipvpn-interworking] specifies the interworking specifies the interworking function between EVPN and IP VPN for
function between EVPN and IPVPN for unicast Inter Subnet Forwarding. unicast inter-subnet forwarding. If inter-subnet multicast
If Inter Subnet Multicast Forwarding is also needed across an IPVPN forwarding is also needed across an IP VPN WAN, [BESS-EVPN-IRB-MCAST]
Wide Area Network, [I-D.ietf-bess-evpn-irb-mcast] describes the describes the required interworking between EVPN and MVPNs.
required interworking between EVPN and MVPN (Multicast Virtual
Private Networks).
5. Security Considerations 5. Security Considerations
This document does not introduce any new procedure or additional This document does not introduce any new procedure or additional
signaling in EVPN, and relies on the security considerations of the signaling in EVPN and relies on the security considerations of the
individual specifications used as a reference throughout the individual specifications used as a reference throughout the
document. In particular, and as mentioned in [RFC7432], control document. In particular, and as mentioned in [RFC7432], control
plane and forwarding path protection are aspects to secure in any plane and forwarding path protection are aspects to secure in any
EVPN domain, when applied to NVO3 networks. EVPN domain when applied to NVO3 networks.
[RFC7432] mentions security techniques such as those discussed in [RFC7432] mentions security techniques such as those discussed in
[RFC5925] to authenticate BGP messages, and those included in [RFC5925] to authenticate BGP messages, and those included in
[RFC4271], [RFC4272] and [RFC6952] to secure BGP are relevant for [RFC4271], [RFC4272], and [RFC6952] to secure BGP are relevant for
EVPN in NVO3 networks as well. EVPN in NVO3 networks as well.
6. IANA Considerations 6. IANA Considerations
None. This document has no IANA actions.
7. References 7. References
7.1. Normative References 7.1. Normative References
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/info/rfc7432>.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
2014, <https://www.rfc-editor.org/info/rfc7365>.
[RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L., [RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
Kreeger, L., and M. Napierala, "Problem Statement: Kreeger, L., and M. Napierala, "Problem Statement:
Overlays for Network Virtualization", RFC 7364, Overlays for Network Virtualization", RFC 7364,
DOI 10.17487/RFC7364, October 2014, DOI 10.17487/RFC7364, October 2014,
<https://www.rfc-editor.org/info/rfc7364>. <https://www.rfc-editor.org/info/rfc7364>.
7.2. Informative References [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
[RFC9136] Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
A. Sajassi, "IP Prefix Advertisement in Ethernet VPN 2014, <https://www.rfc-editor.org/info/rfc7365>.
(EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
<https://www.rfc-editor.org/info/rfc9136>.
[RFC9135] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J. [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Rabadan, "Integrated Routing and Bridging in Ethernet VPN Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
(EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021, Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
<https://www.rfc-editor.org/info/rfc9135>. 2015, <https://www.rfc-editor.org/info/rfc7432>.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 7.2. Informative References
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., [BESS-EVPN-GENEVE]
"Geneve: Generic Network Virtualization Encapsulation", Boutros, S., Ed., Sajassi, A., Drake, J., Rabadan, J., and
RFC 8926, DOI 10.17487/RFC8926, November 2020, S. Aldrin, "EVPN control plane for Geneve", Work in
<https://www.rfc-editor.org/info/rfc8926>. Progress, Internet-Draft, draft-ietf-bess-evpn-geneve-06,
26 May 2023, <https://datatracker.ietf.org/doc/html/draft-
ietf-bess-evpn-geneve-06>.
[I-D.ietf-nvo3-encap] [BESS-EVPN-IPVPN-INTERWORKING]
Boutros, S. and D. E. Eastlake, "Network Virtualization Rabadan, J., Ed., Sajassi, A., Ed., Rosen, E., Drake, J.,
Overlays (NVO3) Encapsulation Considerations", Work in Lin, W., Uttaro, J., and A. Simpson, "EVPN Interworking
Progress, Internet-Draft, draft-ietf-nvo3-encap-09, 7 with IPVPN", Work in Progress, Internet-Draft, draft-ietf-
October 2022, <https://datatracker.ietf.org/doc/html/ bess-evpn-ipvpn-interworking-08, 5 July 2023,
draft-ietf-nvo3-encap-09>. <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-ipvpn-interworking-08>.
[RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, [BESS-EVPN-IRB-MCAST]
"The BGP Tunnel Encapsulation Attribute", RFC 9012, Lin, W., Zhang, Z., Drake, J., Rosen, E., Ed., Rabadan,
DOI 10.17487/RFC9012, April 2021, J., and A. Sajassi, "EVPN Optimized Inter-Subnet Multicast
<https://www.rfc-editor.org/info/rfc9012>. (OISM) Forwarding", Work in Progress, Internet-Draft,
draft-ietf-bess-evpn-irb-mcast-09, 21 February 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-irb-mcast-09>.
[I-D.ietf-bess-evpn-lsp-ping] [BESS-EVPN-LSP-PING]
Jain, P., Sajassi, A., Salam, S., Boutros, S., and G. Jain, P., Sajassi, A., Salam, S., Boutros, S., and G.
Mirsky, "LSP-Ping Mechanisms for EVPN and PBB-EVPN", Work Mirsky, "LSP-Ping Mechanisms for EVPN and PBB-EVPN", Work
in Progress, Internet-Draft, draft-ietf-bess-evpn-lsp- in Progress, Internet-Draft, draft-ietf-bess-evpn-lsp-
ping-09, 10 December 2022, ping-11, 29 May 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess- <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-lsp-ping-09>. evpn-lsp-ping-11>.
[RFC9161] Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G., [BESS-EVPN-MVPN-SEAMLESS-INTEROP]
and T. King, "Operational Aspects of Proxy ARP/ND in Sajassi, A., Thiruvenkatasamy, K., Thoria, S., Gupta, A.,
Ethernet Virtual Private Networks", RFC 9161, and L. Jalil, "Seamless Multicast Interoperability between
DOI 10.17487/RFC9161, January 2022, EVPN and MVPN PEs", Work in Progress, Internet-Draft,
<https://www.rfc-editor.org/info/rfc9161>. draft-ietf-bess-evpn-mvpn-seamless-interop-05, 13 March
2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
bess-evpn-mvpn-seamless-interop-05>.
[RFC9251] Sajassi, A., Thoria, S., Mishra, M., Patel, K., Drake, J., [BESS-EVPN-OPTIMIZED-IR]
and W. Lin, "Internet Group Management Protocol (IGMP) and Rabadan, J., Ed., Sathappan, S., Lin, W., Katiyar, M., and
Multicast Listener Discovery (MLD) Proxies for Ethernet A. Sajassi, "Optimized Ingress Replication Solution for
VPN (EVPN)", RFC 9251, DOI 10.17487/RFC9251, June 2022, Ethernet VPN (EVPN)", Work in Progress, Internet-Draft,
<https://www.rfc-editor.org/info/rfc9251>. draft-ietf-bess-evpn-optimized-ir-12, 25 January 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-optimized-ir-12>.
[I-D.skr-bess-evpn-pim-proxy] [BESS-EVPN-PIM-PROXY]
Rabadan, J., Kotalwar, J., Sathappan, S., Zhang, Z. J., Rabadan, J., Ed., Kotalwar, J., Sathappan, S., Zhang, Z.,
and A. Sajassi, "PIM Proxy in EVPN Networks", Work in and A. Sajassi, "PIM Proxy in EVPN Networks", Work in
Progress, Internet-Draft, draft-skr-bess-evpn-pim-proxy- Progress, Internet-Draft, draft-skr-bess-evpn-pim-proxy-
01, 30 October 2017, 01, 30 October 2017,
<https://datatracker.ietf.org/doc/html/draft-skr-bess- <https://datatracker.ietf.org/doc/html/draft-skr-bess-
evpn-pim-proxy-01>. evpn-pim-proxy-01>.
[I-D.ietf-bess-evpn-optimized-ir] [BESS-EVPN-PREF-DF]
Rabadan, J., Sathappan, S., Lin, W., Katiyar, M., and A. Rabadan, J., Ed., Sathappan, S., Lin, W., Drake, J., and
Sajassi, "Optimized Ingress Replication Solution for A. Sajassi, "Preference-based EVPN DF Election", Work in
Ethernet VPN (EVPN)", Work in Progress, Internet-Draft, Progress, Internet-Draft, draft-ietf-bess-evpn-pref-df-11,
draft-ietf-bess-evpn-optimized-ir-12, 25 January 2022, 6 July 2023, <https://datatracker.ietf.org/doc/html/draft-
ietf-bess-evpn-pref-df-11>.
[BESS-RFC7432BIS]
Sajassi, A., Burdet, L., Drake, J., and J. Rabadan, "BGP
MPLS-Based Ethernet VPN", Work in Progress, Internet-
Draft, draft-ietf-bess-rfc7432bis-07, 13 March 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess- <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-optimized-ir-12>. rfc7432bis-07>.
[RFC8584] Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake, [BESS-SECURE-EVPN]
J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis,
VPN Designated Forwarder Election Extensibility", B., and J. Drake, "Secure EVPN", Work in Progress,
RFC 8584, DOI 10.17487/RFC8584, April 2019, Internet-Draft, draft-ietf-bess-secure-evpn-00, 20 June
<https://www.rfc-editor.org/info/rfc8584>. 2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
bess-secure-evpn-00>.
[I-D.ietf-bess-evpn-pref-df] [CLOS1953] Clos, C., "A study of non-blocking switching networks",
Rabadan, J., Sathappan, S., Lin, W., Drake, J., and A. The Bell System Technical Journal, Vol. 32, Issue 2,
Sajassi, "Preference-based EVPN DF Election", Work in DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953,
Progress, Internet-Draft, draft-ietf-bess-evpn-pref-df-10, <https://ieeexplore.ieee.org/document/6770468>.
2 September 2022, <https://datatracker.ietf.org/doc/html/
draft-ietf-bess-evpn-pref-df-10>.
[I-D.ietf-bess-evpn-irb-mcast] [IEEE.802.1AX_2014]
Lin, W., Zhang, Z. J., Drake, J., Rosen, E. C., Rabadan, IEEE, "IEEE Standard for Local and metropolitan area
J., and A. Sajassi, "EVPN Optimized Inter-Subnet Multicast networks -- Link Aggregation", IEEE Std 802.1AX-2014,
(OISM) Forwarding", Work in Progress, Internet-Draft, DOI 10.1109/IEEESTD.2014.7055197, December 2014,
draft-ietf-bess-evpn-irb-mcast-09, 21 February 2023, <https://doi.org/10.1109/IEEESTD.2014.7055197>.
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-irb-mcast-09>.
[RFC9014] Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi, [NVO3-ENCAP]
A., and J. Drake, "Interconnect Solution for Ethernet VPN Boutros, S., Ed. and D. Eastlake 3rd, Ed., "Network
(EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014, Virtualization Overlays (NVO3) Encapsulation
May 2021, <https://www.rfc-editor.org/info/rfc9014>. Considerations", Work in Progress, Internet-Draft, draft-
ietf-nvo3-encap-09, 7 October 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-
encap-09>.
[I-D.ietf-bess-evpn-ipvpn-interworking] [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
Rabadan, J., Sajassi, A., Rosen, E. C., Drake, J., Lin, Border Gateway Protocol 4 (BGP-4)", RFC 4271,
W., Uttaro, J., and A. Simpson, "EVPN Interworking with DOI 10.17487/RFC4271, January 2006,
IPVPN", Work in Progress, Internet-Draft, draft-ietf-bess- <https://www.rfc-editor.org/info/rfc4271>.
evpn-ipvpn-interworking-07, 6 July 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess- [RFC4272] Murphy, S., "BGP Security Vulnerabilities Analysis",
evpn-ipvpn-interworking-07>. RFC 4272, DOI 10.17487/RFC4272, January 2006,
<https://www.rfc-editor.org/info/rfc4272>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <https://www.rfc-editor.org/info/rfc4364>.
[RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
"Multiprotocol Extensions for BGP-4", RFC 4760,
DOI 10.17487/RFC4760, January 2007,
<https://www.rfc-editor.org/info/rfc4760>.
[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP
Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
June 2010, <https://www.rfc-editor.org/info/rfc5925>.
[RFC6952] Jethanandani, M., Patel, K., and L. Zheng, "Analysis of
BGP, LDP, PCEP, and MSDP Issues According to the Keying
and Authentication for Routing Protocols (KARP) Design
Guide", RFC 6952, DOI 10.17487/RFC6952, May 2013,
<https://www.rfc-editor.org/info/rfc6952>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3 Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>. <https://www.rfc-editor.org/info/rfc7348>.
[RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
"Encapsulating MPLS in UDP", RFC 7510, "Encapsulating MPLS in UDP", RFC 7510,
DOI 10.17487/RFC7510, April 2015, DOI 10.17487/RFC7510, April 2015,
<https://www.rfc-editor.org/info/rfc7510>. <https://www.rfc-editor.org/info/rfc7510>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February Uttaro, J., and W. Henderickx, "A Network Virtualization
2006, <https://www.rfc-editor.org/info/rfc4364>. Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
[CLOS1953] Clos, C., "A Study of Non-Blocking Switching Networks", <https://www.rfc-editor.org/info/rfc8365>.
March 1953.
[I-D.ietf-bess-evpn-geneve]
Boutros, S., Sajassi, A., Drake, J., Rabadan, J., and S.
Aldrin, "EVPN control plane for Geneve", Work in Progress,
Internet-Draft, draft-ietf-bess-evpn-geneve-05, 23
November 2022, <https://datatracker.ietf.org/doc/html/
draft-ietf-bess-evpn-geneve-05>.
[I-D.ietf-bess-evpn-mvpn-seamless-interop] [RFC8584] Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
Sajassi, A., Thiruvenkatasamy, K., Thoria, S., Gupta, A., J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
and L. Jalil, "Seamless Multicast Interoperability between VPN Designated Forwarder Election Extensibility",
EVPN and MVPN PEs", Work in Progress, Internet-Draft, RFC 8584, DOI 10.17487/RFC8584, April 2019,
draft-ietf-bess-evpn-mvpn-seamless-interop-05, 13 March <https://www.rfc-editor.org/info/rfc8584>.
2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
bess-evpn-mvpn-seamless-interop-05>.
[I-D.sajassi-bess-secure-evpn] [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis, "Geneve: Generic Network Virtualization Encapsulation",
B., and J. Drake, "Secure EVPN", Work in Progress, RFC 8926, DOI 10.17487/RFC8926, November 2020,
Internet-Draft, draft-sajassi-bess-secure-evpn-06, 13 <https://www.rfc-editor.org/info/rfc8926>.
March 2023, <https://datatracker.ietf.org/doc/html/draft-
sajassi-bess-secure-evpn-06>.
[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP [RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
Authentication Option", RFC 5925, DOI 10.17487/RFC5925, "The BGP Tunnel Encapsulation Attribute", RFC 9012,
June 2010, <https://www.rfc-editor.org/info/rfc5925>. DOI 10.17487/RFC9012, April 2021,
<https://www.rfc-editor.org/info/rfc9012>.
[RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A [RFC9014] Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
Border Gateway Protocol 4 (BGP-4)", RFC 4271, A., and J. Drake, "Interconnect Solution for Ethernet VPN
DOI 10.17487/RFC4271, January 2006, (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014,
<https://www.rfc-editor.org/info/rfc4271>. May 2021, <https://www.rfc-editor.org/info/rfc9014>.
[RFC4272] Murphy, S., "BGP Security Vulnerabilities Analysis", [RFC9135] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
RFC 4272, DOI 10.17487/RFC4272, January 2006, Rabadan, "Integrated Routing and Bridging in Ethernet VPN
<https://www.rfc-editor.org/info/rfc4272>. (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
<https://www.rfc-editor.org/info/rfc9135>.
[RFC6952] Jethanandani, M., Patel, K., and L. Zheng, "Analysis of [RFC9136] Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
BGP, LDP, PCEP, and MSDP Issues According to the Keying A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
and Authentication for Routing Protocols (KARP) Design (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
Guide", RFC 6952, DOI 10.17487/RFC6952, May 2013, <https://www.rfc-editor.org/info/rfc9136>.
<https://www.rfc-editor.org/info/rfc6952>.
[RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter, [RFC9161] Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G.,
"Multiprotocol Extensions for BGP-4", RFC 4760, and T. King, "Operational Aspects of Proxy ARP/ND in
DOI 10.17487/RFC4760, January 2007, Ethernet Virtual Private Networks", RFC 9161,
<https://www.rfc-editor.org/info/rfc4760>. DOI 10.17487/RFC9161, January 2022,
<https://www.rfc-editor.org/info/rfc9161>.
[I-D.ietf-bess-rfc7432bis] [RFC9251] Sajassi, A., Thoria, S., Mishra, M., Patel, K., Drake, J.,
Sajassi, A., Burdet, L. A., Drake, J., and J. Rabadan, and W. Lin, "Internet Group Management Protocol (IGMP) and
"BGP MPLS-Based Ethernet VPN", Work in Progress, Internet- Multicast Listener Discovery (MLD) Proxies for Ethernet
Draft, draft-ietf-bess-rfc7432bis-07, 13 March 2023, VPN (EVPN)", RFC 9251, DOI 10.17487/RFC9251, June 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess- <https://www.rfc-editor.org/info/rfc9251>.
rfc7432bis-07>.
[I-D.ietf-rtgwg-bgp-pic] [RTGWG-BGP-PIC]
Bashandy, A., Filsfils, C., and P. Mohapatra, "BGP Prefix Bashandy, A., Ed., Filsfils, C., and P. Mohapatra, "BGP
Independent Convergence", Work in Progress, Internet- Prefix Independent Convergence", Work in Progress,
Draft, draft-ietf-rtgwg-bgp-pic-19, 1 April 2023, Internet-Draft, draft-ietf-rtgwg-bgp-pic-19, 1 April 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg- <https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-
bgp-pic-19>. bgp-pic-19>.
[IEEE.802.1AX_2014] Acknowledgments
IEEE, "IEEE Standard for Local and metropolitan area
networks -- Link Aggregation", 24 December 2014.
Appendix A. Acknowledgments
The authors want to thank Aldrin Isaac for his comments.
Appendix B. Contributors
Appendix C. Authors' Addresses The authors thank Aldrin Isaac for his comments.
Authors' Addresses Authors' Addresses
Jorge Rabadan (editor) Jorge Rabadan (editor)
Nokia Nokia
520 Almanor Ave 520 Almanor Ave
Sunnyvale, CA 94085 Sunnyvale, CA 94085
United States of America United States of America
Email: jorge.rabadan@nokia.com Email: jorge.rabadan@nokia.com
 End of changes. 235 change blocks. 
793 lines changed or deleted 778 lines changed or added

This html diff was produced by rfcdiff 1.48.