Network Working Group Huajin Jeng Internet Draft AT&T Intended status: Standards Track Expires: April 2013 James Uttaro AT&T Luay Jalil Verizon Bruno Decraene France Telecom Yakov Rekhter Juniper Networks Rahul Aggarwal Arktan October 15 2012 Virtual Hub-and-Spoke in BGP/MPLS VPNs draft-ietf-l3vpn-virtual-hub-02.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Jeng [Page 1] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Abstract With BGP/MPLS VPNs any-to-any connectivity among sites of a given Virtual Private Network would require each Provider Edge router that has one or more of these sites connected to it to hold all the routes of that Virtual Private Network. The approach described in this document allows to reduce the number of Provider Edge routers that have to maintain all these routes by requiring only a subset of these routers to maintain all these routes. Furthermore, when Provider Edge routers use ingress replication to carry multicast traffic of VPN customers, the approach described in this document could allow to reduce bandwidth inefficiency associated with ingress replication, and to redistribute the replication load among Provider Edge routers. Jeng [Page 2] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 Table of Contents 1 Specification of requirements ......................... 4 2 Overview .............................................. 4 3 Routing Information Exchange .......................... 5 4 Forwarding Considerations ............................. 6 5 Internet Connectivity ................................. 9 6 Deployment Considerations ............................. 11 7 Multicast Considerations .............................. 12 7.1 Terminology ........................................... 12 7.2 Eligible Upstream Multicast Hop (UMH) routes .......... 13 7.3 Originating VPN-IP default route by a V-hub ........... 13 7.4 Handling C-multicast routes ........................... 13 7.5 Originating I-PMSI/S-PMSI/SA A-D routes by V-spoke .... 14 7.6 Originating I-PMSI/S-PMSI/SA A-D routes by V-hub ...... 14 7.7 Receiving I-PMSI/S-PMSI/SA A-D routes by V-spoke ...... 15 7.8 Receiving I-PMSI/S-PMSI/SA A-D routes by V-hub ........ 15 7.8.1 Case 1 ................................................ 15 7.8.2 Case 2 ................................................ 16 7.9 Use of Ingress Replication with I-PMSI A-D routes ..... 17 8 An example of RT provisioning ......................... 18 8.1 Unicast routing ....................................... 18 8.2 Multicast routing ..................................... 19 9 Further refinements ................................... 20 10 IANA Considerations ................................... 20 11 Security Considerations ............................... 20 12 Acknowledgements ...................................... 20 13 Normative References .................................. 21 14 Non-normative References .............................. 21 15 Authors' Addresses .................................... 21 Jeng [Page 3] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 1. Specification of requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 2. Overview With BGP/MPLS VPNs [rfc4364] providing any-to-any connectivity among sites of a given Virtual Private Network (VPN) is usually accomplished by requiring each Provider Edge router (PE) that has one or more of these sites connected to it to hold all the routes of that VPN. The approach described in this document allows to reduce the number of PEs that have to maintain all these routes by requiring only a subset of such PEs to maintain all these routes. Consider a VPN that has its sites connected to a set of PEs. In the context of this VPN we designate a subset of these PEs as "Virtual Spoke" PEs (or just Virtual Spokes), while some other (non- overlapping) subset of these PEs will be "Virtual Hub" PEs (or just Virtual Hubs), with the rest of the PEs in the set will be "vanilla" PEs (PEs that implement the [rfc4364] procedures, but do not implement procedures specified in this document). For the sake of brevity we will use the term "V-hub" to denote a Virtual Hub, and "V-spoke" to denote a Virtual Spoke. For a given VPN, its set of V-hubs could include not only the PEs that have sites of that VPN connected to them, but also PEs that have no sites of that VPN connected to them. Note that while in the context of one VPN a given PE may act as a V- hub, in the context of another VPN the same PE may act as a V-spoke, and vice versa. Thus a given PE may act as a V-hub only for some, but not all the VPNs present on that PE. Likewise, a given PE may act as a V-spoke only for some, but not all the VPNs present on that PE. For a given VPN each V-spoke of that VPN is "associated" with two or more V-hubs of that VPN (we use two V-hubs for redundancy to avoid a single point of failure). Note that a given V-hub may have no V- spokes associated with it. A PE that acts as a V-hub of a given VPN maintains all the routes of that VPN. A PE that acts as a V-spoke of a given VPN needs to maintain only the routes originated by the sites of that VPN connected to that PE, plus two or more VPN-IP default routes whose Next Hops are the addresses of the V-hubs associated with that V- Jeng [Page 4] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 spoke. This way, only a subset of PEs that have sites of a given VPN, namely only the PEs acting as V-hubs of that VPN, have to maintain all the routes of that VPN. PEs acting as V-spokes of that VPN need to maintain only a (small) subset of the routes of that VPN. When PEs use ingress replication to carry multicast traffic of VPN customers, the approach described in this document could allow to reduce bandwidth inefficiency associated with ingress replication, and to redistribute the replication load among the PEs. This is because a PE that acts as a V-spoke of a given VPN would need to replicate multicast traffic only to other V-hubs (while other V-hubs would replicate this traffic to the V-spokes associated with these V- hubs), rather than to all the PEs of that VPN. Likewise, a PE that acts as a V-hub of a given VPN would need to replicate multicast traffic to other V-hubs, and the V-spokes, but only to the V-spokes associated with that V-hub, rather than replicating the traffic to all the PEs of that VPN. Limiting replication could be especially beneficial if a PE that does the replication has limited replication capabilities and/or has links with limited bandwidth. 3. Routing Information Exchange Routing information exchange among all the PEs of a given VPN is subject to the following rules. A PE that has sites of a given VPN connected to it has to retain routing information received from the VPN sites connected to it. This is irrespective of whether this PE acts as a V-hub or a V-spoke of that VPN. A PE that has sites of a given VPN connected to it has to export (as VPN-IP routes) all the routes received from these sites. This is irrespective of whether this PE acts as a V-hub or a V-spoke of that VPN. In addition, a V-hub of a given VPN has to export a VPN-IP default route for that VPN. This route SHOULD be exported to only the V- spokes of that VPN that are associated with that V-hub. To enable a V-spoke of a given VPN to share its outbound traffic load among the V-hubs associated with that V-spoke, each V-hub of that VPN, when originating a VPN-IP default route MUST use a distinct RD (per V-hub, per VPN). If a V-spoke imports several VPN-IP default routes, each originated by its own V-hub, and these routes have the same preference, then traffic from the V-spoke to other sites of that VPN would be load Jeng [Page 5] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 shared among the V-hubs. A V-hub of a given VPN has to import all the non-default VPN-IP routes originated by all other PEs that have sites of that VPN connected to them (irrespective of whether these other PEs act as V- hubs or V-spokes for that VPN). A V-hub of a given VPN MUST NOT import a VPN-IP default route unless this is the Internet VPN-IP default route (for the definition of "Internet VPN-IP default route" see section "Internet Connectivity"). A V-spoke of a given VPN has to import a VPN-IP default route for that VPN that has been originated by the V-hubs of that VPN that are associated with that V-spoke. In addition, a V-spoke of a given VPN MAY import VPN-IP routes for that VPN that have been originated by some other V-spokes of that VPN. The above rules that govern the routing information exchange among all the PEs of a given VPN are realized using Route Target (RT) extended communities [rfc4360] and VRF export/import policies based on these RTs. One possible scheme that enables to realize such rules is described in Section "Deployment Considerations". This document does not preclude use of other schemes, as long as such schemes would support the above rules. 4. Forwarding Considerations This section describes changes/modifications to the forwarding procedures specified in [rfc4364]. The MPLS label that the V-hub advertises with a VPN-IP default route MUST be the label that points to the VRF of the V-hub. This way, whenever the V-hub receives packets carrying this label, the V-hub forwards the packets by performing IP lookup in the VRF identified by the label. When a V-hub of a given VPN originates a VPN-IP default route for that VPN, the V-hub MUST NOT install in its VRF of that VPN a default route, unless this route has been originated either (a) as a result of the V-hub receiving an IP default route from one of the CEs of that VPN connected to it, or (b) as a result of the V-hub receiving (and importing) a VPN-IP default route from some other PE, or (c) the VRF being provisioned with a default route pointing to the routing table on the same PE that maintains the Internet routes. Jeng [Page 6] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 In the absence of V-hubs and V-spokes, support for IBGP/EBGP load balancing in the presence of any-to-any connectivity uses the following rule. When a PE receives a packet from another PE, the PE that receives the packet determines (using the MPLS label carried in the packet) the VRF that should be used for further forwarding the packet. If (using the information present in the VRF) the PE determines that the packet could be forwarded over one of the attachment circuits of that VRF (for the definition of "VRF attachment circuits" see [rfc4364]), then the PE MUST forward the packet over one of these circuits, and MUST NOT forward the packet to another PE. In order to support IBGP/EBGP load balancing on a V-hub the above rule is modified as follows. When a V-hub receives a packet from another PE scenario, the V-hub identifies (using the MPLS label carried in the packet) the VRF that should be used for further forwarding the packet. If the MPLS label carried in the packet is the label that the V-hub advertised in the VPN-IP default route, then the V-hub is not restricted to use only one of the VRF attachment circuits for forwarding the packet - the V-hub may forward the packet to another PE. To illustrate this consider the following example: <- RD:0/0 RD:0/0-> <- RD:10.2.1 <-10.2.1/24 CE1----PE-S-------------PE-H----------------PE-S1-------------CE2 / | | | | 10.2.1/24 | | CE4 CE3 A multi-homed site (not shown in the above figure) is connect via CE2 and CE3. Thus both CE2 and and CE3 advertise a route to 10.2.1/24. CE2 advertises this route to PE-S1, which in turn originates a VPN-IP route RD:10.2.1. CE3 advertises this route to PE-H. PE-H is a V-hub, while PE-S and PE-S1 are V-spokes associated with that V-hub. Thus PE-H originates a VPN-IP default route (RD:0/0), and both PE-S and PE-S1 import that route. PE-H receives from PE-S1 a VPN-IP route to RD:10.2.1 and from CE3 a plain IP route to 10.2.1. Thus the VRF entry on PE-H has two possible next hops for 10.2.1: CE3 and PE-S1 (the latter is an indirect next hop). Jeng [Page 7] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 Now consider what happens when CE1 originates a packet destined to 10.2.1.1. When PE-S receives this packet, PE-S (following the VPN-IP default route) forwards the packet to PE-H. In the absence of the modification specified above, when PE-H receives this packet, PE-H will be forced to forward the packet to CE3 (as PE-H can reach CE3 via a VRF attachment circuit). However, that would prevent IBGP/EBGP load balancing, as it would preclude the possibility of forwarding this packet via PE-S1 and CE2. With the modified rule specified above, PE-H determines that the MPLS label in the packet is the MPLS label that PE-H advertised to PE-S in the VPN-IP default route. Thus PE-H may forward the packet either via CE3 or via PE-S1 (with PE-S1 subsequently forwarding the packet to CE2), resulting in preserving IBGP/EBGP load balancing. Likewise, if CE4 originates a packet destined to 10.2.1.1, PE-H may forward the packet either via CE3 or via PE-S1 (with PE-S1 subsequently forwarding the traffic to CE2), resulting in preserving IBGP/EBGP load balancing. Note however, that if there is some other CE, CE5, connected to PE- S1, and that CE5 sends a packet to 10.2.1.1, then (due to the IP longest match rule) PE-S1 will always forward this packet to CE2. Thus IBGP/EBGP load balancing will not be available for the destinations that have been learned locally by a V-spoke. Moreover, if CE3 advertises 10.2.1.0/25 and 10.2.1/24, while CE2 advertises 10.2.1.128/25 and 10.2.1/24 (which is yet another form of load balancing for a multi-homed site), then when CE5 sends a packet to 10.2.1.1, then (due to the IP longest match rule) PE-S1 will always forward this packet to CE2, even though the VPN customer would expect this traffic to flow via CE3. This document proposes two options to address this, as well as to make the IBGP/EBGP load balancing available for the destinations that have been learned locally by a V-spoke. One option is to provision PEs that have multi-homed sites connected to them as V-hubs. Another option is for the V-spoke, when it receives an IP route from a CE, not to install this route in its forwarding table, but just re- advertise this route as a VPN-IP route, together with an MPLS label. The packet's next hop of the Next Hop Label Forwarding Entry (NHLFE) [rfc3031] associated with that label MUST point to the CE that advertised the IP route. As a result, when the PE receives data that carries that label, the PE performs no IP lookup on the data, and just forwards the data to the CE. Note that doing this would result in forcing the traffic between a pair of sites connected to the same V-spoke to go through the V-hub of that V-spoke. Jeng [Page 8] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 5. Internet Connectivity In this document we consider two possible scenarios of providing Internet connectivity for a given VPN. The first scenario is when one or more sites of a given VPN are connected to a PE, and that PE also maintains Internet routes. In this case the Internet connectivity for that VPN is provided by provisioning in the VPN's VRF on that PE a default route pointing to the routing table on that PE that maintains the Internet routes. This PE MUST act as a V-hub for that VPN, and therefore MUST originate a VPN-IP default route. The second scenario is when a site of a given VPN has connection to the Internet, and a CE of that site advertises an IP default route to the PE connected to that CE. This scenario has two sub-cases: (a) PE is provisioned as a V-hub, and (b) PE is provisioned as a V-spoke. If a PE is provisioned as a V-hub, then the PE re-advertises this IP default route as a VPN-IP default route, and install in its VRF an IP default route with the next hop pointing to the CE(s) that advertise the IP default route to the PE. If a PE is provisioned as a V-spoke, then the PE MUST NOT install in its VRF an IP default route. The PE MUST originate a VPN-IP default route with a (non-null) MPLS label. The packet's next hop of the Next Hop Label Forwarding Entry (NHLFE) [rfc3031] associated with that label MUST point to the CE that advertises the IP default route. As a result, when the PE receives data that carries that label, the PE performs no IP lookup on the data, and just forwards the data to the CE. Note that in this case the VRF on the PE is going to have an IP default route, but this route would be created as a result of receiving a VPN-IP default route from one of the V-hubs of that PE (and not as a result of receiving the IP default route from the CE). Note also that if this PE has other sites of that VPN connected to it, then traffic from these sites to the Internet would go to that PE, then to the V-hub selected by the PE, then from that V-hub back to the PE, and then to the CE that advertises IP default route to the PE. If a PE is provisioned as a V-spoke of a given VPN, and if a CE of that VPN advertises an IP default route to the PE (as the CE belongs to the site that provides the Internet connectivity for the VPN), then the PE can not advertise an IP default route back to that CE. Yet, the CE has to point to the PE as the next hop for all the traffic to other sites of that VPN. This document proposes the following options to accomplish this. The first option is to require the V-spoke to implement procedures of section "Further refinements". Jeng [Page 9] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 The second option is for the V-spoke to advertise to the CE 10/8, 172.16/12, and 192.168/16. Note that this option is applicable only if the service provider knows that the VPN customer uses only the [rfc1918] addresses. The third option is to re-provision this PE as a V-hub. In all of the above we refer to the originated VPN-IP default route as the "Internet VPN-IP default route". Note that if a V-hub originates the Internet VPN-IP default route for a given VPN, the V-hub does not originate any other VPN-IP default routes for that VPN. If a V-hub originates the Internet VPN-IP default route, then all the V-spokes associated with that V-hub MUST import that route. In addition (and in contrast with a non-Internet VPN-IP default route), other V-hubs MAY import that route. A V-hub MAY also import the Internet VPN-IP default routes originated by V-spoke(s). A V-spoke MUST NOT import the Internet VPN-IP default route originated by any other V-spoke. Such a route MAY be imported only by V-hubs. The above rules that govern exchange of the Internet VPN-IP default route among all the PEs of a given VPN are realized using Route Target (RT) extended communities [rfc4360] and VRF export/import policies based on these RTs. One possible scheme that enables to realize such rules is described in Section "Deployment Considerations". This document does not preclude use of other schemes, as long as such schemes would support the above rules. If the Internet VPN-IP default route originated by a V-hub has the same preference as the (non Internet) VPN-IP default route originated by some other V-hub, then a V-spoke that imports VPN-IP default routes originated by both of these V-hubs would load share the outgoing Internet traffic between these two V-hubs (and thus some of the outgoing Internet traffic from that V-spoke will first be routed to the V-hub that does not originate the Internet VPN-IP default route, and then from that V-hub to the V-hub that does originate the Internet VPN-IP default route). If taking an extra-hub hop for the Internet traffic is viewed as undesirable, then it is RECOMMENDED that the Internet VPN-IP default route be of higher preference than a (non-Internet) VPN-IP default route originated by some other V-hub. However, in this case the traffic from the V-spokes to other sites of that VPN will not be load Jeng [Page 10] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 shared between these two V-hubs. 6. Deployment Considerations For a given VPN a V-hub and a set of V-spokes associated with that V- hub should be chosen in a way that minimizes the additional network distance/latency penalty, given that VPN geographic footprint. For a given VPN some/all of its V-spokes could be grouped into geographically-based clusters with any-any connectivity within each cluster. Note that the V-spokes within a given cluster need not be associated with the same V-hub(s). Likewise, not all V-spokes associated with a given V-hub need to be in the same cluster. A use case for this would be VPN for large retail chain in which data traffic is hub/spoke between each store and centralized data-centers but there is need for direct VoIP traffic between stores within same geographical area. The following describes one possible scheme that enables to realize the rules for exchanging routing information among PEs, as specified in Section "Routing Information Exchange". Consider a "vanilla" any-to-any VPN. All the PEs of such VPN are provisioned with the same export and import RT - we will refer to this RT as RT-VPN. To evolve this VPN into V-hubs and V-spokes we keep the same export RT-VPN on all the PEs of that VPN (both V-hubs and V-spokes). This RT-VPN is attached to all the VPN-IP routes that PEs originate as a result of receiving corresponding IP routes from their CEs. We also keep the same import RT-VPN on all the V-hubs. In addition, each V-hub is provisioned with its own import RT, called RT-VH. This RT-VH MUST be different from the export RT provisioned on that V-hub. For a given PE this RT-VH has to be distinct for every V-hub present on that PE. To avoid the situation where within a given VPN all the V-spokes would be associated with every V-hub (in other words, to partition V-spokes among V-hubs), different V-hubs within that VPN MAY use different RT-VHs. At one extreme every V-hub may use a distinct RT-VH. However, it is also possible for several V-hubs to use the same RT-VH, in which case all these V-hubs are used by the same set of V-spokes. Use of IP-address specific RTs may be an attractive option for RT-VHs. When a V-hub originates a (non-Internet) VPN-IP default route, then the V-hub attaches RT-VH to that route. Thus this route is imported by all the V-spokes associated with the V-hub. Jeng [Page 11] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 If a V-hub originates the Internet VPN-IP default route, then the V- hub attaches both RT-VH and RT-VPN to that route. Thus this route is imported by all the V-spokes associated with the V-hub, as well as by other V-hubs. If a V-spoke originates the Internet VPN-IP default route, then the V-spoke attaches only RT-VPN to that route. Thus this route is not imported by any of the V-spokes, but is imported by V-hubs. A given V-spoke is provisioned to import only VPN-IP routes that carry RT-VHs of the V-hubs associated with that V-spoke (thus associating a new V-spoke with a given V-hub requires provisioning only on that V-spoke - no provisioning changes are required on the V- hub). A V-spoke may be provisioned to export VPN-IP routes not just to the V-hubs, but also to the V-spokes that import the same VPN-IP default route(s) as the V-spoke itself. The V-spoke accomplishes this by adding its import RT-VH(s) to the VPN-IP routes exported by the V- spoke. Use of RT Constrains [rfc4684] may further facilitate/optimize routing exchange in support of V-hubs and V-spokes. 7. Multicast Considerations This section describes procedures for supporting MVPN in the presence of Virtual Hub-and-Spoke. The procedures rely on MVPN specifications as defined in [MVPN], [BGP-MVPN]. The procedures assume that for the purpose of ensuring non- duplication both V-hubs and V-spokes can discard packets from a "wrong" PE, as specified in [MVPN]. This applies for all types of P- tunnels, including Ingress Replication. 7.1. Terminology When Ingress Replication is used to realize P-tunnels, a P-tunnel being "bound" to a particular I-PMSI/S-PMSI A-D route means the set of (unicast) tunnels used to carry traffic from the PE originating the route to the PEs that import the route. When Ingress Replication is used to realize P-tunnels, traffic received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A- D received by the PE means the traffic received on the (unicast) tunnel that the PE originating the route uses to send traffic to the Jeng [Page 12] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 PE that receives the route. 7.2. Eligible Upstream Multicast Hop (UMH) routes On a V-spoke the set of Eligible UMH routes consists of all the unicast VPN-IP routes received by the V-spoke, including the default VPN-IP routes received from its V-hub(s). Note that such routes may include VPN-IP routes received from other V-spokes. 7.3. Originating VPN-IP default route by a V-hub A V-hub, when originating a VPN-IP default route, follows the procedures of [BGP-MVPN]. Specifically the V-hub must add the VRF Route Import extended community that embeds the V-hub's IP address. The route also must include the Source AS extended community. 7.4. Handling C-multicast routes Origination of C-multicast routes follow the procedures of [BGP-MVPN] (this is irrespective of whether these routes are originated by a V- hub or by a V-spoke). Note that for a given V-spoke its set of eligible UMH routes may contains not only the VPN-IP default routes advertised by its V-hubs, but also VPN-IP routes advertised by some other V-spokes. When a V-hub receives a C-multicast route, the V-hub determines whether the C-RP/C-S of the route is reachable via one of its VRF interfaces, and if yes, then the V-hub follows the [BGP-MVPN] procedures. Otherwise, the C-RP/C-S of the route is reachable via some other PE, in which case the V-hub uses the type (Source Tree Join vs Shared Tree Join), the Multicast Source, and Multicast Group from the received route to construct a new route. The hub constructs the rest of the new route following procedures specified in "11.1.3. Constructing the rest of the C-multicast route" of [BGP-MVPN]. The hub also creates the appropriate (C-*, C-G)/(C-S, C-G) state in its MVPN-TIB. Jeng [Page 13] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 7.5. Originating I-PMSI/S-PMSI/SA A-D routes by V-spoke When a V-spoke originates an I-PMSI/S-PMSI/SA A-D route, the V-spoke follows the procedures of [BGP-MVPN], including the procedures for constructing RT(s) carried by the route. Note that as a result, such a route will be imported by the V-hubs. The P-tunnel bound to this route is used to carry to these V-hubs traffic originated by the sites connected to the V-spoke. This route may also be imported by some other V-spokes - the V-spokes that import the (unicast) VPN-IP routes originated by the V-spoke that originates the I-PMSI/S-PMSI/SA A-D route. In this case the route would carry not one, but two RTs: the first one results in importing the route by V-hubs, the second results in importing the route by the V-spokes (the second RT is the RT that V-spoke use for importing the VPN-IP default route). In this case the P-tunnel bound to this route is also used to carry traffic originated by the sites connected to the V-spoke that originates the route to these other V- spokes. 7.6. Originating I-PMSI/S-PMSI/SA A-D routes by V-hub When a V-hub originates an I-PMSI/S-PMSI/SA A-D route, the V-hub follows the procedures of [BGP-MVPN], except that in addition to the RT(s) constructed following these procedures, the route also carries the RT of the VPN-IP default route advertised by the V-hub. Note that as a result, such a route will be imported by other V-hubs, and also by the V-spokes, but only by the V-spokes that import the VPN-IP default route originated by the V-hub. The P-tunnel bound to this route is used to carry to these other V-hubs and V-spokes the traffic originated by the sites connected to the V-hub that originates the route. In addition, a V-hub originates another I-PMSI A-D route - we'll refer to this route as a "Default I-PMSI A-D route". The RT carried by this route is the RT that is carried in the VPN-IP default route advertised by the V-hub (RT-VH). Therefore, this route will be imported only by the V-spokes that import the VPN-IP default route advertised by this V-hub. The P-tunnel bound to this route is used to carry to these V-spokes traffic originated by either (a) other V- hubs, or (b) V-spokes, including the V-spokes that import the VPN-IP default route from the V-hub. More details on the use of this P- tunnel is described later. As a result, a V-hub originates not one, but two I-PMSI A-D routes. Each of these routes MUST have a distinct RD. Jeng [Page 14] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 When a V-hub receives traffic from one of the sites connected to the V-hub, and the V-hub determines (using some local policies) that this traffic should be transmitted using an I-PMSI, the V-hub forwards this traffic on the P-tunnel bound to the first (non Default) I-PMSI A-D route, but not on the P-tunnel bound to the second, the Default I-PMSI A-D route. 7.7. Receiving I-PMSI/S-PMSI/SA A-D routes by V-spoke When a V-spoke receives an I-PMSI/S-PMSI/SA A-D route, the V-spoke follows the procedures of [BGP-MVPN]. As a result, a V-spoke that imports the VPN-IP default route originated by a given V-hub will also import I-PMSI/S-PMSI/SA A-D routes originated by that V-hub. Specifically, the V-spoke will import both the non Default I-PMSI A-D route and the Default I-PMSI A-D route originated by the V-hub. In addition, if a V-spoke imports the (unicast) VPN-IP routes originated by some other V-spokes, then the V-spoke will also import I-PMSI/S-PMSI/SA A-D routes originated by these other V-spokes. 7.8. Receiving I-PMSI/S-PMSI/SA A-D routes by V-hub The following describe procedures that a V-hub must follow when it receives an I-PMSI/S-PMSI/SA A-D route. 7.8.1. Case 1 This is the case where the received I-PMSI/S-PMSI/SA A-D route was originated by a V-spoke that imports the VPN-IP default route originated by the V-hub, and the received route is imported not just by the V-hub (as well as by all other V-hubs), but also by other V- spokes, but only by the V-spokes that import the VPN-IP default route originated by the V-hub. This is also the case where the received route was originated by a V-hub that uses the same RT as the received V-hub for advertising the VPN-IP default route. In this case the received I-PMSI/S-PMSI/SA A-D route carries more than one RT. One of these RTs results in importing this route by the V-hubs. Another of these RTs is the RT that the V-hub uses when advertising its VPN-IP default route (RT-VH). This RT results in importing the received I-PMSI/S-PMSI/SA A-D route o all the V-spokes that import the VPN-IP default route originated by the V-hub. In handling such I-PMSI/S-PMSI/SA A-D route the V-hub follows the Jeng [Page 15] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 [BGP-MVPN] procedures. Specifically, in contrast to Case 2 below, the V-hub MUST NOT re-advertise this route. The V-hub may forward traffic received on a P-tunnel bound to this I- PMSI/S-PMSI A-D route to only the sites connected to that V-hub. The V-hub MUST NOT forward the traffic received on this P-tunnel to any other V-hubs or V-spokes, including the V-spokes that import the VPN- IP default route originated by the V-hub. Specifically, the V-hub MUST NOT forward the traffic received on the P-tunnel advertised in the received I-PMSI A-D route over the P-tunnel that the V-hub binds to its Default I-PMSI A-D route. 7.8.2. Case 2 This is the case where the received I-PMSI/S-PMSI/SA A-D route was originated by either some other V-hub, or by a V-spoke. The I-PMSI/S- PMSI/SA A-D route is imported by the V-hub (as well as by other V- hubs), but not by any of the V-spokes that import the VPN-IP default route originated by the V-hub. In this case the received I-PMSI/S-PMSI/SA A-D route does not carry the RT that the receiving V-hub uses when advertising its VPN-IP default route (RT-VH), although the route may carry more than one RT. In this case the V-hubs follows the procedures of [BGP-MVPN] with the following additions. Once a V-hub accepts an I-PMSI A-D route, when the V-hub receives data on the P-tunnel bound to that I-PMSI A-D route, the V-hub follows procedures in [MVPN], [MVPN-BGP] to determine whether to accept the data. If the data is accepted, then the V-hub further forwards the data over the P-tunnel bound to the Default I-PMSI A-D route originated by the V-hub. Note that in deciding whether to forward the data over the P-tunnel bound to the Default I-PMSI A-D route originated by the V-hub, the V-hub SHOULD take into account the (multicast) state present in its MVPN-TIB that has been created as a result receiving C-multicast routes from the V-spokes associated with the V-hub. Whenever a V-hub accepts an S-PMSI/SA A-D route, the V-hub, in contrast to Case 1 above, MUST re-advertise this route to its V- spokes. To accomplish this, the V-hub changes the RT carried by the route to the RT that the V-hub uses when originating its VPN-IP default route (RT-VH), and sets Next Hop to self. In addition, for S- PMSI/SA A-D routes the V-hub changes the RD of the route to the RD that the hub uses when originating its VPN-IP default route, and for S-PMSI A-D routes the V-hub changes the Originating Router's IP Jeng [Page 16] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 address in the MCAST-VPN NLRI of the route to its own IP address. Moreover, before re-advertising S-PMSI A-D routes to V-spokes, the V- hub modifies the PMSI Tunnel attribute of these routes as appropriate (e.g., by replacing the P-tunnel rooted at the originator of these routes with a P-tunnel rooted at the V-hub). Whenever the V-hub receives data on the P-tunnel bound to the received S-PMSI A-D route, the V-hub follows procedures of [MVPN], [MVPN-BGP] to determine whether to accept the data. If the data is accepted, then the V-hub further forwards it over the P-tunnel bound to the S-PMSI A-D route that has been re-advertised by the V-hub. If multiple S-PMSIs received by the V-hub have been aggregated into the same P-tunnel, then the V-hub, prior to forwarding to the V- spokes the data received on this P-tunnel may de-aggregate and then re-aggregate (in a different way) this data using the state present in its MVPN-TIB that has been created as a result of receiving C- multicast routes from the V-spokes. Even if S-PMSIs received by the V-hub each has its own P-tunnel, the V-hub, prior to forwarding to the V-spokes the data received on these P-tunnels may aggregate these S-PMSIs using the state present in its MVPN-TIB that has been created as a result of receiving C-multicast routes from the V-spokes. 7.9. Use of Ingress Replication with I-PMSI A-D routes The existing procedures for S-PMSI A-D routes are sufficient to discard packets coming from a "wrong" PE for any type of P-tunnels, including Ingress Replication. The following modifications to originating/receiving I-PMSI A-D routes enable to discard packets coming from a "wrong" PE when Ingress Replication is used for I-PMSI P-tunnels (for other types of P-tunnels the procedures specified in [MVPN], [BGP-MVPN] are sufficient). If Ingress Replication is used with I-PMSI A-D routes, then when a PE advertises such routes, the Tunnel Type in the PMSI Tunnel attribute is set to Ingress Replication; the Leaf Information Required flag is set to 1; the the attribute carries no MPLS labels. A PE that receives such I-PMSI A-D route MUST respond with a Leaf A-D route. The PMSI Tunnel attribute of that Leaf A-D route is constructed as follows. The Tunnel Type is set to Ingress Replication. The Tunnel Identifier MUST carry a routable address of the PE that originates the Leaf A-D route. The PMSI Tunnel attribute Jeng [Page 17] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 MUST carry a downstream assigned MPLS label that is used to demultiplex the traffic received over a unicast tunnel by the PE. 8. An example of RT provisioning Consider a VPN A that consists of 9 sites - site-1 through site-9. Each site is connected to its own PE - PE-1 through PE-9. We designate PE-3, PE-6, and PE-9 as V-hubs. To simplify the presentation the following example assumes that each V-spoke is associated with just one V-hub. However, as mentioned earlier, in practice each V-spoke should be associated with two or more V-hubs. PE-1 and PE-2 are V-spokes associated with PE-3. PE-4 and PE-5 are V- spokes associated with PE-6. PE-7 and PE-8 are V-spokes associated with PE-9. 8.1. Unicast routing All the PEs (both V-hubs and V-spokes) are provisioned to export routes using RT-A (just as it would be with "vanilla" any-to-any VPN). All the V-hubs (PE-3, PE-6, and PE-9) are provisioned to import routes with RT-A (just as it would be with "vanilla" any-to-any VPN). In addition, PE-3 is provisioned to originate a VPN-IP default route with RT-A-VH-1 (but not with RT-A), while PE-1 and PE-2 are provisioned to import routes with RT-A-VH-1. Likewise, PE-6 is provisioned to originate a VPN-IP default route with RT-A-VH-2 (but not with RT-A), while PE-4 and PE-5 are provisioned to import routes with RT-A-VH-2. Finally, PE-9 is provisioned to originate a VPN-IP default route with RT-A-VH-3 (but not with RT-A), while PE-7 and PE-8 are provisioned to import routes with RT-A-VH-3. Now let's modify the above a bit by assuming that site-3 has Internet connectivity. Thus site-3 advertises an IP default route to PE-3. PE-3, in turn originates a VPN-IP default route. In this case, the VPN-IP default route carries RT-A and RT-A-VH-1 (rather than just RT- A-VH-1, as before), which results in importing this route to PE-6 and PE-9, as well as to PE-1 and PE-2. Jeng [Page 18] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 If PE-7 and PE-8, in addition to importing VPN-IP default route from PE-9, also want to import each other VPN-IP routes, then PE-7 and PE-8 export their VPN-IP routes with two RTs: RT-A and RT-A-VH-3. 8.2. Multicast routing All the PEs designated as V-spokes (PE-1, PE-2, PE-4, PE-5, PE-7, and PE-8) are provisioned to export their I-PMSI/S-PMSI/SA A-D routes using RT-A (just as it would be with "vanilla" any-to-any MVPN). Thus these routes could be imported by all the V-hubs (PE-3, PE-6, and PE-9). The V-hub on PE-3 is provisioned to export its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-1. Thus these routes could be imported by all the other V-hubs (PE-6 and PE-9), and also by the V- spokes, but only by the V-spokes associated with the V-hub on PE-3 (PE-1 and PE-2). In addition, the V-hub on PE-3 originates the Default I-PMSI A-D route with RT-A-VH-1. This route could be imported only by the V-spokes associated with the V-hub on PE-3 (PE-1 and PE-2). The V-hub on PE-6 is provisioned to export its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-2. Thus these routes could be imported by all the other V-hubs (PE-3 and PE-9), and also by the V- spokes, but only by the V-spokes associated with the V-hub on PE-6 (PE-4 and PE-5). In addition, the V-hub on PE-6 originates the Default I-PMSI A-D route with RT-A-VH-2. This route could be imported only by the V-spokes associated with the V-hub on PE-6 (PE-4 and PE-5). The V-hub on PE-9 is provisioned to export its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-3. Thus these routes could be imported by all the other V-hubs (PE-3 and PE-6), and also by the V- spokes, but only by the V-spokes associated with the V-hub on PE-9 (PE-7 and PE-8). In addition, the V-hub on PE-9 originates the Default I-PMSI A-D route with RT-A-VH-3. This route could be imported only by the V-spokes associated with the V-hub on PE-9 (PE-7 and PE-8). If PE-7 and PE-8, in addition to importing VPN-IP default route from PE-9, also want to import each other VPN-IP routes, then PE-7 and PE-8 export their I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-3. If the V-hub on PE-9 receives an S-PMSI/SA A-D route originated by either some other V-hub (PE-3 or PE-6), or by a V-spoke that is not associated with this V-hub (PE-1, or PE-2, or PE-4, or PE-5), the V- Jeng [Page 19] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 hub, before re-advertising this route replaces the RT(s) carried in this route with just one RT - RT-A-VH-3. Thus, the re-advertised route could be imported only by the V-spokes associated with the V- hub on PE-9 (PE-7 and PE-8). 9. Further refinements In some cases a VPN customer may not want to rely solely on an (IP) default route being advertised from a V-spoke to a CE, but may want CEs to receive all the VPN routes (e.g., for the purpose of faster detection of VPN connectivity failures, and activating some backup connectivity). In this case one possible approach would be to install in the V- spoke's data plane only the default route (following the Virtual Hub and Spoke model, as described above), but keep all the VPN-IP routes in the V-spoke's control plane (and thus being able to advertise these routes from the V-spoke to the CEs). Granted, this would not change control plane resource consumption, but would (significantly) reduce resource consumption on the data plane. 10. IANA Considerations This document introduces no new IANA Considerations. 11. Security Considerations This document introduces no new Security Considerations, above and beyond what is already specified in [rfc4364]. 12. Acknowledgements We would like to acknowledge Han Nguyen (AT&T) for his contributions to this document. We would also like to thank Jeffrey (Zhaohui) Zhang (Juniper) for his review and comments. Jeng [Page 20] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 13. Normative References [rfc1918] Rekhter, Y., et.al., "Address Allocation for Private Internets", RFC 1918, February 1996. [rfc3031] Rosen, E., et. al., "MPLS Architecture", RFC3031, January 2001. [rfc4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended Communities Attribute", RFC 4360, February 2006. [rfc4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, February 2006. [rfc4684] Pedro Marques, et al., "Constrained Route Distribution for Border Gateway Protocol/MultiProtocol Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs)", RFC4684, November 2006 [MVPN] Eric C. Rosen, Rahul Aggarwal, et. al., "Multicast in MPLS/BGP IP VPNs", RFC6513, February 2012 [BGP-MVPN] R. Aggarwal, E. Rosen, T. Morin, Y. Rekhter, "BGP Encodings and Procedures for Multicast in MPLS/BGP IP VPNs", RFC6514, February 2012 14. Non-normative References 15. Authors' Addresses Huajin Jeng AT&T e-mail: hj2387@att.com James Uttaro AT&T e-mail: ju1738@att.com Luay Jalil Verizon e-mail: luay.jalil@verizon.com Bruno Decraene France Telecom e-mail: bruno.decraene@orange.com Rahul Aggarwal Jeng [Page 21] Internet Draft draft-ietf-l3vpn-virtual-hub-02.txt October 2012 e-mail: raggarwa_1@yahoo.com Yakov Rekhter Juniper Networks, Inc. e-mail: yakov@juniper.net Jeng [Page 22]