rfc9198.original   rfc9198.txt 
Network Working Group J. Alvarez-Hamelin Internet Engineering Task Force (IETF) J. Alvarez-Hamelin
Internet-Draft Universidad de Buenos Aires Request for Comments: 9198 Universidad de Buenos Aires
Updates: 2330 (if approved) A. Morton Updates: 2330 A. Morton
Intended status: Standards Track AT&T Labs Category: Standards Track AT&T Labs
Expires: February 14, 2021 J. Fabini ISSN: 2070-1721 J. Fabini
TU Wien TU Wien
C. Pignataro C. Pignataro
Cisco Systems, Inc. Cisco Systems, Inc.
R. Geib R. Geib
Deutsche Telekom Deutsche Telekom
August 13, 2020 May 2022
Advanced Unidirectional Route Assessment (AURA) Advanced Unidirectional Route Assessment (AURA)
draft-ietf-ippm-route-10
Abstract Abstract
This memo introduces an advanced unidirectional route assessment This memo introduces an advanced unidirectional route assessment
(AURA) metric and associated measurement methodology, based on the IP (AURA) metric and associated measurement methodology based on the IP
Performance Metrics (IPPM) Framework RFC 2330. This memo updates RFC Performance Metrics (IPPM) framework (RFC 2330). This memo updates
2330 in the areas of path-related terminology and path description, RFC 2330 in the areas of path-related terminology and path
primarily to include the possibility of parallel subpaths between a description, primarily to include the possibility of parallel
given Source and Destination pair, owing to the presence of multi- subpaths between a given Source and Destination pair, owing to the
path technologies. presence of multipath technologies.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This is an Internet Standards Track document.
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.
This Internet-Draft will expire on February 14, 2021. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9198.
Copyright Notice Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Revised BSD License text as described in Section 4.e of the
the Trust Legal Provisions and are provided without warranty as Trust Legal Provisions and are provided without warranty as described
described in the Simplified BSD License. in the Revised BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction
1.1. Issues with Earlier Work to Define a Route Metric . . . . 3 1.1. Issues with Earlier Work to Define a Route Metric
1.2. Requirements Language . . . . . . . . . . . . . . . . . . 4 1.2. Requirements Language
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope
3. Route Metric Specifications . . . . . . . . . . . . . . . . . 5 3. Route Metric Specifications
3.1. Terms and Definitions . . . . . . . . . . . . . . . . . . 5 3.1. Terms and Definitions
3.2. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 3.2. Formal Name
3.3. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 3.3. Parameters
3.4. Metric Definitions . . . . . . . . . . . . . . . . . . . 7 3.4. Metric Definitions
3.5. Related Round-Trip Delay and Loss Definitions . . . . . . 9 3.5. Related Round-Trip Delay and Loss Definitions
3.6. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 3.6. Discussion
3.7. Reporting the Metric . . . . . . . . . . . . . . . . . . 10 3.7. Reporting the Metric
4. Route Assessment Methodologies . . . . . . . . . . . . . . . 11 4. Route Assessment Methodologies
4.1. Active Methodologies . . . . . . . . . . . . . . . . . . 11 4.1. Active Methodologies
4.1.1. Temporal Composition for Route Metrics . . . . . . . 13 4.1.1. Temporal Composition for Route Metrics
4.1.2. Routing Class Identification . . . . . . . . . . . . 15 4.1.2. Routing Class Identification
4.1.3. Intermediate Observation Point Route Measurement . . 16 4.1.3. Intermediate Observation Point Route Measurement
4.2. Hybrid Methodologies . . . . . . . . . . . . . . . . . . 16 4.2. Hybrid Methodologies
4.3. Combining Different Methods . . . . . . . . . . . . . . . 17 4.3. Combining Different Methods
5. Background on Round-Trip Delay Measurement Goals . . . . . . 17 5. Background on Round-Trip Delay Measurement Goals
6. RTD Measurements Statistics . . . . . . . . . . . . . . . . . 18 6. RTD Measurements Statistics
7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 7. Security Considerations
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 8. IANA Considerations
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 9. References
10. Appendix I MPLS Methods for Route Assessment . . . . . . . . 21 9.1. Normative References
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 9.2. Informative References
11.1. Normative References . . . . . . . . . . . . . . . . . . 22 Appendix A. MPLS Methods for Route Assessment
11.2. Informative References . . . . . . . . . . . . . . . . . 24 Acknowledgements
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses
1. Introduction 1. Introduction
The IETF IP Performance Metrics (IPPM) working group first created a The IETF IP Performance Metrics (IPPM) Working Group first created a
framework for metric development in [RFC2330]. This framework has framework for metric development in [RFC2330]. This framework has
stood the test of time and enabled development of many fundamental stood the test of time and enabled development of many fundamental
metrics. It has been updated in the area of metric composition metrics. It has been updated in the area of metric composition
[RFC5835] and in several areas related to active stream measurement
[RFC5835], and in several areas related to active stream measurement
of modern networks with reactive properties [RFC7312]. of modern networks with reactive properties [RFC7312].
The [RFC2330] framework motivated the development of "performance and The framework in [RFC2330] motivated the development of "performance
reliability metrics for paths through the Internet," and Section 5 of and reliability metrics for paths through the Internet"; Section 5 of
[RFC2330] defines terms that support description of a path under [RFC2330] defines terms that support description of a path under
test. However, metrics for assessment of paths and related test. However, metrics for assessment of paths and related
performance aspects had not been attempted in IPPM when the [RFC2330] performance aspects had not been attempted in IPPM when the framework
framework was written. in [RFC2330] was written.
This memo takes up the route measurement challenge and specifies a This memo takes up the Route measurement challenge and specifies a
new route metric, two practical frameworks for methods of measurement new Route metric, two practical frameworks for methods of measurement
(using either active or hybrid active-passive methods [RFC7799]), and (using either active or hybrid active-passive methods [RFC7799]), and
Round-Trip Delay and link information discovery using the results of Round-Trip Delay and link information discovery using the results of
measurements. All route measurements are limited by the willingness measurements. All Route measurements are limited by the willingness
of hosts along the path to be discovered, to cooperate with the of Hosts along the path to be discovered, to cooperate with the
methods used, or to recognize that the measurement operation is methods used, or to recognize that the measurement operation is
taking place (such as when tunnels are present). taking place (such as when tunnels are present).
1.1. Issues with Earlier Work to Define a Route Metric 1.1. Issues with Earlier Work to Define a Route Metric
Section 7 of [RFC2330] presented a simple example of a "route" metric Section 7 of [RFC2330] presents a simple example of a "Route" metric
along with several other examples. The example is reproduced below along with several other examples. The example is reproduced below
(where the reference is to Section 5 of [RFC2330]): (where the reference is to Section 5 of [RFC2330]):
"route: The path, as defined in Section 5, from A to B at a given | route: The path, as defined in Section 5, from A to B at a given
time." | time.
This example provides a starting point to develop a more complete This example provides a starting point to develop a more complete
definition of route. Areas needing clarification include: definition of Route. Areas needing clarification include:
Time: In practice, the route will be assessed over a time interval, Time: In practice, the Route will be assessed over a time interval
because active path detection methods like Paris Traceroute [PT] because active path detection methods like Paris-traceroute [PT]
rely on hop limits for their operation and cannot accomplish rely on Hop Limits for their operation and cannot accomplish
discovery of all hosts using a single packet. discovery of all Hosts using a single packet.
Type-P: The legacy route definition lacks the option to cater for Type-P: The legacy Route definition lacks the option to cater for
packet-dependent routing. In this memo, we assess the route for a packet-dependent routing. In this memo, we assess the Route for a
specific packet of Type-P, and reflect this in the metric specific packet of Type-P and reflect this in the metric
definition. The methods of measurement determine the specific definition. The methods of measurement determine the specific
Type-P used. Type-P used.
Parallel Paths: Parallel paths are a reality of the Internet and a Parallel Paths: Parallel paths are a reality of the Internet and a
strength of advanced route assessment methods, so the metric must strength of advanced Route assessment methods, so the metric must
acknowledge this possibility. Use of Equal Cost Multi-Path (ECMP) acknowledge this possibility. Use of Equal-Cost Multipath (ECMP)
and Unequal Cost Multi-Path (UCMP) technologies are common sources and Unequal-Cost Multipath (UCMP) technologies are common sources
of parallel subpaths. of parallel subpaths.
Cloud Subpath: May contain hosts that do not decrement hop limit, Cloud Subpath: Cloud subpaths may contain Hosts that do not
but may have two or more exchange links connecting "discoverable" decrement the Hop Limit but may have two or more exchange links
hosts or routers. Parallel subpaths contained within clouds connecting "discoverable" Hosts or routers. Parallel subpaths
cannot be discovered. The assessment methods only discover hosts contained within clouds cannot be discovered. The assessment
or routers on the path that decrement hop limit, or cooperate with methods only discover Hosts or routers on the path that decrement
interrogation protocols. The presence of tunnels and nested Hop Limit or cooperate with interrogation protocols. The presence
tunnels further complicate assessment by hiding hops. of tunnels and nested tunnels further complicate assessment by
hiding Hops.
Hop: Although the [RFC2330] definition of a hop was a link-host Hop: The definition of Hop in [RFC2330] was a link-Host pair.
pair, only hosts that are discoverable or have the capability to However, only Hosts that were discoverable and cooperated with
cooperate with interrogation protocols where link information may interrogation protocols (where link information may be exposed)
be exposed. provided both link and Host information.
The refined definition of Route metrics begins in the sections that Note that the actual definitions appear in Section 3.1.
follow.
1.2. Requirements Language 1.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in BCP
14[RFC2119] [RFC8174] when, and only when, they appear in all 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
2. Scope 2. Scope
The purpose of this memo is to add new route metrics and methods of The purpose of this memo is to add new Route metrics and methods of
measurement to the existing set of IPPM metrics. measurement to the existing set of IPPM metrics.
The scope is to define route metrics that can identify the path taken The scope is to define Route metrics that can identify the path taken
by a packet or a flow traversing the Internet between two hosts. by a packet or a flow traversing the Internet between two Hosts.
Although primarily intended for hosts communicating on the Internet, Although primarily intended for Hosts communicating on the Internet,
the definitions and metrics are constructed to be applicable to other the definitions and metrics are constructed to be applicable to other
network domains, if desired. The methods of measurement to assess network domains, if desired. The methods of measurement to assess
the path may not be able to discover all hosts comprising the path, the path may not be able to discover all Hosts comprising the path,
but such omissions are often deterministic and explainable sources of but such omissions are often deterministic and explainable sources of
error. error.
This memo also specifies a framework for active methods of This memo also specifies a framework for active methods of
measurement which uses the techniques described in [PT], as well as a measurement that uses the techniques described in [PT] as well as a
framework for hybrid active-passive methods of measurement such as framework for hybrid active-passive methods of measurement, such as
the Hybrid Type I method [RFC7799] described in the Hybrid Type I method [RFC7799] described in [RFC9197]. Methods
[I-D.ietf-ippm-ioam-data]. Methods using [I-D.ietf-ippm-ioam-data] using [RFC9197] are intended only for single administrative domains
are intended only for single administrative domains that provide a that provide a protocol for explicit interrogation of Nodes on a
protocol for explicit interrogation of nodes on a path. Combinations path. Combinations of active methods and hybrid active-passive
of active methods and hybrid active-passive methods are also in- methods are also in scope.
scope.
Further, this memo provides additional analysis of the round-trip Further, this memo provides additional analysis of the Round-Trip
delay measurements made possible by the methods, in an effort to Delay measurements made possible by the methods in an effort to
discover more details about the path, such as the link technology in discover more details about the path, such as the link technology in
use. use.
This memo updates Section 5 of [RFC2330] in the areas of path-related This memo updates Section 5 of [RFC2330] in the areas of path-related
terminology and path description, primarily to include the terminology and path description, primarily to include the
possibility of parallel subpaths between a given Source and possibility of parallel subpaths between a given Source and
Destination address pair (possibly resulting from Equal Cost Multi- Destination address pair (possibly resulting from ECMP and UCMP
Path (ECMP) and Unequal Cost Multi-Path (UCMP) technologies). technologies).
There are several simple non-goals of this memo. There is no attempt There are several simple non-goals of this memo. There is no attempt
to assess the reverse path from any host on the path to the host to assess the reverse path from any Host on the path to the Host
attempting the path measurement. The reverse path contribution to attempting the path measurement. The reverse path contribution to
delay will be that experienced by ICMP packets (in active methods), delay will be that experienced by ICMP packets (in active methods)
and may be different from delays experienced by UDP or TCP packets. and may be different from delays experienced by UDP or TCP packets.
Also, the round trip delay will include an unknown contribution of Also, the Round-Trip Delay will include an unknown contribution of
processing time at the host that generates the ICMP response. processing time at the Host that generates the ICMP response.
Therefore, the ICMP-based active methods are not supposed to yield Therefore, the ICMP-based active methods are not supposed to yield
accurate, reproducible estimations of the Round-Trip Delay that UDP accurate, reproducible estimations of the Round-Trip Delay that UDP
or TCP packets will experience. or TCP packets will experience.
3. Route Metric Specifications 3. Route Metric Specifications
This section sets requirements for the components of the Route This section sets requirements for the components of the route
Metric. metric.
3.1. Terms and Definitions 3.1. Terms and Definitions
Host A Host as defined in [RFC2330] (a computer capable of IP Host
communication, includes routers), a.k.a. RFC 2330 Host. A Host (as defined in [RFC2330]) is a computer capable of IP
communication, including routers (aka an RFC 2330 Host).
Node A Node is any network function on the path capable of IP-layer Node
Communication, includes RFC 2330 Hosts. A Node is any network function on the path capable of IP-layer
Communication, including RFC 2330 Hosts.
Node Identity The unique address for Nodes communicating within the Node Identity
network domain. For Nodes communicating on the Internet with IP, The Node identity is the unique address for Nodes communicating
it is the globally routable IP address which the Node uses when within the network domain. For Nodes communicating on the
communicating with other Nodes under normal or error conditions. Internet with IP, it is the globally routable IP address that the
The Node Identity revealed (and its connection to a Node Name Node uses when communicating with other Nodes under normal or
through reverse DNS) determines whether interfaces to parallel error conditions. The Node identity revealed (and its connection
links can be associated with a single Node, or appear to identify to a Node name through reverse DNS) determines whether interfaces
unique Nodes. to parallel links can be associated with a single Node or appear
to identify unique Nodes.
Discoverable Node Nodes that convey their Node Identity according to Discoverable Node
the requirements of their network domain, such as when error Discoverable Nodes are Nodes that convey their Node identity
conditions are detected by that Node. For Nodes communicating according to the requirements of their network domain, such as
with IP packets, compliance with Section 3.2.2.4 of [RFC1122] when when error conditions are detected by that Node. For Nodes
discarding a packet due to TTL or Hop Limit Exceeded condition, communicating with IP packets, compliance with Section 3.2.2.4 of
MUST result in sending the corresponding Time Exceeded message [RFC1122], when discarding a packet due to TTL or Hop Limit
(containing a form of Node identity) to the source. This Exceeded condition, MUST result in sending the corresponding Time
requirement is also consistent with section 5.3.1 of [RFC1812] for Exceeded message (containing a form of Node identity) to the
routers. source. This requirement is also consistent with Section 5.3.1 of
[RFC1812] for routers.
Cooperating Node Nodes that respond to direct queries for their Node Cooperating Node
identity as part of a previously agreed and established Cooperating Nodes are Nodes that respond to direct queries for
interrogation protocol. Nodes SHOULD also provide information their Node identity as part of a previously established and agreed
such as arrival/departure interface identification, arrival upon interrogation protocol. Nodes SHOULD also provide
timestamp, and any relevant information about the Node or specific information such as arrival/departure interface identification,
link which delivered the query to the Node. arrival timestamp, and any relevant information about the Node or
specific link that delivered the query to the Node.
Hop specification A Hop specification MUST contain a Node Identity, Hop specification
and MAY contain arrival and/or departure interface identification, A Hop specification MUST contain a Node identity and MAY contain
round trip delay, and an arrival timestamp. arrival and/or departure interface identification, Round-Trip
Delay, and an arrival timestamp.
Routing Class A route that treats equally a class of different types Routing Class
of packets, designated "C" (unrelated to address classes of the Routing Class is a Route that treats a class of different types of
past) [RFC2330] [RFC8468]. Knowledge of such a class allows any packets, designated "C" (unrelated to address classes of the past)
one of the types of packets within that class to be used for equally ([RFC2330] [RFC8468]). Knowledge of such a class allows
subsequent measurement of the route. The designator "class C" is any one of the types of packets within that class to be used for
used for historical reasons, see [RFC2330]. subsequent measurement of the Route. The designator "class C" is
used for historical reasons; see [RFC2330].
3.2. Formal Name 3.2. Formal Name
The formal name of the metric is: The formal name of the metric is:
Type-P-Route-Ensemble-Method-Variant Type-P-Route-Ensemble-Method-Variant
abbreviated as Route Ensemble. abbreviated as Route Ensemble.
Note that Type-P depends heavily on the chosen method and variant. Note that Type-P depends heavily on the chosen method and variant.
3.3. Parameters 3.3. Parameters
This section lists the REQUIRED input factors to define and measure a This section lists the REQUIRED input factors to define and measure a
Route metric, as specified in this memo. Route metric, as specified in this memo.
o Src, the address of a Node (such as the globally routable IP Src: the address of a Node (such as the globally routable IP
address). address).
o Dst, the address of a Node (such as the globally routable IP Dst: the address of a Node (such as the globally routable IP
address). address).
o i, the limit on the number of Hops a specific packet may visit as i: the limit on the number of Hops a specific packet may visit as it
it traverses from the Node at Src to the Node at Dst (such as the traverses from the Node at Src to the Node at Dst (such as the TTL
TTL or Hop Limit). or Hop Limit).
o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). MaxHops: the maximum value of i used (i=1,2,3,...MaxHops).
o T0, a time (start of measurement interval) T0: a time (start of measurement interval).
o Tf, a time (end of measurement interval) Tf: a time (end of measurement interval).
o MP(address), Measurement Point at address, such as Src or Dst, MP(address): the Measurement Point at address, such as Src or Dst,
usually at the same node stack layer as "address". usually at the same Node stack layer as "address".
o T, the Node time of a packet as measured at MP(Src), meaning T: the Node time of a packet as measured at MP(Src), meaning
Measurement Point at the Source. Measurement Point at the Source.
o Ta, the Node time of a reply packet's *arrival* as measured at Ta: the Node time of a reply packet's *arrival* as measured at
MP(Src), assigned to packets that arrive within a "reasonable" MP(Src), assigned to packets that arrive within a "reasonable"
time (see parameter below). time (see parameter below).
o Tmax, a maximum waiting time for reply packets to return to the Tmax: a maximum waiting time for reply packets to return to the
source, set sufficiently long to disambiguate packets with long source, set sufficiently long to disambiguate packets with long
delays from packets that are discarded (lost), such that the delays from packets that are discarded (lost), such that the
distribution of Round-Trip Delay is not truncated. distribution of Round-Trip Delay is not truncated.
o F, the number of different flows simulated by the method and F: the number of different flows simulated by the method and
variant. variant.
o flow, the stream of packets with the same n-tuple of designated flow: the stream of packets with the same n-tuple of designated
header fields that (when held constant) result in identical header fields that (when held constant) result in identical
treatment in a multi-path decision (such as the decision taken in treatment in a multipath decision (such as the decision taken in
load balancing). Note: The IPv6 flow label MAY be included in the load balancing). Note: The IPv6 flow label MAY be included in the
flow definition if the MP(Src) is a Tunnel End Point (TEP) flow definition if the MP(Src) is a Tunnel Endpoint (TEP)
complying with [RFC6438] guidelines. complying with the guidelines in [RFC6438].
o Type-P, the complete description of the packets for which this Type-P: the complete description of the packets for which this
assessment applies (including the flow-defining fields). assessment applies (including the flow-defining fields).
3.4. Metric Definitions 3.4. Metric Definitions
This section defines the REQUIRED measurement components of the Route This section defines the REQUIRED measurement components of the Route
metrics (unless otherwise indicated): metrics (unless otherwise indicated):
M, the total number of packets sent between T0 and Tf. M: the total number of packets sent between T0 and Tf.
N, the smallest value of i needed for a packet to be received at Dst N: the smallest value of i needed for a packet to be received at Dst
(sent between T0 and Tf). (sent between T0 and Tf).
Nmax, the largest value of i needed for a packet to be received at Nmax: the largest value of i needed for a packet to be received at
Dst (sent between T0 and Tf). Nmax may be equal to N. Dst (sent between T0 and Tf). Nmax may be equal to N.
Next define a *singleton* definition for a Node on the path, with Next, define a *singleton* for a Node on the path with sufficient
sufficient indexes to identify all Nodes identified in a measurement indexes to identify all Nodes identified in a measurement interval
interval (where *singleton* is part of the IPPM Framework [RFC2330]). (where *singleton* is part of the IPPM Framework [RFC2330]).
A Hop Specification, designated h(i,j), the IP address and/or singleton: A Hop specification, designated h(i,j), the IP address
identity of Discoverable Nodes (or Cooperating Nodes) that are i hops and/or identity of Discoverable Nodes (or Cooperating Nodes) that
away from the Node with address = Src and part of Route j during the are i Hops away from the Node with address = Src and part of Route
measurement interval, T0 to Tf. As defined here, a Hop singleton j during the measurement interval T0 to Tf. As defined here, a
measurement MUST contain a Node Identity, hid(i,j), and MAY contain Hop singleton measurement MUST contain a Node identity, hid(i,j),
one or more of the following attributes: and MAY contain one or more of the following attributes:
o a(i,j) Arrival Interface ID (e.g., when [RFC5837] is supported) * a(i,j) Arrival Interface ID (e.g., when [RFC5837] is supported)
o d(i,j) Departure Interface ID (e.g., when [RFC5837] is supported) * d(i,j) Departure Interface ID (e.g., when [RFC5837] is supported)
o t(i,j) Arrival Timestamp, where t(i,j) is ideally supplied by the * t(i,j) arrival timestamp, where t(i,j) is ideally supplied by the
Hop. (Note that t(i,j) might be approximated from the sending time Hop (note that t(i,j) might be approximated from the sending time
of the packet that revealed the Hop, e.g., when the round trip of the packet that revealed the Hop, e.g., when the round-trip
response time is available and divided by 2.) response time is available and divided by 2)
o Measurements of Round-Trip Delay (for each packet that reveals the * Measurements of Round-Trip Delay (for each packet that reveals the
same Node Identity and flow attributes, then this attribute is same Node identity and flow attributes, then this attribute is
computed, see next section) computed; see next section)
Node Identities and related information can be ordered by their Node identities and related information can be ordered by their
distance from the Node with address Src in Hops h(i,j). Based on distance from the Node with address Src in Hops h(i,j). Based on
this, two forms of Routes are distinguished: this, two forms of Routes are distinguished:
A Route Ensemble is defined as the combination of all routes A Route Ensemble is defined as the combination of all Routes
traversed by different flows from the Node at Src address to the Node traversed by different flows from the Node at Src address to the Node
at Dst address. A single Route traversed by a single flow at Dst address. A single Route traversed by a single flow
(determined by an unambiguous tuple of addresses Src and Dst, and (determined by an unambiguous tuple of addresses Src and Dst and
other identical flow criteria) is a member of the Route Ensemble and other identical flow criteria) is a member of the Route Ensemble and
called a Member Route. called a Member Route.
Using h(i,j) and components and parameters, further define: Using h(i,j) and components and parameters further define:
When considering the set of Hops in the context of a single flow, a When considering the set of Hops in the context of a single flow, a
Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1, Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1,
j) and h(i, j) are 1 hop away from each other and Nj satisfying j) and h(i, j) are one Hop away from each other and Nj satisfying
h(Nj,j)=Dst is the minimum count of Hops needed by the packet on h(Nj,j)=Dst is the minimum count of Hops needed by the packet on
Member Route j to reach Dst. Member Routes must be unique. The member Route j to reach Dst. Member Routes must be unique. The
uniqueness property requires that any two Member routes j and k that uniqueness property requires that any two Member Routes, j and k,
are part of the same Route Ensemble differ either in terms of minimum that are part of the same Route Ensemble differ either in terms of
hop count Nj and Nk to reach the destination Dst, or, in the case of minimum Hop count Nj and Nk to reach the destination Dst or, in the
identical hop count Nj=Nk, they have at least one distinct Hop: case of identical Hop count Nj=Nk, they have at least one distinct
h(i,j) != h(i,k) for at least one i (i=1..Nj). Hop: h(i,j) != h(i,k) for at least one i (i=1..Nj).
All the optional information collected to describe a Member Route, All the optional information collected to describe a Member Route,
such as the arrival interface, departure interface, and Round Trip such as the arrival interface, departure interface, and Round-Trip
Delay at each Hop, turns each list item into a rich structure. There Delay at each Hop, turns each list item into a rich structure. There
may be information on the links between Hops, possibly information on may be information on the links between Hops, possible information on
the routing (arrival interface and departure interface), an estimate the routing (arrival interface and departure interface), an estimate
of distance between Hops based on Round-Trip Delay measurements and of distance between Hops based on Round-Trip Delay measurements and
calculations, and a time stamp indicating when all these additional calculations, and a timestamp indicating when all these additional
details were valid. details were measured.
The Route Ensemble from Src to Dst, during the measurement interval The Route Ensemble from Src to Dst, during the measurement interval
T0 to Tf, is the aggregate of all m distinct Member Routes discovered T0 to Tf, is the aggregate of all m distinct Member Routes discovered
between the two Nodes with Src and Dst addresses. More formally, between the two Nodes with Src and Dst addresses. More formally,
with the Node having address Src omitted: with the Node having address Src omitted:
Route Ensemble = { Route Ensemble = {
{h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst},
{h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst},
... ...
{h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst}
} }
where the following conditions apply: i <= Nj <= Nmax (j=1..m) where the following conditions apply: i <= Nj <= Nmax (j=1..m)
Note that some h(i,j) may be empty (null) in the case that systems do Note that some h(i,j) may be empty (null) in the case that systems do
not reply (not discoverable, or not cooperating). not reply (not discoverable or not cooperating).
h(i-1,j) and h(i,j) are the Hops on the same Member Route one hop h(i-1,j) and h(i,j) are the Hops on the same Member Route one Hop
away from each other. away from each other.
Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l ; which Hop h(i,j) may be identical with h(k,l) for i!=k and j!=l, which
means there may be portions shared among different Member Routes means there may be portions shared among different Member Routes
(parts of Member Routes may overlap). (parts of Member Routes may overlap).
3.5. Related Round-Trip Delay and Loss Definitions 3.5. Related Round-Trip Delay and Loss Definitions
RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-Trip RTD(i,j,T) is defined as a singleton of the [RFC2681] Round-Trip
Delay between the Node with address = Src and the Node at Hop h(i,j) Delay between the Node with address = Src and the Node at Hop h(i,j)
at time T. at time T.
RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-trip Loss RTL(i,j,T) is defined as a singleton of the [RFC6673] Round-Trip Loss
between the Node with address = Src and the Node at Hop h(i,j) at between the Node with address = Src and the Node at Hop h(i,j) at
time T. time T.
3.6. Discussion 3.6. Discussion
Depending on the way that Node Identity is revealed, it may be Depending on the way that the Node identity is revealed, it may be
difficult to determine parallel subpaths between the same pair of difficult to determine parallel subpaths between the same pair of
Nodes (i.e. multiple parallel links). It is easier to detect Nodes (i.e., multiple parallel links). It is easier to detect
parallel subpaths involving different Nodes. parallel subpaths involving different Nodes.
o If a pair of discovered Nodes identify two different addresses (IP * If a pair of discovered Nodes identify two different addresses (IP
or not), then they will appear to be different Nodes. See item or not), then they will appear to be different Nodes. See item
below. below.
o If a pair of discovered Nodes identify two different IP addresses, * If a pair of discovered Nodes identify two different IP addresses
and the IP addresses resolve to the same Node name (in the DNS), and the IP addresses resolve to the same Node name (in the DNS),
then they will appear to be the same Nodes. then they will appear to be the same Node.
o If a discovered Node always replies using the same network * If a discovered Node always replies using the same network
address, regardless of the interface a packet arrives on, then address, regardless of the interface a packet arrives on, then
multiple parallel links cannot be detected in that network domain. multiple parallel links cannot be detected in that network domain.
This condition may apply to traceroute-style methods, but may not This condition may apply to traceroute-style methods but may not
apply to other hybrid methods based on In-situ Operations, apply to other hybrid methods based on In situ Operations,
Administration, and Maintenance (IOAM). For example, if the Administration, and Maintenance (IOAM). For example, if the ICMP
[RFC5837] ICMP extension mechanism is implemented, then parallel extension mechanism described in [RFC5837] is implemented, then
links can be detected with the discovery traceroute-style methods. parallel links can be detected with the discovery traceroute-style
methods.
o If parallel links between routers are aggregated below the IP * If parallel links between routers are aggregated below the IP
layer, then from Node point of view, all these links share the layer, then, from the Node's point of view, all these links share
same pair of IP addresses. The existence of these parallel links the same pair of IP addresses. The existence of these parallel
can't be detected at the IP layer. This applies to other network links can't be detected at the IP layer. This applies to other
domains with layers below them, as well. This condition may apply network domains with layers below them as well. This condition
to traceroute-style methods, but may not apply to other hybrid may apply to traceroute-style methods but may not apply to other
methods based on IOAM. hybrid methods based on IOAM.
When a route assessment employs IP packets (for example), the reality When a Route assessment employs IP packets (for example), the reality
of flow assignment to parallel subpaths involves layers above IP. of flow assignment to parallel subpaths involves layers above IP.
Thus, the measured Route Ensemble is applicable to IP and higher Thus, the measured Route Ensemble is applicable to IP and higher
layers (as described in the methodology's packet of Type-P and flow layers (as described in the methodology's packet of Type-P and flow
parameters). parameters).
3.7. Reporting the Metric 3.7. Reporting the Metric
An Information Model and an XML Data Model for Storing Traceroute An Information Model and an XML Data Model for Storing Traceroute
Measurements is available in [RFC5388]. The measured information at Measurements is available in [RFC5388]. The measured information at
each hop includes four pieces of information: a one-dimensional hop each Hop includes four pieces of information: a one-dimensional Hop
index, Node symbolic address, Node IP address, and RTD for each index, Node symbolic address, Node IP address, and RTD for each
response. response.
The description of Hop information that may be collected according to The description of Hop information that may be collected according to
this memo covers more dimensions, as defined in Section 3.4 above. this memo covers more dimensions, as defined in Section 3.4. For
example, the Hop index is two-dimensional to capture the complexity
For example, the Hop index is two-dimensional to capture the of a Route Ensemble, and it contains corresponding Node identities at
complexity of a Route Ensemble, and it contains corresponding Node a minimum. The models need to be expanded to include these features
identities at a minimum. The models need to be expanded to include as well as Arrival Interface ID, Departure Interface ID, and arrival
these features, as well as Arrival Interface ID, Departure Interface timestamp, when available. The original sending Timestamp from the
ID, and Arrival Timestamp, when available. The original sending Src Node anchors a particular measurement in time.
Timestamp from the Src Node anchors a particular measurement in time.
4. Route Assessment Methodologies 4. Route Assessment Methodologies
There are two classes of methods described in this section, active There are two classes of methods described in this section, active
methods relying on the reaction to TTL or Hop Limit Exceeded methods relying on the reaction to TTL or Hop Limit Exceeded
condition to discover Nodes on a path, and Hybrid active-passive condition to discover Nodes on a path and hybrid active-passive
methods that involve direct interrogation of cooperating Nodes methods that involve direct interrogation of Cooperating Nodes
(usually within a single domain). Description of these methods (usually within a single domain). Description of these methods
follow. follow.
4.1. Active Methodologies 4.1. Active Methodologies
This section describes the method employed by current open source This section describes the method employed by current open-source
tools, thereby providing a practical framework for further advanced tools, thereby providing a practical framework for further advanced
techniques to be included as method variants. This method is techniques to be included as method variants. This method is
applicable for use across multiple administrative domains. applicable for use across multiple administrative domains.
Internet routing is complex because it depends on the policies of Internet routing is complex because it depends on the policies of
thousands of Autonomous Systems (AS). Most routers perform load thousands of Autonomous Systems (ASes). Most routers perform load
balancing on flows using a form of Equal Cost Multiple Path (ECMP). balancing on flows using a form of ECMP. [RFC2991] describes a
[RFC2991] describes a number of flow-based or hashed approaches number of flow-based or hashed approaches (e.g., Modulo-N Hash, Hash-
(e.g., Modulo-N Hash, Hash-Threshold, Highest Random Weight (HRW)), Threshold, and Highest Random Weight (HRW)) and makes some good
and makes some good suggestions. Flow-based ECMP avoids increased suggestions. Flow-based ECMP avoids increased packet Delay Variation
packet delay variation and possibly overwhelming levels of packet and possibly overwhelming levels of packet reordering in flows.
reordering in flows.
A few routers still divide the workload through packet-based A few routers still divide the workload through packet-based
techniques, such as a round-robin scheme to distribute every new techniques, such as a round-robin scheme to distribute every new
outgoing packet to multiple links, as explained in [RFC2991]. The outgoing packet to multiple links, as explained in [RFC2991]. The
methods described in this section assume flow-based ECMP. methods described in this section assume flow-based ECMP.
Taking into account that Internet protocol was designed under the Taking into account that Internet protocol was designed under the
"end-to-end" principle, the IP payload and its header do not provide "end-to-end" principle, the IP payload and its header do not provide
any information about the routes or path necessary to reach some any information about the Routes or path necessary to reach some
destination. For this reason, the popular tool traceroute was destination. For this reason, the popular tool, traceroute, was
developed to gather the IP addresses of each hop along a path using developed to gather the IP addresses of each Hop along a path using
the ICMP protocol [RFC0792]. Traceroute also measures RTD from each ICMP [RFC0792]. Traceroute also measures RTD from each Hop. However,
hop. However, the growing complexity of the Internet makes it more the growing complexity of the Internet makes it more challenging to
challenging to develop an accurate traceroute implementation. For develop an accurate traceroute implementation. For instance, the
instance, the early traceroute tools would be inaccurate in the early traceroute tools would be inaccurate in the current network,
current network, mainly because they were not designed to retain a mainly because they were not designed to retain a flow state.
flow state. However, evolved traceroute tools, such as Paris- However, evolved traceroute tools, such as Paris-traceroute ([PT]
traceroute [PT] [MLB] and Scamper [SCAMPER], expect to encounter ECMP [MLB]) and Scamper ([SCAMPER]), expect to encounter ECMP and achieve
and achieve more accurate results when they do, where Scamper ensures more accurate results when they do, where Scamper ensures traceroute
traceroute packets will follow the same path in 98% of packets will follow the same path in 98% of cases ([SCAMPER]).
cases[SCAMPER].
Today's traceroute tools send Type-P of packets, either ICMP, UDP, or Today's traceroute tools send Type-P of packets, which are either
TCP. UDP and TCP are used when a particular characteristic needs to ICMP, UDP, or TCP. UDP and TCP are used when a particular
be verified, such as filtering or traffic shaping on specific ports characteristic needs to be verified, such as filtering or traffic
(i.e., services). UDP and TCP traceroute are also used when ICMP shaping on specific ports (i.e., services). UDP and TCP traceroute
responses are not received. [SCAMPER] supports IPv6 traceroute are also used when ICMP responses are not received. [SCAMPER]
measurements, keeping the FlowLabel constant in all packets. supports IPv6 traceroute measurements, keeping the Flow Label
constant in all packets.
Paris-traceroute allows its users to measure the RTD to every Node of Paris-traceroute allows its users to measure the RTD to every Node of
the path for a particular flow. Furthermore, either Paris-traceroute the path for a particular flow. Furthermore, either Paris-traceroute
or Scamper is capable of unveiling the many available paths between a or Scamper is capable of unveiling the many available paths between a
source and destination (which are visible to active methods). This source and destination (which are visible to active methods). This
task is accomplished by repeating complete traceroute measurements task is accomplished by repeating complete traceroute measurements
with different flow parameters for each measurement; Paris-traceroute with different flow parameters for each measurement; Paris-traceroute
provides "exhaustive" mode while scamper provides "tracelb" (stands provides an "exhaustive" mode, while Scamper provides "tracelb"
for traceroute load balance). The Framework for IP Performance (which stands for "traceroute load balance"). "Framework for IP
Metrics (IPPM) ([RFC2330] updated by[RFC7312]) has the flexibility to Performance Metrics" [RFC2330], updated by [RFC7312], has the
require that the Round-Trip Delay measurement [RFC2681] uses packets flexibility to require that the Round-Trip Delay measurement
with the constraints to assure that all packets in a single [RFC2681] uses packets with the constraints to assure that all
measurement appear as the same flow. This flexibility covers ICMP, packets in a single measurement appear as the same flow. This
UDP, and TCP. The accompanying methodology of [RFC2681] needs to be flexibility covers ICMP, UDP, and TCP. The accompanying methodology
expanded to report the sequential hop identifiers along with RTD of [RFC2681] needs to be expanded to report the sequential Hop
measurements, but no new metric definition is needed. identifiers along with RTD measurements, but no new metric definition
is needed.
The advanced route assessment methods used in Paris-traceroute [PT] The advanced Route assessment methods used in Paris-traceroute [PT]
keep the critical fields constant for every packet to maintain the keep the critical fields constant for every packet to maintain the
appearance of the same flow. When considering IPv6 headers, it is appearance of the same flow. When considering IPv6 headers, it is
necessary to ensure that the IP source and destination addresses and necessary to ensure that the IP Source and Destination addresses and
the FlowLabel are constant (but note that many routers ignore the Flow Label are constant (but note that many routers ignore the Flow
FlowLabel field at this time), see [RFC6437]. Use of IPv6 Extension Label field at this time); see [RFC6437]. Use of IPv6 Extension
Headers may add critical fields, and SHOULD be avoided. In IPv4, Headers may add critical fields and SHOULD be avoided. In IPv4,
certain fields of the IP header and the first four bytes of the IP certain fields of the IP header and the first 4 bytes of the IP
payload should remain constant in a flow. In the IPv4 header, the IP payload should remain constant in a flow. In the IPv4 header, the IP
source and destination addresses, protocol number, and Diffserv Source and Destination addresses, protocol number, and Diffserv
fields identify flows. The first four payload bytes include the UDP fields identify flows. The first 4 payload bytes include the UDP and
and TCP ports, and the ICMP type, code, and checksum fields. TCP ports and the ICMP type, code, and checksum fields.
Maintaining a constant ICMP checksum in IPv4 is most challenging, as Maintaining a constant ICMP checksum in IPv4 is most challenging, as
the ICMP sequence number or identifier fields will usually change for the ICMP sequence number or identifier fields will usually change for
different probes of the same path. Probes should use arbitrary bytes different probes of the same path. Probes should use arbitrary bytes
in the ICMP data field to offset changes to sequence number and in the ICMP data field to offset changes to the sequence number and
identifier, thus keeping the checksum constant. identifier, thus keeping the checksum constant.
Finally, it is also essential to route the resulting ICMP Time Finally, it is also essential to Route the resulting ICMP Time
Exceeded messages along a consistent path. In IPv6, the fields above Exceeded messages along a consistent path. In IPv6, the fields above
are sufficient. In IPv4, the ICMP Time Exceeded message will contain are sufficient. In IPv4, the ICMP Time Exceeded message will contain
the IP header and the first eight bytes of the IP payload, which the IP header and the first 8 bytes of the IP payload, both of which
affects its ICMP checksum. The TCP sequence number, UDP Length, and affect its ICMP checksum calculation. The TCP sequence number, UDP
UDP checksum will affect this value, and should remain constant. length, and UDP checksum will affect this value and should remain
constant.
Formally, to maintain the same flow in the measurements to a Formally, to maintain the same flow in the measurements to a
particular hop, the Type-P-Route-Ensemble-Method-Variant packets particular Hop, the Type-P-Route-Ensemble-Method-Variant packets
should be[PT]: should have the following attributes (see [PT]):
o TCP case: For IPv4, the fields Src, Dst, port-Src, port_Dst, TCP case: For IPv4, the fields Src, Dst, port-Src, port_Dst,
sequence number, and Diffserv Field SHOULD be the same. For IPv6, sequence number, and Diffserv SHOULD be the same. For IPv6, the
the field FlowLabel, Src and Dst SHOULD be the same. fields Flow Label, Src, and Dst SHOULD be the same.
o UDP case: For IPv4, the fields Src, Dst, port-Src, port-Dst, UDP case: For IPv4, the fields Src, Dst, port-Src, port-Dst, and
Diffserv should be the same, and the UDP-checksum SHOULD change to Diffserv should be the same, and the UDP checksum SHOULD change to
keep the IP checksum of the ICMP time exceeded reply constant. keep the IP checksum of the ICMP Time Exceeded reply constant.
Then, the data length should be fixed, and the data field is used Then, the data length should be fixed, and the data field is used
to make it so (consider that ICMP checksum uses its data field, to make it so (consider that ICMP checksum uses its data field,
which contains the original IP header plus 8 bytes of UDP, where which contains the original IP header plus 8 bytes of UDP, where
TTL, IP identification, IP checksum, and UDP checksum changes). TTL, IP identification, IP checksum, and UDP checksum changes).
For IPv6, the field FlowLabel, and Source and Destination For IPv6, the field Flow Label and Source and Destination
addresses SHOULD be the same. addresses SHOULD be the same.
o ICMP case: For IPv4, the Data field SHOULD compensate variations ICMP case: For IPv4, the data field SHOULD compensate variations on
on TTL or Hop Limit, IP identification, and IP checksum for every TTL or Hop Limit, IP identification, and IP checksum for every
packet. There is no need to consider ICMPv6 because only packet. There is no need to consider ICMPv6 because only Flow
FlowLabel of IPv6 and Source and Destination addresses are used, Label of IPv6 and Source and Destination addresses are used, and
and all of them SHOULD be constant. all of them SHOULD be constant.
Then, the way to identify different hops and attempts of the same Then, the way to identify different Hops and attempts of the same
IPv4 flow is: IPv4 flow is:
o TCP case: The IP identification field. TCP case: The IP identification field.
o UDP case: The IP identification field. UDP case: The IP identification field.
o ICMP case: The IP identification field, and ICMP Sequence number. ICMP case: The IP identification field and ICMP sequence number.
4.1.1. Temporal Composition for Route Metrics 4.1.1. Temporal Composition for Route Metrics
The Active Route Assessment Methods described above have the ability The active Route assessment methods described above have the ability
to discover portions of a path where ECMP load balancing is present, to discover portions of a path where ECMP load balancing is present,
observed as two or more unique Member Routes having one or more observed as two or more unique Member Routes having one or more
distinct Hops which are part of the Route Ensemble. Likewise, distinct Hops that are part of the Route Ensemble. Likewise,
attempts to deliberately vary the flow characteristics to discover attempts to deliberately vary the flow characteristics to discover
all Member Routes will reveal portions of the path which are flow- all Member Routes will reveal portions of the path that are flow
invariant. invariant.
Section 9.2 of [RFC2330] describes Temporal Composition of metrics, Section 9.2 of [RFC2330] describes the Temporal Composition of
and introduces the possibility of a relationship between earlier metrics and introduces the possibility of a relationship between
measurement results and the results for measurement at the current earlier measurement results and the results for measurement at the
time (for a given metric). There is value in establishing a Temporal current time (for a given metric). There is value in establishing a
Composition relationship for Route Metrics. However, this Temporal Composition relationship for Route metrics; however, this
relationship does not represent a forecast of future route conditions relationship does not represent a forecast of future Route conditions
in any way. in any way.
For Route Metric measurements, the value of Temporal Composition is For Route-metric measurements, the value of Temporal Composition is
to reduce the measurement iterations required with repeated to reduce the measurement iterations required with repeated
measurements. Reduced iterations are possible by inferring that measurements. Reduced iterations are possible by inferring that
current measurements using fixed and previously measured flow current measurements using fixed and previously measured flow
characteristics: characteristics:
o will have many common hops with previous measurements. * will have many common Hops with previous measurements.
o will have relatively time-stable results at the ingress and egress * will have relatively time-stable results at the ingress and egress
portions of the path when measured from user locations, as opposed portions of the path when measured from user locations, as opposed
to measurements of backbone networks and across inter-domain to measurements of backbone networks and across inter-domain
gateways. gateways.
o may have greater potential for time-variation in path portions * may have greater potential for time variation in path portions
where ECMP load balancing is observed (because increasing or where ECMP load balancing is observed (because increasing or
decreasing the pool of links changes the hash calculations). decreasing the pool of links changes the hash calculations).
Optionally, measurement systems may take advantage of the inferences Optionally, measurement systems may take advantage of the inferences
above when seeking to reduce measurement iterations, after exhaustive above when seeking to reduce measurement iterations after exhaustive
measurements indicate that the time-stable properties are present. measurements indicate that the time-stable properties are present.
Repetitive Active Route measurement systems: Repetitive active Route measurement systems:
1. SHOULD occasionally check path portions which have exhibited 1. SHOULD occasionally check path portions that have exhibited
stable results over time, particularly ingress and egress stable results over time, particularly ingress and egress
portions of the path (e.g., daily checks if measuring many times portions of the path (e.g., daily checks if measuring many times
during a day). during a day).
2. SHOULD continue testing portions of the path that have previously 2. SHOULD continue testing portions of the path that have previously
exhibited ECMP load balancing. exhibited ECMP load balancing.
3. SHALL trigger re-assessment of the complete path and Route 3. SHALL trigger reassessment of the complete path and Route
Ensemble, if any change in hops is observed for a specific (and Ensemble if any change in Hops is observed for a specific (and
previously tested) flow. previously tested) flow.
4.1.2. Routing Class Identification 4.1.2. Routing Class Identification
There is an opportunity to apply the [RFC2330] notion of equal There is an opportunity to apply the notion from [RFC2330] of equal
treatment for a class of packets, "...very useful to know if a given treatment for a class of packets, "...very useful to know if a given
Internet component treats equally a class C of different types of Internet component treats equally a class C of different types of
packets", as it applies to Route measurements. The notion of class C packets", as it applies to Route measurements. The notion of class C
was examined further in [RFC8468] as it applied to load-balancing was examined further in [RFC8468] as it applied to load-balancing
flows over parallel paths, which is the case we develop here. flows over parallel paths, which is the case we develop here.
Knowledge of class C parameters (unrelated to address classes of the Knowledge of class C parameters (unrelated to address classes of the
past) on a path potentially reduces the number of flows required for past) on a path potentially reduces the number of flows required for
a given method to assess a Route Ensemble over time. a given method to assess a Route Ensemble over time.
First, recognize that each Member Route of a Route Ensemble will have First, recognize that each Member Route of a Route Ensemble will have
a corresponding class C. Class C can be discovered by testing with a corresponding class C. Class C can be discovered by testing with
multiple flows, all of which traverse the unique set of hops that multiple flows, all of which traverse the unique set of Hops that
comprise a specific Member Route. comprise a specific Member Route.
Second, recognize that the different classes depend primarily on the Second, recognize that the different classes depend primarily on the
hash functions used at each instance of ECMP load balancing on the hash functions used at each instance of ECMP load balancing on the
path. path.
Third, recognize the synergy with Temporal Composition methods Third, recognize the synergy with Temporal Composition methods
(described above), where evaluation intends to discover time-stable (described above), where evaluation intends to discover time-stable
portions of each Member Route, so that more emphasis can be placed on portions of each Member Route so that more emphasis can be placed on
ECMP portions that also determine class C. ECMP portions that also determine class C.
The methods to assess the various class C characteristics benefit The methods to assess the various class C characteristics benefit
from the following measurement capabilities: from the following measurement capabilities:
o flows designed to determine which n-tuple header fields are * flows designed to determine which n-tuple header fields are
considered by a given hash function and ECMP hop on the path, and considered by a given hash function and ECMP Hop on the path and
which are not. This operation immediately narrows the search which are not. This operation immediately narrows the search
space, where possible, and partially defines a class C. space, where possible, and partially defines a class C.
o a priori knowledge of the possible types of hash functions in use * a priori knowledge of the possible types of hash functions in use
also helps to design the flows for testing (major router vendors also helps to design the flows for testing (major router vendors
publish information about these hash functions, examples are here publish information about these hash functions; examples are in
[LOAD_BALANCE]. [LOAD_BALANCE]).
o ability to direct the emphasis of current measurements on ECMP * ability to direct the emphasis of current measurements on ECMP
portions of the path, based on recent past measurement results portions of the path, based on recent past measurement results
(the Routing Class of some portions of the path is essentially (the Routing Class of some portions of the path is essentially
"all packets"). "all packets").
4.1.3. Intermediate Observation Point Route Measurement 4.1.3. Intermediate Observation Point Route Measurement
There are many examples where passive monitoring of a flow at an There are many examples where passive monitoring of a flow at an
Observation Point within the network can detect unexpected Round Trip Observation Point within the network can detect unexpected Round-Trip
Delay or Delay Variation. But how can the cause of the anomalous Delay or Delay Variation. But how can the cause of the anomalous
delay be investigated further *from the Observation Point* possibly delay be investigated further *from the Observation Point* possibly
located at an intermediate point on the path? located at an intermediate point on the path?
In this case, knowledge that the flow of interest belongs to a In this case, knowledge that the flow of interest belongs to a
specific Routing Class C will enable measurement of the route where specific Routing Class C will enable measurement of the Route where
anomalous delay has been observed. Specifically, Round-Trip Delay anomalous delay has been observed. Specifically, Round-Trip Delay
assessment to each Hop on the path between the Observation Point and assessment to each Hop on the path between the Observation Point and
the Destination for the flow of interest may discover high or the Destination for the flow of interest may discover high or
variable delay on a specific link and Hop combination. variable delay on a specific link and Hop combination.
The determination of a Routing Class C which includes the flow of The determination of a Routing Class C that includes the flow of
interest is as described in the section above, aided by computation interest is as described in the section above, aided by computation
of the relevant hash function output as the target. of the relevant hash function output as the target.
4.2. Hybrid Methodologies 4.2. Hybrid Methodologies
The Hybrid Type I methods provide an alternative method for Route The Hybrid Type I methods provide an alternative for Route
Member assessment. As mentioned in the Scope section, assessment. The "Scope, Applicability, and Assumptions" section of
[I-D.ietf-ippm-ioam-data] provides a possible set of data fields that [RFC9197] provides one possible set of data fields that would support
would support route identification. Route identification.
In general, nodes in the measured domain would be equipped with In general, Nodes in the measured domain would be equipped with
specific abilities: specific abilities:
o Store the identity of nodes that a packet has visited in header * Store the identity of Nodes that a packet has visited in header
data fields, in the order the packet visited the nodes. data fields in the order the packet visited the Nodes.
o Support of a "Loopback" capability, where a copy of the packet is * Support of a "Loopback" capability where a copy of the packet is
returned to the encapsulating node, and the packet is processed returned to the encapsulating Node and the packet is processed
like any other IOAM packet on the return transfer. like any other IOAM packet on the return transfer.
In addition to node identity, nodes may also identify the ingress and In addition to Node identity, Nodes may also identify the ingress and
egress interfaces utilized by the tracing packet, the absolute time egress interfaces utilized by the tracing packet, the absolute time
when the packet was processed, and other generic data (as described when the packet was processed, and other generic data (as described
in section 4 of [I-D.ietf-ippm-ioam-data]). Interface identification in Section 3 of [RFC9197]). Interface identification isn't
isn't necessarily limited to IP, i.e. different links in a bundle necessarily limited to IP, i.e., different links in a bundle (Link
(LACP) could be identified. Equally well, links without explicit IP Aggregation Control Protocol (LACP)) could be identified. Equally
addresses can be identified (like with unnumbered interfaces in an well, links without explicit IP addresses can be identified (like
IGP deployment). with unnumbered interfaces in an IGP deployment).
Note that the Type-P packet specification for this method will likely Note that the Type-P packet specification for this method will likely
be a partial specification, because most of the packet fields are be a partial specification because most of the packet fields are
determined by the user traffic. The packet (encapsulation) header(s) determined by the user traffic. The packet encapsulation header or
added by the Hybrid method can certainly be specified in Type-P, in headers added by the hybrid method can certainly be specified in
unpopulated form. Type-P, in unpopulated form.
4.3. Combining Different Methods 4.3. Combining Different Methods
In principle, there are advantages if the entity conducting Route In principle, there are advantages if the entity conducting Route
measurements can utilize both forms of advanced methods (active and measurements can utilize both forms of advanced methods (active and
hybrid), and combine the results. For example, if there are Nodes hybrid) and combine the results. For example, if there are Nodes
involved in the path that qualify as Cooperating Nodes, but not as involved in the path that qualify as Cooperating Nodes but not as
Discoverable Nodes, then a more complete view of Hops on the path is Discoverable Nodes, then a more complete view of Hops on the path is
possible when a hybrid method (or interrogation protocol) is applied possible when a hybrid method (or interrogation protocol) is applied
and the results are combined with the active method results collected and the results are combined with the active method results collected
across all other domains. across all other domains.
In order to combine the results of active and hybrid/interrogation In order to combine the results of active and hybrid/interrogation
methods, the network Nodes that are part of a domain supporting an methods, the network Nodes that are part of a domain supporting an
interrogation protocol have the following attributes: interrogation protocol have the following attributes:
1. Nodes at the ingress to the domain SHOULD be both Discoverable 1. Nodes at the ingress to the domain SHOULD be both Discoverable
and Cooperating. and Cooperating.
2. Any Nodes within the domain that are both Discoverable and 2. Any Nodes within the domain that are both Discoverable and
Cooperating SHOULD reveal the same Node Identity in response to Cooperating SHOULD reveal the same Node identity in response to
both active and hybrid methods. both active and hybrid methods.
3. Nodes at the egress to the domain SHOULD be both Discoverable and 3. Nodes at the egress to the domain SHOULD be both Discoverable and
Cooperating, and SHOULD reveal the same Node Identity in response Cooperating and SHOULD reveal the same Node identity in response
to both active and hybrid methods. to both active and hybrid methods.
When Nodes follow these requirements, it becomes a simple matter to When Nodes follow these requirements, it becomes a simple matter to
match single domain measurements with the overlapping results from a match single-domain measurements with the overlapping results from a
multidomain measurement. multidomain measurement.
In practice, Internet users do not typically have the ability to In practice, Internet users do not typically have the ability to
utilize the OAM capabilities of networks that their packets traverse, utilize the Operations, Administrations, and Maintenance (OAM)
so the results from a remote domain supporting an interrogation capabilities of networks that their packets traverse, so the results
protocol would not normally be accessible. However, a network from a remote domain supporting an interrogation protocol would not
operator could combine interrogation results from their access domain normally be accessible. However, a network operator could combine
with other measurements revealing the path outside their domain. interrogation results from their access domain with other
measurements revealing the path outside their domain.
5. Background on Round-Trip Delay Measurement Goals 5. Background on Round-Trip Delay Measurement Goals
The aim of this method is to use packet probes to unveil the paths The aim of this method is to use packet probes to unveil the paths
between any two end-Nodes of the network. Moreover, information between any two End-Nodes of the network. Moreover, information
derived from RTD measurements might be meaningful to identify: derived from RTD measurements might be meaningful to identify:
1. Intercontinental submarine links 1. Intercontinental submarine links
2. Satellite communications 2. Satellite communications
3. Congestion 3. Congestion
4. Inter-domain paths 4. Inter-domain paths
This categorization is widely accepted in the literature and among This categorization is widely accepted in the literature and among
operators alike, and it can be trusted with empirical data and operators alike, and it can be trusted with empirical data and
several sources as ground of truth (e.g., [RTTSub] ) but it is an several sources as ground of truth (e.g., [RTTSub]), but it is an
inference measurement nonetheless [bdrmap][IDCong]. inference measurement nonetheless [bdrmap][IDCong].
The first two categories correspond to the physical distance The first two categories correspond to the physical distance
dependency on Round-Trip Delay (RTD), the next one binds RTD with dependency on RTD, the next one binds RTD with queuing delay on
queuing delay on routers, and the last one helps to identify routers, and the last one helps to identify different ASes using
different ASes using traceroutes. Due to the significant traceroutes. Due to the significant contribution of propagation
contribution of propagation delay in long-distance hops, RTD will be delay in long-distance Hops, RTD will be on the order of 100 ms on
on the order of 100ms on transatlantic hops, depending on the transatlantic Hops, depending on the geolocation of the vantage
geolocation of the vantage points. Moreover, RTD is typically higher points. Moreover, RTD is typically higher than 480 ms when two Hops
than 480ms when two hops are connected using geostationary satellite are connected using geostationary satellite technology (i.e., their
technology (i.e., their orbit is at 36000km). Detecting congestion orbit is at 36000 km). Detecting congestion with latency implies
with latency implies deeper mathematical understanding since network deeper mathematical understanding, since network traffic load is not
traffic load is not stationary. Nonetheless, as the first approach, stationary. Nonetheless, as the first approach, a link seems to be
a link seems to be congested if observing different/varying congested if observing different/varying statistical results after
statistical results after sending several traceroute probes (e.g., sending several traceroute probes (e.g., see [IDCong]). Finally, to
see [IDCong]). Finally, to recognize distinctive ASes in the same recognize distinctive ASes in the same traceroute path is challenging
traceroute path is challenging, because more data is needed, like AS because more data is needed, like AS relationships and Regional
relationships and RIR delegations among other (for more detail, Internet Registry (RIR) delegations among others (for more details,
please consult [bdrmap]). please consult [bdrmap]).
6. RTD Measurements Statistics 6. RTD Measurements Statistics
Several articles have shown that network traffic presents a self- Several articles have shown that network traffic presents a self-
similar nature [SSNT] [MLRM] which is accountable for filling the similar nature [SSNT] [MLRM] that is accountable for filling the
queues of the routers. Moreover, router queues are designed to queues of the routers. Moreover, router queues are designed to
handle traffic bursts, which is one of the most remarkable features handle traffic bursts, which is one of the most remarkable features
of self-similarity. Naturally, while queue length increases, the of self-similarity. Naturally, while queue length increases, the
delay to traverse the queue increases as well and leads to an delay to traverse the queue increases as well and leads to an
increase on RTD. Due to traffic bursts generating short-term increase on RTD. Due to traffic bursts generating short-term
overflow on buffers (spiky patterns), every RTD only depicts the overflow on buffers (spiky patterns), every RTD only depicts the
queueing status on the instant when that packet probe was in transit. queueing status on the instant when that packet probe was in transit.
For this reason, several RTD measurements during a time window could For this reason, several RTD measurements during a time window could
begin to describe the random behavior of latency. Loss must also be begin to describe the random behavior of latency. Loss must also be
accounted for in the methodology. accounted for in the methodology.
To understand the ongoing process, examining the quartiles provides a To understand the ongoing process, examining the quartiles provides a
non-parametric way of analysis. Quartiles are defined by five nonparametric way of analysis. Quartiles are defined by five values:
values: minimum RTD (m), RTD value of the 25% of the Empirical minimum RTD (m), RTD value of the 25% of the Empirical Cumulative
Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), Distribution Function (ECDF) (Q1), the median value (Q2), the RTD
the RTD value of the 75% of the ECDF (Q3) and the maximum RTD (M). value of the 75% of the ECDF (Q3), and the maximum RTD (M).
Congestion can be inferred when RTD measurements are spread apart, Congestion can be inferred when RTD measurements are spread apart;
and consequently, the Inter-Quartile Range (IQR), the distance consequently, the Interquartile Range (IQR), i.e., the distance
between Q3 and Q1, increases its value. between Q3 and Q1, increases its value.
This procedure requires the algorithm presented in [P2] to compute This procedure requires the algorithm presented in [P2] to compute
quartile values "on the fly". quartile values "on the fly".
This procedure allows us to update the quartiles value whenever a new This procedure allows us to update the quartile values whenever a new
measurement arrives, which is radically different from classic measurement arrives, which is radically different from classic
methods of computing quartiles because they need to use the whole methods of computing quartiles, because they need to use the whole
dataset to compute the values. This way of calculus provides savings dataset to compute the values. This way of calculus provides savings
in memory and computing time. in memory and computing time.
To sum up, the proposed measurement procedure consists of performing To sum up, the proposed measurement procedure consists of performing
traceroutes several times to obtain samples of the RTD in every hop traceroutes several times to obtain samples of the RTD in every Hop
from a path, during a time window (W), and compute the quartiles for from a path during a time window (W) and compute the quartiles for
every hop. This procedure could be done for a single Member Route every Hop. This procedure could be done for a single Member Route
flow, a non-exhaustive search with parameter E (defined below) set as flow, for a non-exhaustive search with parameter E (defined below)
False, or for every detected Route Ensemble flow (E=True). set to False, or for every detected Route Ensemble flow (E=True).
The identification of a specific Hop in traceroute is based on the IP The identification of a specific Hop in a traceroute is based on the
origin address of the returned ICMP Time Exceeded packet, and on the IP origin address of the returned ICMP Time Exceeded packet and on
distance identified by the value set in the TTL (or Hop Limit) field the distance identified by the value set in the TTL (or Hop Limit)
inserted by traceroute. As this specific Hop can be reached by field inserted by traceroute. As this specific Hop can be reached by
different paths, also the IP source and destination addresses of the different paths, the IP Source and Destination addresses of the
traceroute packet need to be recorded. Finally, different return traceroute packet also need to be recorded. Finally, different
paths are distinguished by evaluating the ICMP Time Exceeded TTL (or return paths are distinguished by evaluating the ICMP Time Exceeded
Hop Limit) of the reply message: if this TTL (or Hop Limit) is TTL (or Hop Limit) of the reply message; if this TTL (or Hop Limit)
constant for different paths containing the same Hop, the return is constant for different paths containing the same Hop, the return
paths have the same distance. Moreover, this distance can be paths have the same distance. Moreover, this distance can be
estimated considering that the TTL (or Hop Limit) value is normally estimated considering that the TTL (or Hop Limit) value is normally
initialized with values 64, 128, or 255. The 5-tuple (origin IP, initialized with values 64, 128, or 255. The 5-tuple (origin IP,
destination IP, reply IP, distance, response TTL or Hop Limit) destination IP, reply IP, distance, and response TTL or Hop Limit)
unequivocally identifies every measurement. unequivocally identifies every measurement.
This algorithm below runs in the origin of the traceroute. It This algorithm below runs in the origin of the traceroute. It
returns the Qs quartiles for every Hop and Alt (alternative paths returns the Qs quartiles for every Hop and Alt (alternative paths
because of balancing). Notice that the "Alt" parameter condenses the because of balancing). Notice that the "Alt" parameter condenses the
parameters of the 5-tuple (origin IP, destination IP, reply IP, parameters of the 5-tuple (origin IP, destination IP, reply IP,
distance, response TTL), i.e., one for each possible combination. distance, and response TTL), i.e., one for each possible combination.
================================================================ ================================================================
0 input: W (window time of the measurement) 0 input: W (window time of the measurement)
1 i_t (time between two measurements, set the i_t time 1 i_t (time between two measurements, set the i_t time
2 long enough to avoid incomplete results) 2 long enough to avoid incomplete results)
3 E (True: exhaustive, False: a single path) 3 E (True: exhaustive, False: a single path)
4 Dst (destination IP address) 4 Dst (destination IP address)
5 output: Qs (quartiles for every Hop and Alt) 5 output: Qs (quartiles for every Hop and Alt)
---------------------------------------------------------------- ----------------------------------------------------------------
6 T := start_timer(W) 6 T := start_timer(W)
skipping to change at page 20, line 26 skipping to change at line 919
9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E) 9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E)
10 | for each Hop and Alt in RTD do: 10 | for each Hop and Alt in RTD do:
11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt)) 11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt))
12 | done 12 | done
13 | wait until i_t timer is expired 13 | wait until i_t timer is expired
14 done 14 done
15 return (Qs) 15 return (Qs)
================================================================ ================================================================
During the time W, lines 6 and 7 assure that the measurement loop is During the time W, lines 6 and 7 assure that the measurement loop is
made. Line 8 and 13 set a timer for each cycle of measurements. A made. Lines 8 and 13 set a timer for each cycle of measurements. A
cycle comprises the traceroutes packets, considering every possible cycle comprises the traceroutes packets, considering every possible
Hop and the alternatives paths in the Alt variable (ensured in lines Hop and the alternatives paths in the Alt variable (ensured in lines
9-12). In line 9, the advance-traceroute could be either Paris- 9-12). In line 9, the advanced-traceroute could be either Paris-
traceroute or Scamper, which will use the "exhaustive" mode or traceroute or Scamper, which will use the "exhaustive" mode or
"tracelb" option if E is set True, respectively. The procedure "tracelb" option if E is set to True, respectively. The procedure
returns a list of tuples (m,Q1,Q2,Q3,M) for each intermediate hop, or returns a list of tuples (m, Q1, Q2, Q3, and M) for each intermediate
"Alt" in as a function of the 5-tuple, in the path towards the Dst. Hop, or "Alt" in as a function of the 5-tuple, in the path towards
Finally, lines 10 through 12 stores each measurement into the real- the Dst. Finally, lines 10 through 12 store each measurement into the
time quartiles computation. real-time quartiles computation.
Notice there are cases where the even having a unique hop at distance Notice there are cases where even having a unique Hop at distance h
h from the Src to Dst, the returning path could have several from the Src to Dst, the returning path could have several
possibilities, yielding in different total paths. In this situation, possibilities, yielding different total paths. In this situation,
the algorithm will return more "Alt" for this particular hop. the algorithm will return another "Alt" for this particular Hop.
7. Security Considerations 7. Security Considerations
The security considerations that apply to any active measurement of The security considerations that apply to any active measurement of
live paths are relevant here as well. See [RFC4656] and [RFC5357]. live paths are relevant here as well. See [RFC4656] and [RFC5357].
The active measurement process of "changing several fields to keep The active measurement process of changing several fields to keep the
the checksum of different packets identical" does not require special checksum of different packets identical does not require special
security considerations because it is part of synthetic traffic security considerations because it is part of synthetic traffic
generation, and is designed to have minimal to zero impact on network generation and is designed to have minimal to zero impact on network
processing (to process the packets for ECMP). processing (to process the packets for ECMP).
Some of the protocols used (e.g., ICMP) do not provide cryptographic Some of the protocols used (e.g., ICMP) do not provide cryptographic
protection for the requested/returned data, and there are risks of protection for the requested/returned data, and there are risks of
processing untrusted data in general, but these are limitations of processing untrusted data in general, but these are limitations of
the existing protocols where we are applying new methods. the existing protocols where we are applying new methods.
For applicable Hybrid methods, the security considerations For applicable hybrid methods, the security considerations in
in[I-D.ietf-ippm-ioam-data] apply. [RFC9197] apply.
When considering privacy of those involved in measurement or those When considering the privacy of those involved in measurement or
whose traffic is measured, the sensitive information available to those whose traffic is measured, the sensitive information available
potential observers is greatly reduced when using active techniques to potential observers is greatly reduced when using active
which are within this scope of work. Passive observations of user techniques that are within this scope of work. Passive observations
traffic for measurement purposes raise many privacy issues. We refer of user traffic for measurement purposes raise many privacy issues.
the reader to the privacy considerations described in the Large Scale We refer the reader to the privacy considerations described in the
Measurement of Broadband Performance (LMAP) Framework [RFC7594], Large-scale Measurement of Broadband Performance (LMAP) Framework
which covers active and passive techniques. [RFC7594], which covers active and passive techniques.
8. IANA Considerations 8. IANA Considerations
This memo makes no requests of IANA. We thank the good folks at IANA This document has no IANA actions.
for having checked this section anyway.
9. Acknowledgements
The original 3 authors (Ignacio, Al, Joachim) acknowledge Ruediger
Geib, for his penetrating comments on the initial draft, and his
initial text for the Appendix on MPLS. Carlos Pignataro challenged
the authors to consider a wider scope, and applied his substantial
expertise with many technologies and their measurement features in
his extensive comments. Frank Brockners also shared useful comments,
so did Footer Foote. We thank them all!
10. Appendix I MPLS Methods for Route Assessment
A Node assessing an MPLS path must be part of the MPLS domain where
the path is implemented. When this condition is met, RFC 8029
provides a powerful set of mechanisms to detect "correct operation of
the data plane, as well as a mechanism to verify the data plane
against the control plane" [RFC8029].
MPLS routing is based on the presence of a Forwarding Equivalence
Class (FEC) Stack in all visited Nodes. Selecting one of several
Equal Cost Multi Path (ECMP) is however based on information hidden
deeper in the stack. Late deployments may support a so called
"Entropy label" for this purpose. State of the art deployments base
their choice of an ECMP member interface on the complete MPLS label
stack and on IP addresses up to the complete 5 tuple IP header
information (see Section 2.4 of [RFC7325]). Load Sharing based on IP
information decouples this function from the actual MPLS routing
information. Thus, an MPLS traceroute is able to check how packets
with a contiguous number of ECMP relevant IP addresses (and an
identical MPLS label stack) are forwarded by a particular router.
The minimum number of equivalent MPLS paths traceable at a router
should be 32. Implementations supporting more paths are available.
The MPLS echo request and reply messages offering this feature must
support the Downstream Detailed Mapping TLV (was Downstream Mapping
initially, but the latter has been deprecated). The MPLS echo
response includes the incoming interface where a router received the
MPLS Echo request. The MPLS Echo reply further informs which of the
n addresses relevant for the load sharing decision results in a
particular next hop interface and contains the next hop's interface
address (if available). This ensures that the next hop will receive
a properly coded MPLS Echo request in the next step route of
assessment.
[RFC8403] explains how a central Path Monitoring System could be used
to detect arbitrary MPLS paths between any routers within a single
MPLS domain. The combination of MPLS forwarding, Segment Routing and
MPLS traceroute offers a simple architecture and a powerful mechanism
to detect and validate (segment routed) MPLS paths.
11. References
11.1. Normative References 9. References
[I-D.ietf-ippm-ioam-data] 9.1. Normative References
Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields
for In-situ OAM", draft-ietf-ippm-ioam-data-10 (work in
progress), July 2020.
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, DOI 10.17487/RFC0792, September 1981, RFC 792, DOI 10.17487/RFC0792, September 1981,
<https://www.rfc-editor.org/info/rfc792>. <https://www.rfc-editor.org/info/rfc792>.
[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, Communication Layers", STD 3, RFC 1122,
DOI 10.17487/RFC1122, October 1989, DOI 10.17487/RFC1122, October 1989,
<https://www.rfc-editor.org/info/rfc1122>. <https://www.rfc-editor.org/info/rfc1122>.
skipping to change at page 24, line 11 skipping to change at line 1037
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>. May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V.
Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for
the IP Performance Metrics (IPPM) Framework", RFC 8468, the IP Performance Metrics (IPPM) Framework", RFC 8468,
DOI 10.17487/RFC8468, November 2018, DOI 10.17487/RFC8468, November 2018,
<https://www.rfc-editor.org/info/rfc8468>. <https://www.rfc-editor.org/info/rfc8468>.
11.2. Informative References [RFC9197] Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi,
Ed., "Data Fields for In Situ Operations, Administration,
and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197,
May 2022, <https://www.rfc-editor.org/info/rfc9197>.
9.2. Informative References
[bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and [bdrmap] Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., and
KC. Claffy, "bdrmap: Inference of Borders Between IP KC. Claffy, "bdrmap: Inference of Borders Between IP
Networks", In Proceedings of the 2016 ACM on Internet Networks", Proceedings of the 2016 ACM on Internet
Measurement Conference, pp. 381-396. ACM, 2016. Measurement Conference, pp. 381-396,
DOI 10.1145/2987443.2987467, November 2016,
<https://doi.org/10.1145/2987443.2987467>.
[IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker, [IDCong] Luckie, M., Dhamdhere, A., Clark, D., and B. Huffaker,
"Challenges in inferring Internet interdomain "Challenges in Inferring Internet Interdomain Congestion",
congestion", In Proceedings of the 2014 Conference on Proceedings of the 2014 Conference on Internet Measurement
Internet Measurement Conference, pp. 15-22. ACM, 2014. Conference, pp. 15-22, DOI 10.1145/2663716.2663741,
November 2014, <https://doi.org/10.1145/2663716.2663741>.
[LOAD_BALANCE] [LOAD_BALANCE]
Sanguanpong, S., Pittayapitak, W., and K. Kasom Koht-Arsa, Sanguanpong, S., Pittayapitak, W., and K. Kasom Koht-Arsa,
"COMPARISON OF HASH STRATEGIES FOR FLOW-BASED LOAD "COMPARISON OF HASH STRATEGIES FOR FLOW-BASED LOAD
BALANCING", International Journal of Electronic Commerce BALANCING", International Journal of Electronic Commerce
Studies, Vol.6, No.2, pp.259-268. Studies, Vol.6, No.2, pp.259-268, DOI 10.7903/ijecs.1346,
http://dx.doi.org/10.7903/ijecs.1346, 2015. December 2015, <https://doi.org/10.7903/ijecs.1346>.
[MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring [MLB] Augustin, B., Friedman, T., and R. Teixeira, "Measuring
load-balanced paths in the Internet", Proceedings of the load-balanced paths in the internet", Proceedings of the
7th ACM SIGCOMM conference on Internet measurement, pp. 7th ACM SIGCOMM conference on Internet measurement, pp.
149-160. ACM, 2007., 2007. 149-160, DOI 10.1145/1298306.1298329, October 2007,
<https://doi.org/10.1145/1298306.1298329>.
[MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical [MLRM] Fontugne, R., Mazel, J., and K. Fukuda, "An empirical
mixture model for large-scale RTT measurements", 2015 mixture model for large-scale RTT measurements", 2015 IEEE
IEEE Conference on Computer Communications (INFOCOM), pp. Conference on Computer Communications (INFOCOM), pp.
2470-2478. IEEE, 2015., 2015. 2470-2478, DOI 10.1109/INFOCOM.2015.7218636, April 2015,
<https://doi.org/10.1109/INFOCOM.2015.7218636>.
[P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic [P2] Jain, R. and I. Chlamtac, "The P 2 algorithm for dynamic
calculation of quartiles and histograms without storing calculation of quartiles and histograms without storing
observations", Communications of the ACM 28.10 (1985): observations", Communications of the ACM 28.10 (1985):
1076-1085, 2015. 1076-1085, DOI 10.1145/4372.4378, October 1985,
<https://doi.org/10.1145/4372.4378>.
[PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F., [PT] Augustin, B., Cuvellier, X., Orgogozo, B., Viger, F.,
Friedman, T., Latapy, M., Magnien, C., and R. Teixeira, Friedman, T., Latapy, M., Magnien, C., and R. Teixeira,
"Avoiding traceroute anomalies with Paris traceroute", "Avoiding traceroute anomalies with Paris traceroute",
Proceedings of the 6th ACM SIGCOMM conference on Internet Proceedings of the 6th ACM SIGCOMM conference on Internet
measurement, pp. 153-158. ACM, 2006., 2006. measurement, pp. 153-158, DOI 10.1145/1177080.1177100,
October 2006, <https://doi.org/10.1145/1177080.1177100>.
[RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and
Multicast Next-Hop Selection", RFC 2991, Multicast Next-Hop Selection", RFC 2991,
DOI 10.17487/RFC2991, November 2000, DOI 10.17487/RFC2991, November 2000,
<https://www.rfc-editor.org/info/rfc2991>. <https://www.rfc-editor.org/info/rfc2991>.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, DOI 10.17487/RFC5357, October 2008, RFC 5357, DOI 10.17487/RFC5357, October 2008,
<https://www.rfc-editor.org/info/rfc5357>. <https://www.rfc-editor.org/info/rfc5357>.
skipping to change at page 26, line 6 skipping to change at line 1135
Measurement of Broadband Performance (LMAP)", RFC 7594, Measurement of Broadband Performance (LMAP)", RFC 7594,
DOI 10.17487/RFC7594, September 2015, DOI 10.17487/RFC7594, September 2015,
<https://www.rfc-editor.org/info/rfc7594>. <https://www.rfc-editor.org/info/rfc7594>.
[RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N.
Kumar, "A Scalable and Topology-Aware MPLS Data-Plane Kumar, "A Scalable and Topology-Aware MPLS Data-Plane
Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July
2018, <https://www.rfc-editor.org/info/rfc8403>. 2018, <https://www.rfc-editor.org/info/rfc8403>.
[RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of [RTTSub] Bischof, Z., Rula, J., and F. Bustamante, "In and out of
Cuba: Characterizing Cuba's connectivity", In Proceedings Cuba: Characterizing Cuba's Connectivity", Proceedings of
of the 2015 ACM Conference on Internet Measurement the 2015 ACM Conference on Internet Measurement
Conference, pp. 487-493. ACM, 2015. Conference, pp. 487-493, DOI 10.1145/2815675.2815718,
October 2015, <https://doi.org/10.1145/2815675.2815718>.
[SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible [SCAMPER] Matthew Luckie, M., "Scamper: a scalable and extensible
packet prober for active measurement of the packet prober for active measurement of the internet",
Internet", Proceedings of the 10th ACM SIGCOMM conference Proceedings of the 10th ACM SIGCOMM conference on Internet
on Internet measurement, pp. 239-245. ACM, 2010., 2010. measurement, pp. 239-245, DOI 10.1145/1879141.1879171,
November 2010, <https://doi.org/10.1145/1879141.1879171>.
[SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic [SSNT] Park, K. and W. Willinger, "Self-Similar Network Traffic
and Performance Evaluation (1st ed.)", John Wiley & Sons, and Performance Evaluation (1st ed.)",
Inc., New York, NY, USA, 2000. DOI 10.1002/047120644X, John Wiley & Sons, Inc., New
York, NY, USA, August 2000,
<https://doi.org/10.1002/047120644X>.
Appendix A. MPLS Methods for Route Assessment
A Node assessing an MPLS path must be part of the MPLS domain where
the path is implemented. When this condition is met, [RFC8029]
provides a powerful set of mechanisms to detect "correct operation of
the data plane, as well as a mechanism to verify the data plane
against the control plane".
MPLS routing is based on the presence of a Forwarding Equivalence
Class (FEC) Stack in all visited Nodes. Selecting one of several
Equal-Cost Multipaths (ECMPs) is, however, based on information
hidden deeper in the stack. Late deployments may support a so-called
"Entropy label" for this purpose. State-of-the-art deployments base
their choice of an ECMP member interface on the complete MPLS label
stack and on IP addresses up to the complete 5-tuple IP header
information (see Section 2.4 of [RFC7325]). Load sharing based on IP
information decouples this function from the actual MPLS routing
information. Thus, an MPLS traceroute is able to check how packets
with a contiguous number of ECMP-relevant IP addresses (and an
identical MPLS label stack) are forwarded by a particular router.
The minimum number of equivalent MPLS paths traceable at a router
should be 32. Implementations supporting more paths are available.
The MPLS echo request and reply messages offering this feature must
support the Downstream Detailed Mapping TLV (was Downstream Mapping
initially, but the latter has been deprecated). The MPLS echo
response includes the incoming interface where a router received the
MPLS echo request. The MPLS echo reply further informs which of the
n addresses relevant for the load-sharing decision results in a
particular next-hop interface and contains the next Hop's interface
address (if available). This ensures that the next Hop will receive
a properly coded MPLS echo request in the next step Route of
assessment.
[RFC8403] explains how a central Path Monitoring System could be used
to detect arbitrary MPLS paths between any routers within a single
MPLS domain. The combination of MPLS forwarding, Segment Routing,
and MPLS traceroute offers a simple architecture and a powerful
mechanism to detect and validate (segment-routed) MPLS paths.
Acknowledgements
The original three authors (Ignacio, Al, Joachim) acknowledge
Ruediger Geib for his penetrating comments on the initial document
and his initial text for the appendix on MPLS. Carlos Pignataro
challenged the authors to consider a wider scope and applied his
substantial expertise with many technologies and their measurement
features in his extensive comments. Frank Brockners also shared
useful comments and so did Footer Foote. We thank them all!
Authors' Addresses Authors' Addresses
J. Ignacio Alvarez-Hamelin J. Ignacio Alvarez-Hamelin
Universidad de Buenos Aires Universidad de Buenos Aires
Av. Paseo Colon 850 Av. Paseo Colón 850
Buenos Aires C1063ACV C1063ACV Buenos Aires
Argentina Argentina
Phone: +54 11 5285-0716 Phone: +54 11 5285-0716
Email: ihameli@cnet.fi.uba.ar Email: ihameli@cnet.fi.uba.ar
URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ URI: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown, NJ 07748 Middletown, NJ 07748
USA United States of America
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192
Email: acm@research.att.com Email: acm@research.att.com
Joachim Fabini Joachim Fabini
TU Wien TU Wien
Gusshausstrasse 25/E389 Gusshausstrasse 25/E389
Vienna 1040 1040 Vienna
Austria Austria
Phone: +43 1 58801 38813 Phone: +43 1 58801 38813
Fax: +43 1 58801 38898
Email: Joachim.Fabini@tuwien.ac.at Email: Joachim.Fabini@tuwien.ac.at
URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/ URI: http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/
Carlos Pignataro Carlos Pignataro
Cisco Systems, Inc. Cisco Systems, Inc.
7200-11 Kit Creek Road 7200-11 Kit Creek Road
Research Triangle Park, NC 27709 Research Triangle Park, NC 27709
USA United States of America
Email: cpignata@cisco.com Email: cpignata@cisco.com
Ruediger Geib Ruediger Geib
Deutsche Telekom Deutsche Telekom
Heinrich Hertz Str. 3-7 Heinrich Hertz Str. 3-7
Darmstadt 64295 64295 Darmstadt
Germany Germany
Phone: +49 6151 5812747 Phone: +49 6151 5812747
Email: Ruediger.Geib@telekom.de Email: Ruediger.Geib@telekom.de
 End of changes. 192 change blocks. 
539 lines changed or deleted 547 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/