<?xml version="1.0"encoding="US-ASCII"?>encoding="UTF-8"?> <!DOCTYPE rfcSYSTEM "rfc2629.dtd"> <?rfc toc="yes"?> <?rfc tocompact="yes"?> <?rfc tocdepth="3"?> <?rfc tocindent="yes"?> <?rfc symrefs="yes"?> <?rfc sortrefs="yes"?> <?rfc comments="yes"?> <?rfc inline="yes"?> <?rfc compact="yes"?> <?rfc subcompact="no"?>[ <!ENTITY nbsp " "> <!ENTITY zwsp "​"> <!ENTITY nbhy "‑"> <!ENTITY wj "⁠"> ]> <rfccategory="std"xmlns:xi="http://www.w3.org/2001/XInclude" docName="draft-ietf-ippm-route-10" number="9198" ipr="trust200902"updates="2330">updates="2330" obsoletes="" submissionType="IETF" category="std" consensus="true" xml:lang="en" tocInclude="true" tocDepth="3" symRefs="true" sortRefs="true" version="3"> <!-- xml2rfc v2v3 conversion 3.1.1 --> <front> <title abbrev="AURA Metrics & Methods">Advanced Unidirectional Route Assessment (AURA)</title> <seriesInfo name="RFC" value="9198"/> <author fullname="J. Ignacio Alvarez-Hamelin"initials="J.I."initials="J" surname="Alvarez-Hamelin"> <organization>Universidad de Buenos Aires</organization> <address> <postal> <street>Av. PaseoColónColón 850</street> <city>Buenos Aires</city> <code>C1063ACV</code> <country>Argentina</country> </postal> <phone>+54 11 5285-0716</phone> <email>ihameli@cnet.fi.uba.ar</email> <uri>http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/</uri> </address> </author> <author fullname="Al Morton" initials="A." surname="Morton"> <organization>AT&T Labs</organization> <address> <postal> <street>200 Laurel Avenue South</street> <city>Middletown</city> <region>NJ</region> <code>07748</code><country>USA</country><country>United States of America</country> </postal> <phone>+1 732 420 1571</phone><facsimile>+1 732 368 1192</facsimile><email>acm@research.att.com</email> <uri/> </address> </author> <author fullname="Joachim Fabini" initials="J." surname="Fabini"> <organization>TU Wien</organization> <address> <postal> <street>Gusshausstrasse 25/E389</street> <city>Vienna</city> <region/> <code>1040</code> <country>Austria</country> </postal> <phone>+43 1 58801 38813</phone><facsimile>+43 1 58801 38898</facsimile><email>Joachim.Fabini@tuwien.ac.at</email> <uri>http://www.tc.tuwien.ac.at/about-us/staff/joachim-fabini/</uri> </address> </author> <author fullname="Carlos Pignataro" initials="C." surname="Pignataro"> <organization>Cisco Systems, Inc.</organization> <address> <postal> <street>7200-11 Kit Creek Road</street> <city>Research Triangle Park</city> <region>NC</region> <code>27709</code><country>USA</country><country>United States of America</country> </postal> <phone/><facsimile/><email>cpignata@cisco.com</email> <uri/> </address> </author> <author fullname="Ruediger Geib" initials="R." surname="Geib"> <organization>Deutsche Telekom</organization> <address> <postal> <street>Heinrich Hertz Str. 3-7</street> <city>Darmstadt</city> <region/> <code>64295</code> <country>Germany</country> </postal> <phone>+49 6151 5812747</phone><facsimile/><email>Ruediger.Geib@telekom.de</email> <uri/> </address> </author> <dateday="13" month="August" year="2020"/>month="May" year="2022"/> <keyword>Performance</keyword> <keyword>Metrics</keyword> <keyword>IPPM</keyword> <keyword>path</keyword> <keyword>parallel paths</keyword> <abstract> <t>This memo introduces an advanced unidirectional route assessment (AURA) metric and associated measurementmethodology,methodology based on the IP Performance Metrics (IPPM)Framework RFC 2330.framework (RFC 2330). This memo updates RFC 2330 in the areas of path-related terminology and path description, primarily to include the possibility of parallel subpaths between a given Source and Destination pair, owing to the presence ofmulti-pathmultipath technologies.</t> </abstract> </front> <middle> <sectiontitle="Introduction">numbered="true" toc="default"> <name>Introduction</name> <t>The IETF IP Performance Metrics (IPPM)working groupWorking Group first created a framework for metric development in <xreftarget="RFC2330"/>.target="RFC2330" format="default"/>. This framework has stood the test of time and enabled development of many fundamental metrics. It has been updated in the area of metric composition <xreftarget="RFC5835"/>,target="RFC5835" format="default"/> and in several areas related to active stream measurement of modern networks with reactive properties <xreftarget="RFC7312"/>.</t>target="RFC7312" format="default"/>.</t> <t>The<xref target="RFC2330"/>framework in <xref target="RFC2330" format="default"/> motivated the development of "performance and reliability metrics for paths through theInternet," and Section 5 ofInternet"; <xreftarget="RFC2330"/>target="RFC2330" section="5" sectionFormat="of"/> defines terms that support description of a path under test. However, metrics for assessment of paths and related performance aspects had not been attempted in IPPM when the<xref target="RFC2330"/>framework in <xref target="RFC2330" format="default"/> was written.</t> <t>This memo takes up therouteRoute measurement challenge and specifies a newrouteRoute metric, two practical frameworks for methods of measurement (using either active or hybrid active-passive methods <xreftarget="RFC7799"/>),target="RFC7799" format="default"/>), and Round-Trip Delay and link information discovery using the results of measurements. AllrouteRoute measurements are limited by the willingness ofhostsHosts along the path to be discovered, to cooperate with the methods used, or to recognize that the measurement operation is taking place (such as when tunnels are present).</t> <sectiontitle="Issuesnumbered="true" toc="default"> <name>Issues with Earlier Work to Define a RouteMetric"> <t>Section 7 of <xref target="RFC2330"/> presentedMetric</name> <t><xref target="RFC2330" section="7" sectionFormat="of"/> presents a simple example of a"route""Route" metric along with several other examples. The example is reproduced below (where the reference is toSection 5 of<xreftarget="RFC2330"/>):</t> <t>"route: Thetarget="RFC2330" section="5" sectionFormat="of"/>):</t> <blockquote> <dl newline="false" spacing="normal"> <dt>route:</dt> <dd>The path, as defined in Section5,<xref target="RFC2330" section="5" sectionFormat="bare"/>, from A to B at a giventime."</t>time.</dd> </dl> </blockquote> <t>This example provides a starting point to develop a more complete definition ofroute.Route. Areas needing clarification include:</t><t><list style="hanging"> <t hangText="Time:">In<dl newline="false" spacing="normal"> <dt>Time:</dt> <dd>In practice, therouteRoute will be assessed over a timeinterval,interval because active path detection methods likeParis TracerouteParis-traceroute <xreftarget="PT"/>target="PT" format="default"/> rely onhop limitsHop Limits for their operation and cannot accomplish discovery of allhostsHosts using a singlepacket.</t> <t hangText="Type-P:">Thepacket.</dd> <dt>Type-P:</dt> <dd>The legacyrouteRoute definition lacks the option to cater for packet-dependent routing. In this memo, we assess therouteRoute for a specific packet ofType-P,Type-P and reflect this in the metric definition. The methods of measurement determine the specific Type-Pused.</t> <t hangText="Parallel Paths:">Parallelused.</dd> <dt>Parallel Paths:</dt> <dd>Parallel paths are a reality of the Internet and a strength of advancedrouteRoute assessment methods, so the metric must acknowledge this possibility. Use ofEqual Cost Multi-PathEqual-Cost Multipath (ECMP) andUnequal Cost Multi-PathUnequal-Cost Multipath (UCMP) technologies are common sources of parallelsubpaths.</t> <t hangText="Cloud Subpath:">Maysubpaths.</dd> <dt>Cloud Subpath:</dt> <dd>Cloud subpaths may containhostsHosts that do not decrementhop limit,the Hop Limit but may have two or more exchange links connecting "discoverable"hostsHosts or routers. Parallel subpaths contained within clouds cannot be discovered. The assessment methods only discoverhostsHosts or routers on the path that decrementhop limit,Hop Limit or cooperate with interrogation protocols. The presence of tunnels and nested tunnels further complicate assessment by hidinghops.</t> <t hangText="Hop:">Although the <xref target="RFC2330"/>Hops.</dd> <dt>Hop:</dt> <dd>The definition ofa hopHop in <xref target="RFC2330" format="default"/> was alink-host pair,link-Host pair. However, onlyhostsHosts thatarewere discoverableor have the capability to cooperateand cooperated with interrogation protocolswhere(where link information may beexposed.</t> </list>The refined definition of Route metrics begins in the sectionsexposed) provided both link and Host information.</dd> </dl> <t>Note thatfollow.</t>the actual definitions appear in <xref target="terms"/>.</t> </section> <sectiontitle="Requirements Language">numbered="true" toc="default"> <name>Requirements Language</name> <t>The key words"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY","<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>", "<bcp14>MAY</bcp14>", and"OPTIONAL""<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as described in BCP14<xref target="RFC2119"/>14 <xreftarget="RFC8174"/>target="RFC2119" format="default"/> <xref target="RFC8174" format="default"/> when, and only when, they appear in all capitals, as shown here.</t> </section> </section> <section anchor="Scope"title="Scope">numbered="true" toc="default"> <name>Scope</name> <t>The purpose of this memo is to add newrouteRoute metrics and methods of measurement to the existing set of IPPM metrics.</t> <t>The scope is to definerouteRoute metrics that can identify the path taken by a packet or a flow traversing the Internet between twohosts.Hosts. Although primarily intended forhostsHosts communicating on the Internet, the definitions and metrics are constructed to be applicable to other network domains, if desired. The methods of measurement to assess the path may not be able to discover allhostsHosts comprising the path, but such omissions are often deterministic and explainable sources of error.</t> <t>This memo also specifies a framework for active methods of measurementwhichthat uses the techniques described in <xreftarget="PT"/>,target="PT" format="default"/> as well as a framework for hybrid active-passive methods ofmeasurementmeasurement, such as the Hybrid Type I method <xreftarget="RFC7799"/>target="RFC7799" format="default"/> described in <xreftarget="I-D.ietf-ippm-ioam-data"/>.target="RFC9197" format="default"/>. Methods using <xreftarget="I-D.ietf-ippm-ioam-data"/>target="RFC9197" format="default"/> are intended only for single administrative domains that provide a protocol for explicit interrogation ofnodesNodes on a path. Combinations of active methods and hybrid active-passive methods are alsoin-scope.</t>in scope.</t> <t>Further, this memo provides additional analysis of theround-trip delayRound-Trip Delay measurements made possible by themethods,methods in an effort to discover more details about the path, such as the link technology in use.</t> <t>This memo updatesSection 5 of<xreftarget="RFC2330"/>target="RFC2330" section="5" sectionFormat="of"/> in the areas of path-related terminology and path description, primarily to include the possibility of parallel subpaths between a given Source and Destination address pair (possibly resulting fromEqual Cost Multi-Path (ECMP)ECMP andUnequal Cost Multi-Path (UCMP)UCMP technologies).</t> <t>There are several simple non-goals of this memo. There is no attempt to assess the reverse path from anyhostHost on the path to thehostHost attempting the path measurement. The reverse path contribution to delay will be that experienced by ICMP packets (in activemethods),methods) and may be different from delays experienced by UDP or TCP packets. Also, theround trip delayRound-Trip Delay will include an unknown contribution of processing time at thehostHost that generates the ICMP response. Therefore, the ICMP-based active methods are not supposed to yield accurate, reproducible estimations of the Round-Trip Delay that UDP or TCP packets will experience.</t> </section> <sectiontitle="Routenumbered="true" toc="default"> <name>Route MetricSpecifications">Specifications</name> <t>This section sets requirements for the components of theRoute Metric.</t>route metric.</t> <sectiontitle="Termsnumbered="true" toc="default" anchor="terms"> <name>Terms andDefinitions">Definitions</name> <t/><t><list style="hanging"> <t hangText="Host">A<dl newline="true" spacing="normal"> <dt>Host</dt> <dd>A Hostas(as defined in <xreftarget="RFC2330"/> (atarget="RFC2330" format="default"/>) is a computer capable of IP communication,includes routers), a.k.a.including routers (aka an RFC 2330Host.</t> <t hangText="Node ">AHost).</dd> <dt>Node </dt> <dd>A Node is any network function on the path capable of IP-layer Communication,includesincluding RFC 2330Hosts.</t> <t hangText="Node Identity">TheHosts.</dd> <dt>Node Identity</dt> <dd>The Node identity is the unique address for Nodes communicating within the network domain. For Nodes communicating on the Internet with IP, it is the globally routable IP addresswhichthat the Node uses when communicating with other Nodes under normal or error conditions. The NodeIdentityidentity revealed (and its connection to a NodeNamename through reverse DNS) determines whether interfaces to parallel links can be associated with a singleNode,Node or appear to identify uniqueNodes.</t> <t hangText="Discoverable Node">NodesNodes.</dd> <dt>Discoverable Node</dt> <dd>Discoverable Nodes are Nodes that convey their NodeIdentityidentity according to the requirements of their network domain, such as when error conditions are detected by that Node. For Nodes communicating with IP packets, compliance withSection 3.2.2.4 of<xreftarget="RFC1122"/>target="RFC1122" section="3.2.2.4" sectionFormat="of"/>, when discarding a packet due to TTL or Hop Limit Exceeded condition,MUST<bcp14>MUST</bcp14> result in sending the corresponding Time Exceeded message (containing a form of Node identity) to the source. This requirement is also consistent withsection 5.3.1 of<xreftarget="RFC1812"/> for routers.</t> <t hangText="Cooperating Node">Nodestarget="RFC1812" sectionFormat="of" section="5.3.1"/> for routers.</dd> <dt>Cooperating Node</dt> <dd>Cooperating Nodes are Nodes that respond to direct queries for their Node identity as part of a previouslyagreed andestablished and agreed upon interrogation protocol. NodesSHOULD<bcp14>SHOULD</bcp14> also provide information such as arrival/departure interface identification, arrival timestamp, and any relevant information about the Node or specific linkwhichthat delivered the query to theNode.</t> <t hangText="Hop specification">ANode.</dd> <dt>Hop specification</dt> <dd>A Hop specificationMUST<bcp14>MUST</bcp14> contain a NodeIdentity,identity andMAY<bcp14>MAY</bcp14> contain arrival and/or departure interface identification,round trip delay,Round-Trip Delay, and an arrivaltimestamp.</t> <t hangText="Routing Class">A routetimestamp.</dd> <dt>Routing Class</dt> <dd>Routing Class is a Route that treatsequallya class of different types of packets, designated "C" (unrelated to address classes of the past) equally (<xref target="RFC2330" format="default"/> <xreftarget="RFC2330"/> <xref target="RFC8468"/>.target="RFC8468" format="default"/>). Knowledge of such a class allows any one of the types of packets within that class to be used for subsequent measurement of theroute.Route. The designator "class C" is used for historicalreasons,reasons; see <xreftarget="RFC2330"/>.</t> </list></t>target="RFC2330" format="default"/>.</dd> </dl> </section> <sectiontitle="Formal Name">numbered="true" toc="default"> <name>Formal Name</name> <t>The formal name of the metric is:</t> <t> Type-P-Route-Ensemble-Method-Variant </t> <t>abbreviated as Route Ensemble.</t> <t>Note that Type-P depends heavily on the chosen method and variant.</t> </section> <sectiontitle="Parameters">numbered="true" toc="default"> <name>Parameters</name> <t>This section lists theREQUIRED<bcp14>REQUIRED</bcp14> input factors to define and measure a Route metric, as specified in thismemo.<list style="symbols"> <t>Src,memo.</t> <dl spacing="normal"> <dt>Src:</dt> <dd> the address of a Node (such as the globally routable IPaddress).</t> <t>Dst,address).</dd> <dt>Dst:</dt> <dd> the address of a Node (such as the globally routable IPaddress).</t> <t>i,address).</dd> <dt>i:</dt> <dd> the limit on the number of Hops a specific packet may visit as it traverses from the Node at Src to the Node at Dst (such as the TTL or HopLimit).</t> <t>MaxHops,Limit).</dd> <dt>MaxHops:</dt> <dd> the maximum value of iused, (i=1,2,3,...MaxHops).</t> <t>T0, aused (i=1,2,3,...MaxHops).</dd> <dt>T0:</dt> <dd>a time (start of measurementinterval)</t> <t>Tf, ainterval).</dd> <dt>Tf:</dt> <dd>a time (end of measurementinterval)</t> <t>MP(address),interval).</dd> <dt>MP(address):</dt> <dd>the Measurement Point at address, such as Src or Dst, usually at the samenodeNode stack layer as"address".</t> <t>T,"address".</dd> <dt>T:</dt> <dd> the Node time of a packet as measured at MP(Src), meaning Measurement Point at theSource.</t> <t>Ta,Source.</dd> <dt>Ta:</dt> <dd> the Node time of a reply packet's*arrival*<strong>arrival</strong> as measured at MP(Src), assigned to packets that arrive within a "reasonable" time (see parameterbelow).</t> <t>Tmax,below).</dd> <dt>Tmax:</dt> <dd> a maximum waiting time for reply packets to return to the source, set sufficiently long to disambiguate packets with long delays from packets that are discarded (lost), such that the distribution of Round-Trip Delay is nottruncated.</t> <t>F,truncated.</dd> <dt>F:</dt> <dd> the number of different flows simulated by the method andvariant.</t> <t>flow,variant.</dd> <dt>flow:</dt> <dd> the stream of packets with the same n-tuple of designated header fields that (when held constant) result in identical treatment in amulti-pathmultipath decision (such as the decision taken in load balancing). Note: The IPv6 flow labelMAY<bcp14>MAY</bcp14> be included in the flow definition if the MP(Src) is a TunnelEnd PointEndpoint (TEP) complying with the guidelines in <xreftarget="RFC6438"/> guidelines.</t> <t>Type-P,target="RFC6438" format="default"/>.</dd> <dt>Type-P:</dt> <dd> the complete description of the packets for which this assessment applies (including the flow-definingfields).</t> </list></t>fields).</dd> </dl> </section> <sectiontitle="Metric Definitions">numbered="true" toc="default" anchor="Metric"> <name>Metric Definitions</name> <t>This section defines theREQUIRED<bcp14>REQUIRED</bcp14> measurement components of the Route metrics (unless otherwise indicated):</t><t>M,<dl spacing="normal"> <dt>M:</dt> <dd> the total number of packets sent between T0 andTf.</t> <t>N,Tf.</dd> <dt>N:</dt> <dd> the smallest value of i needed for a packet to be received at Dst (sent between T0 andTf).</t> <t>Nmax,Tf).</dd> <dt>Nmax:</dt> <dd> the largest value of i needed for a packet to be received at Dst (sent between T0 and Tf). Nmax may be equal toN.</t> <t>NextN.</dd> </dl> <t>Next, define a*singleton* definition<strong>singleton</strong> for a Node on thepath,path with sufficient indexes to identify all Nodes identified in a measurement interval (where*singleton*<strong>singleton</strong> is part of the IPPM Framework <xreftarget="RFC2330"/>).</t> <t>Atarget="RFC2330" format="default"/>).</t> <dl spacing="normal"> <dt>singleton:</dt> <dd>A HopSpecification,specification, designated h(i,j), the IP address and/or identity of Discoverable Nodes (or Cooperating Nodes) that are ihopsHops away from the Node with address = Src and part of Route j during the measurementinterval,interval T0 to Tf. As defined here, a Hop singleton measurementMUST<bcp14>MUST</bcp14> contain a NodeIdentity,identity, hid(i,j), andMAY<bcp14>MAY</bcp14> contain one or more of the followingattributes:<list style="symbols"> <t>a(i,j)attributes:</dd> </dl> <ul spacing="normal"> <li>a(i,j) Arrival Interface ID (e.g., when <xreftarget="RFC5837"/>target="RFC5837" format="default"/> issupported)</t> <t>d(i,j)supported)</li> <li>d(i,j) Departure Interface ID (e.g., when <xreftarget="RFC5837"/>target="RFC5837" format="default"/> issupported)</t> <t>t(i,j) Arrival Timestamp,supported)</li> <li>t(i,j) arrival timestamp, where t(i,j) is ideally supplied by theHop. (NoteHop (note that t(i,j) might be approximated from the sending time of the packet that revealed the Hop, e.g., when theround tripround-trip response time is available and divided by2.)</t> <t>Measurements2)</li> <li>Measurements of Round-Trip Delay (for each packet that reveals the same NodeIdentityidentity and flow attributes, then this attribute iscomputed,computed; see nextsection)</t> </list></t>section)</li> </ul> <t>NodeIdentitiesidentities and related information can be ordered by their distance from the Node with address Src in Hops h(i,j). Based on this, two forms of Routes are distinguished:</t> <t>A Route Ensemble is defined as the combination of allroutesRoutes traversed by different flows from the Node at Src address to the Node at Dst address. A single Route traversed by a single flow (determined by an unambiguous tuple of addresses Src andDst,Dst and other identical flow criteria) is a member of the Route Ensemble and called a Member Route.</t> <t>Using h(i,j) and components andparameters,parameters further define:</t> <t>When considering the set of Hops in the context of a single flow, a Member Route j is an ordered list {h(1,j), ... h(Nj, j)} where h(i-1, j) and h(i, j) are1 hopone Hop away from each other and Nj satisfying h(Nj,j)=Dst is the minimum count of Hops needed by the packet onMembermember Route j to reach Dst. Member Routes must be unique. The uniqueness property requires that any two MemberroutesRoutes, j andkk, that are part of the same Route Ensemble differ either in terms of minimumhopHop count Nj and Nk to reach the destinationDst,Dst or, in the case of identicalhopHop count Nj=Nk, they have at least one distinct Hop: h(i,j) != h(i,k) for at least one i (i=1..Nj).</t> <t>All the optional information collected to describe a Member Route, such as the arrival interface, departure interface, andRound TripRound-Trip Delay at each Hop, turns each list item into a rich structure. There may be information on the links between Hops,possiblypossible information on the routing (arrival interface and departure interface), an estimate of distance between Hops based on Round-Trip Delay measurements and calculations, and atime stamptimestamp indicating when all these additional details werevalid.</t>measured.</t> <t>The Route Ensemble from Src to Dst, during the measurement interval T0 to Tf, is the aggregate of all m distinct Member Routes discovered between the two Nodes with Src and Dst addresses. More formally, with the Node having address Src omitted:</t><t><figure> <artwork><![CDATA[Route<sourcecode name=""><![CDATA[ Route Ensemble = { {h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, {h(1,2), h(2,2), h(3,2),..., h(N2,2)=Dst}, ... {h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} }]]></artwork> </figure>where]]></sourcecode> <t>where the following conditions apply: i <= Nj <= Nmax (j=1..m)</t> <t>Note that some h(i,j) may be empty (null) in the case that systems do not reply (notdiscoverable,discoverable or not cooperating).</t> <t>h(i-1,j) and h(i,j) are the Hops on the same Member Route onehopHop away from each other.</t> <t>Hop h(i,j) may be identical with h(k,l) for i!=k andj!=l ;j!=l, which means there may be portions shared among different Member Routes (parts of Member Routes may overlap).</t> </section> <sectiontitle="Relatednumbered="true" toc="default"> <name>Related Round-Trip Delay and LossDefinitions">Definitions</name> <t>RTD(i,j,T) is defined as a singleton of the <xreftarget="RFC2681"/>target="RFC2681" format="default"/> Round-Trip Delay between the Node with address = Src and the Node at Hop h(i,j) at time T.</t> <t>RTL(i,j,T) is defined as a singleton of the <xreftarget="RFC6673"/> Round-triptarget="RFC6673" format="default"/> Round-Trip Loss between the Node with address = Src and the Node at Hop h(i,j) at time T.</t> </section> <sectiontitle="Discussion">numbered="true" toc="default"> <name>Discussion</name> <t>Depending on the way that the NodeIdentityidentity is revealed, it may be difficult to determine parallel subpaths between the same pair of Nodes(i.e.(i.e., multiple parallel links). It is easier to detect parallel subpaths involving differentNodes.<list style="symbols"> <t>IfNodes.</t> <ul spacing="normal"> <li>If a pair of discovered Nodes identify two different addresses (IP or not), then they will appear to be different Nodes. See itembelow.</t> <t>Ifbelow.</li> <li>If a pair of discovered Nodes identify two different IPaddresses,addresses and the IP addresses resolve to the same Node name (in the DNS), then they will appear to be the sameNodes.</t> <t>IfNode.</li> <li>If a discovered Node always replies using the same network address, regardless of the interface a packet arrives on, then multiple parallel links cannot be detected in that network domain. This condition may apply to traceroute-stylemethods,methods but may not apply to other hybrid methods based onIn-situIn situ Operations, Administration, and Maintenance (IOAM). For example, if the<xref target="RFC5837"/>ICMP extension mechanism described in <xref target="RFC5837" format="default"/> is implemented, then parallel links can be detected with the discovery traceroute-style methods.</t> <t>If</li> <li>If parallel links between routers are aggregated below the IP layer,thenthen, fromNodethe Node's point of view, all these links share the same pair of IP addresses. The existence of these parallel links can't be detected at the IP layer. This applies to other network domains with layers belowthem,them as well. This condition may apply to traceroute-stylemethods,methods but may not apply to other hybrid methods based onIOAM.</t> </list></t>IOAM.</li> </ul> <t>When arouteRoute assessment employs IP packets (for example), the reality of flow assignment to parallel subpaths involves layers above IP. Thus, the measured Route Ensemble is applicable to IP and higher layers (as described in the methodology's packet of Type-P and flow parameters).</t> </section> <sectiontitle="Reportingnumbered="true" toc="default"> <name>Reporting theMetric">Metric</name> <t>An Information Model and an XML Data Model for Storing Traceroute Measurements is available in <xreftarget="RFC5388"/>.target="RFC5388" format="default"/>. The measured information at eachhopHop includes four pieces of information: a one-dimensionalhopHop index, Node symbolic address, Node IP address, and RTD for each response.</t> <t>The description of Hop information that may be collected according to this memo covers more dimensions, as defined inSection 3.4 above.<xref target="Metric"/>. For example, the Hop index is two-dimensional to capture the complexity of a Route Ensemble, and it contains corresponding Node identities at a minimum. The models need to be expanded to include thesefeatures,features as well as Arrival Interface ID, Departure Interface ID, andArrival Timestamp,arrival timestamp, when available. The original sending Timestamp from the Src Node anchors a particular measurement in time.</t> </section> </section> <section anchor="Methodologies"title="Routenumbered="true" toc="default"> <name>Route AssessmentMethodologies">Methodologies</name> <t>There are two classes of methods described in this section, active methods relying on the reaction to TTL or Hop Limit Exceeded condition to discover Nodes on apath,path andHybridhybrid active-passive methods that involve direct interrogation ofcooperatingCooperating Nodes (usually within a single domain). Description of these methods follow.</t> <sectiontitle="Active Methodologies">numbered="true" toc="default"> <name>Active Methodologies</name> <t>This section describes the method employed by currentopen sourceopen-source tools, thereby providing a practical framework for further advanced techniques to be included as method variants. This method is applicable for use across multiple administrative domains.</t> <t>Internet routing is complex because it depends on the policies of thousands of Autonomous Systems(AS).(ASes). Most routers perform load balancing on flows using a form ofEqual Cost Multiple Path (ECMP).ECMP. <xreftarget="RFC2991"/>target="RFC2991" format="default"/> describes a number of flow-based or hashed approaches (e.g., Modulo-N Hash, Hash-Threshold, and Highest Random Weight(HRW)),(HRW)) and makes some good suggestions. Flow-based ECMP avoids increased packetdelay variationDelay Variation and possibly overwhelming levels of packet reordering in flows.</t> <t>A few routers still divide the workload through packet-based techniques, such as a round-robin scheme to distribute every new outgoing packet to multiple links, as explained in <xreftarget="RFC2991"/>.target="RFC2991" format="default"/>. The methods described in this section assume flow-based ECMP.</t> <t>Taking into account that Internet protocol was designed under the“end-to-end”"end-to-end" principle, the IP payload and its header do not provide any information about theroutesRoutes or path necessary to reach some destination. For this reason, the populartool traceroutetool, traceroute, was developed to gather the IP addresses of eachhopHop along a path usingtheICMPprotocol<xreftarget="RFC0792"/>.target="RFC0792" format="default"/>. Traceroute also measures RTD from eachhop.Hop. However, the growing complexity of the Internet makes it more challenging to develop an accurate traceroute implementation. For instance, the early traceroute tools would be inaccurate in the current network, mainly because they were not designed to retain a flow state. However, evolved traceroute tools, such as Paris-traceroute (<xref target="PT" format="default"/> <xreftarget="PT"/> <xref target="MLB"/>target="MLB" format="default"/>) and Scamper<xref target="SCAMPER"/>,(<xref target="SCAMPER" format="default"/>), expect to encounter ECMP and achieve more accurate results when they do, where Scamper ensures traceroute packets will follow the same path in 98% ofcases<xref target="SCAMPER"/>.</t>cases (<xref target="SCAMPER" format="default"/>).</t> <t>Today's traceroute tools send Type-P of packets, which are either ICMP, UDP, or TCP. UDP and TCP are used when a particular characteristic needs to be verified, such as filtering or traffic shaping on specific ports (i.e., services). UDP and TCP traceroute are also used when ICMP responses are not received. <xreftarget="SCAMPER"/>target="SCAMPER" format="default"/> supports IPv6 traceroute measurements, keeping theFlowLabelFlow Label constant in all packets.</t> <t>Paris-traceroute allows its users to measure the RTD to every Node of the path for a particular flow. Furthermore, either Paris-traceroute or Scamper is capable of unveiling the many available paths between a source and destination (which are visible to active methods). This task is accomplished by repeating complete traceroute measurements with different flow parameters for each measurement; Paris-traceroute provides“exhaustive” modean "exhaustive" mode, whilescamperScamper provides“tracelb” (stands"tracelb" (which stands fortraceroute"traceroute loadbalance). The Frameworkbalance"). <xref target="RFC2330" format="default">"Framework for IP PerformanceMetrics (IPPM) (<xref target="RFC2330"/>Metrics"</xref>, updatedby<xref target="RFC7312"> </xref>)by <xref target="RFC7312" format="default"> </xref>, has the flexibility to require that the Round-Trip Delay measurement <xreftarget="RFC2681"/>target="RFC2681" format="default"/> uses packets with the constraints to assure that all packets in a single measurement appear as the same flow. This flexibility covers ICMP, UDP, and TCP. The accompanying methodology of <xreftarget="RFC2681"/>target="RFC2681" format="default"/> needs to be expanded to report the sequentialhopHop identifiers along with RTD measurements, but no new metric definition is needed.</t> <t>The advancedrouteRoute assessment methods used in Paris-traceroute <xreftarget="PT"/>target="PT" format="default"/> keep the critical fields constant for every packet to maintain the appearance of the same flow. When considering IPv6 headers, it is necessary to ensure that the IPsourceSource anddestinationDestination addresses andthe FlowLabelFlow Label are constant (but note that many routers ignore theFlowLabelFlow Label field at thistime),time); see <xreftarget="RFC6437"/>.target="RFC6437" format="default"/>. Use of IPv6 Extension Headers may add criticalfields,fields andSHOULD<bcp14>SHOULD</bcp14> be avoided. In IPv4, certain fields of the IP header and the firstfour4 bytes of the IP payload should remain constant in a flow. In the IPv4 header, the IPsourceSource anddestinationDestination addresses, protocol number, and Diffserv fields identify flows. The firstfour4 payload bytes include the UDP and TCPports,ports and the ICMP type, code, and checksum fields.</t> <t>Maintaining a constant ICMP checksum in IPv4 is most challenging, as the ICMP sequence number or identifier fields will usually change for different probes of the same path. Probes should use arbitrary bytes in the ICMP data field to offset changes to the sequence number and identifier, thus keeping the checksum constant.</t> <t>Finally, it is also essential torouteRoute the resulting ICMP Time Exceeded messages along a consistent path. In IPv6, the fields above are sufficient. In IPv4, the ICMP Time Exceeded message will contain the IP header and the firsteight8 bytes of the IP payload, both of whichaffectsaffect its ICMPchecksum.checksum calculation. The TCP sequence number, UDPLength,length, and UDP checksum will affect thisvalue,value and should remain constant.</t> <t>Formally, to maintain the same flow in the measurements to a particularhop,Hop, the Type-P-Route-Ensemble-Method-Variant packets shouldbe<xref target="PT"/>:</t> <t><list style="symbols"> <t>TCP case:have the following attributes (see <xref target="PT" format="default"/>):</t> <dl spacing="normal"> <dt>TCP case:</dt> <dd> For IPv4, the fields Src, Dst, port-Src, port_Dst, sequence number, and DiffservField SHOULD<bcp14>SHOULD</bcp14> be the same. For IPv6, thefield FlowLabel, Srcfields Flow Label, Src, and DstSHOULD<bcp14>SHOULD</bcp14> be thesame.</t> <t>UDP case:same.</dd> <dt>UDP case:</dt> <dd> For IPv4, the fields Src, Dst, port-Src, port-Dst, and Diffserv should be the same, and theUDP-checksum SHOULDUDP checksum <bcp14>SHOULD</bcp14> change to keep the IP checksum of the ICMPtime exceededTime Exceeded reply constant. Then, the data length should be fixed, and the data field is used to make it so (consider that ICMP checksum uses its data field, which contains the original IP header plus 8 bytes of UDP, where TTL, IP identification, IP checksum, and UDP checksum changes). For IPv6, the fieldFlowLabel,Flow Label and Source and Destination addressesSHOULD<bcp14>SHOULD</bcp14> be thesame.</t> <t>ICMP case:same.</dd> <dt>ICMP case:</dt> <dd> For IPv4, theDatadata fieldSHOULD<bcp14>SHOULD</bcp14> compensate variations on TTL or Hop Limit, IP identification, and IP checksum for every packet. There is no need to consider ICMPv6 because onlyFlowLabelFlow Label of IPv6 and Source and Destination addresses are used, and all of themSHOULD<bcp14>SHOULD</bcp14> beconstant.</t> </list></t>constant.</dd> </dl> <t>Then, the way to identify differenthopsHops and attempts of the same IPv4 flow is:</t><t><list style="symbols"> <t>TCP case:<dl spacing="normal"> <dt>TCP case:</dt> <dd> The IP identificationfield.</t> <t>UDP case:field.</dd> <dt>UDP case:</dt> <dd> The IP identificationfield.</t> <t>ICMP case:field.</dd> <dt>ICMP case:</dt> <dd> The IP identificationfield,field and ICMPSequence number.</t> </list></t>sequence number.</dd> </dl> <sectiontitle="Temporalnumbered="true" toc="default"> <name>Temporal Composition for RouteMetrics">Metrics</name> <t>TheActiveactive RouteAssessment Methodsassessment methods described above have the ability to discover portions of a path where ECMP load balancing is present, observed as two or more unique Member Routes having one or more distinct Hopswhichthat are part of the Route Ensemble. Likewise, attempts to deliberately vary the flow characteristics to discover all Member Routes will reveal portions of the pathwhichthat areflow-invariant.</t> <t>Section 9.2 of <xref target="RFC2330"/>flow invariant.</t> <t><xref target="RFC2330" section="9.2" sectionFormat="of"/> describes the Temporal Composition ofmetrics,metrics and introduces the possibility of a relationship between earlier measurement results and the results for measurement at the current time (for a given metric). There is value in establishing a Temporal Composition relationship for RouteMetrics. However,metrics; however, this relationship does not represent a forecast of futurerouteRoute conditions in any way.</t> <t>ForRoute MetricRoute-metric measurements, the value of Temporal Composition is to reduce the measurement iterations required with repeated measurements. Reduced iterations are possible by inferring that current measurements using fixed and previously measured flowcharacteristics:<list style="symbols"> <t>willcharacteristics:</t> <ul spacing="normal"> <li>will have many commonhopsHops with previousmeasurements.</t> <t>willmeasurements.</li> <li>will have relatively time-stable results at the ingress and egress portions of the path when measured from user locations, as opposed to measurements of backbone networks and across inter-domaingateways.</t> <t>maygateways.</li> <li>may have greater potential fortime-variationtime variation in path portions where ECMP load balancing is observed (because increasing or decreasing the pool of links changes the hashcalculations).</t> </list></t>calculations).</li> </ul> <t>Optionally, measurement systems may take advantage of the inferences above when seeking to reduce measurementiterations,iterations after exhaustive measurements indicate that the time-stable properties are present. RepetitiveActiveactive Route measurementsystems:<list style="numbers"> <t>SHOULDsystems:</t> <ol spacing="normal" type="1"> <li><bcp14>SHOULD</bcp14> occasionally check path portionswhichthat have exhibited stable results over time, particularly ingress and egress portions of the path (e.g., daily checks if measuring many times during aday).</t> <t>SHOULDday).</li> <li><bcp14>SHOULD</bcp14> continue testing portions of the path that have previously exhibited ECMP loadbalancing.</t> <t>SHALLbalancing.</li> <li><bcp14>SHALL</bcp14> triggerre-assessmentreassessment of the complete path and RouteEnsemble,Ensemble if any change inhopsHops is observed for a specific (and previously tested)flow.</t> </list></t>flow.</li> </ol> <t/> </section> <sectiontitle="Routingnumbered="true" toc="default"> <name>Routing ClassIdentification">Identification</name> <t>There is an opportunity to apply the<xref target="RFC2330"/>notion from <xref target="RFC2330" format="default"/> of equal treatment for a class of packets, "...very useful to know if a given Internet component treats equally a class C of different types of packets", as it applies to Route measurements. The notion of class C was examined further in <xreftarget="RFC8468"/>target="RFC8468" format="default"/> as it applied to load-balancing flows over parallel paths, which is the case we develop here. Knowledge of class C parameters (unrelated to address classes of the past) on a path potentially reduces the number of flows required for a given method to assess a Route Ensemble over time.</t> <t>First, recognize that each Member Route of a Route Ensemble will have a corresponding class C. Class C can be discovered by testing with multiple flows, all of which traverse the unique set ofhopsHops that comprise a specific Member Route.</t> <t>Second, recognize that the different classes depend primarily on the hash functions used at each instance of ECMP load balancing on the path.</t> <t>Third, recognize the synergy with Temporal Composition methods (described above), where evaluation intends to discover time-stable portions of each MemberRoute,Route so that more emphasis can be placed on ECMP portions that also determine class C.</t> <t>The methods to assess the various class C characteristics benefit from the following measurementcapabilities:<list style="symbols"> <t>flowscapabilities:</t> <ul spacing="normal"> <li>flows designed to determine which n-tuple header fields are considered by a given hash function and ECMPhopHop on thepath,path and which are not. This operation immediately narrows the search space, where possible, and partially defines a classC.</t> <t>aC.</li> <li>a priori knowledge of the possible types of hash functions in use also helps to design the flows for testing (major router vendors publish information about these hashfunctions,functions; examples areherein <xreftarget="LOAD_BALANCE"/>.</t> <t>abilitytarget="LOAD_BALANCE" format="default"/>).</li> <li>ability to direct the emphasis of current measurements on ECMP portions of the path, based on recent past measurement results (the Routing Class of some portions of the path is essentially "allpackets").</t> </list></t>packets").</li> </ul> <t/> </section> <sectiontitle="Intermediatenumbered="true" toc="default"> <name>Intermediate Observation Point RouteMeasurement">Measurement</name> <t>There are many examples where passive monitoring of a flow at an Observation Point within the network can detect unexpectedRound TripRound-Trip Delay or Delay Variation. But how can the cause of the anomalous delay be investigated further*from<strong>from the ObservationPoint*Point</strong> possibly located at an intermediate point on the path?</t> <t>In this case, knowledge that the flow of interest belongs to a specific Routing Class C will enable measurement of therouteRoute where anomalous delay has been observed. Specifically, Round-Trip Delay assessment to each Hop on the path between the Observation Point and the Destination for the flow of interest may discover high or variable delay on a specific link and Hop combination.</t> <t>The determination of a Routing Class Cwhichthat includes the flow of interest is as described in the section above, aided by computation of the relevant hash function output as the target.</t> <t/> </section> </section> <sectiontitle="Hybrid Methodologies">numbered="true" toc="default"> <name>Hybrid Methodologies</name> <t>The Hybrid Type I methods provide an alternativemethodfor RouteMemberassessment.As mentioned in the Scope section,The "Scope, Applicability, and Assumptions" section of <xreftarget="I-D.ietf-ippm-ioam-data"/>target="RFC9197" format="default"/> providesaone possible set of data fields that would supportrouteRoute identification.</t> <t>In general,nodesNodes in the measured domain would be equipped with specificabilities:<list style="symbols"> <t>Storeabilities:</t> <ul spacing="normal"> <li>Store the identity ofnodesNodes that a packet has visited in header datafields,fields in the order the packet visited thenodes.</t> <t>SupportNodes.</li> <li>Support of a "Loopback"capability,capability where a copy of the packet is returned to the encapsulatingnode,Node and the packet is processed like any other IOAM packet on the returntransfer.</t> </list></t>transfer.</li> </ul> <t>In addition tonodeNode identity,nodesNodes may also identify the ingress and egress interfaces utilized by the tracing packet, the absolute time when the packet was processed, and other generic data (as described insection 4 of<xreftarget="I-D.ietf-ippm-ioam-data"/>).target="RFC9197" sectionFormat="of" section="3"/>). Interface identification isn't necessarily limited to IP,i.e.i.e., different links in a bundle(LACP)(Link Aggregation Control Protocol (LACP)) could be identified. Equally well, links without explicit IP addresses can be identified (like with unnumbered interfaces in an IGP deployment).</t> <t>Note that the Type-P packet specification for this method will likely be a partialspecification,specification because most of the packet fields are determined by the user traffic. The packet(encapsulation) header(s)encapsulation header or headers added by theHybridhybrid method can certainly be specified in Type-P, in unpopulated form.</t> </section> <sectiontitle="Combiningnumbered="true" toc="default"> <name>Combining DifferentMethods">Methods</name> <t>In principle, there are advantages if the entity conducting Route measurements can utilize both forms of advanced methods (active andhybrid),hybrid) and combine the results. For example, if there are Nodes involved in the path that qualify as CooperatingNodes,Nodes but not as Discoverable Nodes, then a more complete view of Hops on the path is possible when a hybrid method (or interrogation protocol) is applied and the results are combined with the active method results collected across all other domains.</t> <t>In order to combine the results of active and hybrid/interrogation methods, the network Nodes that are part of a domain supporting an interrogation protocol have the following attributes:<list style="numbers"> <t>Nodes</t> <ol spacing="normal" type="1"> <li>Nodes at the ingress to the domainSHOULD<bcp14>SHOULD</bcp14> be both Discoverable andCooperating.</t> <t>AnyCooperating.</li> <li>Any Nodes within the domain that are both Discoverable and CooperatingSHOULD<bcp14>SHOULD</bcp14> reveal the same NodeIdentityidentity in response to both active and hybridmethods.</t> <t>Nodesmethods.</li> <li>Nodes at the egress to the domainSHOULD<bcp14>SHOULD</bcp14> be both Discoverable andCooperating,Cooperating andSHOULD<bcp14>SHOULD</bcp14> reveal the same NodeIdentityidentity in response to both active and hybridmethods.</t> </list></t>methods.</li> </ol> <t>When Nodes follow these requirements, it becomes a simple matter to matchsingle domainsingle-domain measurements with the overlapping results from a multidomain measurement.</t> <t>In practice, Internet users do not typically have the ability to utilize theOAMOperations, Administrations, and Maintenance (OAM) capabilities of networks that their packets traverse, so the results from a remote domain supporting an interrogation protocol would not normally be accessible. However, a network operator could combine interrogation results from their access domain with other measurements revealing the path outside their domain.</t> </section> </section> <section anchor="Cases"title="Backgroundnumbered="true" toc="default"> <name>Background on Round-Trip Delay MeasurementGoals">Goals</name> <t>The aim of this method is to use packet probes to unveil the paths between any twoend-NodesEnd-Nodes of the network. Moreover, information derived from RTD measurements might be meaningful to identify:</t><t><list style="numbers"> <t>Intercontinental<ol spacing="normal" type="1"> <li>Intercontinental submarinelinks</t> <t>Satellite communications</t> <t>Congestion</t> <t>Inter-domain paths</t> </list></t>links</li> <li>Satellite communications</li> <li>Congestion</li> <li>Inter-domain paths</li> </ol> <t>This categorization is widely accepted in the literature and among operators alike, and it can be trusted with empirical data and several sources as ground of truth (e.g., <xreftarget="RTTSub"/> )target="RTTSub" format="default"/>), but it is an inference measurement nonetheless <xreftarget="bdrmap"/><xref target="IDCong"/>.</t>target="bdrmap" format="default"/><xref target="IDCong" format="default"/>.</t> <t>The first two categories correspond to the physical distance dependency onRound-Trip Delay (RTD),RTD, the next one binds RTD with queuing delay on routers, and the last one helps to identify different ASes using traceroutes. Due to the significant contribution of propagation delay in long-distancehops,Hops, RTD will be on the order of100ms100 ms on transatlantichops,Hops, depending on the geolocation of the vantage points. Moreover, RTD is typically higher than480ms480 ms when twohopsHops are connected using geostationary satellite technology (i.e., their orbit is at36000km).36000 km). Detecting congestion with latency implies deeper mathematicalunderstandingunderstanding, since network traffic load is not stationary. Nonetheless, as the first approach, a link seems to be congested if observing different/varying statistical results after sending several traceroute probes (e.g., see <xreftarget="IDCong"/>).target="IDCong" format="default"/>). Finally, to recognize distinctive ASes in the same traceroute path ischallenging,challenging because more data is needed, like AS relationships andRIRRegional Internet Registry (RIR) delegations amongotherothers (for moredetail,details, please consult <xreftarget="bdrmap"/>).</t>target="bdrmap" format="default"/>).</t> </section> <section anchor="Statistics"title="RTDnumbered="true" toc="default"> <name>RTD MeasurementsStatistics">Statistics</name> <t>Several articles have shown that network traffic presents a self-similar nature <xreftarget="SSNT"/>target="SSNT" format="default"/> <xreftarget="MLRM"/> whichtarget="MLRM" format="default"/> that is accountable for filling the queues of the routers. Moreover, router queues are designed to handle traffic bursts, which is one of the most remarkable features of self-similarity. Naturally, while queue length increases, the delay to traverse the queue increases as well and leads to an increase on RTD. Due to traffic bursts generating short-term overflow on buffers (spiky patterns), every RTD only depicts the queueing status on the instant when that packet probe was in transit. For this reason, several RTD measurements during a time window could begin to describe the random behavior of latency. Loss must also be accounted for in the methodology.</t> <t>To understand the ongoing process, examining the quartiles provides anon-parametricnonparametric way of analysis. Quartiles are defined by five values: minimum RTD (m), RTD value of the 25% of the Empirical Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), the RTD value of the 75% of the ECDF(Q3)(Q3), and the maximum RTD (M). Congestion can be inferred when RTD measurements are spreadapart, andapart; consequently, theInter-QuartileInterquartile Range (IQR), i.e., the distance between Q3 and Q1, increases its value.</t> <t>This procedure requires the algorithm presented in <xreftarget="P2"/>target="P2" format="default"/> to compute quartile values "on thefly”.</t>fly".</t> <t>This procedure allows us to update thequartiles valuequartile values whenever a new measurement arrives, which is radically different from classic methods of computingquartilesquartiles, because they need to use the whole dataset to compute the values. This way of calculus provides savings in memory and computing time.</t> <t>To sum up, the proposed measurement procedure consists of performing traceroutes several times to obtain samples of the RTD in everyhopHop from apath,path during a time window(W),(W) and compute the quartiles for everyhop.Hop. This procedure could be done for a single Member Route flow, for a non-exhaustive search with parameter E (defined below) setasto False, or for every detected Route Ensemble flow (E=True).</t> <t>The identification of a specific Hop in a traceroute is based on the IP origin address of the returned ICMP Time Exceededpacket,packet and on the distance identified by the value set in the TTL (or Hop Limit) field inserted by traceroute. As this specific Hop can be reached by different paths,alsothe IPsourceSource anddestinationDestination addresses of the traceroute packet also need to be recorded. Finally, different return paths are distinguished by evaluating the ICMP Time Exceeded TTL (or Hop Limit) of the replymessage:message; if this TTL (or Hop Limit) is constant for different paths containing the same Hop, the return paths have the same distance. Moreover, this distance can be estimated considering that the TTL (or Hop Limit) value is normally initialized with values 64, 128, or 255. The 5-tuple (origin IP, destination IP, reply IP, distance, and response TTL or Hop Limit) unequivocally identifies every measurement.</t> <t>This algorithm below runs in the origin of the traceroute. It returns the Qs quartiles for every Hop and Alt (alternative paths because of balancing). Notice that the "Alt" parameter condenses the parameters of the 5-tuple (origin IP, destination IP, reply IP, distance, and response TTL), i.e., one for each possible combination.</t><t><figure> <artwork><![CDATA[================================================================<sourcecode name="" type="pseudocode"><![CDATA[ ================================================================ 0 input: W (window time of the measurement) 1 i_t (time between two measurements, set the i_t time 2 long enough to avoid incomplete results) 3 E (True: exhaustive, False: a single path) 4 Dst (destination IP address) 5 output: Qs (quartiles for every Hop and Alt) ---------------------------------------------------------------- 6 T := start_timer(W) 7 while T is not finished do: 8 | start_timer(i_t) 9 | RTD(Hop,Alt) = advanced-traceroute(Dst,E) 10 | for each Hop and Alt in RTD do: 11 | | Qs[Dst,Hop,Alt] := ComputeQs(RTD(Hop,Alt)) 12 | done 13 | wait until i_t timer is expired 14 done 15 return (Qs)================================================================]]></artwork> </figure></t>================================================================ ]]> </sourcecode> <t>During the time W, lines 6 and 7 assure that the measurement loop is made.LineLines 8 and 13 set a timer for each cycle of measurements. A cycle comprises the traceroutes packets, considering every possible Hop and the alternatives paths in the Alt variable (ensured in lines 9-12). In line 9, theadvance-tracerouteadvanced-traceroute could be either Paris-traceroute or Scamper, which will use the“exhaustive”"exhaustive" mode or“tracelb”"tracelb" option if E is set to True, respectively. The procedure returns a list of tuples(m,Q1,Q2,Q3,M)(m, Q1, Q2, Q3, and M) for each intermediatehop,Hop, or "Alt" in as a function of the 5-tuple, in the path towards the Dst. Finally, lines 10 through 12storesstore each measurement into the real-time quartiles computation.</t> <t>Notice there are cases wheretheeven having a uniquehopHop at distance h from the Src to Dst, the returning path could have several possibilities, yieldingindifferent total paths. In this situation, the algorithm will returnmoreanother "Alt" for this particularhop.</t>Hop.</t> </section> <section anchor="Security"title="Security Considerations">numbered="true" toc="default"> <name>Security Considerations</name> <t>The security considerations that apply to any active measurement of live paths are relevant here as well. See <xreftarget="RFC4656"/>target="RFC4656" format="default"/> and <xreftarget="RFC5357"/>.</t>target="RFC5357" format="default"/>.</t> <t>The active measurement process of"changingchanging several fields to keep the checksum of different packetsidentical"identical does not require special security considerations because it is part of synthetic trafficgeneration,generation and is designed to have minimal to zero impact on network processing (to process the packets for ECMP).</t> <t>Some of the protocols used (e.g., ICMP) do not provide cryptographic protection for the requested/returned data, and there are risks of processing untrusted data in general, but these are limitations of the existing protocols where we are applying new methods.</t> <t>For applicableHybridhybrid methods, the security considerationsin<xref target="I-D.ietf-ippm-ioam-data"/>in <xref target="RFC9197" format="default"/> apply.</t> <t>When considering the privacy of those involved in measurement or those whose traffic is measured, the sensitive information available to potential observers is greatly reduced when using active techniqueswhichthat are within this scope of work. Passive observations of user traffic for measurement purposes raise many privacy issues. We refer the reader to the privacy considerations described in theLarge ScaleLarge-scale Measurement of Broadband Performance (LMAP) Framework <xreftarget="RFC7594"/>,target="RFC7594" format="default"/>, which covers active and passive techniques.</t> </section> <section anchor="IANA"title="IANA Considerations">numbered="true" toc="default"> <name>IANA Considerations</name> <t>Thismemo makesdocument has norequests of IANA. We thank the good folks atIANAfor having checked this section anyway.</t> </section> <section anchor="Acknowledgements" title="Acknowledgements"> <t>The original 3 authors (Ignacio, Al, Joachim) acknowledge Ruediger Geib, for his penetrating comments on the initial draft, and his initial text for the Appendix on MPLS. Carlos Pignataro challenged the authors to consider a wider scope, and applied his substantial expertise with many technologies and their measurement features in his extensive comments. Frank Brockners also shared useful comments, so did Footer Foote. We thank them all!</t> </section> <section title="Appendix I MPLS Methods for Route Assessment"> <t>A Node assessing an MPLS path must be part of the MPLS domain where the path is implemented. When this condition is met, RFC 8029 provides a powerful set of mechanisms to detect “correct operation of the data plane, as well as a mechanism to verify the data plane against the control plane” <xref target="RFC8029"/>.</t> <t>MPLS routing is based on the presence of a Forwarding Equivalence Class (FEC) Stack in all visited Nodes. Selecting one of several Equal Cost Multi Path (ECMP) is however based on information hidden deeper in the stack. Late deployments may support a so called "Entropy label" for this purpose. State of the art deployments base their choice of an ECMP member interface on the complete MPLS label stack and on IP addresses up to the complete 5 tuple IP header information (see Section 2.4 of <xref target="RFC7325"/>). Load Sharing based on IP information decouples this function from the actual MPLS routing information. Thus, an MPLS traceroute is able to check how packets with a contiguous number of ECMP relevant IP addresses (and an identical MPLS label stack) are forwarded by a particular router. The minimum number of equivalent MPLS paths traceable at a router should be 32. Implementations supporting more paths are available.</t> <t>The MPLS echo request and reply messages offering this feature must support the Downstream Detailed Mapping TLV (was Downstream Mapping initially, but the latter has been deprecated). The MPLS echo response includes the incoming interface where a router received the MPLS Echo request. The MPLS Echo reply further informs which of the n addresses relevant for the load sharing decision results in a particular next hop interface and contains the next hop’s interface address (if available). This ensures that the next hop will receive a properly coded MPLS Echo request in the next step route of assessment.</t> <t><xref target="RFC8403"/> explains how a central Path Monitoring System could be used to detect arbitrary MPLS paths between any routers within a single MPLS domain. The combination of MPLS forwarding, Segment Routing and MPLS traceroute offers a simple architecture and a powerful mechanism to detect and validate (segment routed) MPLS paths.</t>actions.</t> </section> </middle> <back><references title="Normative References"> <?rfc include='reference.RFC.0792'?> <?rfc include='reference.RFC.1122'?> <?rfc include='reference.RFC.1812'?> <?rfc include='reference.RFC.2119'?> <?rfc include='reference.RFC.2330'?> <?rfc include='reference.RFC.2681'?> <?rfc ?> <?rfc include='reference.RFC.4656'?> <?rfc include='reference.RFC.5388'?> <?rfc include='reference.RFC.6438'?> <?rfc include='reference.RFC.6673'?> <?rfc include='reference.RFC.7799'?> <?rfc include='reference.RFC.8029'?> <?rfc include='reference.RFC.8174'?> <?rfc include='reference.RFC.8468'?> <?rfc include='reference.I-D.ietf-ippm-ioam-data'?><references> <name>References</name> <references> <name>Normative References</name> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.0792.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1122.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1812.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2330.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2681.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4656.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5388.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6438.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6673.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7799.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8029.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8468.xml"/> <reference anchor='RFC9197' target='https://www.rfc-editor.org/info/rfc9197'> <front> <title>Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)</title> <author initials='F' surname='Brockners' fullname='Frank Brockners' role="editor"> </author> <author initials='S' surname='Bhandari' fullname='Shwetha Bhandari' role="editor"> </author> <author initials='T' surname='Mizrahi' fullname='Tal Mizrahi' role="editor"> </author> <date month='May' year='2022' /> </front> <seriesInfo name="RFC" value="9197"/> <seriesInfo name="DOI" value="10.17487/RFC9197"/> </reference> </references><references title="Informative References"> <?rfc include='reference.RFC.2991'?> <?rfc include='reference.RFC.5357'?> <?rfc include='reference.RFC.5835'?> <?rfc include='reference.RFC.5837'?> <?rfc include='reference.RFC.6437'?> <?rfc include='reference.RFC.7312'?> <?rfc include='reference.RFC.7325'?> <?rfc include='reference.RFC.7594'?> <?rfc include='reference.RFC.8403'?><references> <name>Informative References</name> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2991.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5357.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5835.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5837.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6437.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7312.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7325.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7594.xml"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8403.xml"/> <reference anchor="PT"> <front> <title>Avoiding traceroute anomalies with Paris traceroute</title> <author fullname="Brice Augustin"initials="B.A"initials="B" surname="Augustin"><!-- fullname="B. Augustin"--><organization/> </author> <author fullname="Xavier Cuvellier"initials="X.C."initials="X" surname="Cuvellier"><!-- fullname="X. Cuvellier" --><organization/> </author> <author fullname="Benjamin Orgogozo"initials="B.O."initials="B" surname="Orgogozo"><!-- fullname="B. Orgogozo" --><organization/> </author> <author fullname="Fabien Viger"initials="F.V."initials="F" surname="Viger"><!-- fullname="F. Viger" --><organization/> </author> <author fullname="Timur Friedman"initials="T.F."initials="T" surname="Friedman"><!-- fullname="T. Friedman" --><organization/> </author> <author fullname="Matthieu Latapy"initials="M.L."initials="M" surname="Latapy"><!-- fullname="M. Latapy" --><organization/> </author> <author fullname="Clmence Magnien"initials="C.M."initials="C" surname="Magnien"><!-- fullname="C. Magnien" --><organization/> </author> <author fullname="Renata Teixeira"initials="R.T."initials="R" surname="Teixeira"><!-- fullname="R. Teixeira" --><organization/> </author> <dateday="" month=""month="October" year="2006"/> </front> <seriesInfoname="" value=" Proceedingsname="DOI" value="10.1145/1177080.1177100"/> <refcontent>Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, pp.153-158. ACM, 2006."/>153-158</refcontent> </reference> <reference anchor="LOAD_BALANCE"> <front> <title>COMPARISON OF HASH STRATEGIES FOR FLOW-BASED LOAD BALANCING</title> <author fullname="Surasak Sanguanpong" initials="S." surname="Sanguanpong"><!-- fullname="Surasak Sanguanpong"--><organization/> </author> <author fullname="Witsarut Pittayapitak" initials="W." surname="Pittayapitak"><!-- fullname="Witsarut Pittayapitak"--><organization/> </author> <author fullname="Kasom Koht-Arsa" initials="K." surname="Kasom Koht-Arsa"><!-- fullname="Kasom Koht-Arsa"--><organization/> </author> <dateday="" month=""month="December" year="2015"/> </front> <seriesInfoname="" value="Internationalname="DOI" value="10.7903/ijecs.1346"/> <refcontent>International Journal of Electronic Commerce Studies, Vol.6, No.2,pp.259-268. http://dx.doi.org/10.7903/ijecs.1346"/>pp.259-268</refcontent> </reference> <reference anchor="MLB"> <front> <title>Measuring load-balanced paths in theInternet</title>internet</title> <author fullname="Brice Augustin"initials="B.A"initials="B" surname="Augustin"><!-- fullname="B. Augustin"--><organization/> </author> <author fullname="Timur Friedman"initials="T.F."initials="T" surname="Friedman"><!-- fullname="T. Friedman" --><organization/> </author> <author fullname="Renata Teixeira"initials="R.T."initials="R" surname="Teixeira"><!-- fullname="R. Teixeira" --><organization/> </author> <dateday="" month=""month="October" year="2007"/> </front> <seriesInfoname="" value=" Proceedingsname="DOI" value="10.1145/1298306.1298329"/> <refcontent>Proceedings of the 7th ACM SIGCOMM conference on Internet measurement, pp.149-160. ACM, 2007."/>149-160</refcontent> </reference> <reference anchor="SCAMPER"> <front> <title>Scamper: a scalable and extensible packet prober for active measurement of theInternet</title>internet</title> <author fullname="Matthew Luckie"initials="M.L."initials="M" surname="Matthew Luckie"><!-- fullname="M. Luckie"--><organization/> </author> <dateday="" month=""month="November" year="2010"/> </front> <seriesInfoname="" value="Proceedingsname="DOI" value="10.1145/1879141.1879171"/> <refcontent>Proceedings of the 10th ACM SIGCOMM conference on Internet measurement, pp.239-245. ACM, 2010."/>239-245</refcontent> </reference> <reference anchor="SSNT"> <front> <title>Self-Similar Network Traffic and Performance Evaluation (1st ed.)</title> <author fullname="Kihong Park"initials="K.P."initials="K" surname="Park"><!-- fullname="K. Park"--><organization/> </author> <author fullname="Walter Willinger"initials="W.W."initials="W" surname="Willinger"><!-- fullname="W. Willinger"--><organization/> </author> <dateday="" month=""month="August" year="2000"/> </front> <seriesInfo name="DOI" value="10.1002/047120644X"/> <seriesInfo name="" value="John Wiley & Sons, Inc., New York, NY, USA"/> </reference> <reference anchor="MLRM"> <front> <title>An empirical mixture model for large-scale RTT measurements</title> <author fullname="Romain Fontugne"initials="R.F."initials="R" surname="Fontugne"><!-- fullname="R. Fontugne"--><organization/> </author> <author fullname="Johan Mazel"initials="J.M."initials="J" surname="Mazel"><!-- fullname="J. Mazel" --><organization/> </author> <author fullname="Kensuke Fukuda"initials="K.F."initials="K" surname="Fukuda"><!-- fullname="K. Fukuda" --><organization/> </author> <dateday="" month=""month="April" year="2015"/> </front> <seriesInfoname="" value="2015name="DOI" value="10.1109/INFOCOM.2015.7218636"/> <refcontent>2015 IEEE Conference on Computer Communications (INFOCOM), pp.2470-2478. IEEE, 2015."/>2470-2478</refcontent> </reference> <reference anchor="P2"> <front> <title>The P 2 algorithm for dynamic calculation of quartiles and histograms without storing observations</title> <author fullname="Raj Jain"initials="R.J."initials="R" surname="Jain"><!-- fullname="R. Jain"--><organization/> </author> <author fullname="Imrich Chlamtac"initials="I.C."initials="I" surname="Chlamtac"><!-- fullname="I. Chlamtac" --><organization/> </author> <dateday="" month="" year="2015"/>month="October" year="1985"/> </front> <seriesInfoname="" value="Communicationsname="DOI" value="10.1145/4372.4378"/> <refcontent>Communications of the ACM 28.10 (1985):1076-1085"/>1076-1085</refcontent> </reference> <reference anchor="IDCong"> <front> <title>Challenges ininferringInferring Internetinterdomain congestion</title>Interdomain Congestion</title> <author fullname="Matthew Luckie" initials="M." surname="Luckie"><!-- fullname="M. Luckie"--><organization/> </author> <author fullname="Amogh Dhamdhere" initials="A." surname="Dhamdhere"><!-- fullname="A. Dhamdhere" --><organization/> </author> <author fullname="David Clark" initials="D." surname="Clark"><!-- fullname="D. Clark" --><organization/> </author> <author fullname="Bradley Huffaker" initials="B." surname="Huffaker"><!-- fullname="B. Huffaker" --><organization>bdrmap: Inference of Borders Between IP Networks</organization> </author> <dateday="" month=""month="November" year="2014"/> </front> <seriesInfoname="" value="In Proceedingsname="DOI" value="10.1145/2663716.2663741"/> <refcontent>Proceedings of the 2014 Conference on Internet Measurement Conference, pp.15-22. ACM"/>15-22</refcontent> </reference> <reference anchor="bdrmap"> <front> <title>bdrmap: Inference of Borders Between IP Networks</title> <author fullname="Matthew Luckie" initials="M." surname="Luckie"><!-- fullname="M. Luckie"--><organization/> </author> <author fullname="Amogh Dhamdhere" initials="A." surname="Dhamdhere"><!-- fullname="A. Dhamdhere" --><organization/> </author> <author fullname="Bradley Huffaker" initials="B." surname="Huffaker"><!-- fullname="B. Huffaker" --><organization/> </author> <author fullname="David Clark" initials="D." surname="Clark"><!-- fullname="D. Clark" --><organization/> </author> <author fullname="KC" initials="KC." surname="Claffy"><!-- fullname="D. Clark" --><organization/> </author> <dateday="" month=""month="November" year="2016"/> </front> <seriesInfoname="" value="In Proceedingsname="DOI" value="10.1145/2987443.2987467"/> <refcontent>Proceedings of the 2016 ACM on Internet Measurement Conference, pp.381-396. ACM"/>381-396</refcontent> </reference> <reference anchor="RTTSub"> <front> <title>In and out of Cuba: Characterizing Cuba'sconnectivity</title>Connectivity</title> <author fullname="Zachary S. Bischof"initials="Z.S."initials="Z" surname="Bischof"><!-- fullname="M. Luckie"--><organization/> </author> <author fullname="John P. Rula"initials="J.P."initials="J" surname="Rula"><!-- fullname="A. Dhamdhere" --><organization/> </author> <author fullname="Fabian E. Bustamante"initials="F.E."initials="F" surname="Bustamante"><!-- fullname="D. Clark" --><organization/> </author> <dateday="" month=""month="October" year="2015"/> </front> <seriesInfoname="" value="In Proceedingsname="DOI" value="10.1145/2815675.2815718"/> <refcontent>Proceedings of the 2015 ACM Conference on Internet Measurement Conference, pp.487-493. ACM"/>487-493</refcontent> </reference> </references> </references> <section numbered="true" toc="default"> <name>MPLS Methods for Route Assessment</name> <t>A Node assessing an MPLS path must be part of the MPLS domain where the path is implemented. When this condition is met, <xref target="RFC8029" format="default"/> provides a powerful set of mechanisms to detect "correct operation of the data plane, as well as a mechanism to verify the data plane against the control plane".</t> <t>MPLS routing is based on the presence of a Forwarding Equivalence Class (FEC) Stack in all visited Nodes. Selecting one of several Equal-Cost Multipaths (ECMPs) is, however, based on information hidden deeper in the stack. Late deployments may support a so-called "Entropy label" for this purpose. State-of-the-art deployments base their choice of an ECMP member interface on the complete MPLS label stack and on IP addresses up to the complete 5-tuple IP header information (see <xref target="RFC7325" section="2.4" sectionFormat="of"/>). Load sharing based on IP information decouples this function from the actual MPLS routing information. Thus, an MPLS traceroute is able to check how packets with a contiguous number of ECMP-relevant IP addresses (and an identical MPLS label stack) are forwarded by a particular router. The minimum number of equivalent MPLS paths traceable at a router should be 32. Implementations supporting more paths are available.</t> <t>The MPLS echo request and reply messages offering this feature must support the Downstream Detailed Mapping TLV (was Downstream Mapping initially, but the latter has been deprecated). The MPLS echo response includes the incoming interface where a router received the MPLS echo request. The MPLS echo reply further informs which of the n addresses relevant for the load-sharing decision results in a particular next-hop interface and contains the next Hop's interface address (if available). This ensures that the next Hop will receive a properly coded MPLS echo request in the next step Route of assessment.</t> <t><xref target="RFC8403" format="default"/> explains how a central Path Monitoring System could be used to detect arbitrary MPLS paths between any routers within a single MPLS domain. The combination of MPLS forwarding, Segment Routing, and MPLS traceroute offers a simple architecture and a powerful mechanism to detect and validate (segment-routed) MPLS paths.</t> </section> <section anchor="Acknowledgements" numbered="false" toc="default"> <name>Acknowledgements</name> <t>The original three authors (Ignacio, Al, Joachim) acknowledge <contact fullname="Ruediger Geib"/> for his penetrating comments on the initial document and his initial text for the appendix on MPLS. <contact fullname="Carlos Pignataro"/> challenged the authors to consider a wider scope and applied his substantial expertise with many technologies and their measurement features in his extensive comments. <contact fullname="Frank Brockners"/> also shared useful comments and so did <contact fullname="Footer Foote"/>. We thank them all!</t> </section> </back> </rfc>