rfc9378.original   rfc9378.txt 
ippm F. Brockners, Ed. Internet Engineering Task Force (IETF) F. Brockners, Ed.
Internet-Draft Cisco Request for Comments: 9378 Cisco
Intended status: Informational S. Bhandari, Ed. Category: Informational S. Bhandari, Ed.
Expires: 8 July 2023 Thoughtspot ISSN: 2070-1721 Thoughtspot
D. Bernier D. Bernier
Bell Canada Bell Canada
T. Mizrahi, Ed. T. Mizrahi, Ed.
Huawei Huawei
4 January 2023 April 2023
In-situ OAM Deployment In Situ Operations, Administration, and Maintenance (IOAM) Deployment
draft-ietf-ippm-ioam-deployment-05
Abstract Abstract
In-situ Operations, Administration, and Maintenance (IOAM) collects In situ Operations, Administration, and Maintenance (IOAM) collects
operational and telemetry information in the packet while the packet operational and telemetry information in the packet while the packet
traverses a path between two points in the network. This document traverses a path between two points in the network. This document
provides a framework for IOAM deployment and provides IOAM deployment provides a framework for IOAM deployment and provides IOAM deployment
considerations and guidance. considerations and guidance.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on 8 July 2023. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9378.
Copyright Notice Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. Code Components carefully, as they describe your rights and restrictions with respect
extracted from this document must include Revised BSD License text as to this document. Code Components extracted from this document must
described in Section 4.e of the Trust Legal Provisions and are include Revised BSD License text as described in Section 4.e of the
provided without warranty as described in the Revised BSD License. Trust Legal Provisions and are provided without warranty as described
in the Revised BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction
2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Conventions
3. IOAM Deployment: Domains And Nodes . . . . . . . . . . . . . 3 3. IOAM Deployment: Domains and Nodes
4. Types of IOAM . . . . . . . . . . . . . . . . . . . . . . . . 5 4. Types of IOAM
4.1. Per-hop Tracing IOAM . . . . . . . . . . . . . . . . . . 6 4.1. Per-Hop Tracing IOAM
4.2. Proof of Transit IOAM . . . . . . . . . . . . . . . . . . 8 4.2. Proof of Transit IOAM
4.3. Edge to Edge IOAM . . . . . . . . . . . . . . . . . . . . 8 4.3. E2E IOAM
4.4. Direct Export IOAM . . . . . . . . . . . . . . . . . . . 8 4.4. Direct Export IOAM
5. IOAM Encapsulations . . . . . . . . . . . . . . . . . . . . . 8 5. IOAM Encapsulations
5.1. IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.1. IPv6
5.2. NSH . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2. NSH
5.3. BIER . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.3. BIER
5.4. GRE . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.4. GRE
5.5. Geneve . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.5. Geneve
5.6. Segment Routing . . . . . . . . . . . . . . . . . . . . . 9 5.6. Segment Routing
5.7. Segment Routing for IPv6 . . . . . . . . . . . . . . . . 9 5.7. Segment Routing for IPv6
5.8. VXLAN-GPE . . . . . . . . . . . . . . . . . . . . . . . . 9 5.8. VXLAN-GPE
6. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 9 6. IOAM Data Export
7. IOAM Deployment Considerations . . . . . . . . . . . . . . . 11 7. IOAM Deployment Considerations
7.1. IOAM Namespaces . . . . . . . . . . . . . . . . . . . . . 11 7.1. IOAM-Namespaces
7.2. IOAM Layering . . . . . . . . . . . . . . . . . . . . . . 12 7.2. IOAM Layering
7.3. IOAM Trace Option Types . . . . . . . . . . . . . . . . . 13 7.3. IOAM Trace Option-Types
7.4. Traffic-sets that IOAM Is Applied To . . . . . . . . . . 15 7.4. Traffic-Sets That IOAM Is Applied To
7.5. IOAM Loopback Mode . . . . . . . . . . . . . . . . . . . 15 7.5. Loopback Flag
7.6. IOAM Active Mode . . . . . . . . . . . . . . . . . . . . 16 7.6. Active Flag
7.7. Brown Field Deployments: IOAM Unaware Nodes . . . . . . . 17 7.7. Brown Field Deployments: IOAM-Unaware Nodes
8. IOAM Manageability . . . . . . . . . . . . . . . . . . . . . 17 8. IOAM Manageability
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 9. IANA Considerations
10. Security Considerations . . . . . . . . . . . . . . . . . . . 18 10. Security Considerations
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 11. Informative References
12. Informative References . . . . . . . . . . . . . . . . . . . 19 Acknowledgements
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 Authors' Addresses
1. Introduction 1. Introduction
"In-situ" Operations, Administration, and Maintenance (IOAM) collects In situ Operations, Administration, and Maintenance (IOAM) collects
OAM information within the packet while the packet traverses a OAM information within the packet while the packet traverses a
particular network domain. The term "in-situ" refers to the fact particular network domain. The term "in situ" refers to the fact
that the OAM data is added to the data packets rather than is being that the OAM data is added to the data packets rather than being sent
sent within packets specifically dedicated to OAM. IOAM is to within packets specifically dedicated to OAM. IOAM complements
complement mechanisms such as Ping, Traceroute, or other active mechanisms such as Ping, Traceroute, or other active probing
probing mechanisms. In terms of "active" or "passive" OAM, "in-situ" mechanisms. In terms of "active" or "passive" OAM, IOAM can be
OAM can be considered a hybrid OAM type. "In-situ" mechanisms do not considered a hybrid OAM type. In situ mechanisms do not require
require extra packets to be sent. IOAM adds information to the extra packets to be sent. IOAM adds information to the already
already available data packets and therefore cannot be considered available data packets and, therefore, cannot be considered passive.
passive. In terms of the classification given in [RFC7799] IOAM In terms of the classification given in [RFC7799], IOAM could be
could be portrayed as Hybrid Type I. IOAM mechanisms can be portrayed as Hybrid Type I. IOAM mechanisms can be leveraged where
leveraged where mechanisms using e.g., ICMP do not apply or do not mechanisms using, e.g., ICMP do not apply or do not offer the desired
offer the desired results, such as proving that a certain traffic results. These situations could include:
flow takes a pre-defined path, SLA verification for the live data
traffic, detailed statistics on traffic distribution paths in * proving that a certain traffic flow takes a predefined path,
networks that distribute traffic across multiple paths, or scenarios
in which probe traffic is potentially handled differently from * verifying the Service Level Agreement (SLA) verification for the
regular data traffic by the network devices. live data traffic,
* providing detailed statistics on traffic distribution paths in
networks that distribute traffic across multiple paths, or
* providing scenarios in which probe traffic is potentially handled
differently from regular data traffic by the network devices.
2. Conventions 2. Conventions
Abbreviations used in this document: Abbreviations used in this document:
BIER: Bit Index Explicit Replication [RFC8279] BIER: Bit Index Explicit Replication [RFC8279]
Geneve: Generic Network Virtualization Encapsulation [RFC8926] Geneve: Generic Network Virtualization Encapsulation [RFC8926]
GRE: Generic Routing Encapsulation [RFC2784] GRE: Generic Routing Encapsulation [RFC2784]
IOAM: In-situ Operations, Administration, and Maintenance IOAM: In situ Operations, Administration, and Maintenance
MTU: Maximum Transmit Unit MTU: Maximum Transmission Unit
NSH: Network Service Header [RFC8300] NSH: Network Service Header [RFC8300]
OAM: Operations, Administration, and Maintenance OAM: Operations, Administration, and Maintenance
POT: Proof of Transit POT: Proof of Transit
VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol VXLAN-GPE: Virtual eXtensible Local Area Network - Generic Protocol
Extension [I-D.ietf-nvo3-vxlan-gpe] Extension [VXLAN-GPE]
3. IOAM Deployment: Domains And Nodes 3. IOAM Deployment: Domains and Nodes
IOAM is focused on "limited domains" as defined in [RFC8799]. IOAM [RFC9197] defines the scope of IOAM as well as the different types of
IOAM nodes. For improved readability, this section provides a brief
overview of where IOAM applies, along with explaining the main roles
of nodes that employ IOAM. Please refer to [RFC9197] for further
details.
IOAM is focused on "limited domains", as defined in [RFC8799]. IOAM
is not targeted for a deployment on the global Internet. The part of is not targeted for a deployment on the global Internet. The part of
the network which employs IOAM is referred to as the "IOAM-Domain". the network that employs IOAM is referred to as the "IOAM-Domain".
For example, an IOAM-domain can include an enterprise campus using For example, an IOAM-Domain can include an enterprise campus using
physical connections between devices or an overlay network using physical connections between devices or an overlay network using
virtual connections / tunnels for connectivity between said devices. virtual connections or tunnels for connectivity between said devices.
An IOAM-domain is defined by its perimeter or edge. The operator of An IOAM-Domain is defined by its perimeter or edge. The operator of
an IOAM-domain is expected to put provisions in place to ensure that an IOAM-Domain is expected to put provisions in place to ensure that
packets which contain IOAM data fields do not leak beyond the edge of packets that contain IOAM data fields do not leak beyond the edge of
an IOAM domain, e.g., using for example packet filtering methods. an IOAM-Domain, e.g., using packet filtering methods. The operator
The operator should consider the potential operational impact of IOAM should consider the potential operational impact of IOAM on
to mechanisms such as ECMP load-balancing schemes (e.g., load- mechanisms such as ECMP load-balancing schemes (e.g., load-balancing
balancing schemes based on packet length could be impacted by the schemes based on packet length could be impacted by the increased
increased packet size due to IOAM), path MTU (i.e., ensure that the packet size due to IOAM), path MTU (i.e., ensure that the MTU of all
MTU of all links within a domain is sufficiently large to support the links within a domain is sufficiently large enough to support the
increased packet size due to IOAM) and ICMP message handling. increased packet size due to IOAM), and ICMP message handling.
An IOAM-Domain consists of "IOAM encapsulating nodes", "IOAM An IOAM-Domain consists of "IOAM encapsulating nodes", "IOAM
decapsulating nodes" and "IOAM transit nodes". The role of a node decapsulating nodes", and "IOAM transit nodes". The role of a node
(i.e., encapsulating, transit, decapsulating) is defined within an (i.e., encapsulating, transit, decapsulating) is defined within an
IOAM-Namespace (see below). A node can have different roles in IOAM-Namespace (see below). A node can have different roles in
different IOAM-Namespaces. different IOAM-Namespaces.
An "IOAM encapsulating node" incorporates one or more IOAM-Option- An IOAM encapsulating node incorporates one or more IOAM Option-Types
Types into packets that IOAM is enabled for. If IOAM is enabled for into packets that IOAM is enabled for. If IOAM is enabled for a
a selected subset of the traffic, the IOAM encapsulating node is selected subset of the traffic, the IOAM encapsulating node is
responsible for applying the IOAM functionality to the selected responsible for applying the IOAM functionality to the selected
subset. subset.
An "IOAM transit node" updates one or more of the IOAM-Data-Fields. An IOAM transit node updates one or more of the IOAM-Data-Fields. If
If both the Pre-allocated and the Incremental Trace Option-Types are both the Pre-allocated and the Incremental Trace Option-Types are
present in the packet, each IOAM transit node will update at most one present in the packet, each IOAM transit node will update at most one
of these Option-Types. Note that in case both Trace Option-Types are of these Option-Types. Note that in case both Trace Option-Types are
present in a packet, it is up to the IOAM data processing systems present in a packet, it is up to the IOAM data processing systems
(see Section 6) to integrate the data from the two Trace Option-Types (see Section 6) to integrate the data from the two Trace Option-Types
to form a view of the entire journey of the packet. A transit node to form a view of the entire journey of the packet. A transit node
does not add new IOAM-Option-Types to a packet, and does not change does not add new IOAM Option-Types to a packet and does not change
the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type. the IOAM-Data-Fields of an IOAM Edge-to-Edge (E2E) Option-Type.
An "IOAM decapsulating node" removes IOAM-Option-Type(s) from An IOAM decapsulating node removes any IOAM Option-Types from
packets. packets.
The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating The role of an IOAM encapsulating, IOAM transit, or IOAM
node is always performed within a specific IOAM-Namespace. This decapsulating node is always performed within a specific IOAM-
means that an IOAM node which is e.g., an IOAM-decapsulating node for Namespace. This means that an IOAM node that is, e.g., an IOAM
IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove decapsulating node for IOAM-Namespace "A" but not for IOAM-Namespace
the IOAM-Option-Types for IOAM-Namespace "A" from the packet. An "B" will only remove the IOAM Option-Types for IOAM-Namespace "A"
IOAM decapsulating node situated at the edge of an IOAM domain from the packet. An IOAM decapsulating node situated at the edge of
removes all IOAM-Option-Types and associated encapsulation headers an IOAM-Domain removes all IOAM Option-Types and associated
for all IOAM-Namespaces from the packet. encapsulation headers for all IOAM-Namespaces from the packet.
IOAM-Namespaces allow for a namespace-specific definition and IOAM-Namespaces allow for a namespace-specific definition and
interpretation of IOAM-Data-Fields. Please refer to Section 7.1 for interpretation of IOAM-Data-Fields. Please refer to Section 7.1 for
a discussion of IOAM-Namespaces. a discussion of IOAM-Namespaces.
Export of Export of Export of Export of Export of Export of Export of Export of
IOAM data IOAM data IOAM data IOAM data IOAM data IOAM data IOAM data IOAM data
(optional) (optional) (optional) (optional) (optional) (optional) (optional) (optional)
^ ^ ^ ^ ^ ^ ^ ^
| | | | | | | |
| | | | | | | |
User +---+----+ +---+----+ +---+----+ +---+----+ User +---+----+ +---+----+ +---+----+ +---+----+
packets |Encapsu-| | Transit| | Transit| |Decapsu-| packets |Encapsu-| | Transit| | Transit| |Decapsu-|
-------->|lating |====>| Node |====>| Node |====>|lating |--> -------->|lating |====>| Node |====>| Node |====>|lating |-->
|Node | | A | | B | |Node | |Node | | A | | B | |Node |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 1: Roles of IOAM nodes Figure 1: Roles of IOAM Nodes
IOAM nodes which add or remove the IOAM-Data-Fields can also update IOAM nodes that add or remove the IOAM-Data-Fields can also update
the IOAM-Data-Fields at the same time. Or in other words, IOAM the IOAM-Data-Fields at the same time. Or, in other words, IOAM
encapsulating or decapsulating nodes can also serve as IOAM transit encapsulating or decapsulating nodes can also serve as IOAM transit
nodes at the same time. Note that not every node in an IOAM domain nodes at the same time. Note that not every node in an IOAM-Domain
needs to be an IOAM transit node. For example, a deployment might needs to be an IOAM transit node. For example, a deployment might
require that packets traverse a set of firewalls which support IOAM. require that packets traverse a set of firewalls that support IOAM.
In that case, only the set of firewall nodes would be IOAM transit In that case, only the set of firewall nodes would be IOAM transit
nodes rather than all nodes. nodes rather than all nodes.
4. Types of IOAM 4. Types of IOAM
IOAM supports different modes of operation, which are differentiated IOAM supports different modes of operation. These modes are
by the type of IOAM data fields being carried in the packet, the data differentiated by the type of IOAM data fields that are being carried
being collected, the type of nodes which collect or update data as in the packet, the data being collected, the type of nodes that
well as whether and how nodes export IOAM information. collect or update data, and if and how nodes export IOAM information.
* Per-hop tracing: OAM information about each IOAM node a packet Per-hop tracing: OAM information about each IOAM node a packet
traverses is collected and stored within the user data packet as traverses is collected and stored within the user data packet as
the packet progresses through the IOAM domain. Potential uses of the packet progresses through the IOAM-Domain. Potential uses of
IOAM per-hop tracing include: IOAM per-hop tracing include:
- Understand the different paths different packets traverse * Understanding the different paths that different packets
between an IOAM encapsulating and an IOAM decapsulating node in traverse between an IOAM encapsulating node and an IOAM
a network that uses load balancing such as Equal Cost Multi- decapsulating node in a network that uses load balancing, such
Path (ECMP). This information could be used to tune the as Equal Cost Multi-Path (ECMP). This information could be
algorithm for ECMP for optimized network resource usage. used to tune the algorithm for ECMP for optimized network
resource usage.
- Operations/Troubleshooting: Understand which path a particular * With regard to operations and troubleshooting, understanding
packet or set of packets take as well as what amount of jitter which path a particular packet or set of packets take as well
and delay different IOAM nodes in the path contribute to the as what amount of jitter and delay different IOAM nodes in the
overall delay and jitter between the IOAM encapsulating node path contribute to the overall delay and jitter between the
and the IOAM decapsulating node. IOAM encapsulating node and the IOAM decapsulating node.
* Proof-of-transit: Information that a verifier node can use to Proof of Transit: Information that a verifier node can use to verify
verify whether a packet has traversed all nodes that is supposed whether a packet has traversed all nodes that it is supposed to
to traverse is stored within the user data packet. Proof-of- traverse is stored within the user data packet. For example,
transit could for example be used to verify that a packet indeed Proof of Transit could be used to verify that a packet indeed
passes through all services of a service function chain (e.g., passes through all services of a service function chain (e.g.,
verify whether a packet indeed traversed the set of firewalls that verify whether a packet indeed traversed the set of firewalls that
it is expected to traverse), or whether a packet indeed took the it is expected to traverse) or whether a packet indeed took the
expected path. expected path.
* Edge-to-edge: OAM information which is specific to the edges of an Edge-to-Edge (E2E): OAM information that is specific to the edges of
IOAM domain is collected and stored within the user data packet. an IOAM-Domain is collected and stored within the user data
Edge-to-Edge OAM could be used to gather operational information packet. E2E OAM could be used to gather operational information
about a particular network domain, such as the delay and jitter about a particular network domain, such as the delay and jitter
incurred by that network domain or the traffic matrix of the incurred by that network domain or the traffic matrix of the
network domain. network domain.
* Direct export: OAM information about each IOAM node a packet Direct Export: OAM information about each IOAM node a packet
traverses is collected and immediately exported to a collector. traverses is collected and immediately exported to a collector.
Direct export could be used in situations where per-hop tracing Direct Export could be used in situations where per-hop tracing
information is desired, but gathering the information within the information is desired, but gathering the information within the
packet - as with per-hop tracing - is not feasible. Rather than packet -- as with per-hop tracing -- is not feasible. Rather than
automatically correlating the per-hop tracing information, as done automatically correlating the per-hop tracing information, as done
with per-hop tracing, direct export requires a collector to with per-hop tracing, Direct Export requires a collector to
correlate the information from the individual nodes. In addition, correlate the information from the individual nodes. In addition,
all nodes enabled for direct export need to be capable to export all nodes enabled for Direct Export need to be capable of
the IOAM information to the collector. exporting the IOAM information to the collector.
4.1. Per-hop Tracing IOAM 4.1. Per-Hop Tracing IOAM
"IOAM tracing data" is expected to be collected at every IOAM transit "IOAM tracing data" is expected to be collected at every IOAM transit
node that a packet traverses to ensure visibility into the entire node that a packet traverses to ensure visibility into the entire
path a packet takes within an IOAM-Domain. I.e., in a typical path that a packet takes within an IOAM-Domain. In other words, in a
deployment all nodes in an IOAM-Domain would participate in IOAM and typical deployment, all nodes in an IOAM-Domain would participate in
thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating IOAM and, thus, be IOAM transit nodes, IOAM encapsulating nodes, or
nodes. If not all nodes within a domain are IOAM capable, IOAM IOAM decapsulating nodes. If not all nodes within a domain are IOAM
tracing information (i.e., node data, see below) will only be capable, IOAM tracing information (i.e., node data, see below) will
collected on those nodes which are IOAM capable. Nodes which are not only be collected on those nodes that are IOAM capable. Nodes that
IOAM capable will forward the packet without any changes to the IOAM- are not IOAM capable will forward the packet without any changes to
Data-Fields. The maximum number of hops and the minimum path MTU of the IOAM-Data-Fields. The maximum number of hops and the minimum
the IOAM domain is assumed to be known. path MTU of the IOAM-Domain are assumed to be known.
IOAM offers two different trace Option-Types, the "incremental" IOAM offers two different Trace Option-Types: the "Incremental" Trace
Option-Type as well as the "pre-allocated" Option-Type. For a Option-Type and the "Pre-allocated" Trace Option-Type. For a
discussion which of the two option types is the most suitable for an discussion about which of the two option types is the most suitable
implementation and/or deployment, see Section 7.3. for an implementation and/or deployment, see Section 7.3.
Every node data entry holds information for a particular IOAM transit Every node data entry holds information for a particular IOAM transit
node that is traversed by a packet. The IOAM decapsulating node node that is traversed by a packet. The IOAM decapsulating node
removes the IOAM-Option-Type(s) and processes and/or exports the removes any IOAM Option-Types and processes and/or exports the
associated data. All IOAM-Data-Fields are defined in the context of associated data. All IOAM-Data-Fields are defined in the context of
an IOAM-Namespace. an IOAM-Namespace.
IOAM tracing can for example collect the following types of IOAM tracing can, for example, collect the following types of
information: information:
* Identification of the IOAM node. An IOAM node identifier can * Identification of the IOAM node. An IOAM node identifier can
match to a device identifier or a particular control point or match to a device identifier or a particular control point or
subsystem within a device. subsystem within a device.
* Identification of the interface that a packet was received on, * Identification of the interface that a packet was received on,
i.e. ingress interface. i.e., ingress interface.
* Identification of the interface that a packet was sent out on, * Identification of the interface that a packet was sent out on,
i.e. egress interface. i.e., egress interface.
* Time of day when the packet was processed by the node as well as * Time of day when the packet was processed by the node as well as
the transit delay. Different definitions of processing time are the transit delay. Different definitions of processing time are
feasible and expected, though it is important that all devices of feasible and expected, though it is important that all devices of
an in-situ OAM domain follow the same definition. an IOAM-Domain follow the same definition.
* Generic data: Format-free information where syntax and semantic of * Generic data, which is format-free information, where the syntax
the information is defined by the operator in a specific and semantics of the information are defined by the operator in a
deployment. For a specific IOAM-Namespace, all IOAM nodes should specific deployment. For a specific IOAM-Namespace, all IOAM
interpret the generic data the same way. Examples for generic nodes should interpret the generic data the same way. Examples
IOAM data include geolocation information (location of the node at for generic IOAM data include geolocation information (location of
the time the packet was processed), buffer queue fill level or the node at the time the packet was processed), buffer queue fill
cache fill level at the time the packet was processed, or even a level or cache fill level at the time the packet was processed, or
battery charge level. even a battery charge level.
* Information to detect whether IOAM trace data was added at every * Information to detect whether IOAM trace data was added at every
hop or whether certain hops in the domain weren't IOAM transit hop or whether certain hops in the domain weren't IOAM transit
nodes. nodes.
* Data that relates to how the packet traversed a node (transit * Data that relates to how the packet traversed a node (transit
delay, buffer occupancy in case the packet was buffered, queue delay, buffer occupancy in case the packet was buffered, and queue
depth in case the packet was queued) depth in case the packet was queued).
The Option-Types of incremental tracing and pre-allocated tracing are The Incremental Trace Option-Type and Pre-allocated Trace Option-Type
defined in [RFC9197]. are defined in [RFC9197].
4.2. Proof of Transit IOAM 4.2. Proof of Transit IOAM
IOAM Proof of Transit Option-Type is to support path or service The IOAM Proof of Transit Option-Type is to support path or service
function chain [RFC7665] verification use cases. Proof-of-transit function chain [RFC7665] verification use cases. Proof of transit
could use methods like nested hashing or nested encryption of the could use methods like nested hashing or nested encryption of the
IOAM data. IOAM data.
The IOAM Proof of Transit Option-Type consist of a fixed size "IOAM The IOAM Proof of Transit Option-Type consists of a fixed-size "IOAM
proof of transit option header" and "IOAM proof of transit option Proof of Transit Option header" and "IOAM Proof of Transit Option
data fields". For details see [RFC9197]. data fields". For details, see [RFC9197].
4.3. Edge to Edge IOAM 4.3. E2E IOAM
The IOAM Edge-to-Edge Option-Type is to carry data that is added by The IOAM E2E Option-Type is to carry the data that is added by the
the IOAM encapsulating node and interpreted by IOAM decapsulating IOAM encapsulating node and interpreted by IOAM decapsulating node.
node. The IOAM transit nodes may process the data but must not The IOAM transit nodes may process the data but must not modify it.
modify it.
The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- The IOAM E2E Option-Type consists of a fixed-size "IOAM Edge-to-Edge
to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data Option-Type header" and "IOAM Edge-to-Edge Option-Type data fields".
fields". For details see [RFC9197]. For details, see [RFC9197].
4.4. Direct Export IOAM 4.4. Direct Export IOAM
Direct Export is an IOAM mode of operation within which IOAM data to Direct Export is an IOAM mode of operation within which IOAM data are
be directly exported to a collector rather than being collected to be directly exported to a collector rather than be collected
within the data packets. The IOAM Direct Export Option-Type consist within the data packets. The IOAM Direct Export Option-Type consists
of a fixed size "IOAM direct export option header". Direct Export of a fixed-size "IOAM direct export option header". Direct Export
for IOAM is defined in [RFC9326]. for IOAM is defined in [RFC9326].
5. IOAM Encapsulations 5. IOAM Encapsulations
IOAM data fields and associated data types for in-situ OAM are IOAM data fields and associated data types for IOAM are defined in
defined in [RFC9197]. The in-situ OAM data field can be transported [RFC9197]. The IOAM data field can be transported by a variety of
by a variety of transport protocols, including NSH, Segment Routing, transport protocols, including NSH, Segment Routing, Geneve, BIER,
Geneve, BIER, IPv6, etc. IPv6, etc.
5.1. IPv6 5.1. IPv6
IOAM encapsulation for IPv6 is defined in IOAM encapsulation for IPv6 is defined in [IOAM-IPV6-OPTIONS], which
[I-D.ietf-ippm-ioam-ipv6-options], which also discussed IOAM also discusses IOAM deployment considerations for IPv6 networks.
deployment considerations for IPv6 networks
5.2. NSH 5.2. NSH
IOAM encapsulation for NSH is defined in [I-D.ietf-sfc-ioam-nsh]. IOAM encapsulation for NSH is defined in [IOAM-NSH].
5.3. BIER 5.3. BIER
IOAM encapsulation for BIER is defined in [I-D.xzlnp-bier-ioam]. IOAM encapsulation for BIER is defined in [BIER-IOAM].
5.4. GRE 5.4. GRE
IOAM encapsulation for GRE is outlined as part of the "EtherType IOAM encapsulation for GRE is outlined as part of the "EtherType
Protocol Identification of In-situ OAM Data" in Protocol Identification of In-situ OAM Data" in [IOAM-ETH].
[I-D.weis-ippm-ioam-eth].
5.5. Geneve 5.5. Geneve
IOAM encapsulation for Geneve is defined in IOAM encapsulation for Geneve is defined in [IOAM-GENEVE].
[I-D.brockners-ippm-ioam-geneve].
5.6. Segment Routing 5.6. Segment Routing
IOAM encapsulation for Segment Routing is defined in IOAM encapsulation for Segment Routing is defined in [MPLS-IOAM].
[I-D.gandhi-spring-ioam-sr-mpls].
5.7. Segment Routing for IPv6 5.7. Segment Routing for IPv6
IOAM encapsulation for Segment Routing over IPv6 is defined in IOAM encapsulation for Segment Routing over IPv6 is defined in
[I-D.ali-spring-ioam-srv6]. [IOAM-SRV6].
5.8. VXLAN-GPE 5.8. VXLAN-GPE
IOAM encapsulation for VXLAN-GPE is defined in IOAM encapsulation for VXLAN-GPE is defined in [IOAM-VXLAN-GPE].
[I-D.brockners-ippm-ioam-vxlan-gpe].
6. IOAM Data Export 6. IOAM Data Export
IOAM nodes collect information for packets traversing a domain that IOAM nodes collect information for packets traversing a domain that
supports IOAM. IOAM decapsulating nodes as well as IOAM transit supports IOAM. IOAM decapsulating nodes, as well as IOAM transit
nodes can choose to retrieve IOAM information from the packet, nodes, can choose to retrieve IOAM information from the packet,
process the information further and export the information using process the information further, and export the information using,
e.g., IPFIX. e.g., IP Flow Information Export (IPFIX).
Raw data export of IOAM data using IPFIX is discussed in Raw data export of IOAM data using IPFIX is discussed in
[I-D.spiegel-ippm-ioam-rawexport]. "Raw export of IOAM data" refers [IOAM-RAWEXPORT]. "Raw export of IOAM data" refers to a mode of
to a mode of operation where a node exports the IOAM data as it is operation where a node exports the IOAM data as it is received in the
received in the packet. The exporting node neither interprets, packet. The exporting node does not interpret, aggregate, or
aggregates nor reformats the IOAM data before it is exported. Raw reformat the IOAM data before it is exported. Raw export of IOAM
export of IOAM data is to support an operational model where the data is to support an operational model where the processing and
processing and interpretation of IOAM data is decoupled from the interpretation of IOAM data is decoupled from the operation of
operation of encapsulating/updating/decapsulating IOAM data, which is encapsulating/updating/decapsulating IOAM data, which is also
also referred to as IOAM data-plane operation. The figure below referred to as "IOAM data-plane operation". Figure 2 shows the
shows the separation of concerns for IOAM export: Exporting IOAM data separation of concerns for IOAM export. Exporting IOAM data is
is performed by the "IOAM node" which performs IOAM data-plane performed by the "IOAM node", which performs IOAM data-plane
operation, whereas the interpretation of IOAM data is performed by operation, whereas the interpretation of IOAM data is performed by
one or several IOAM data processing systems. The separation of one or several IOAM data processing systems. The separation of
concerns is to off-load interpretation, aggregation and formatting of concerns is to offload interpretation, aggregation, and formatting of
IOAM data from the node which performs data-plane operations. In IOAM data from the node that performs data-plane operations. In
other words, a node which is focused on data-plane operations, i.e. other words, a node that is focused on data-plane operations, i.e.,
forwarding of packets and handling IOAM data will not be tasked to forwarding of packets and handling IOAM data, will not be tasked to
also interpret the IOAM data, but can leave this task to another also interpret the IOAM data. Instead, that node can leave this task
system or a set of systems. For scalability reasons, a single IOAM to another system or a set of systems. For scalability reasons, a
node could choose to export IOAM data to several IOAM data processing single IOAM node could choose to export IOAM data to several systems
systems. Similarly, there several monitoring systems or analytics that process IOAM data. Similarly, several monitoring systems or
systems can be used to further process the data received from the analytics systems can be used to further process the data received
IOAM preprocessing systems. Figure 2 shows an overview of IOAM from the IOAM preprocessing systems. Figure 2 shows an overview of
export, including IOAM data processing systems and monitoring/ IOAM export, including IOAM data processing systems and monitoring
analytics systems. and analytics systems.
+--------------+ +--------------+
+-------------+ | +-------------+ |
| Monitoring/ | | | Monitoring/ | |
| Analytics | | | Analytics | |
| system(s) |-+ | system(s) |-+
+-------------+ +-------------+
^ ^
| Processed/interpreted/ | Processed/interpreted/
| aggregated IOAM data | aggregated IOAM data
skipping to change at page 10, line 49 skipping to change at line 474
| |
+--------------+-------+------+--------------+ +--------------+-------+------+--------------+
| | | | | | | |
| | | | | | | |
User +---+----+ +---+----+ +---+----+ +---+----+ User +---+----+ +---+----+ +---+----+ +---+----+
packets |Encapsu-| | Transit| | Transit| |Decapsu-| packets |Encapsu-| | Transit| | Transit| |Decapsu-|
-------->|lating |====>| Node |====>| Node |====>|lating |--> -------->|lating |====>| Node |====>| Node |====>|lating |-->
|Node | | A | | B | |Node | |Node | | A | | B | |Node |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 2: IOAM framework with data export Figure 2: IOAM Framework with Data Export
7. IOAM Deployment Considerations 7. IOAM Deployment Considerations
This section describes several concepts of IOAM, and provides This section describes several concepts of IOAM and provides
considerations that need to be taken to account when implementing considerations that need to be taken into account when implementing
IOAM in a network domain. This includes concepts like IOAM IOAM in a network domain. This includes concepts like IOAM-
Namespaces, IOAM Layering, traffic-sets that IOAM is applied to and Namespaces, IOAM Layering, traffic-sets that IOAM is applied to, and
IOAM loopback mode. For a definition of IOAM Namespaces and IOAM IOAM Loopback. For a definition of IOAM-Namespaces and IOAM
layering, please refer to [RFC9197]. IOAM loopback mode is defined Layering, please refer to [RFC9197]. IOAM Loopback is defined in
in [RFC9322]. [RFC9322].
7.1. IOAM Namespaces 7.1. IOAM-Namespaces
IOAM-Namespaces add further context to IOAM-Option-Types and IOAM-Namespaces add further context to IOAM Option-Types and
associated IOAM-Data-Fields. IOAM-Namespaces are defined in associated IOAM-Data-Fields. IOAM-Namespaces are defined in
Section 4.3 of [RFC9197]. The Namespace-ID is part of the IOAM Section 4.3 of [RFC9197]. The Namespace-ID is part of the IOAM
Option-Type definition, see e.g., Section 4.4 of [RFC9197] for IOAM Option-Type definition. See Section 4.4 of [RFC9197] for IOAM Trace
Trace Option-Types or Section 4.6 of [RFC9197] for the IOAM Edge-to- Option-Types or Section 4.6 of [RFC9197] for the IOAM E2E Option-
Edge Option-Type. IOAM-Namespaces support several uses: Type. IOAM-Namespaces support several uses:
* IOAM-Namespaces can be used by an operator to distinguish * IOAM-Namespaces can be used by an operator to distinguish between
different operational domains. Devices at domain edges can filter different operational domains. Devices at domain edges can filter
on Namespace-IDs to provide for proper IOAM-Domain isolation. on Namespace-IDs to provide for proper IOAM-Domain isolation.
* IOAM-Namespaces provide additional context for IOAM-Data-Fields * IOAM-Namespaces provide additional context for IOAM-Data-Fields;
and thus ensure that IOAM-Data-Fields are unique and can be thus, they ensure that IOAM-Data-Fields are unique and can be
interpreted properly by management stations or network interpreted properly by management stations or network
controllers. While, for example, the node identifier field does controllers. While, for example, the node identifier field does
not need to be unique in a deployment (e.g., an operator may wish not need to be unique in a deployment (e.g., an operator may wish
to use different node identifiers for different IOAM layers, even to use different node identifiers for different IOAM layers, even
within the same device; or node identifiers might not be unique within the same device; or node identifiers might not be unique
for other organizational reasons, such as after a merger of two for other organizational reasons, such as after a merger of two
formerly separated organizations), the combination of node_id and formerly separated organizations), the combination of node_id and
Namespace-ID should always be unique. Similarly, IOAM-Namespaces Namespace-ID should always be unique. Similarly, IOAM-Namespaces
can be used to define how certain IOAM-Data-Fields are can be used to define how certain IOAM-Data-Fields are
interpreted: IOAM offers three different timestamp format options. interpreted. IOAM offers three different timestamp format
The Namespace-ID can be used to determine the timestamp format. options. The Namespace-ID can be used to determine the timestamp
IOAM-Data-Fields (e.g., buffer occupancy) which do not have a unit format. IOAM-Data-Fields (e.g., buffer occupancy) that do not
associated are to be interpreted within the context of a IOAM- have a unit associated are to be interpreted within the context of
Namespace. The Namespace-ID could also be used to distinguish an IOAM-Namespace. The Namespace-ID could also be used to
between different types of interfaces: An interface-id could for distinguish between different types of interfaces. An interface-
example point to a physical interface (e.g., to understand which id could, for example, point to a physical interface (e.g., to
physical interface of an aggregated link is used when receiving or understand which physical interface of an aggregated link is used
transmitting a packet) whereas in another case it could refer to a when receiving or transmitting a packet). Whereas, in another
logical interface (e.g., in case of tunnels). Namespace-IDs could case, an interface-id could refer to a logical interface (e.g., in
be used to distinguish between the different types of interfaces. case of tunnels). Namespace-IDs could be used to distinguish
between the different types of interfaces.
* IOAM-Namespaces can be used to identify different sets of devices * IOAM-Namespaces can be used to identify different sets of devices
(e.g., different types of devices) in a deployment: If an operator (e.g., different types of devices) in a deployment. If an
desires to insert different IOAM-Data-Fields based on the device, operator desires to insert different IOAM-Data-Fields based on the
the devices could be grouped into multiple IOAM-Namespaces. This device, the devices could be grouped into multiple IOAM-
could be due to the fact that the IOAM feature set differs between Namespaces. This could be due to the fact that the IOAM feature
different sets of devices, or it could be for reasons of optimized set differs between different sets of devices, or it could be for
space usage in the packet header. It could also stem from reasons of optimized space usage in the packet header. It could
hardware or operational limitations on the size of the trace data also stem from hardware or operational limitations on the size of
that can be added and processed, preventing collection of a full the trace data that can be added and processed, preventing
trace for a flow. collection of a full trace for a flow.
- Assigning different IOAM Namespace-IDs to different sets of - Assigning different IOAM Namespace-IDs to different sets of
nodes or network partitions and using the Namespace-ID as a nodes or network partitions and using the Namespace-ID as a
selector at the IOAM encapsulating node, a full trace for a selector at the IOAM encapsulating node, a full trace for a
flow could be collected and constructed via partial traces in flow could be collected and constructed via partial traces in
different packets of the same flow. Example: An operator could different packets of the same flow. For example, an operator
choose to group the devices of a domain into two IOAM- could choose to group the devices of a domain into two IOAM-
Namespaces, in a way that at average, only every second hop Namespaces in a way that, on average, only every second hop
would be recorded by any device. To retrieve a full view of would be recorded by any device. To retrieve a full view of
the deployment, the captured IOAM-Data-Fields of the two IOAM- the deployment, the captured IOAM-Data-Fields of the two IOAM-
Namespaces need to be correlated. Namespaces need to be correlated.
- Assigning different IOAM Namespace-IDs to different sets of - Assigning different IOAM Namespace-IDs to different sets of
nodes or network partitions and using a separate instance of an nodes or network partitions and using a separate instance of an
IOAM-Option-Type for each Namespace-ID, a full trace for a flow IOAM Option-Type for each Namespace-ID, a full trace for a flow
could be collected and constructed via partial traces from each could be collected and constructed via partial traces from each
IOAM-Option-Type in each of the packets in the flow. Example: IOAM Option-Type in each of the packets in the flow. For
An operator could choose to group the devices of a domain into example, an operator could choose to group the devices of a
two IOAM-Namespaces, in a way that each IOAM-Namespace is domain into two IOAM-Namespaces in a way that each IOAM-
represented by one of two IOAM-Option-Types in the packet. Namespace is represented by one of two IOAM Option-Types in the
Each node would record data only for the IOAM-Namespace that it packet. Each node would record data only for the IOAM-
belongs to, ignoring the other IOAM-Option-Type with a IOAM- Namespace that it belongs to, ignoring the other IOAM Option-
Namespace to which it doesn't belong. To retrieve a full view Type with an IOAM-Namespace it doesn't belong to. To retrieve
of the deployment, the captured IOAM-Data-Fields of the two a full view of the deployment, the captured IOAM-Data-Fields of
IOAM-Namespaces need to be correlated. the two IOAM-Namespaces need to be correlated.
7.2. IOAM Layering 7.2. IOAM Layering
If several encapsulation protocols (e.g., in case of tunneling) are If several encapsulation protocols (e.g., in case of tunneling) are
stacked on top of each other, IOAM-Data-Fields could be present in stacked on top of each other, IOAM-Data-Fields could be present in
different protocol fields at different layers. Layering allows different protocol fields at different layers. Layering allows
operators to instrument the protocol layer they want to measure. The operators to instrument the protocol layer they want to measure. The
behavior follows the ships-in-the-night model, i.e., IOAM-Data-Fields behavior follows the ships-in-the-night model, i.e., IOAM-Data-Fields
in one layer are independent of IOAM-Data-Fields in another layer. in one layer are independent of IOAM-Data-Fields in another layer.
Or in other words: Even though the term "layering" often implies some Or in other words, even though the term "layering" often implies
form of hierarchy and relationship, in IOAM, layers are independent there is some form of hierarchy and relationship, in IOAM, layers are
of each other and don't assume any relationship among them. The independent of each other and don't assume any relationship among
different layers could, but do not have to share the same IOAM them. The different layers could, but do not have to, share the same
encapsulation mechanisms. Similarly, the semantics of the IOAM-Data- IOAM encapsulation mechanisms. Similarly, the semantics of the IOAM-
Fields, can, but do not have to be associated to cross different Data-Fields can, but do not have to, be associated to cross different
layers. For example, a node which inserts node-id information into layers. For example, a node that inserts node-id information into
two different layers could use "node-id=10" for one layer and "node- two different layers could use "node-id=10" for one layer and "node-
id=1000" for the second layer. id=1000" for the second layer.
Figure 3 shows an example of IOAM layering. The figure shows a Figure 3 shows an example of IOAM Layering. The figure shows a
Geneve tunnel carried over IPv6 which starts at node A and ends at Geneve tunnel carried over IPv6, which starts at node A and ends at
node D. IOAM information is encapsulated in IPv6 as well as in node D. IOAM information is encapsulated in IPv6 as well as in
Geneve. At the IPv6 layer, node A is the IOAM encapsulating node Geneve. At the IPv6 layer, node A is the IOAM encapsulating node
(into IPv6), node D is the IOAM decapsulating node and node B and (into IPv6), node D is the IOAM decapsulating node, and nodes B and C
node C are IOAM transit nodes. At the Geneve layer, node A is the are IOAM transit nodes. At the Geneve layer, node A is the IOAM
IOAM encapsulating node (into Geneve) and node D is the IOAM encapsulating node (into Geneve), and node D is the IOAM
decapsulating node (from Geneve). The use of IOAM at both layers as decapsulating node (from Geneve). The use of IOAM at both layers, as
shown in the example here could be used to reveal which nodes of an shown in the example here, could be used to reveal which nodes of an
underlay (here the IPv6 network) are traversed by tunneled packet in underlay (here the IPv6 network) are traversed by a tunneled packet
an overlay (here the Geneve network) - which assumes that the IOAM in an overlay (here the Geneve network) -- which assumes that the
information encapsulated by nodes A and D into Geneve and IPv6 is IOAM information encapsulated by nodes A and D into Geneve and IPv6
associated to each other. is associated to each other.
+---+----+ +---+----+ +---+----+ +---+----+
User |Geneve | |Geneve | User |Geneve | |Geneve |
packets |Encapsu-| |Decapsu-| packets |Encapsu-| |Decapsu-|
-------->|lating |==================================>|lating |--> -------->|lating |==================================>|lating |-->
| Node | | Node | | Node | | Node |
| A | | D | | A | | D |
+--------+ +--------+ +--------+ +--------+
^ ^ ^ ^
| | | |
v v v v
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
|IPv6 | | IPv6 | | IPv6 | |IPv6 | |IPv6 | | IPv6 | | IPv6 | |IPv6 |
|Encapsu-| | Transit| | Transit| |Decapsu-| |Encapsu-| | Transit| | Transit| |Decapsu-|
|lating |====>| Node |====>| Node |====>|lating | |lating |====>| Node |====>| Node |====>|lating |
| Node | | | | | | Node | | Node | | | | | | Node |
| A | | B | | C | | D | | A | | B | | C | | D |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 3: IOAM layering example Figure 3: IOAM Layering Example
7.3. IOAM Trace Option Types 7.3. IOAM Trace Option-Types
IOAM offers two different IOAM Option-Types for tracing: IOAM offers two different IOAM Option-Types for tracing:
"Incremental" Trace-Option-Type and "Pre-allocated" Trace-Option- "Incremental" Trace Option-Type and "Pre-allocated" Trace Option-
Type. "Incremental" refers to a mode of operation where the packet Type. "Incremental" refers to a mode of operation where the packet
is expanded at every IOAM node that adds IOAM-Data-Fields. "Pre- is expanded at every IOAM node that adds IOAM-Data-Fields. "Pre-
Allocated" describes a mode of operation where the IOAM encapsulating allocated" describes a mode of operation where the IOAM encapsulating
node allocates room for all IOAM-Data-Fields in the entire IOAM node allocates room for all IOAM-Data-Fields in the entire IOAM-
domain. More specifically: Domain. More specifically:
Pre-allocated Trace-Option: This trace option is defined as a Pre-allocated Trace Option: This trace option is defined as a
container of node data fields with pre-allocated space for each container of node data fields with pre-allocated space for each
node to populate its information. This option is useful for node to populate its information. This option is useful for
implementations where it is efficient to allocate the space once implementations where it is efficient to allocate the space once
and index into the array to populate the data during transit and index into the array to populate the data during transit
(e.g., software forwarders often fall into this class). (e.g., software forwarders often fall into this class).
Incremental Trace-Option: This trace option is defined as a Incremental Trace Option: This trace option is defined as a
container of node data fields where each node allocates and pushes container of node data fields where each node allocates and pushes
its node data immediately following the option header. its node data immediately following the option header.
Which IOAM Trace-Option-Types can be supported is not only a function Which IOAM Trace Option-Types can be supported is not only a function
of operator-defined configuration, but may also be limited by of operator-defined configuration but may also be limited by protocol
protocol constraints unique to a given encapsulating protocol. For constraints unique to a given encapsulating protocol. For
encapsulating protocols which support both IOAM Trace-Option-Types, encapsulating protocols that support both IOAM Trace Option-Types,
the operator decides by means of configuration which Trace-Option- the operator decides, by means of configuration, which Trace Option-
Type(s) will be used for a particular domain. In this case, Type(s) will be used for a particular domain. In this case,
deployments can mix devices which include either the Incremental deployments can mix devices that include either the Incremental Trace
Trace-Option-Type or the Pre-allocated Trace-Option-Type. If for Option-Type or the Pre-allocated Trace Option-Type. For example, if
example different types of packet forwarders and associated different different types of packet forwarders and associated different types
types of IOAM implementations exist in a deployment and the of IOAM implementations exist in a deployment and the encapsulating
encapsulating protocol supports both IOAM Trace-Option-Types, a protocol supports both IOAM Trace Option-Types, a deployment can mix
deployment can mix devices which include either the Incremental devices that include either the Incremental Trace Option-Type or the
Trace-Option-Type or the Pre-allocated Trace-Option-Type. As a Pre-allocated Trace Option-Type. As a result, both Option-Types can
result, both Option-Types can be present in a packet. IOAM be present in a packet. IOAM decapsulating nodes remove both types
decapsulating nodes remove both types of Trace-Option-Types from the of Trace Option-Types from the packet.
packet.
The two different Option-Types cater to different packet forwarding The two different Option-Types cater to different packet-forwarding
infrastructures and are to allow an optimized implementation of IOAM infrastructures and allow an optimized implementation of IOAM
tracing: tracing:
Pre-allocated Trace-Option: For some implementations of packet Pre-allocated Trace Option: For some implementations of packet
forwarders it is efficient to allocate the space for the maximum forwarders, it is efficient to allocate the space for the maximum
number of nodes that IOAM Trace Data-Fields should be collected number of nodes that IOAM Trace Data-Fields should be collected
from and insert/update information in the packet at flexible from and insert/update information in the packet at flexible
locations, based on a pointer retrieved from a field in the locations based on a pointer retrieved from a field in the packet.
packet. The IOAM encapsulating node allocates an array of the The IOAM encapsulating node allocates an array of the size of the
size of the maximum number of nodes that IOAM Trace Data-Fields maximum number of nodes that IOAM Trace Data-Fields should be
should be collected from. IOAM transit nodes index into the array collected from. IOAM transit nodes index into the array to
to populate the data during transit. Software forwarders often populate the data during transit. Software forwarders often fall
fall into this class of packet forwarder implementations. The into this class of packet-forwarder implementations. The maximum
maximum number of nodes that IOAM information could be collected number of nodes that IOAM information could be collected from is
from is configured by the operator on the IOAM encapsulating node. configured by the operator on the IOAM encapsulating node. The
The operator has to ensure that the packet with the pre-allocated operator has to ensure that the packet with the pre-allocated
array that carries the IOAM Data-Fields does not exceed the MTU of array that carries the IOAM Data-Fields does not exceed the MTU of
any of the links in the IOAM domain. any of the links in the IOAM-Domain.
Incremental Trace-Option: Looking up a pointer contained in the Incremental Trace Option: Looking up a pointer contained in the
packet and inserting/updating information at a flexible location packet and inserting/updating information at a flexible location
in the packet as a result of the pointer lookup is costly for some in the packet as a result of the pointer lookup is costly for some
forwarding infrastructures. Hardware-based packet forwarding forwarding infrastructures. Hardware-based packet-forwarding
infrastructures often fall into this category. Consequently, infrastructures often fall into this category. Consequently,
hardware-based packet forwarders could choose to support the hardware-based packet forwarders could choose to support the IOAM
incremental IOAM-Trace-Option-Type. The incremental IOAM-Trace- Incremental Trace Option-Type. The IOAM Incremental Trace Option-
Option-Type eliminates the need for the IOAM transit nodes to read Type eliminates the need for the IOAM transit nodes to read the
the full array in the Trace-Option-Type and allows packets to grow full array in the Trace Option-Type and allows packets to grow to
to the size of the MTU of the IOAM domain. IOAM transit nodes the size of the MTU of the IOAM-Domain. IOAM transit nodes will
will expand the packet and insert the IOAM-Data-Fields as long as expand the packet and insert the IOAM-Data-Fields as long as there
there is space available in the packet, i.e. as long as the size is space available in the packet, i.e., as long as the size of the
of the packet stays within the bounds of the MTU of the links in packet stays within the bounds of the MTU of the links in the
the IOAM domain. There is no need for the operator to configure IOAM-Domain. There is no need for the operator to configure the
the IOAM encapsulation node with the maximum number of nodes that IOAM encapsulation node with the maximum number of nodes that IOAM
IOAM information could be collected from. The operator has to information could be collected from. The operator has to ensure
ensure that the minimum MTU of the links in the IOAM domain is that the minimum MTU of the links in the IOAM-Domain is known to
known to all IOAM transit nodes. all IOAM transit nodes.
7.4. Traffic-sets that IOAM Is Applied To 7.4. Traffic-Sets That IOAM Is Applied To
IOAM can be deployed on all or only on subsets of the live user IOAM can be deployed on all or only on subsets of the live user
traffic, e.g., per interface, based on an access control list or flow traffic, e.g., per interface, based on an access control list or flow
specification defining a specific set of traffic, etc. specification defining a specific set of traffic, etc.
7.5. IOAM Loopback Mode 7.5. Loopback Flag
IOAM Loopback is used to trigger each transit device along the path IOAM Loopback is used to trigger each transit device along the path
of a packet to send a copy of the data packet back to the source. of a packet to send a copy of the data packet back to the source.
Loopback allows an IOAM encapsulating node to trace the path to a Loopback allows an IOAM encapsulating node to trace the path to a
given destination, and to receive per-hop data about both the forward given destination and to receive per-hop data about both the forward
and the return path. Loopback is enabled by the encapsulating node and the return path. Loopback is enabled by the encapsulating node
setting the loopback flag. Looped-back packets use the source setting the Loopback flag. Looped-back packets use the source
address of the original packet as destination address and the address address of the original packet as a destination address and the
of the node which performs the loopback operation as source address. address of the node that performs the Loopback operation as source
Nodes which loop back a packet clear the loopback flag before sending address. Nodes that loop back a packet clear the Loopback flag
the copy back towards the source. Loopack applies to IOAM before sending the copy back towards the source. Loopack applies to
deployments where the encapsulating node is either a host or the IOAM deployments where the encapsulating node is either a host or the
start of a tunnel: For details on IOAM loopback, please refer to start of a tunnel. For details on IOAM Loopback, please refer to
[RFC9322]. [RFC9322].
7.6. IOAM Active Mode 7.6. Active Flag
The IOAM active mode flag indicates that a packet is an active OAM The Active flag indicates that a packet is an active OAM packet as
packet as opposed to regular user data traffic. Active mode is opposed to regular user data traffic. Active flag is expected to be
expected to be used for active measurement using IOAM. For details used for active measurement using IOAM. For details on the Active
on IOAM active mode, please refer to [RFC9322]. flag, please refer to [RFC9322].
Example use-cases for IOAM Active Mode include: Example use cases for the Active flag include:
* Endpoint detailed active measurement: Synthetic probe packets are Endpoint detailed active measurement: Synthetic probe packets are
sent between the source and destination. These probe packets sent between the source and destination. These probe packets
include a Trace Option-Type (i.e., either incremental or pre- include a Trace Option-Type (i.e., either incremental or pre-
allocated). Since the probe packets are sent between the allocated). Since the probe packets are sent between the
endpoints, these packets are treated as data packets by the IOAM endpoints, these packets are treated as data packets by the IOAM-
domain, and do not require special treatment at the IOAM layer. Domain and do not require special treatment at the IOAM layer.
The source, which is also the IOAM encapsulating node can choose The source, which is also the IOAM encapsulating node, can choose
to set the Active flag, providing an explicit indication that to set the Active flag, providing an explicit indication that
these probe packets are meant for telemetry collection. these probe packets are meant for telemetry collection.
* IOAM active measurement using probe packets: Probe packets are IOAM active measurement using probe packets: Probe packets are
generated and transmitted by an IOAM encapsulating node towards a generated and transmitted by an IOAM encapsulating node towards a
destination which is also the IOAM decapsulating node. Probe destination that is also the IOAM decapsulating node. Probe
packets include a Trace Option-Type (i.e., either incremental or packets include a Trace Option-Type (i.e., either incremental or
pre-allocated) which has its Active flag set. pre-allocated) that has its Active flag set.
* IOAM active measurement using replicated data packets: Probe IOAM active measurement using replicated data packets: Probe packets
packets are created by an IOAM encapsulating node by selecting are created by an IOAM encapsulating node by selecting some or all
some or all of the en route data packets and replicating them. A of the en route data packets and replicating them. A selected
selected data packet that is replicated, and its (possibly data packet that is replicated and its (possibly truncated) copy
truncated) copy is forwarded with one or more IOAM option, while are forwarded with one or more IOAM options, while the original
the original packet is forwarded normally, without IOAM options. packet is forwarded, normally, without IOAM options. To the
To the extent possible, the original data packet and its replica extent possible, the original data packet and its replica are
are forwarded through the same path. The replica includes a Trace forwarded through the same path. The replica includes a Trace
Option-Type that has its Active flag set, indicating that the IOAM Option-Type that has its Active flag set, indicating that the IOAM
decapsulating node should terminate it. In this case the IOAM decapsulating node should terminate it. In this case, the IOAM
Active flag ensures that the replicated traffic is not forwarded Active flag ensures that the replicated traffic is not forwarded
beyond the IOAM domain. beyond the IOAM-Domain.
7.7. Brown Field Deployments: IOAM Unaware Nodes 7.7. Brown Field Deployments: IOAM-Unaware Nodes
A network can consist of a mix of IOAM aware and IOAM unaware nodes. A network can consist of a mix of IOAM-aware and IOAM-unaware nodes.
The encapsulation of IOAM-Data-Fields into different protocols (see The encapsulation of IOAM-Data-Fields into different protocols (see
also Section 5) are defined such that data packets that include IOAM- also Section 5) are defined such that data packets that include IOAM-
Data-Fields do not get dropped by IOAM unaware nodes. For example, Data-Fields do not get dropped by IOAM-unaware nodes. For example,
packets which contain the IOAM-Trace-Option-Types in IPv6 Hop by Hop packets that contain the IOAM Trace Option-Types in IPv6 Hop-by-Hop
extension headers are defined with bits to indicate "00 - skip over extension headers are defined with bits to indicate "00 - skip over
this option and continue processing the header". This will ensure this option and continue processing the header". This will ensure
that when a node that is IOAM unaware receives a packet with IOAM- that when an IOAM-unaware node receives a packet with IOAM-Data-
Data-Fields included, does not drop the packet. Fields included, it does not drop the packet.
Deployments which leverage the IOAM-Trace-Option-Type(s) could Deployments that leverage the IOAM Trace Option-Type(s) could benefit
benefit from the ability to detect the presence of IOAM unaware from the ability to detect the presence of IOAM-unaware nodes, i.e.,
nodes, i.e. nodes which forward the packet but do not update/add nodes that forward the packet but do not update or add IOAM-Data-
IOAM-Data-Fields in IOAM-Trace-Option-Type(s). The node data that is Fields in IOAM Trace Option-Types. The node data that is defined as
defined as part of the IOAM-Trace-Option-Type(s) includes a Hop_Lim part of the IOAM Trace Option-Type(s) includes a Hop_Lim field
field associated to the node identifier to detect missed nodes, i.e. associated to the node identifier to detect missed nodes, i.e.,
"holes" in the trace. Monitoring/Analytics system(s) could utilize "holes" in the trace. Monitoring/Analytics systems could utilize
this information to account for the presence of IOAM unaware nodes in this information to account for the presence of IOAM-unaware nodes in
the network. the network.
8. IOAM Manageability 8. IOAM Manageability
The YANG model for configuring IOAM in network nodes which support The YANG model for configuring IOAM in network nodes that support
IOAM is defined in [I-D.zhou-ippm-ioam-yang]. IOAM is defined in [IOAM-YANG].
A deployment can leverage IOAM profiles to limit the scope of IOAM A deployment can leverage IOAM profiles to limit the scope of IOAM
features, allowing simpler implementation, verification, and features, allowing simpler implementation, verification, and
interoperability testing in the context of specific use cases that do interoperability testing in the context of specific use cases that do
not require the full functionality of IOAM. An IOAM profile defines not require the full functionality of IOAM. An IOAM profile defines
a use case or a set of use cases for IOAM, and an associated set of a use case or a set of use cases for IOAM and an associated set of
rules that restrict the scope and features of the IOAM specification, rules that restrict the scope and features of the IOAM specification,
thereby limiting it to a subset of the full functionality. IOAM thereby limiting it to a subset of the full functionality. IOAM
profiles are defined in [I-D.mizrahi-ippm-ioam-profile]. profiles are defined in [IOAM-PROFILES].
For deployments where the IOAM capabilities of a node are unknown, For deployments where the IOAM capabilities of a node are unknown,
[I-D.ietf-ippm-ioam-conf-state] could be used to discover the enabled [RFC9359] could be used to discover the enabled IOAM capabilities of
IOAM capabilities of nodes. nodes.
9. IANA Considerations 9. IANA Considerations
This document does not request any IANA actions. This document has no IANA actions.
10. Security Considerations 10. Security Considerations
As discussed in [RFC7276], a successful attack on an OAM protocol in As discussed in [RFC7276], a successful attack on an OAM protocol in
general, and specifically on IOAM, can prevent the detection of general and, specifically, on IOAM can prevent the detection of
failures or anomalies, or create a false illusion of nonexistent failures or anomalies or can create a false illusion of nonexistent
ones. ones.
The Proof of Transit Option-Type (Section 4.2) is used for verifying The Proof of Transit Option-Type (Section 4.2) is used for verifying
the path of data packets. The security considerations of POT are the path of data packets. The security considerations of POT are
further discussed in [I-D.ietf-sfc-proof-of-transit]. further discussed in [PROOF-OF-TRANSIT].
Security considerations related to the use of IOAM flags, in Security considerations related to the use of IOAM flags,
particular the loopback flag are found in [RFC9322]. particularly the Loopback flag, are found in [RFC9322].
IOAM data can be subject to eavesdropping. Although the IOAM data can be subject to eavesdropping. Although the
confidentiality of the user data is not at risk in this context, the confidentiality of the user data is not at risk in this context, the
IOAM data elements can be used for network reconnaissance, allowing IOAM data elements can be used for network reconnaissance, allowing
attackers to collect information about network paths, performance, attackers to collect information about network paths, performance,
queue states, buffer occupancy and other information. Recon is an queue states, buffer occupancy, and other information. Recon is an
improbable security threat in an IOAM deployment that is within a improbable security threat in an IOAM deployment that is within a
confined physical domain. However, in deployments that are not confined physical domain. However, in deployments that are not
confined to a single LAN, but span multiple interconnected sites (for confined to a single LAN but span multiple interconnected sites (for
example, using an overlay network), the inter-site links are expected example, using an overlay network), the inter-site links are expected
to be secured (e.g., by IPsec) in order to avoid external to be secured (e.g., by IPsec) in order to avoid external
eavesdropping and introduction of malicious or false data. Another eavesdropping and introduction of malicious or false data. Another
possible mitigation approach is to use the "direct exporting" mode possible mitigation approach is to use "Direct Exporting" [RFC9326].
[RFC9326]. In this case the IOAM related trace information would not In this case, the IOAM-related trace information would not be
be available in the customer data packets, but would trigger available in the customer data packets but would trigger the
exporting of (secured) packet-related IOAM information at every node. exporting of (secured) packet-related IOAM information at every node.
IOAM data export and securing IOAM data export is outside the scope IOAM data export and securing IOAM data export is outside the scope
of this document. of this document.
IOAM can be used as a means for implementing Denial of Service (DoS) IOAM can be used as a means for implementing or amplifying Denial-of-
attacks, or for amplifying them. For example, a malicious attacker Service (DoS) attacks. For example, a malicious attacker can add an
can add an IOAM header to packets or modify an IOAM header in en IOAM header to packets or modify an IOAM header in en route packets
route packets in order to consume the resources of network devices in order to consume the resources of network devices that take part
that take part in IOAM or collectors that analyze the IOAM data. in IOAM or collectors that analyze the IOAM data. Another example is
Another example is a packet length attack, in which an attacker a packet-length attack, in which an attacker pushes headers
pushes headers associated with IOAM Option-Types into data packets, associated with IOAM Option-Types into data packets, causing these
causing these packets to be increased beyond the MTU size, resulting packets to be increased beyond the MTU size, resulting in
in fragmentation or in packet drops. Such DoS attacks can be fragmentation or in packet drops. Such DoS attacks can be mitigated
mitigated by deploying IOAM in confined administrative domains, and by deploying IOAM in confined administrative domains and by limiting
by limiting the rate and/or the percentage of packets that an IOAM the rate and/or the percentage of packets that an IOAM encapsulating
encapsulating node adds IOAM information to, as well as limiting rate node adds IOAM information to as well as limiting rate and/or
and/or percentage of packets that an IOAM transit or an IOAM percentage of packets that an IOAM transit or an IOAM decapsulating
decapsulating node creates to export IOAM information extracted from node creates to export IOAM information extracted from the data
the data packets that carry IOAM information. packets that carry IOAM information.
Even though IOAM focused on limited domains [RFC8799], there might be Even though IOAM focused on limited domains [RFC8799], there might be
deployments for which it is important for IOAM transit nodes and IOAM deployments for which it is important for IOAM transit nodes and IOAM
decapsulating nodes to know that the data received hadn't been decapsulating nodes to know that the data received haven't been
tampered with. In those cases, the IOAM data should be integrity tampered with. In those cases, the IOAM data should be integrity
protected. Integrity protection of IOAM data fields is described in protected. Integrity protection of IOAM data fields is described in
[I-D.ietf-ippm-ioam-data-integrity]. In addition, since IOAM options [IOAM-DATA-INTEGRITY]. In addition, since IOAM options may include
may include timestamps, if network devices use synchronization timestamps, if network devices use synchronization protocols, then
protocols then any attack on the time protocol [RFC7384] can any attack on the time protocol [RFC7384] can compromise the
compromise the integrity of the timestamp-related data fields. integrity of the timestamp-related data fields. Synchronization
Synchronization attacks can be mitigated by combining a secured time attacks can be mitigated by combining a secured time distribution
distribution scheme, e.g., [RFC8915], and by using redundant clock scheme, e.g., [RFC8915], and by using redundant clock sources
sources [RFC5905] and/or redundant network paths for the time [RFC5905] and/or redundant network paths for the time distribution
distribution protocol [RFC8039]. protocol [RFC8039].
At the management plane, attacks may be implemented by misconfiguring At the management plane, attacks may be implemented by misconfiguring
or by maliciously configuring IOAM-enabled nodes in a way that or by maliciously configuring IOAM-enabled nodes in a way that
enables other attacks. Thus, IOAM configuration should be secured in enables other attacks. Thus, IOAM configuration should be secured in
a way that authenticates authorized users and verifies the integrity a way that authenticates authorized users and verifies the integrity
of configuration procedures. of configuration procedures.
Notably, IOAM is expected to be deployed in limited network domains Notably, IOAM is expected to be deployed in limited network domains
([RFC8799]), thus confining the potential attack vectors to within [RFC8799], thus, confining the potential attack vectors within the
the limited domain. Indeed, in order to limit the scope of threats limited domain. Indeed, in order to limit the scope of threats
to within the current network domain the network operator is expected within the current network domain, the network operator is expected
to enforce policies that prevent IOAM traffic from leaking outside to enforce policies that prevent IOAM traffic from leaking outside
the IOAM domain, and prevent an attacker from introducing malicious the IOAM-Domain and prevent an attacker from introducing malicious or
or false IOAM data to be processed and used within the IOAM domain. false IOAM data to be processed and used within the IOAM-Domain.
IOAM data leakage could lead to privacy issues. Consider an IOAM IOAM data leakage could lead to privacy issues. Consider an IOAM
encapsulating node that is a home gateway in an operator's network. encapsulating node that is a home gateway in an operator's network.
A home gateway is often identified with an individual, and revealing A home gateway is often identified with an individual. Revealing
IOAM data such as "IOAM node identifier" or geolocation information IOAM data, such as "IOAM node identifier" or geolocation information
outside of the limited domain could be harmful for that user. Note outside of the limited domain, could be harmful for that user. Note
that the Direct Export mode [RFC9326] can mitigate the potential that Direct Exporting [RFC9326] can mitigate the potential threat of
threat of IOAM data leaking through data packets. IOAM data leaking through data packets.
11. Acknowledgements
The authors would like to thank Tal Mizrahi, Eric Vyncke, Nalini
Elkins, Srihari Raghavan, Ranganathan T S, Barak Gafni, Karthik Babu
Harichandra Babu, Akshaya Nadahalli, LJ Wobker, Erik Nordmark,
Vengada Prasad Govindan, Andrew Yourtchenko, Aviv Kfir, Tianran Zhou,
Zhenbin (Robin), Joe Clarke, Al Morton, Tom Herbet, Haoyu song, and
Mickey Spiegel for the comments and advice on IOAM.
12. Informative References
[I-D.ali-spring-ioam-srv6]
Ali, Z., Gandhi, R., Filsfils, C., Brockners, F., Nainar,
N. K., Pignataro, C., Li, C., Chen, M., and G. Dawra,
"Segment Routing Header encapsulation for In-situ OAM
Data", Work in Progress, Internet-Draft, draft-ali-spring-
ioam-srv6-06, 10 July 2022,
<https://www.ietf.org/archive/id/draft-ali-spring-ioam-
srv6-06.txt>.
[I-D.brockners-ippm-ioam-geneve]
Brockners, F., Bhandari, S., Govindan, V. P., Pignataro,
C., Nainar, N. K., Gredler, H., Leddy, J., Youell, S.,
Mizrahi, T., Lapukhov, P., Gafni, B., Kfir, A., and M.
Spiegel, "Geneve encapsulation for In-situ OAM Data", Work
in Progress, Internet-Draft, draft-brockners-ippm-ioam-
geneve-05, 19 November 2020,
<https://www.ietf.org/archive/id/draft-brockners-ippm-
ioam-geneve-05.txt>.
[I-D.brockners-ippm-ioam-vxlan-gpe]
Brockners, F., Bhandari, S., Govindan, V. P., Pignataro,
C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir,
A., Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE
Encapsulation for In-situ OAM Data", Work in Progress,
Internet-Draft, draft-brockners-ippm-ioam-vxlan-gpe-03, 4
November 2019, <https://www.ietf.org/archive/id/draft-
brockners-ippm-ioam-vxlan-gpe-03.txt>.
[I-D.gandhi-spring-ioam-sr-mpls] 11. Informative References
Gandhi, R., Ali, Z., Filsfils, C., Brockners, F., Wen, B.,
and V. Kozak, "Segment Routing with MPLS Data Plane
Encapsulation for In-situ OAM Data", Work in Progress,
Internet-Draft, draft-gandhi-spring-ioam-sr-mpls-02, 22
August 2019, <https://www.ietf.org/archive/id/draft-
gandhi-spring-ioam-sr-mpls-02.txt>.
[I-D.ietf-ippm-ioam-conf-state] [BIER-IOAM]
Min, X., Mirsky, G., and L. Bo, "Echo Request/Reply for Min, X., Zhang, Z., Liu, Y., Nainar, N., and C. Pignataro,
Enabled In-situ OAM Capabilities", Work in Progress, "BIER Encapsulation for IOAM Data", Work in Progress,
Internet-Draft, draft-ietf-ippm-ioam-conf-state-10, 21 Internet-Draft, draft-xzlnp-bier-ioam-05, 27 January 2023,
November 2022, <https://www.ietf.org/archive/id/draft- <https://datatracker.ietf.org/doc/html/draft-xzlnp-bier-
ietf-ippm-ioam-conf-state-10.txt>. ioam-05>.
[I-D.ietf-ippm-ioam-data-integrity] [IOAM-DATA-INTEGRITY]
Brockners, F., Bhandari, S., Mizrahi, T., and J. Iurman, Brockners, F., Bhandari, S., Mizrahi, T., and J. Iurman,
"Integrity of In-situ OAM Data Fields", Work in Progress, "Integrity of In-situ OAM Data Fields", Work in Progress,
Internet-Draft, draft-ietf-ippm-ioam-data-integrity-03, 24 Internet-Draft, draft-ietf-ippm-ioam-data-integrity-03, 24
November 2022, <https://www.ietf.org/archive/id/draft- November 2022, <https://datatracker.ietf.org/doc/html/
ietf-ippm-ioam-data-integrity-03.txt>. draft-ietf-ippm-ioam-data-integrity-03>.
[I-D.ietf-ippm-ioam-ipv6-options] [IOAM-ETH] Weis, B., Ed., Brockners, F., Ed., Hill, C., Bhandari, S.,
Bhandari, S. and F. Brockners, "In-situ OAM IPv6 Options", Govindan, V., Pignataro, C., Ed., Nainar, N., Ed.,
Work in Progress, Internet-Draft, draft-ietf-ippm-ioam- Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A.,
ipv6-options-09, 11 October 2022, Gafni, B., Lapukhov, P., and M. Spiegel, "EtherType
<https://www.ietf.org/archive/id/draft-ietf-ippm-ioam- Protocol Identification of In-situ OAM Data", Work in
ipv6-options-09.txt>. Progress, Internet-Draft, draft-weis-ippm-ioam-eth-05, 21
February 2022, <https://datatracker.ietf.org/doc/html/
draft-weis-ippm-ioam-eth-05>.
[I-D.ietf-nvo3-vxlan-gpe] [IOAM-GENEVE]
Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol Brockners, F., Ed., Bhandari, S., Govindan, V., Pignataro,
Extension for VXLAN (VXLAN-GPE)", Work in Progress, C., Ed., Nainar, N., Ed., Gredler, H., Leddy, J., Youell,
Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12, 22 September S., Mizrahi, T., Lapukhov, P., Gafni, B., Kfir, A., and M.
2021, <https://www.ietf.org/archive/id/draft-ietf-nvo3- Spiegel, "Geneve encapsulation for In-situ OAM Data", Work
vxlan-gpe-12.txt>. in Progress, Internet-Draft, draft-brockners-ippm-ioam-
geneve-05, 19 November 2020,
<https://datatracker.ietf.org/doc/html/draft-brockners-
ippm-ioam-geneve-05>.
[I-D.ietf-sfc-ioam-nsh] [IOAM-IPV6-OPTIONS]
Brockners, F. and S. Bhandari, "Network Service Header Bhandari, S., Ed. and F. Brockners, Ed., "In-situ OAM IPv6
(NSH) Encapsulation for In-situ OAM (IOAM) Data", Work in Options", Work in Progress, Internet-Draft, draft-ietf-
Progress, Internet-Draft, draft-ietf-sfc-ioam-nsh-11, 30 ippm-ioam-ipv6-options-10, 28 February 2023,
September 2022, <https://www.ietf.org/archive/id/draft- <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-
ietf-sfc-ioam-nsh-11.txt>. ioam-ipv6-options-10>.
[I-D.ietf-sfc-proof-of-transit] [IOAM-NSH] Brockners, F., Ed. and S. Bhandari, Ed., "Network Service
Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. Header (NSH) Encapsulation for In-situ OAM (IOAM) Data",
Youell, "Proof of Transit", Work in Progress, Internet- Work in Progress, Internet-Draft, draft-ietf-sfc-ioam-nsh-
Draft, draft-ietf-sfc-proof-of-transit-08, 1 November 11, 30 September 2022,
2020, <https://www.ietf.org/archive/id/draft-ietf-sfc- <https://datatracker.ietf.org/doc/html/draft-ietf-sfc-
proof-of-transit-08.txt>. ioam-nsh-11>.
[I-D.mizrahi-ippm-ioam-profile] [IOAM-PROFILES]
Mizrahi, T., Brockners, F., Bhandari, S., Sivakolundu, R., Mizrahi, T., Brockners, F., Bhandari, S., Ed.,
Pignataro, C., Kfir, A., Gafni, B., Spiegel, M., and T. Sivakolundu, R., Pignataro, C., Kfir, A., Gafni, B.,
Zhou, "In Situ OAM Profiles", Work in Progress, Internet- Spiegel, M., and T. Zhou, "In Situ OAM Profiles", Work in
Draft, draft-mizrahi-ippm-ioam-profile-06, 17 February Progress, Internet-Draft, draft-mizrahi-ippm-ioam-profile-
2022, <https://www.ietf.org/archive/id/draft-mizrahi-ippm- 06, 17 February 2022,
ioam-profile-06.txt>. <https://datatracker.ietf.org/doc/html/draft-mizrahi-ippm-
ioam-profile-06>.
[I-D.spiegel-ippm-ioam-rawexport] [IOAM-RAWEXPORT]
Spiegel, M., Brockners, F., Bhandari, S., and R. Spiegel, M., Brockners, F., Bhandari, S., and R.
Sivakolundu, "In-situ OAM raw data export with IPFIX", Sivakolundu, "In-situ OAM raw data export with IPFIX",
Work in Progress, Internet-Draft, draft-spiegel-ippm-ioam- Work in Progress, Internet-Draft, draft-spiegel-ippm-ioam-
rawexport-06, 21 February 2022, rawexport-06, 21 February 2022,
<https://www.ietf.org/archive/id/draft-spiegel-ippm-ioam- <https://datatracker.ietf.org/doc/html/draft-spiegel-ippm-
rawexport-06.txt>. ioam-rawexport-06>.
[I-D.weis-ippm-ioam-eth] [IOAM-SRV6]
Weis, B., Brockners, F., Hill, C., Bhandari, S., Govindan, Ali, Z., Gandhi, R., Filsfils, C., Brockners, F., Nainar,
V. P., Pignataro, C., Nainar, N. K., Gredler, H., Leddy, N., Pignataro, C., Li, C., Chen, M., and G. Dawra,
J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., "Segment Routing Header encapsulation for In-situ OAM
Lapukhov, P., and M. Spiegel, "EtherType Protocol Data", Work in Progress, Internet-Draft, draft-ali-spring-
Identification of In-situ OAM Data", Work in Progress, ioam-srv6-06, 10 July 2022,
Internet-Draft, draft-weis-ippm-ioam-eth-05, 21 February <https://datatracker.ietf.org/doc/html/draft-ali-spring-
2022, <https://www.ietf.org/archive/id/draft-weis-ippm- ioam-srv6-06>.
ioam-eth-05.txt>.
[I-D.xzlnp-bier-ioam] [IOAM-VXLAN-GPE]
Min, X., Zhang, Z., Liu, Y., Nainar, N. K., and C. Brockners, F., Bhandari, S., Govindan, V., Pignataro, C.,
Pignataro, "Bit Index Explicit Replication (BIER) Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A.,
Encapsulation for In-situ OAM (IOAM) Data", Work in Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE
Progress, Internet-Draft, draft-xzlnp-bier-ioam-04, 25 Encapsulation for In-situ OAM Data", Work in Progress,
July 2022, <https://www.ietf.org/archive/id/draft-xzlnp- Internet-Draft, draft-brockners-ipxpm-ioam-vxlan-gpe-03, 4
bier-ioam-04.txt>. November 2019, <https://datatracker.ietf.org/doc/html/
draft-brockners-ippm-ioam-vxlan-gpe-03>.
[I-D.zhou-ippm-ioam-yang] [IOAM-YANG]
Zhou, T., Guichard, J., Brockners, F., and S. Raghavan, "A Zhou, T., Ed., Guichard, J., Brockners, F., and S.
YANG Data Model for In-Situ OAM", Work in Progress, Raghavan, "A YANG Data Model for In-Situ OAM", Work in
Internet-Draft, draft-zhou-ippm-ioam-yang-08, 30 July Progress, Internet-Draft, draft-ietf-ippm-ioam-yang-06, 27
2020, <https://www.ietf.org/archive/id/draft-zhou-ippm- February 2023, <https://datatracker.ietf.org/doc/html/
ioam-yang-08.txt>. draft-ietf-ippm-ioam-yang-06>.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., Traina, P., [MPLS-IOAM]
and RFC Publisher, "Generic Routing Encapsulation (GRE)", Gandhi, R., Ed., Brockners, F., Wen, B., Decraene, B., and
RFC 2784, DOI 10.17487/RFC2784, March 2000, H. Song, "MPLS Data Plane Encapsulation for In Situ OAM
Data", Work in Progress, Internet-Draft, draft-gandhi-
mpls-ioam-10, 10 March 2023,
<https://datatracker.ietf.org/doc/html/draft-gandhi-mpls-
ioam-10>.
[PROOF-OF-TRANSIT]
Brockners, F., Ed., Bhandari, S., Ed., Mizrahi, T., Ed.,
Dara, S., and S. Youell, "Proof of Transit", Work in
Progress, Internet-Draft, draft-ietf-sfc-proof-of-transit-
08, 31 October 2020,
<https://datatracker.ietf.org/doc/html/draft-ietf-sfc-
proof-of-transit-08>.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
Traina, "Generic Routing Encapsulation (GRE)", RFC 2784,
DOI 10.17487/RFC2784, March 2000,
<https://www.rfc-editor.org/info/rfc2784>. <https://www.rfc-editor.org/info/rfc2784>.
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., Kasch, W., and [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
RFC Publisher, "Network Time Protocol Version 4: Protocol "Network Time Protocol Version 4: Protocol and Algorithms
and Algorithms Specification", RFC 5905, Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
DOI 10.17487/RFC5905, June 2010,
<https://www.rfc-editor.org/info/rfc5905>. <https://www.rfc-editor.org/info/rfc5905>.
[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., Weingarten, Y., [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y.
and RFC Publisher, "An Overview of Operations, Weingarten, "An Overview of Operations, Administration,
Administration, and Maintenance (OAM) Tools", RFC 7276, and Maintenance (OAM) Tools", RFC 7276,
DOI 10.17487/RFC7276, June 2014, DOI 10.17487/RFC7276, June 2014,
<https://www.rfc-editor.org/info/rfc7276>. <https://www.rfc-editor.org/info/rfc7276>.
[RFC7384] Mizrahi, T. and RFC Publisher, "Security Requirements of [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in
Time Protocols in Packet Switched Networks", RFC 7384, Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384,
DOI 10.17487/RFC7384, October 2014, October 2014, <https://www.rfc-editor.org/info/rfc7384>.
<https://www.rfc-editor.org/info/rfc7384>.
[RFC7665] Halpern, J., Ed., Pignataro, C., Ed., and RFC Publisher, [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function
"Service Function Chaining (SFC) Architecture", RFC 7665, Chaining (SFC) Architecture", RFC 7665,
DOI 10.17487/RFC7665, October 2015, DOI 10.17487/RFC7665, October 2015,
<https://www.rfc-editor.org/info/rfc7665>. <https://www.rfc-editor.org/info/rfc7665>.
[RFC7799] Morton, A. and RFC Publisher, "Active and Passive Metrics [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with
and Methods (with Hybrid Types In-Between)", RFC 7799, Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799,
DOI 10.17487/RFC7799, May 2016, May 2016, <https://www.rfc-editor.org/info/rfc7799>.
<https://www.rfc-editor.org/info/rfc7799>.
[RFC8039] Shpiner, A., Tse, R., Schelp, C., Mizrahi, T., and RFC [RFC8039] Shpiner, A., Tse, R., Schelp, C., and T. Mizrahi,
Publisher, "Multipath Time Synchronization", RFC 8039, "Multipath Time Synchronization", RFC 8039,
DOI 10.17487/RFC8039, December 2016, DOI 10.17487/RFC8039, December 2016,
<https://www.rfc-editor.org/info/rfc8039>. <https://www.rfc-editor.org/info/rfc8039>.
[RFC8279] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., [RFC8279] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
Przygienda, T., Aldrin, S., and RFC Publisher, "Multicast Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
Using Bit Index Explicit Replication (BIER)", RFC 8279, Explicit Replication (BIER)", RFC 8279,
DOI 10.17487/RFC8279, November 2017, DOI 10.17487/RFC8279, November 2017,
<https://www.rfc-editor.org/info/rfc8279>. <https://www.rfc-editor.org/info/rfc8279>.
[RFC8300] Quinn, P., Ed., Elzur, U., Ed., Pignataro, C., Ed., and [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,
RFC Publisher, "Network Service Header (NSH)", RFC 8300, "Network Service Header (NSH)", RFC 8300,
DOI 10.17487/RFC8300, January 2018, DOI 10.17487/RFC8300, January 2018,
<https://www.rfc-editor.org/info/rfc8300>. <https://www.rfc-editor.org/info/rfc8300>.
[RFC8799] Carpenter, B., Liu, B., and RFC Publisher, "Limited [RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet
Domains and Internet Protocols", RFC 8799, Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020,
DOI 10.17487/RFC8799, July 2020,
<https://www.rfc-editor.org/info/rfc8799>. <https://www.rfc-editor.org/info/rfc8799>.
[RFC8915] Franke, D., Sibold, D., Teichel, K., Dansarie, M., [RFC8915] Franke, D., Sibold, D., Teichel, K., Dansarie, M., and R.
Sundblad, R., and RFC Publisher, "Network Time Security Sundblad, "Network Time Security for the Network Time
for the Network Time Protocol", RFC 8915, Protocol", RFC 8915, DOI 10.17487/RFC8915, September 2020,
DOI 10.17487/RFC8915, September 2020,
<https://www.rfc-editor.org/info/rfc8915>. <https://www.rfc-editor.org/info/rfc8915>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., Sridhar, T., Ed., and RFC [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
Publisher, "Geneve: Generic Network Virtualization "Geneve: Generic Network Virtualization Encapsulation",
Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November RFC 8926, DOI 10.17487/RFC8926, November 2020,
2020, <https://www.rfc-editor.org/info/rfc8926>. <https://www.rfc-editor.org/info/rfc8926>.
[RFC9197] Brockners, F., Ed., Bhandari, S., Ed., Mizrahi, T., Ed., [RFC9197] Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi,
and RFC Publisher, "Data Fields for In Situ Operations, Ed., "Data Fields for In Situ Operations, Administration,
Administration, and Maintenance (IOAM)", RFC 9197, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197,
DOI 10.17487/RFC9197, May 2022, May 2022, <https://www.rfc-editor.org/info/rfc9197>.
<https://www.rfc-editor.org/info/rfc9197>.
[RFC9322] Mizrahi, T., Brockners, F., Bhandari, S., Gafni, B., [RFC9322] Mizrahi, T., Brockners, F., Bhandari, S., Gafni, B., and
Spiegel, M., and RFC Publisher, "In Situ Operations, M. Spiegel, "In Situ Operations, Administration, and
Administration, and Maintenance (IOAM) Loopback and Active Maintenance (IOAM) Loopback and Active Flags", RFC 9322,
Flags", RFC 9322, DOI 10.17487/RFC9322, November 2022, DOI 10.17487/RFC9322, November 2022,
<https://www.rfc-editor.org/info/rfc9322>. <https://www.rfc-editor.org/info/rfc9322>.
[RFC9326] Song, H., Gafni, B., Brockners, F., Bhandari, S., Mizrahi, [RFC9326] Song, H., Gafni, B., Brockners, F., Bhandari, S., and T.
T., and RFC Publisher, "In Situ Operations, Mizrahi, "In Situ Operations, Administration, and
Administration, and Maintenance (IOAM) Direct Exporting", Maintenance (IOAM) Direct Exporting", RFC 9326,
RFC 9326, DOI 10.17487/RFC9326, November 2022, DOI 10.17487/RFC9326, November 2022,
<https://www.rfc-editor.org/info/rfc9326>. <https://www.rfc-editor.org/info/rfc9326>.
[RFC9359] Min, X., Mirsky, G., and L. Bo, "Echo Request/Reply for
Enabled In Situ OAM (IOAM) Capabilities", RFC 9359,
DOI 10.17487/RFC9359, April 2023,
<https://www.rfc-editor.org/info/rfc9359>.
[VXLAN-GPE]
Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed.,
"Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work
in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12,
22 September 2021, <https://datatracker.ietf.org/doc/html/
draft-ietf-nvo3-vxlan-gpe-12>.
Acknowledgements
The authors would like to thank Tal Mizrahi, Eric Vyncke, Nalini
Elkins, Srihari Raghavan, Ranganathan T S, Barak Gafni, Karthik Babu
Harichandra Babu, Akshaya Nadahalli, LJ Wobker, Erik Nordmark,
Vengada Prasad Govindan, Andrew Yourtchenko, Aviv Kfir, Tianran Zhou,
Zhenbin (Robin), Joe Clarke, Al Morton, Tom Herbet, Haoyu Song, and
Mickey Spiegel for the comments and advice on IOAM.
Authors' Addresses Authors' Addresses
Frank Brockners (editor) Frank Brockners (editor)
Cisco Systems, Inc. Cisco Systems, Inc.
Hansaallee 249, 3rd Floor 3rd Floor
Hansaallee 249
40549 DUESSELDORF 40549 DUESSELDORF
Germany Germany
Email: fbrockne@cisco.com Email: fbrockne@cisco.com
Shwetha Bhandari (editor) Shwetha Bhandari (editor)
Thoughtspot Thoughtspot
3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 3rd Floor, Indiqube Orion
Bangalore, KARNATAKA 560 102 Garden Layout, HSR Layout
24th Main Rd
Bangalore 560 102
KARNATAKA
India India
Email: shwetha.bhandari@thoughtspot.com Email: shwetha.bhandari@thoughtspot.com
Daniel Bernier Daniel Bernier
Bell Canada Bell Canada
Canada Canada
Email: daniel.bernier@bell.ca Email: daniel.bernier@bell.ca
Tal Mizrahi (editor) Tal Mizrahi (editor)
Huawei Huawei
8-2 Matam 8-2 Matam
Haifa 3190501 Haifa 3190501
Israel Israel
Email: tal.mizrahi.phd@gmail.com Email: tal.mizrahi.phd@gmail.com
 End of changes. 170 change blocks. 
621 lines changed or deleted 624 lines changed or added

This html diff was produced by rfcdiff 1.48.