rfc9049.original   rfc9049.txt 
PANRG S. Dawkins, Ed. Internet Research Task Force (IRTF) S. Dawkins, Ed.
Internet-Draft Tencent America Request for Comments: 9049 Tencent America
Intended status: Informational 26 March 2021 Category: Informational June 2021
Expires: 27 September 2021 ISSN: 2070-1721
Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not Path Aware Networking: Obstacles to Deployment
Taken) (A Bestiary of Roads Not Taken)
draft-irtf-panrg-what-not-to-do-19
Abstract Abstract
At the first meeting of the Path Aware Networking Research Group, the This document is a product of the Path Aware Networking Research
research group agreed to catalog and analyze past efforts to develop Group (PANRG). At the first meeting of the PANRG, the Research Group
and deploy Path Aware techniques, most of which were unsuccessful or agreed to catalog and analyze past efforts to develop and deploy Path
at most partially successful, in order to extract insights and Aware techniques, most of which were unsuccessful or at most
lessons for path-aware networking researchers. partially successful, in order to extract insights and lessons for
Path Aware networking researchers.
This document contains that catalog and analysis. This document contains that catalog and analysis.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Research Task Force
and may be updated, replaced, or obsoleted by other documents at any (IRTF). The IRTF publishes the results of Internet-related research
time. It is inappropriate to use Internet-Drafts as reference and development activities. These results might not be suitable for
material or to cite them other than as "work in progress." deployment. This RFC represents the consensus of the Path Aware
Networking Research Group of the Internet Research Task Force (IRTF).
Documents approved for publication by the IRSG are not a candidate
for any level of Internet Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on 27 September 2021. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9049.
Copyright Notice Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. carefully, as they describe your rights and restrictions with respect
to this document.
Table of Contents Table of Contents
1. Introduction 1. Introduction
1.1. What Do "Path" and "Path Awareness" Mean in this Document? 1.1. What Do "Path" and "Path Awareness" Mean in This Document?
2. A Perspective On This Document 2. A Perspective on This Document
2.1. Notes for the Reader 2.1. Notes for the Reader
2.2. A Note About Path-Aware Techniques Included In This 2.2. A Note about Path Aware Techniques Included in This
Document Document
2.3. Venue for Discussion of this Document 2.3. Architectural Guidance
2.4. Architectural Guidance 2.4. Terminology Used in This Document
2.5. Terminology Used in this Document 2.5. Methodology for Contributions
2.6. Methodology for Contributions
3. Applying the Lessons We've Learned 3. Applying the Lessons We've Learned
4. Summary of Lessons Learned 4. Summary of Lessons Learned
4.1. Justifying Deployment 4.1. Justifying Deployment
4.2. Providing Benefits for Early Adopters 4.2. Providing Benefits for Early Adopters
4.3. Providing Benefits During Partial Deployment 4.3. Providing Benefits during Partial Deployment
4.4. Outperforming End-to-end Protocol Mechanisms 4.4. Outperforming End-to-End Protocol Mechanisms
4.5. Paying for Path Aware Techniques 4.5. Paying for Path Aware Techniques
4.6. Impact on Operational Practices 4.6. Impact on Operational Practices
4.7. Per-connection State 4.7. Per-Connection State
4.8. Keeping Traffic on Fast-paths 4.8. Keeping Traffic on Fast Paths
4.9. Endpoints Trusting Intermediate Nodes 4.9. Endpoints Trusting Intermediate Nodes
4.10. Intermediate Nodes Trusting Endpoints 4.10. Intermediate Nodes Trusting Endpoints
4.11. Reacting to Distant Signals 4.11. Reacting to Distant Signals
4.12. Support in Endpoint Protocol Stacks 4.12. Support in Endpoint Protocol Stacks
4.13. Planning For Failure 4.13. Planning for Failure
5. Future Work 5. Future Work
6. Contributions 6. Contributions
6.1. Stream Transport (ST, ST2, ST2+) 6.1. Stream Transport (ST, ST2, ST2+)
6.1.1. Reasons for Non-deployment 6.1.1. Reasons for Non-deployment
6.1.2. Lessons Learned. 6.1.2. Lessons Learned
6.2. Integrated Services (IntServ) 6.2. Integrated Services (IntServ)
6.2.1. Reasons for Non-deployment 6.2.1. Reasons for Non-deployment
6.2.2. Lessons Learned. 6.2.2. Lessons Learned
6.3. Quick-Start TCP 6.3. Quick-Start TCP
6.3.1. Reasons for Non-deployment 6.3.1. Reasons for Non-deployment
6.3.2. Lessons Learned 6.3.2. Lessons Learned
6.4. ICMP Source Quench 6.4. ICMP Source Quench
6.4.1. Reasons for Non-deployment 6.4.1. Reasons for Non-deployment
6.4.2. Lessons Learned 6.4.2. Lessons Learned
6.5. Triggers for Transport (TRIGTRAN) 6.5. Triggers for Transport (TRIGTRAN)
6.5.1. Reasons for Non-deployment 6.5.1. Reasons for Non-deployment
6.5.2. Lessons Learned. 6.5.2. Lessons Learned
6.6. Shim6 6.6. Shim6
6.6.1. Reasons for Non-deployment 6.6.1. Reasons for Non-deployment
6.6.2. Lessons Learned 6.6.2. Lessons Learned
6.6.3. Addendum on MultiPath TCP 6.6.3. Addendum on Multipath TCP
6.7. Next Steps in Signaling (NSIS) 6.7. Next Steps in Signaling (NSIS)
6.7.1. Reasons for Non-deployment 6.7.1. Reasons for Non-deployment
6.7.2. Lessons Learned 6.7.2. Lessons Learned
6.8. IPv6 Flow Label 6.8. IPv6 Flow Labels
6.8.1. Reasons for Non-deployment 6.8.1. Reasons for Non-deployment
6.8.2. Lessons Learned 6.8.2. Lessons Learned
6.9. Explicit Congestion Notification (ECN) 6.9. Explicit Congestion Notification (ECN)
6.9.1. Reasons for Non-deployment 6.9.1. Reasons for Non-deployment
6.9.2. Lessons Learned 6.9.2. Lessons Learned
7. Security Considerations 7. Security Considerations
8. IANA Considerations 8. IANA Considerations
9. Acknowledgments 9. Informative References
10. Informative References Acknowledgments
Author's Address Author's Address
1. Introduction 1. Introduction
This document describes the lessons that IETF participants have This document describes the lessons that IETF participants have
learned (and learned the hard way) about Path Aware Networking over a learned (and learned the hard way) about Path Aware networking over a
period of several decades, and provides an analysis of reasons why period of several decades. It also provides an analysis of reasons
various Path Aware Networking techniques have seen limited or no why various Path Aware techniques have seen limited or no deployment.
deployment.
1.1. What Do "Path" and "Path Awareness" Mean in this Document? This document represents the consensus of the Path Aware Networking
Research Group (PANRG).
1.1. What Do "Path" and "Path Awareness" Mean in This Document?
One of the first questions reviewers of this document have asked is One of the first questions reviewers of this document have asked is
"what's the definition of a path, and what's the definition of path "What's the definition of a Path, and what's the definition of Path
awareness?" That is not an easy question to answer for this Awareness?" That is not an easy question to answer for this
document. document.
These terms have definitions in other [PANRG] documents, and are These terms have definitions in other PANRG documents [PANRG] and are
still the subject of some discussion in the research group, as of the still the subject of some discussion in the Research Group, as of the
date of this document. But because this document reflects work date of this document. But because this document reflects work
performed over several decades, the technologies described in performed over several decades, the technologies described in
Section 6 significantly predate the current definitions of "path" and Section 6 significantly predate the current definitions of "Path" and
"path aware" in use in the Path Aware Networking Research Group, and "Path Aware" in use in the Path Aware Networking Research Group, and
it is unlikely that all the contributors to Section 6 would have had it is unlikely that all the contributors to Section 6 would have had
the same understanding of these terms. Those technologies were the same understanding of these terms. Those technologies were
considered "path aware" in early PANRG discussions, and so are considered "Path Aware" in early PANRG discussions and so are
included in this retrospective document. included in this retrospective document.
It is worth noting that the definitions of "path" and "path aware" in It is worth noting that the definitions of "Path" and "Path Aware" in
[I-D.irtf-panrg-path-properties] would apply to path aware networking [PANRG-PATH-PROPERTIES] would apply to Path Aware techniques at a
techniques at a number of levels of the Internet protocol number of levels of the Internet protocol architecture ([RFC1122],
architecture ([RFC1122], plus several decades of refinements), but plus several decades of refinements), but the contributions received
the contributions received for this document tended to target the for this document tended to target the transport layer and to treat a
Transport Layer, and to treat a "path" constructed by routers as a "Path" constructed by routers as opaque. It would be useful to
"black box". It would be useful to consider how applicable the consider how applicable the Lessons Learned cataloged in this
Lessons Learned cataloged in this document are, at other layers, and document are, at other layers, and that would be a fine topic for
that would be a fine topic for follow-on research. follow-on research.
The current definition of "Path" in the Path Aware Networking The current definition of "Path" in the Path Aware Networking
Research Group appears in Section 2 ("Terminology") in Research Group appears in Section 2 ("Terminology") in
[I-D.irtf-panrg-path-properties]. That definition is included here [PANRG-PATH-PROPERTIES]. That definition is included here as a
as a convenience to the reader. convenience to the reader.
| Path: A sequence of adjacent path elements over which a packet can | Path: A sequence of adjacent path elements over which a packet can
| be transmitted, starting and ending with a node. A path is | be transmitted, starting and ending with a node. A path is
| unidirectional. Paths are time-dependent, i.e., the sequence of | unidirectional. Paths are time-dependent, i.e., the sequence of
| path elements over which packets are sent from one node to another | path elements over which packets are sent from one node to another
| may change. A path is defined between two nodes. For multicast | may change. A path is defined between two nodes. For multicast
| or broadcast, a packet may be sent by one node and received by | or broadcast, a packet may be sent by one node and received by
| multiple nodes. In this case, the packet is sent over multiple | multiple nodes. In this case, the packet is sent over multiple
| paths at once, one path for each combination of sending and | paths at once, one path for each combination of sending and
| receiving node; these paths do not have to be disjoint. Note that | receiving node; these paths do not have to be disjoint. Note that
| an entity may have only partial visibility of the path elements | an entity may have only partial visibility of the path elements
| that comprise a path and visibility may change over time. | that comprise a path and visibility may change over time.
| Different entities may have different visibility of a path and/or | Different entities may have different visibility of a path and/or
| treat path elements at different levels of abstraction. | treat path elements at different levels of abstraction.
The current definition of "Path Awareness", used by the Path Aware The current definition of Path Awareness, used by the Path Aware
Networking Research Group, appears in Section 1.1 ("Definition") in Networking Research Group, appears in Section 1.1 ("Definition") in
[I-D.irtf-panrg-questions]. That definition is included here as a [PANRG-QUESTIONS]. That definition is included here as a convenience
convenience to the reader. to the reader.
| For purposes of this document, "path aware networking" describes | For purposes of this document, "path aware networking" describes
| endpoint discovery of the properties of paths they use for | endpoint discovery of the properties of paths they use for
| communication, and endpoint reaction to these properties that | communication across an internetwork, and endpoint reaction to
| affects routing and/or transmission; note that this can and | these properties that affects routing and/or data transfer. Note
| already does happen to some extent in the current Internet | that this can and already does happen to some extent in the
| architecture. Expanding on this definition, a "path aware | current Internet architecture; this definition expands current
| internetwork" is one in which endpoint discovery of path | techniques of path discovery and manipulation to cross
| properties and endpoint selection of paths used by traffic | administrative domain boundaries and up to the transport and
| exchanged by the endpoint are explicitly supported, regardless of | application layers at the endpoints.
| the specific design of the protocol features which enable this |
| discovery and selection. | Expanding on this definition, a "path aware internetwork" is one
| in which endpoint discovery of path properties and endpoint
| selection of paths used by traffic exchanged by the endpoint are
| explicitly supported, regardless of the specific design of the
| protocol features which enable this discovery and selection.
2. A Perspective On This Document 2. A Perspective on This Document
At the first meeting of the Path Aware Networking Research Group At the first meeting of the Path Aware Networking Research Group
[PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion
of "A Decade of Path Awareness" [PATH-Decade], on attempts, which of "A Decade of Path Awareness" [PATH-Decade], on attempts, which
were mostly unsuccessful for a variety of reasons, to exploit Path were mostly unsuccessful for a variety of reasons, to exploit Path
Aware techniques and achieve a variety of goals over the past decade. Aware techniques and achieve a variety of goals over the past decade.
At the end of that discussion, two things were abundantly clear. At the end of that discussion, two things were abundantly clear.
* The Internet community has accumulated considerable experience * The Internet community has accumulated considerable experience
with many Path Aware techniques over a long period of time, and with many Path Aware techniques over a long period of time, and
* Although some path aware techniques have been deployed (for * Although some Path Aware techniques have been deployed (for
example, Differentiated Services, or DiffServ [RFC2475]), most of example, Differentiated Services, or Diffserv [RFC2475]), most of
these techniques haven't seen widespread adoption and deplyment. these techniques haven't seen widespread adoption and deployment.
Even "successful" techniques like DiffServ can face obstacles that Even "successful" techniques like Diffserv can face obstacles that
prevents wider usage. The reasons for non-adoption and limited prevent wider usage. The reasons for non-adoption and limited
adoption and deployment are many, and are worthy of study. adoption and deployment are many and are worthy of study.
The meta-lessons from that experience were The meta-lessons from that experience were as follows:
* Path aware networking has been more Research than Engineering, so * Path Aware networking has been more Research than Engineering, so
establishing an IRTF Research Group for Path Aware Networking is establishing an IRTF Research Group for Path Aware networking was
the right thing to do [RFC7418]. the right thing to do [RFC7418].
* Analyzing a catalog of past experience to learn the reasons for * Analyzing a catalog of past experience to learn the reasons for
non-adoption would be a great first step for the Research Group. non-adoption would be a great first step for the Research Group.
Allison Mankin, as IRTF Chair, officially chartered the Path Aware Allison Mankin, as IRTF Chair, officially chartered the Path Aware
Networking Research Group in July, 2018. Networking Research Group in July 2018.
This document contains the analysis performed by that research group This document contains the analysis performed by that Research Group
(Section 4), based on that catalog (Section 6). (Section 4), based on that catalog (Section 6).
This document represents the consensus of the Path Aware Networking
Research Group.
2.1. Notes for the Reader 2.1. Notes for the Reader
This Informational document discusses Path Aware protocol mechanisms This Informational document discusses Path Aware protocol mechanisms
considered, and in some cases standardized, by the Internet considered, and in some cases standardized, by the Internet
Engineering Task Force (IETF), and considers Lessons Learned from Engineering Task Force (IETF), and it considers Lessons Learned from
those mechanisms. The intention is to inform the work of protocol those mechanisms. The intention is to inform the work of protocol
designers, whether in the IRTF, the IETF, or elsewhere in the designers, whether in the IRTF, the IETF, or elsewhere in the
Internet ecosystem. Internet ecosystem.
As an Informational document published in the IRTF stream, this As an Informational document published in the IRTF Stream, this
document has no authority beyond the quality of the analysis it document has no authority beyond the quality of the analysis it
contains. contains.
2.2. A Note About Path-Aware Techniques Included In This Document 2.2. A Note about Path Aware Techniques Included in This Document
This document does not catalog every proposed path aware networking This document does not catalog every proposed Path Aware technique
technique that was not adopted and deployed. Instead, we limited our that was not adopted and deployed. Instead, we limited our focus to
focus to technologies that passed through the IETF community, and technologies that passed through the IETF community and still
still identified enough techniques to provide background for the identified enough techniques to provide background for the lessons
lessons included in Section 4 to inform researchers and protocol included in Section 4 to inform researchers and protocol engineers in
engineers in their work. their work.
No shame is intended for the techniques included in this document. No shame is intended for the techniques included in this document.
As shown in Section 4, the quality of specific techniques had little As shown in Section 4, the quality of specific techniques had little
to do with whether they were deployed or not. Based on the to do with whether they were deployed or not. Based on the
techniques cataloged in this document, it is likely that when these techniques cataloged in this document, it is likely that when these
techniques were put forward, the proponents were trying to engineer techniques were put forward, the proponents were trying to engineer
something that could not be engineered without first carrying out something that could not be engineered without first carrying out
research. Actual shame would be failing to learn from experience, research. Actual shame would be failing to learn from experience and
and failing to share that experience with other networking failing to share that experience with other networking researchers
researchers and engineers. and engineers.
2.3. Venue for Discussion of this Document
(RFC Editor: please remove this section before publication)
Discussion of specific contributed experiences and this document in
general should take place on the PANRG mailing list.
2.4. Architectural Guidance 2.3. Architectural Guidance
As background for understanding the Lessons Learned contained in this As background for understanding the Lessons Learned contained in this
document, the reader is encouraged to become familiar with the document, the reader is encouraged to become familiar with the
Internet Architecture Board's documents on "What Makes for a Internet Architecture Board's documents on "What Makes for a
Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption
and Subsequent Transitions" [RFC8170]. and Subsequent Transitions" [RFC8170].
Although these two documents do not specifically target path-aware Although these two documents do not specifically target Path Aware
networking protocols, they are helpful resources for readers seeking networking protocols, they are helpful resources for readers seeking
to improve their understanding of considerations for successful to improve their understanding of considerations for successful
adoption and deployment of any protocol. For example, the Basic adoption and deployment of any protocol. For example, the basic
Success Factors described in Setion 2.1 of [RFC5218] are helpful for success factors described in Section 2.1 of [RFC5218] are helpful for
readers of this document. readers of this document.
Because there is an economic aspect to decisions about deployment, Because there is an economic aspect to decisions about deployment,
the IAB Workshop on Internet Technology Adoption and Transition the IAB Workshop on Internet Technology Adoption and Transition
[ITAT] report [RFC7305] also provides food for thought. [ITAT] report [RFC7305] also provides food for thought.
Several of the Lessons Learned in Section 4 reflect considerations Several of the Lessons Learned in Section 4 reflect considerations
described in [RFC5218], [RFC7305], and [RFC8170]. described in [RFC5218], [RFC7305], and [RFC8170].
2.5. Terminology Used in this Document 2.4. Terminology Used in This Document
The terms Node and Element in this document have the meaning defined The terms "node" and "element" in this document have the meaning
in [I-D.irtf-panrg-path-properties]. defined in [PANRG-PATH-PROPERTIES].
2.6. Methodology for Contributions 2.5. Methodology for Contributions
This document grew out of contributions by various IETF participants This document grew out of contributions by various IETF participants
with experience with one or more Path Aware Networking techniques. with experience with one or more Path Aware techniques.
There are many things that could be said about the Path Aware There are many things that could be said about the Path Aware
networking techniques that have been developed. For the purposes of techniques that have been developed. For the purposes of this
this document, contributors were requested to provide document, contributors were requested to provide
* the name of a technique, including an abbreviation if one was used * the name of a technique, including an abbreviation if one was
used.
* if available, a long-term pointer to the best reference describing * if available, a long-term pointer to the best reference describing
the technique the technique.
* a short description of the problem the technique was intended to * a short description of the problem the technique was intended to
solve solve.
* a short description of the reasons why the technique wasn't * a short description of the reasons why the technique wasn't
adopted adopted.
* a short statement of the lessons that researchers can learn from * a short statement of the lessons that researchers can learn from
our experience with this technique. our experience with this technique.
3. Applying the Lessons We've Learned 3. Applying the Lessons We've Learned
The initial scope for this document was roughly "what mistakes have The initial scope for this document was roughly "What mistakes have
we made in the decade prior to [PANRG-99], that we shouldn't make we made in the decade prior to [PANRG-99], that we shouldn't make
again". Some of the contributions in Section 6 predate the initial again?" Some of the contributions in Section 6 predate the initial
scope. The earliest Path-Aware Networking technique referred to in scope. The earliest Path Aware technique referred to in Section 6 is
Section 6 is Section 6.1, published in the late 1970s. Given that [IEN-119], which was published in the late 1970s; see Section 6.1.
the networking ecosystem has evolved continuously, it seems Given that the networking ecosystem has evolved continuously, it
reasonable to consider how to apply these lessons. seems reasonable to consider how to apply these lessons.
The PANRG Research Group reviewed the Lessons Learned (Section 4) The PANRG reviewed the Lessons Learned (Section 4) contained in the
contained in the May 23, 2019 version of this document at IETF 105 May 23, 2019 draft version of this document at IETF 105
[PANRG-105-Min], and carried out additional discussion at IETF 106 [PANRG-105-Min] and carried out additional discussion at IETF 106
[PANRG-106-Min]. Table 1 provides the "sense of the room" about each [PANRG-106-Min]. Table 1 provides the "sense of the room" about each
lesson after those discussions. The intention was to capture whether lesson after those discussions. The intention was to capture whether
a specific lesson seems to be a specific lesson seems to be
* "Invariant" - well-understood and is likely to be applicable for * "Invariant" - well-understood and is likely to be applicable for
any proposed Path Aware Networking solution. any proposed Path Aware networking solution.
* "Variable" - has impeded deployment in the past, but might not be * "Variable" - has impeded deployment in the past but might not be
applicable in a specific technique. Engineering analysis to applicable in a specific technique. Engineering analysis to
understand whether the lesson is applicable is prudent. understand whether the lesson is applicable is prudent.
* "Not Now" - this characteristic tends to turn up a minefield full * "Not Now" - a characteristic that tends to turn up a minefield
of dragons, and prudent network engineers will wish to avoid full of dragons. Prudent network engineers will wish to avoid
gambling on a technique that relies on this, until something gambling on a technique that relies on this, until something
significant changes significant changes.
Section 6.9 on ECN was added during the review and approval process, Section 6.9 on Explicit Congestion Notification (ECN) was added
based on a question from Martin Duke. That section, along with its during the review and approval process, based on a question from
Lessons Learned and place in the "Invariant"/"Variable"/"Not Now" Martin Duke. Section 6.9, as contained in the March 8, 2021 draft
taxonomy, as contained in the March 8, 2021 version of this document, version of this document, was discussed at [PANRG-110] and is
was discussed at [PANRG-110]. summarized in Section 4.13, describing a new Lesson Learned.
+-----------------------------------------------------+-----------+ +=====================================================+===========+
| Lesson | Category | | Lesson | Category |
+=====================================================+===========+ +=====================================================+===========+
| Justifying Deployment (Section 4.1) | Invariant | | Justifying Deployment (Section 4.1) | Invariant |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Providing Benefits for Early Adopters (Section 4.2) | Invariant | | Providing Benefits for Early Adopters (Section 4.2) | Invariant |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Providing Benefits during Partial Deployment | Invariant | | Providing Benefits during Partial Deployment | Invariant |
| (Section 4.3) | | | (Section 4.3) | |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Outperforming End-to-end Protocol Mechanisms | Variable | | Outperforming End-to-End Protocol Mechanisms | Variable |
| (Section 4.4) | | | (Section 4.4) | |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Paying for Path Aware Techniques (Section 4.5) | Invariant | | Paying for Path Aware Techniques (Section 4.5) | Invariant |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Impact on Operational Practices (Section 4.6) | Invariant | | Impact on Operational Practices (Section 4.6) | Invariant |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Per-connection State (Section 4.7) | Variable | | Per-Connection State (Section 4.7) | Variable |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Keeping Traffic on Fast-paths (Section 4.8) | Variable | | Keeping Traffic on Fast Paths (Section 4.8) | Variable |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now | | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Intermediate Nodes Trusting Endpoints | Not Now | | Intermediate Nodes Trusting Endpoints | Not Now |
| (Section 4.10) | | | (Section 4.10) | |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Reacting to Distant Signals (Section 4.11) | Variable | | Reacting to Distant Signals (Section 4.11) | Variable |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Support in Endpoint Protocol Stacks (Section 4.12) | Variable | | Support in Endpoint Protocol Stacks (Section 4.12) | Variable |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
| Planning for Failure (Section 4.13) | Invariant | | Planning for Failure (Section 4.13) | Invariant |
+-----------------------------------------------------+-----------+ +-----------------------------------------------------+-----------+
Table 1 Table 1
"Justifying Deployment", "Providing Benefits for Early Adopters", "Justifying Deployment", "Providing Benefits for Early Adopters",
"Paying for Path Aware Techniques", "Impact on Operational Practice", "Paying for Path Aware Techniques", "Impact on Operational
and "Planning for Failure" were considered to be invariant - the Practices", and "Planning for Failure" were considered to be
sense of the room was that these would always be considerations for Invariant -- the sense of the room was that these would always be
any proposed Path Aware Technique. considerations for any proposed Path Aware technique.
"Providing Benefits During Partial Deployment" was added after IETF "Providing Benefits during Partial Deployment" was added after IETF
105, during research group last call, and is also considered to be 105, during Research Group Last Call, and is also considered to be
invariant. Invariant.
For "Outperforming End-to-end Protocol Mechanisms", there is a trade- For "Outperforming End-to-End Protocol Mechanisms", there is a trade-
off between improved performance from Path Aware Techniques and off between improved performance from Path Aware techniques and
additional complexity required by some Path Aware Techniques. additional complexity required by some Path Aware techniques.
* For example, if you can obtain the same understanding of path * For example, if you can obtain the same understanding of path
characteristics from measurements obtained over a few more round characteristics from measurements obtained over a few more round
trips, endpoint implementers are unlikely to be eager to add trips, endpoint implementers are unlikely to be eager to add
complexity, and many attributes can be measured from an endpoint, complexity, and many attributes can be measured from an endpoint,
without assistance from intermediate nodes. without assistance from intermediate nodes.
For "Per-connection State", the key questions discussed in the For "Per-Connection State", the key questions discussed in the
research group were "how much state" and "where state is maintained". Research Group were "how much state" and "where state is maintained".
* IntServ (Section 6.2) required state at every intermediate node * Integrated Services (IntServ) (Section 6.2) required state at
for every connection between two endpoints. As the Internet every participating intermediate node for every connection between
ecosystem has evolved, carrying many connections in a tunnel that two endpoints. As the Internet ecosystem has evolved, carrying
appears to intermediate nodes as a single connection has become many connections in a tunnel that appears to intermediate nodes as
more common, so that additional end-to-end connections don't add a single connection has become more common, so that additional
additional state to intermediate nodes between tunnel endpoints. end-to-end connections don't add additional state to intermediate
If these tunnels are encrypted, intermediate nodes between tunnel nodes between tunnel endpoints. If these tunnels are encrypted,
endpoints can't distinguish between connections, even if that were intermediate nodes between tunnel endpoints can't distinguish
desirable. between connections, even if that were desirable.
For "Keeping Traffic on Fast-paths", we noted that this was true for For "Keeping Traffic on Fast Paths", we noted that this was true for
many platforms, but not for all. many platforms, but not for all.
* For backbone routers, this is likely an invariant, but for * For backbone routers, this is likely an Invariant, but for
platforms that rely more on general-purpose computers to make platforms that rely more on general-purpose computers to make
forwarding decisions, this may not be a fatal flaw for Path Aware forwarding decisions, this may not be a fatal flaw for Path Aware
Networking techniques. techniques.
For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes
Trusting Endpoints", these lessons point to the broader need to Trusting Endpoints", these lessons point to the broader need to
revisit the Internet Threat Model. revisit the Internet Threat Model.
* We noted with relief that discussions about this were already * We noted with relief that discussions about this were already
underway in the IETF community at IETF 105 (see the Security Area underway in the IETF community at IETF 105 (see the Security Area
Open Meeting minutes [SAAG-105-Min] for discussion of Open Meeting minutes [SAAG-105-Min] for discussion of
[I-D.arkko-arch-internet-threat-model] and [I-D.farrell-etm]), and [INTERNET-THREAT-MODEL] and [FARRELL-ETM]), and the Internet
the Internet Architecture Board has created a mailing list for Architecture Board has created a mailing list for continued
continued discussions ([model-t]), but we recognize that there are discussions [model-t], but we recognize that there are Path Aware
Path Aware Networking aspects of this effort, requiring research. networking aspects of this effort, requiring research.
For "Reacting to Distant Signals", we noted that not all attributes For "Reacting to Distant Signals", we noted that not all attributes
are equal. are equal.
* If an attribute is stable over an extended period of time, is * If an attribute is stable over an extended period of time, is
difficult to observe via end-to-end mechanisms, and is valuable, difficult to observe via end-to-end mechanisms, and is valuable,
Path Aware Techniques that rely on that attribute to provide a Path Aware techniques that rely on that attribute to provide a
significant benefit become more attractive. significant benefit become more attractive.
* Analysis to help identify attributes that are useful enough to * Analysis to help identify attributes that are useful enough to
justify deployment of Path Aware techniques that make use of those justify deployment of Path Aware techniques that make use of those
attributes would be helpful. attributes would be helpful.
For "Support in Endpoint Protocol Stacks", we noted that Path Aware For "Support in Endpoint Protocol Stacks", we noted that Path Aware
applications must be able to identify and communicate requirements applications must be able to identify and communicate requirements
about path characteristics. about path characteristics.
* The de-facto sockets API has no way of signaling application * The de facto sockets API has no way of signaling application
expectations for the network path to the protocol stack. expectations for the network path to the protocol stack.
4. Summary of Lessons Learned 4. Summary of Lessons Learned
This section summarizes the Lessons Learned from the contributed This section summarizes the Lessons Learned from the contributed
subsections in Section 6. subsections in Section 6.
Each Lesson Learned is tagged with one or more contributions that Each Lesson Learned is tagged with one or more contributions that
encountered this obstacle as a significant impediment to deployment. encountered this obstacle as a significant impediment to deployment.
Other contributed techniques may have also encountered this obstacle, Other contributed techniques may have also encountered this obstacle,
but this obstacle may not have been the biggest impediment to but this obstacle may not have been the biggest impediment to
deployment for those techniques. deployment for those techniques.
It is useful to notice that sometimes an obstacle might impede It is useful to notice that sometimes an obstacle might impede
deployment, while at other times, the same obstacle might prevent deployment, while at other times, the same obstacle might prevent
adoption and deployment entirely. The research group discussed adoption and deployment entirely. The Research Group discussed
distinguishing between obstacles that impede and obstacles that distinguishing between obstacles that impede and obstacles that
prevent, but it appears that the boundary between "impede" and prevent, but it appears that the boundary between "impede" and
"prevent" can shift over time - some of the Lessons Learned are based "prevent" can shift over time -- some of the Lessons Learned are
on both Path Aware techniques that were not deployed, and Path Aware based on both a) Path Aware techniques that were not deployed and b)
techniques that were deployed, but were not deployed widely or Path Aware techniques that were deployed but were not deployed widely
quickly. See Section 6.6 and Section 6.6.3 as one example of this or quickly. See Sections 6.6 and 6.6.3 for examples of this shifting
shifting boundary. boundary.
4.1. Justifying Deployment 4.1. Justifying Deployment
The benefit of Path Awareness must be great enough to justify making The benefit of Path Awareness must be great enough to justify making
changes in an operational network. The colloquial U.S. American changes in an operational network. The colloquial U.S. American
English expression, "If it ain't broke, don't fix it" is a "best English expression, "If it ain't broke, don't fix it" is a "best
current practice" on today's Internet. (See Section 6.3, current practice" on today's Internet. (See Sections 6.3, 6.4, 6.5,
Section 6.4, Section 6.5, and Section 6.9, in addition to [RFC5218]). and 6.9, in addition to [RFC5218].)
4.2. Providing Benefits for Early Adopters 4.2. Providing Benefits for Early Adopters
Providing benefits for early adopters can be key - if everyone must Providing benefits for early adopters can be key -- if everyone must
deploy a technique in order for the technique to provide benefits, or deploy a technique in order for the technique to provide benefits, or
even to work at all, the technique is unlikely to be adopted widely even to work at all, the technique is unlikely to be adopted widely
or quickly. (See Section 6.2 and Section 6.3, in addition to or quickly. (See Sections 6.2 and 6.3, in addition to [RFC5218].)
[RFC5218]).
4.3. Providing Benefits During Partial Deployment 4.3. Providing Benefits during Partial Deployment
Some proposals require that all path elements along the full length Some proposals require that all path elements along the full length
of the path must be upgraded to support a new technique, before any of the path must be upgraded to support a new technique, before any
benefits can be seen. This is likely to require coordination between benefits can be seen. This is likely to require coordination between
operators who control a subset of path elements, and between operators who control a subset of path elements, and between
operators and end users if endpoint upgrades are required. If a operators and end users if endpoint upgrades are required. If a
technique provides benefits when only a part of the path has been technique provides benefits when only a part of the path has been
upgraded, this is likely to encourage adoption and deployment. (See upgraded, this is likely to encourage adoption and deployment. (See
Section 6.2, Section 6.3, and Section 6.9, in addition to [RFC5218]). Sections 6.2, 6.3, and 6.9, in addition to [RFC5218].)
4.4. Outperforming End-to-end Protocol Mechanisms 4.4. Outperforming End-to-End Protocol Mechanisms
Adaptive end-to-end protocol mechanisms may respond to feedback Adaptive end-to-end protocol mechanisms may respond to feedback
quickly enough that the additional realizable benefit from a new Path quickly enough that the additional realizable benefit from a new Path
Aware mechanism that tries to manipulate nodes along a path, or Aware mechanism that tries to manipulate nodes along a path, or
observe the attributes of nodes along a path, may be much smaller observe the attributes of nodes along a path, may be much smaller
than anticipated (See Section 6.3 and Section 6.5). than anticipated. (See Sections 6.3 and 6.5.)
4.5. Paying for Path Aware Techniques 4.5. Paying for Path Aware Techniques
"Follow the money." If operators can't charge for a Path Aware "Follow the money." If operators can't charge for a Path Aware
technique to recover the costs of deploying it, the benefits to the technique to recover the costs of deploying it, the benefits to the
operator must be really significant. Corollary: If operators charge operator must be really significant. Corollary: if operators charge
for a Path Aware technique, the benefits to users of that Path Aware for a Path Aware technique, the benefits to users of that Path Aware
technique must be significant enough to justify the cost. (See technique must be significant enough to justify the cost. (See
Section 6.1, Section 6.2, Section 6.5, and Section 6.9). Sections 6.1, 6.2, 6.5, and 6.9.)
4.6. Impact on Operational Practices 4.6. Impact on Operational Practices
Impact of a Path Aware technique requiring changes to operational The impact of a Path Aware technique requiring changes to operational
practices can affect how quickly or widely a promising technique is practices can affect how quickly or widely a promising technique is
deployed. The impacts of these changes may make deployment more deployed. The impacts of these changes may make deployment more
likely, but often discourage deployment. (See Section 6.6, including likely, but they often discourage deployment. (See Section 6.6,
Section 6.6.3). including Section 6.6.3.)
4.7. Per-connection State 4.7. Per-Connection State
Per-connection state in intermediate nodes has been an impediment to Per-connection state in intermediate nodes has been an impediment to
adoption and deployment in the past, because of added cost and adoption and deployment in the past, because of added cost and
complexity. Often, similar benefits can be achieved with much less complexity. Often, similar benefits can be achieved with much less
finely-grained state. This is especially true as we move from the finely grained state. This is especially true as we move from the
edge of the network, further into the routing core (See Section 6.1 edge of the network, further into the routing core. (See
and Section 6.2). Sections 6.1 and 6.2.)
4.8. Keeping Traffic on Fast-paths 4.8. Keeping Traffic on Fast Paths
Many modern platforms, especially high-end routers, have been Many modern platforms, especially high-end routers, have been
designed with hardware that can make simple per-packet forwarding designed with hardware that can make simple per-packet forwarding
decisions ("fast-paths"), but have not been designed to make heavy decisions ("fast paths") but have not been designed to make heavy use
use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options of in-band mechanisms such as IPv4 and IPv6 Router Alert Options
(RAO) that require more processing to make forwarding decisions. (RAOs) that require more processing to make forwarding decisions.
Packets carrying in-band mechanisms are diverted to other processors Packets carrying in-band mechanisms are diverted to other processors
in the router with much lower packet processing rates. Operators can in the router with much lower packet-processing rates. Operators can
be reluctant to deploy techniques that rely heavily on in-band be reluctant to deploy techniques that rely heavily on in-band
mechanisms because they may significantly reduce packet throughput. mechanisms because they may significantly reduce packet throughput.
(See Section 6.7). (See Section 6.7.)
4.9. Endpoints Trusting Intermediate Nodes 4.9. Endpoints Trusting Intermediate Nodes
If intermediate nodes along the path can't be trusted, it's unlikely If intermediate nodes along the path can't be trusted, it's unlikely
that endpoints will rely on signals from intermediate nodes to drive that endpoints will rely on signals from intermediate nodes to drive
changes to endpoint behaviors. We note that "trust" is not binary - changes to endpoint behaviors. We note that "trust" is not binary --
one, low, level of trust applies when a node issuing a message can one low level of trust applies when a node receiving a message can
confirm that it has visibility of the packets on the path it is confirm that the sender of the message has visibility of the packets
seeking to control [RFC8085] (e.g., an ICMP message included a quoted on the path it is seeking to control [RFC8085] (e.g., an ICMP
packet from the source). A higher level of trust can arise when an Destination Unreachable message [RFC0792] that includes the Internet
endpoint has established a short term, or even long term, trust Header + 64 bits of Original Data Datagram payload from the source).
relationship with network nodes. (See Section 6.4 and Section 6.5). A higher level of trust can arise when an endpoint has established a
short-term, or even long-term, trust relationship with network nodes.
(See Sections 6.4 and 6.5.)
4.10. Intermediate Nodes Trusting Endpoints 4.10. Intermediate Nodes Trusting Endpoints
If the endpoints do not have any trust relationship with the If the endpoints do not have any trust relationship with the
intermediate nodes along a path, operators have been reluctant to intermediate nodes along a path, operators have been reluctant to
deploy techniques that rely on endpoints sending unauthenticated deploy techniques that rely on endpoints sending unauthenticated
control signals to routers. (See Section 6.2 and Section 6.7). (We control signals to routers. (See Sections 6.2 and 6.7.) (We also
also note this still remains a factor hindering deployment of note that this still remains a factor hindering deployment of
DiffServ). Diffserv.)
4.11. Reacting to Distant Signals 4.11. Reacting to Distant Signals
Because the Internet is a distributed system, if the distance that Because the Internet is a distributed system, if the distance that
information from distant path elements travels to a Path Aware host information from distant path elements travels to a Path Aware host
is sufficiently large, the information may no longer accurately is sufficiently large, the information may no longer accurately
represent the state and situation at the distant host or elements represent the state and situation at the distant host or elements
along the path when it is received locally. In this case, the along the path when it is received locally. In this case, the
benefit that a Path Aware technique provides will be inconsistent, benefit that a Path Aware technique provides will be inconsistent and
and may not always be beneficial. (See Section 6.3). may not always be beneficial. (See Section 6.3.)
4.12. Support in Endpoint Protocol Stacks 4.12. Support in Endpoint Protocol Stacks
Just because a protocol stack provides a new feature/signal does not Just because a protocol stack provides a new feature/signal does not
mean that applications will use the feature/signal. Protocol stacks mean that applications will use the feature/signal. Protocol stacks
may not know how to effectively utilize Path-Aware techniques, may not know how to effectively utilize Path Aware techniques,
because the protocol stack may require information from applications because the protocol stack may require information from applications
to permit the technique to work effectively, but applications may not to permit the technique to work effectively, but applications may not
a-priori know that information. Even if the application does know a priori know that information. Even if the application does know
that information, the de-facto sockets API has no way of signaling that information, the de facto sockets API has no way of signaling
application expectations for the network path to the protocol stack. application expectations for the network path to the protocol stack.
In order for applications to provide these expectations to protocol In order for applications to provide these expectations to protocol
stacks, we need an API that signals more than the packets to be sent. stacks, we need an API that signals more than the packets to be sent.
(See Section 6.1 and Section 6.2). (See Sections 6.1 and 6.2.)
4.13. Planning For Failure 4.13. Planning for Failure
If early implementers discover severe problems with a new feature, If early implementers discover severe problems with a new feature,
that feature is likely to be disabled, and convincing implementers to that feature is likely to be disabled, and convincing implementers to
re-enable that feature can be very difficult, and can require years re-enable that feature can be very difficult and can require years or
or decades. In addition to testing, partial deployment for a subset decades. In addition to testing, partial deployment for a subset of
of users, implementing instrumentation that will detect degraded user users, implementing instrumentation that will detect degraded user
experience, and even "failback" to a previous version or "failover" experience, and even "failback" to a previous version or "failover"
to an entirely different implementation are likely to be helpful. to an entirely different implementation are likely to be helpful.
(See Section 6.9). (See Section 6.9.)
5. Future Work 5. Future Work
By its nature, this document has been retrospective. In addition to By its nature, this document has been retrospective. In addition to
considering how the Lessons Learned to date apply to current and considering how the Lessons Learned to date apply to current and
future Path Aware networking proposals, it's also worth considering future Path Aware networking proposals, it's also worth considering
whether there is deeper investigation left to do. whether there is deeper investigation left to do.
* We note that this work was based on contributions from experts on * We note that this work was based on contributions from experts on
various Path Aware networking techniques, and all of the various Path Aware techniques, and all of the contributed
contributed techniques involved unicast protocols. We didn't techniques involved unicast protocols. We didn't consider how
consider how these lessons might apply to multicast, and, given these lessons might apply to multicast, and, given anecdotal
anecdotal reports at the IETF 109 MOPS working group meeting of IP reports at the IETF 109 Media Operations (MOPS) Working Group
multicast offerings within data centers at one or more cloud meeting of IP multicast offerings within data centers at one or
providers ([MOPS-109-Min]), it might be useful to think about path more cloud providers [MOPS-109-Min], it might be useful to think
awareness in multicast, before we have a history of unsuccessful about Path Awareness in multicast, before we have a history of
deployments to document. unsuccessful deployments to document.
* The question of whether a mechanism supports admission control, * The question of whether a mechanism supports admission control,
based on either endpoints or applications, is associated with Path based on either endpoints or applications, is associated with Path
Awareness. One of the motivations of IntServ and a number of Awareness. One of the motivations of IntServ and a number of
other architectures (e.g. Deterministic Networking, [RFC8655]) is other architectures (e.g., Deterministic Networking [RFC8655]) is
the ability to "say no" to an application based on resource the ability to "say no" to an application based on resource
availability on a path, before the application tries to inject availability on a path, before the application tries to inject
traffic onto that path and discovers the path does not have the traffic onto that path and discovers the path does not have the
capacity to sustain enough utility to meet the application's capacity to sustain enough utility to meet the application's
minimum needs. The question of whether admission control is minimum needs. The question of whether admission control is
needed comes up repeatedly, but we have learned a few useful needed comes up repeatedly, but we have learned a few useful
lessons that, while covered implicitly in some of the lessons lessons that, while covered implicitly in some of the Lessons
learned of the document, might be explained explicitly: Learned provided in this document, might be explained explicitly:
- We have gained a lot of experience with application-based - We have gained a lot of experience with application-based
adaptation since the days where applications just injected adaptation since the days where applications just injected
traffic in-elastically into the network. Such adaptations seem traffic inelastically into the network. Such adaptations seem
to work well enough that admission control is of less value to to work well enough that admission control is of less value to
these applications these applications.
- There are end-to-end measurement techniques that can steer - There are end-to-end measurement techniques that can steer
traffic at the application layer (Content Distribution traffic at the application layer (Content Delivery Networks
Networks, multi-CDNs like Conviva [Conviva], etc.) (CDNs), multi-CDNs like Conviva [Conviva], etc.).
- We noted in Section 4.12 that applications often don't know how - We noted in Section 4.12 that applications often don't know how
to utilize Path Aware techniques. This includes not knowing to utilize Path Aware techniques. This includes not knowing
enough about their admission control threshold to be able to enough about their admission control threshold to be able to
ask accurately for the resources they need, whether this is ask accurately for the resources they need, whether this is
because the application itself doesn't know, or because the because the application itself doesn't know or because the
application has no way to signal its expectations to the application has no way to signal its expectations to the
underlying protocol stack. To date, attempts to help them underlying protocol stack. To date, attempts to help them
haven't gotten anywhere (e.g. the multiple-TSPEC additions to haven't gotten anywhere (e.g., the multiple-TSPEC (Traffic
RSVP to attempt to mirror codec selection by applications Specification) additions to RSVP to attempt to mirror codec
[I-D.ietf-tsvwg-intserv-multiple-tspec] expired in 2013). selection by applications [INTSERV-MULTIPLE-TSPEC] expired in
2013).
* We note that this work took the then-current IP network * We note that this work took the then-current IP network
architecture as given, at least at the time each technique was architecture as given, at least at the time each technique was
proposed. It might be useful to consider aspects of the now- proposed. It might be useful to consider aspects of the now-
current IP network architecture that ease, or impede, Path Aware current IP network architecture that ease, or impede, Path Aware
networking techniques. For example, there is limited ability in techniques. For example, there is limited ability in IP to
IP to constrain bidirectional paths to be symmetric, and constrain bidirectional paths to be symmetric, and information-
information-centric networking protocols such as Named Data centric networking protocols such as Named Data Networking (NDN)
Networking (NDN) and Content-Centric Networking (CCNx) ([RFC8793]) and Content-Centric Networking (CCNx) [RFC8793] must force
must force bidirectional path symmetry using protocol-specific bidirectional path symmetry using protocol-specific mechanisms.
mechanisms.
6. Contributions 6. Contributions
Contributions on these Path Aware networking techniques were analyzed Contributions on these Path Aware techniques were analyzed to arrive
to arrive at the Lessons Learned captured in Section 4. at the Lessons Learned captured in Section 4.
Our expectation is that most readers will not need to read through Our expectation is that most readers will not need to read through
this section carefully, but we wanted to record these hard-fought this section carefully, but we wanted to record these hard-fought
lessons as a service to others who may revisit this document, so lessons as a service to others who may revisit this document, so
they'll have the details close at hand. they'll have the details close at hand.
6.1. Stream Transport (ST, ST2, ST2+) 6.1. Stream Transport (ST, ST2, ST2+)
The suggested references for Stream Transport are: The suggested references for Stream Transport are:
* ST - A Proposed Internet Stream Protocol [IEN-119] * "ST - A Proposed Internet Stream Protocol" [IEN-119]
* Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] * "Experimental Internet Stream Protocol: Version 2 (ST-II)"
[RFC1190]
* Internet Stream Protocol Version 2 (ST2) Protocol Specification - * "Internet Stream Protocol Version 2 (ST2) Protocol Specification -
Version ST2+ [RFC1819] Version ST2+" [RFC1819]
The first version of Stream Transport, ST [IEN-119], was published in The first version of Stream Transport, ST [IEN-119], was published in
the late 1970's and was implemented and deployed on the ARPANET at the late 1970s and was implemented and deployed on the ARPANET at
small scale. It was used throughout the 1980's for experimental small scale. It was used throughout the 1980s for experimental
transmission of voice, video, and distributed simulation. transmission of voice, video, and distributed simulation.
The second version of the ST specification (ST2) [RFC1190] [RFC1819] The second version of the ST specification (ST2) [RFC1190] [RFC1819]
was an experimental connection-oriented internetworking protocol that was an experimental connection-oriented internetworking protocol that
operated at the same layer as connectionless IP. ST2 packets could operated at the same layer as connectionless IP. ST2 packets could
be distinguished by their IP header version numbers (IP, at that be distinguished by their IP header version numbers (IP, at that
time, used version number 4, while ST2 used version number 5). time, used version number 4, while ST2 used version number 5).
ST2 used a control plane layered over IP to select routes and reserve ST2 used a control plane layered over IP to select routes and reserve
capacity for real-time streams across a network path, based on a flow capacity for real-time streams across a network path, based on a flow
specification communicated by a separate protocol. The flow specification communicated by a separate protocol. The flow
specification could be associated with QoS state in routers, specification could be associated with QoS state in routers,
producing an experimental resource reservation protocol. This producing an experimental resource reservation protocol. This
allowed ST2 routers along a path to offer end-to-end guarantees, allowed ST2 routers along a path to offer end-to-end guarantees,
primarily to satisfy the QoS requirements for realtime services over primarily to satisfy the QoS requirements for real-time services over
the Internet. the Internet.
6.1.1. Reasons for Non-deployment 6.1.1. Reasons for Non-deployment
Although implemented in a range of equipment, ST2 was not widely used Although implemented in a range of equipment, ST2 was not widely used
after completion of the experiments. It did not offer the after completion of the experiments. It did not offer the
scalability and fate-sharing properties that have come to be desired scalability and fate-sharing properties that have come to be desired
by the Internet community. by the Internet community.
The ST2 protocol is no longer in use. The ST2 protocol is no longer in use.
6.1.2. Lessons Learned. 6.1.2. Lessons Learned
As time passed, the trade-off between router processing and link As time passed, the trade-off between router processing and link
capacity changed. Links became faster and the cost of router capacity changed. Links became faster, and the cost of router
processing became comparatively more expensive. processing became comparatively more expensive.
The ST2 control protocol used "hard state" - once a route was The ST2 control protocol used "hard state" -- once a route was
established, and resources were reserved, routes and resources established, and resources were reserved, routes and resources
existing until they were explicitly released via signaling. A soft- existed until they were explicitly released via signaling. A soft-
state approach was thought superior to this hard-state approach, and state approach was thought superior to this hard-state approach and
led to development of the IntServ model described in Section 6.2. led to development of the IntServ model described in Section 6.2.
6.2. Integrated Services (IntServ) 6.2. Integrated Services (IntServ)
The suggested references for IntServ are: The suggested references for IntServ are:
* RFC 1633 Integrated Services in the Internet Architecture: an * "Integrated Services in the Internet Architecture: an Overview"
Overview [RFC1633] [RFC1633]
* RFC 2211 Specification of the Controlled-Load Network Element * "Specification of the Controlled-Load Network Element Service"
Service [RFC2211] [RFC2211]
* RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] * "Specification of Guaranteed Quality of Service" [RFC2212]
* RFC 2215 General Characterization Parameters for Integrated * "General Characterization Parameters for Integrated Service
Service Network Elements [RFC2215] Network Elements" [RFC2215]
* RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] * "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional
Specification" [RFC2205]
In 1994, when the IntServ architecture document [RFC1633] was In 1994, when the IntServ architecture document [RFC1633] was
published, real-time traffic was first appearing on the Internet. At published, real-time traffic was first appearing on the Internet. At
that time, bandwidth was still a scarce commodity. Internet Service that time, bandwidth was still a scarce commodity. Internet Service
Providers built networks over DS3 (45 Mbps) infrastructure, and sub- Providers built networks over DS3 (45 Mbps) infrastructure, and sub-
rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a rate (< 1 Mbps) access was common. Therefore, the IETF anticipated a
need for a fine-grained QoS mechanism. need for a fine-grained QoS mechanism.
In the IntServ architecture, some applications can require service In the IntServ architecture, some applications can require service
guarantees. Therefore, those applications use the Resource guarantees. Therefore, those applications use RSVP [RFC2205] to
Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations signal QoS reservations across network paths. Every router in the
across network paths. Every router in the network that participates network that participates in IntServ maintains per-flow soft state to
in IntServ maintains per-flow soft-state to a) perform call admission a) perform call admission control and b) deliver guaranteed service.
control and b) deliver guaranteed service.
Applications use Flow Specification (Flow Specs) [RFC2210] to Applications use Flow Specifications (Flow Specs, or FLOWSPECs)
describe the traffic that they emit. RSVP reserves capacity for [RFC2210] to describe the traffic that they emit. RSVP reserves
traffic on a per Flow Spec basis. capacity for traffic on a per-Flow-Spec basis.
6.2.1. Reasons for Non-deployment 6.2.1. Reasons for Non-deployment
Although IntServ has been used in enterprise and government networks, Although IntServ has been used in enterprise and government networks,
IntServ was never widely deployed on the Internet because of its IntServ was never widely deployed on the Internet because of its
cost. The following factors contributed to operational cost: cost. The following factors contributed to operational cost:
* IntServ must be deployed on every router that is on a path where * IntServ must be deployed on every router that is on a path where
IntServ is to be used. Although it is possible to include a IntServ is to be used. Although it is possible to include a
router that does not participate in IntServ along the path being router that does not participate in IntServ along the path being
controlled, if that router is likely to become a bottleneck, controlled, if that router is likely to become a bottleneck,
IntServ cannot be used to avoid that bottleneck along the path IntServ cannot be used to avoid that bottleneck along the path.
* IntServ maintained per flow state * IntServ maintained per-flow state.
As IntServ was being discussed, the following occurred: As IntServ was being discussed, the following occurred:
* For many expected uses, it became more cost effective to solve the * For many expected uses, it became more cost effective to solve the
QoS problem by adding bandwidth. Between 1994 and 2000, Internet QoS problem by adding bandwidth. Between 1994 and 2000, Internet
Service Providers upgraded their infrastructures from DS3 (45 Service Providers upgraded their infrastructures from DS3 (45
Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint
was using IntServ in an IntServ-enabled network, its requests was using IntServ in an IntServ-enabled network, its requests
would rarely, if ever, be denied, so endpoints and Internet would rarely, if ever, be denied, so endpoints and Internet
Service Providers had little reason to enable IntServ. Service Providers had little reason to enable IntServ.
* DiffServ [RFC2475] offered a more cost-effective, albeit less * Diffserv [RFC2475] offered a more cost-effective, albeit less
fine-grained, solution to the QoS problem. fine-grained, solution to the QoS problem.
6.2.2. Lessons Learned. 6.2.2. Lessons Learned
The following lessons were learned: The following lessons were learned:
* Any mechanism that requires every participating onpath router to * Any mechanism that requires every participating on-path router to
maintain per-flow state is not likely to succeed, unless the maintain per-flow state is not likely to succeed, unless the
additional cost for offering the feature can be recovered from the additional cost for offering the feature can be recovered from the
user. user.
* Any mechanism that requires an operator to upgrade all of its * Any mechanism that requires an operator to upgrade all of its
routers is not likely to succeed, unless the additional cost for routers is not likely to succeed, unless the additional cost for
offering the feature can be recovered from the user. offering the feature can be recovered from the user.
In environments where IntServ has been deployed, trust relationships In environments where IntServ has been deployed, trust relationships
with endpoints are very different from trust relationships on the with endpoints are very different from trust relationships on the
Internet itself, and there are often clearly-defined hierarchies in Internet itself. There are often clearly defined hierarchies in
Service Level Agreements (SLAs), and well-defined transport flows Service Level Agreements (SLAs) governing well-defined transport
operating with pre-determined capacity and latency requirements over flows operating with predetermined capacity and latency requirements
paths where capacity or other attributes are constrained. over paths where capacity or other attributes are constrained.
IntServ was never widely deployed to manage capacity across the IntServ was never widely deployed to manage capacity across the
Internet. However, the technique that it produced was deployed for Internet. However, the technique that it produced was deployed for
reasons other than bandwidth management. RSVP is widely deployed as reasons other than bandwidth management. RSVP is widely deployed as
an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter
Specs to distribute firewall filters, although they are called Flow Specs to distribute firewall filters, although they are called "Flow
Spec Component Types in BGP [RFC5575]. Spec Component Types" in BGP [RFC5575].
6.3. Quick-Start TCP 6.3. Quick-Start TCP
The suggested references for Quick-Start TCP are: The suggested references for Quick-Start TCP are:
* Quick-Start for TCP and IP [RFC4782] * "Quick-Start for TCP and IP" [RFC4782]
* Determining an appropriate initial sending rate over an * "Determining an appropriate sending rate over an underutilized
underutilized network path [SAF07] network path" [SAF07]
* Fast Startup Internet Congestion Control for Broadband Interactive * "Fast Startup Internet Congestion Control for Broadband
Applications [Sch11] Interactive Applications" [Sch11]
* Using Quick-Start to enhance TCP-friendly rate control performance * "Using Quick-Start to enhance TCP-friendly rate control
in bidirectional satellite networks [QS-SAT] performance in bidirectional satellite networks" [QS-SAT]
Quick-Start [RFC4782] is an Experimental TCP extension that leverages Quick-Start is defined in an Experimental RFC [RFC4782] and is a TCP
support from the routers on the path to determine an allowed initial extension that leverages support from the routers on the path to
sending rate for a path through the Internet, either at the start of determine an allowed initial sending rate for a path through the
data transfers or after idle periods. Without information about the Internet, either at the start of data transfers or after idle
path, a sender cannot easily determine an appropriate initial sending periods. Without information about the path, a sender cannot easily
rate. The default TCP congestion control therefore uses the safe but determine an appropriate initial sending rate. The default TCP
time-consuming slow-start algorithm [RFC5681]. With Quick-Start, congestion control therefore uses the safe but time-consuming slow-
connections are allowed to use higher initial sending rates if there start algorithm [RFC5681]. With Quick-Start, connections are allowed
is significant unused bandwidth along the path, and if the sender and to use higher initial sending rates if there is significant unused
all of the routers along the path approve the request. bandwidth along the path and if the sender and all of the routers
along the path approve the request.
By examining the Time To Live (TTL) field in Quick-Start packets, a By examining the Time To Live (TTL) field in Quick-Start packets, a
sender can determine if routers on the path have approved the Quick- sender can determine if routers on the path have approved the Quick-
Start request. However, this method is unable to take into account Start request. However, this method is unable to take into account
the routers hidden by tunnels or other network nodes invisible at the the routers hidden by tunnels or other network nodes invisible at the
IP layer. IP layer.
The protocol also includes a nonce that provides protection against The protocol also includes a nonce that provides protection against
cheating routers and receivers. If the Quick-Start request is cheating routers and receivers. If the Quick-Start request is
explicitly approved by all routers along the path, the TCP host can explicitly approved by all routers along the path, the TCP host can
send at up to the approved rate; otherwise TCP would use the default send at up to the approved rate; otherwise, TCP would use the default
congestion control. Quick-Start requires modifications in the congestion control. Quick-Start requires modifications in the
involved end-systems as well in routers. Due to the resulting involved end systems as well as in routers. Due to the resulting
deployment challenges, Quick-Start was only proposed in [RFC4782] for deployment challenges, Quick-Start was only proposed in [RFC4782] for
controlled environments. controlled environments.
The Quick-Start mechanism is a lightweight, coarse-grained, in-band, The Quick-Start mechanism is a lightweight, coarse-grained, in-band,
network-assisted fast startup mechanism. The benefits are studied by network-assisted fast startup mechanism. The benefits are studied by
simulation in a research paper [SAF07] that complements the protocol simulation in a research paper [SAF07] that complements the protocol
specification. The study confirms that Quick-Start can significantly specification. The study confirms that Quick-Start can significantly
speed up mid-sized data transfers. That paper also presents router speed up mid-sized data transfers. That paper also presents router
algorithms that do not require keeping per-flow state. Later studies algorithms that do not require keeping per-flow state. Later studies
[Sch11] comprehensively analyzes Quick-Start with a full Linux [Sch11] comprehensively analyze Quick-Start with a full Linux
implementation and with a router fast path prototype using a network implementation and with a router fast-path prototype using a network
processor. In both cases, Quick-Start could be implemented with processor. In both cases, Quick-Start could be implemented with
limited additional complexity. limited additional complexity.
6.3.1. Reasons for Non-deployment 6.3.1. Reasons for Non-deployment
However, experiments with Quick-Start in [Sch11] revealed several However, experiments with Quick-Start in [Sch11] revealed several
challenges: challenges:
* Having information from the routers along the path can reduce the * Having information from the routers along the path can reduce the
risk of congestion, but cannot avoid it entirely. Determining risk of congestion but cannot avoid it entirely. Determining
whether there is unused capacity is not trivial in actual router whether there is unused capacity is not trivial in actual router
and host implementations. Data about available capacity visible and host implementations. Data about available capacity visible
at the IP layer may be imprecise, and due to the propagation at the IP layer may be imprecise, and due to the propagation
delay, information can already be outdated when it reaches a delay, information can already be outdated when it reaches a
sender. There is a trade-off between the speedup of data sender. There is a trade-off between the speedup of data
transfers and the risk of congestion even with Quick-Start. This transfers and the risk of congestion even with Quick-Start. This
could be mitigated by only allowing Quick-Start to access a could be mitigated by only allowing Quick-Start to access a
proportion of the unused capacity along a path. proportion of the unused capacity along a path.
* For scalable router fast path implementation, it is important to * For scalable router fast-path implementations, it is important to
enable parallel processing of packets, as this is a widely used enable parallel processing of packets, as this is a widely used
method e.g. in network processors. One challenge is method, e.g., in network processors. One challenge is
synchronization of information between packets that are processed synchronization of information between packets that are processed
in parallel, which should be avoided as much as possible. in parallel, which should be avoided as much as possible.
* Only some types of application traffic can benefit from Quick- * Only some types of application traffic can benefit from Quick-
Start. Capacity needs to be requested and discovered. The Start. Capacity needs to be requested and discovered. The
discovered capacity needs to be utilized by the flow, or it discovered capacity needs to be utilized by the flow, or it
implicitly becomes available for other flows. Failing to use the implicitly becomes available for other flows. Failing to use the
requested capacity may have already reduced the pool of Quick- requested capacity may have already reduced the pool of Quick-
Start capacity that was made available to other competing Quick- Start capacity that was made available to other competing Quick-
Start requests. The benefit is greatest when senders use this Start requests. The benefit is greatest when senders use this
only for bulk flows and avoid sending unnecessary Quick-Start only for bulk flows and avoid sending unnecessary Quick-Start
requests, e.g. for flows that only send a small amount of data. requests, e.g., for flows that only send a small amount of data.
Choosing an appropriate request size requires application-internal Choosing an appropriate request size requires application-internal
knowledge that is not commonly expressed by the transport API. knowledge that is not commonly expressed by the transport API.
How a sender can determine the rate for an initial Quick-Start How a sender can determine the rate for an initial Quick-Start
request is still a largely unsolved problem. request is still a largely unsolved problem.
There is no known deployment of Quick-Start for TCP or other IETF There is no known deployment of Quick-Start for TCP or other IETF
transports. transports.
6.3.2. Lessons Learned 6.3.2. Lessons Learned
Some lessons can be learned from Quick-Start. Despite being a very Some lessons can be learned from Quick-Start. Despite being a very
light-weight protocol, Quick-Start suffers from poor incremental lightweight protocol, Quick-Start suffers from poor incremental
deployment properties, both regarding the required modifications in deployment properties regarding both a) the required modifications in
network infrastructure as well as its interactions with applications. network infrastructure and b) its interactions with applications.
Except for corner cases, congestion control can be quite efficiently Except for corner cases, congestion control can be quite efficiently
performed end-to-end in the Internet, and in modern stacks there is performed end to end in the Internet, and in modern stacks there is
not much room for significant improvement by additional network not much room for significant improvement by additional network
support. support.
After publication of the Quick-Start specification, there have been After publication of the Quick-Start specification, there have been
large-scale experiments with an initial window of up to 10 MSS large-scale experiments with an initial window of up to 10 segments
[RFC6928]. This alternative "IW10" approach can also ramp-up data [RFC6928]. This alternative "IW10" approach can also ramp up data
transfers faster than the standard congestion control, but it only transfers faster than the standard congestion control, but it only
requires sender-side modifications. As a result, this approach can requires sender-side modifications. As a result, this approach can
be easier and incrementally deployed in the Internet. While be easier and incrementally deployed in the Internet. While
theoretically Quick-Start can outperform "IW10", the improvement in theoretically Quick-Start can outperform "IW10", the improvement in
completion time for data transfer times can, in many cases, be small. completion time for data transfer times can, in many cases, be small.
After publication of [RFC6928], most modern TCP stacks have increased After publication of [RFC6928], most modern TCP stacks have increased
their default initial window. their default initial window.
6.4. ICMP Source Quench 6.4. ICMP Source Quench
The suggested references for ICMP Source Quench are: The suggested reference for ICMP Source Quench is:
* INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] * "Internet Control Message Protocol" [RFC0792]
The ICMP Source Quench message [RFC0792] allowed an on-path router to The ICMP Source Quench message [RFC0792] allowed an on-path router to
request the source of a flow to reduce its sending rate. This method request the source of a flow to reduce its sending rate. This method
allowed a router to provide an early indication of impending allowed a router to provide an early indication of impending
congestion on a path to the sources that contribute to that congestion on a path to the sources that contribute to that
congestion. congestion.
6.4.1. Reasons for Non-deployment 6.4.1. Reasons for Non-deployment
This method was deployed in Internet routers over a period of time, This method was deployed in Internet routers over a period of time;
the reaction of endpoints to receiving this signal has varied. For the reaction of endpoints to receiving this signal has varied. For
low speed links, with low multiplexing of flows the method could be low-speed links, with low multiplexing of flows the method could be
used to regulate (momentarily reduce) the transmission rate. used to regulate (momentarily reduce) the transmission rate.
However, the simple signal does not scale with link speed, or the However, the simple signal does not scale with link speed or with the
number of flows sharing a link. number of flows sharing a link.
The approach was overtaken by the evolution of congestion control The approach was overtaken by the evolution of congestion control
methods in TCP [RFC2001], and later also by other IETF transports. methods in TCP [RFC2001], and later also by other IETF transports.
Because these methods were based upon measurement of the end-to-end Because these methods were based upon measurement of the end-to-end
path and an algorithm in the endpoint, they were able to evolve and path and an algorithm in the endpoint, they were able to evolve and
mature more rapidly than methods relying on interactions between mature more rapidly than methods relying on interactions between
operational routers and endpoint stacks. operational routers and endpoint stacks.
After ICMP Source Quench was specified, the IETF began to recommend After ICMP Source Quench was specified, the IETF began to recommend
that transports provide end-to-end congestion control [RFC2001]. The that transports provide end-to-end congestion control [RFC2001]. The
Source Quench method has been obsoleted by the IETF [RFC6633], and Source Quench method has been obsoleted by the IETF [RFC6633], and
both hosts and routers must now silently discard this message. both hosts and routers must now silently discard this message.
6.4.2. Lessons Learned 6.4.2. Lessons Learned
This method had several problems: This method had several problems.
First, [RFC0792] did not sufficiently specify how the sender would First, [RFC0792] did not sufficiently specify how the sender would
react to the ICMP Source Quench signal from the path (e.g., react to the ICMP Source Quench signal from the path (e.g.,
[RFC1016]). There was ambiguity in how the sender should utilize [RFC1016]). There was ambiguity in how the sender should utilize
this additional information. This could lead to unfairness in the this additional information. This could lead to unfairness in the
way that receivers (or routers) responded to this message. way that receivers (or routers) responded to this message.
Second, while the message did provide additional information, the Second, while the message did provide additional information, the
Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a
more robust and informative signal for network nodes to provide early more robust and informative signal for network nodes to provide early
indication that a path has become congested. indication that a path has become congested.
The mechanism originated at a time when the Internet trust model was The mechanism originated at a time when the Internet trust model was
very different. Most endpoint implementations did not attempt to very different. Most endpoint implementations did not attempt to
verify that the message originated from an on-path node before they verify that the message originated from an on-path node before they
utilized the message. This made it vulnerable to denial of service utilized the message. This made it vulnerable to Denial-of-Service
attacks. In theory, routers might have chosen to use the quoted (DoS) attacks. In theory, routers might have chosen to use the
packet contained in the ICMP payload to validate that the message quoted packet contained in the ICMP payload to validate that the
originated from an on-path node, but this would have increased per- message originated from an on-path node, but this would have
packet processing overhead for each router along the path, would have increased per-packet processing overhead for each router along the
required transport functionality in the router to verify whether the path and would have required transport functionality in the router to
quoted packet header corresponded to a packet the router had sent. verify whether the quoted packet header corresponded to a packet the
In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on router had sent. In addition, Section 5.2 of [RFC4443] noted
hosts that would also have threatened routers processing ICMPv6 ICMPv6-based attacks on hosts that would also have threatened routers
Source Quench payloads. As time passed, it became increasingly processing ICMPv6 Source Quench payloads. As time passed, it became
obvious that the lack of validation of the messages exposed receivers increasingly obvious that the lack of validation of the messages
to a security vulnerability where the messages could be forged to exposed receivers to a security vulnerability where the messages
create a tangible denial of service opportunity. could be forged to create a tangible DoS opportunity.
6.5. Triggers for Transport (TRIGTRAN) 6.5. Triggers for Transport (TRIGTRAN)
The suggested references for TRIGTRAN are: The suggested references for TRIGTRAN are:
* TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55]
* TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56]
TCP [RFC0793] has a well-known weakness - the end-to-end flow control TCP [RFC0793] has a well-known weakness -- the end-to-end flow
mechanism has only a single signal, the loss of a segment, and TCP control mechanism has only a single signal, the loss of a segment,
implementations since the late 1980s have interpreted the loss of a detected when no acknowledgment for the lost segment is received at
segment as evidence that the path between two endpoints may have the sender. There are multiple reasons why the sender might not have
become congested enough to exhaust buffers on intermediate hops, so received an acknowledgment for the segment. To name several, the
that the TCP sender should "back off" - reduce its sending rate until segment could have been trapped in a routing loop, damaged in
it knows that its segments are now being delivered without loss transmission and failed checksum verification at the receiver, or
[RFC5681]. More modern TCP stacks have added a growing array of lost because some intermediate device discarded the packet, or any of
strategies about how to establish the sending rate [RFC5681], but a variety of other things could have happened to the acknowledgment
when a path is no longer operational, TCP would continue to retry on the way back from the receiver to the sender. TCP implementations
transmissions, which would fail, again, and double their since the late 1980s have made the "safe" decision and have
Retransmission Time Out (RTO) timers with each failed transmission, interpreted the loss of a segment as evidence that the path between
with the result that TCP would wait many seconds before retrying a two endpoints may have become congested enough to exhaust buffers on
segment, even if the path becomes operational while the sender is intermediate hops, so that the TCP sender should "back off" -- reduce
waiting for its next retry. its sending rate until it knows that its segments are now being
delivered without loss [RFC5681].
The thinking behind TRIGTRAN was that if a path completely stopped The thinking behind TRIGTRAN was that if a path completely stopped
working because a link along the path was "down", somehow something working because a link along the path was "down", somehow something
along the path could signal TCP when that link returned to service, along the path could signal TCP when that link returned to service,
and the sending TCP could retry immediately, without waiting for a and the sending TCP could retry immediately, without waiting for a
full retransmission timeout (RTO) period. full retransmission timeout (RTO) period.
6.5.1. Reasons for Non-deployment 6.5.1. Reasons for Non-deployment
The early dreams for TRIGTRAN were dashed because of an assumption The early dreams for TRIGTRAN were dashed because of an assumption
that TRIGTRAN triggers would be unauthenticated. This meant that any that TRIGTRAN triggers would be unauthenticated. This meant that any
"safe" TRIGTRAN mechanism would have relied on a mechanism such as "safe" TRIGTRAN mechanism would have relied on a mechanism such as
setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing
that it was 254 upon receipt, so that a receiver could verify that a that it was 254 upon receipt, so that a receiver could verify that a
signal was generated by an adjacent sender known to be on the path signal was generated by an adjacent sender known to be on the path
being used, and not some unknown sender which might not even be on being used and not some unknown sender that might not even be on the
the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" path (e.g., "The Generalized TTL Security Mechanism (GTSM)"
[RFC5082]). This situation is very similar to the case for ICMP [RFC5082]). This situation is very similar to the case for ICMP
Source Quench messages as described in Section 6.4, which were also Source Quench messages as described in Section 6.4, which were also
unauthenticated, and could be sent by an off-path attacker, resulting unauthenticated and could be sent by an off-path attacker, resulting
in deprecation of ICMP Source Quench message processing [RFC6633]. in deprecation of ICMP Source Quench message processing [RFC6633].
TRIGTRAN's scope shrunk from "the path is down" to "the first-hop TRIGTRAN's scope shrunk from "the path is down" to "the first-hop
link is down". link is down."
But things got worse. But things got worse.
Because TRIGTRAN triggers would only be provided when the first-hop Because TRIGTRAN triggers would only be provided when the first-hop
link was "down", TRIGTRAN triggers couldn't replace normal TCP link was "down", TRIGTRAN triggers couldn't replace normal TCP
retransmission behavior if the path failed because some link further retransmission behavior if the path failed because some link further
along the network path was "down". So TRIGTRAN triggers added along the network path was "down". So TRIGTRAN triggers added
complexity to an already complex TCP state machine, and did not allow complexity to an already-complex TCP state machine and did not allow
any existing complexity to be removed. any existing complexity to be removed.
There was also an issue that the TRIGTRAN signal was not sent in There was also an issue that the TRIGTRAN signal was not sent in
response to a specific host that had been sending packets, and was response to a specific host that had been sending packets and was
instead a signal that stimulated a response by any sender on the instead a signal that stimulated a response by any sender on the
link. This needs to scale when there are multiple flows trying to link. This needs to scale when there are multiple flows trying to
use the same resource, yet the sender of a trigger has no use the same resource, yet the sender of a trigger has no
understanding how many of the potential traffic sources will respond understanding of how many of the potential traffic sources will
by sending packets - if recipients of the signal back-off their respond by sending packets -- if recipients of the signal "back off"
responses to a trigger to improve scaling, then that immediately their responses to a trigger to improve scaling, then that
mitigates the benefit of the signal. immediately mitigates the benefit of the signal.
Finally, intermediate forwarding nodes required modification to Finally, intermediate forwarding nodes required modification to
provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN
triggers, so there was no way to recover the cost of modifying, triggers, so there was no way to recover the cost of modifying,
testing, and deploying updated intermediate nodes. testing, and deploying updated intermediate nodes.
Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56
[TRIGTRAN-56], but this work was not chartered, and there was no [TRIGTRAN-56], but this work was not chartered, and there was no
interest in deploying TRIGTRAN unless it was chartered and interest in deploying TRIGTRAN unless it was chartered and
standardized in the IETF. standardized in the IETF.
6.5.2. Lessons Learned. 6.5.2. Lessons Learned
The reasons why this work was not chartered, much less deployed, The reasons why this work was not chartered, much less deployed,
provide several useful lessons for researchers. provide several useful lessons for researchers.
* TRIGTRAN started with a plausible value proposition, but * TRIGTRAN started with a plausible value proposition, but
networking realities in the early 2000s forced reductions in scope networking realities in the early 2000s forced reductions in scope
that led directly to reductions in potential benefits, but no that led directly to reductions in potential benefits but no
corresponding reductions in costs and complexity. corresponding reductions in costs and complexity.
* These reductions in scope were the direct result of an inability * These reductions in scope were the direct result of an inability
for hosts to trust or authenticate TRIGTRAN signals they received for hosts to trust or authenticate TRIGTRAN signals they received
from the network. from the network.
* Operators did not believe they could charge for TRIGTRAN * Operators did not believe they could charge for TRIGTRAN
signaling, because first-hop links didn't fail frequently, and signaling, because first-hop links didn't fail frequently and
TRIGTRAN provided no reduction in operating expenses, so there was TRIGTRAN provided no reduction in operating expenses, so there was
little incentive to purchase and deploy TRIGTRAN-capable network little incentive to purchase and deploy TRIGTRAN-capable network
equipment. equipment.
It is also worth noting that the targeted environment for TRIGTRAN in It is also worth noting that the targeted environment for TRIGTRAN in
the late 1990s contained links with a relatively small number of the late 1990s contained links with a relatively small number of
directly-connected hosts - for instance, cellular or satellite links. directly connected hosts -- for instance, cellular or satellite
The transport community was well aware of the dangers of sender links. The transport community was well aware of the dangers of
synchronization based on multiple senders receiving the same stimulus sender synchronization based on multiple senders receiving the same
at the same time, but the working assumption for TRIGTRAN was that stimulus at the same time, but the working assumption for TRIGTRAN
there wouldn't be enough senders for this to be a meaningful problem. was that there wouldn't be enough senders for this to be a meaningful
In the 2010s, it is common for a single "link" to support many problem. In the 2010s, it was common for a single "link" to support
senders and receivers on a single link, likely requiring TRIGTRAN many senders and receivers, likely requiring TRIGTRAN senders to wait
senders to wait some random amount of time before sending after some random amount of time before sending after receiving a TRIGTRAN
receiving a TRIGTRAN signal, which would have reduced the benefits of signal, which would have reduced the benefits of TRIGTRAN even more.
TRIGTRAN even more.
6.6. Shim6 6.6. Shim6
The suggested references for Shim6 are: The suggested reference for Shim6 is:
* Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] * "Shim6: Level 3 Multihoming Shim Protocol for IPv6" [RFC5533]
The IPv6 routing architecture [RFC1887] assumed that most sites on The IPv6 routing architecture [RFC1887] assumed that most sites on
the Internet would be identified by Provider Assigned IPv6 prefixes, the Internet would be identified by Provider Assigned IPv6 prefixes,
so that Default-Free Zone routers only contained routes to other so that Default-Free Zone routers only contained routes to other
providers, resulting in a very small IPv6 global routing table. providers, resulting in a very small IPv6 global routing table.
For a single-homed site, this could work well. A multihomed site For a single-homed site, this could work well. A multihomed site
with only one upstream provider could also work well, although BGP with only one upstream provider could also work well, although BGP
multihoming from a single upstream provider was often a premium multihoming from a single upstream provider was often a premium
service (costing more than twice as much as two single-homed sites), service (costing more than twice as much as two single-homed sites),
and if the single upstream provider went out of service, all of the and if the single upstream provider went out of service, all of the
multihomed paths could fail simultaneously. multihomed paths could fail simultaneously.
IPv4 sites often multihomed by obtaining Provider Independent IPv4 sites often multihomed by obtaining Provider Independent
prefixes, and advertising these prefixes through multiple upstream prefixes and advertising these prefixes through multiple upstream
providers. With the assumption that any multihomed IPv4 site would providers. With the assumption that any multihomed IPv4 site would
also multihome in IPv6, it seemed likely that IPv6 routing would be also multihome in IPv6, it seemed likely that IPv6 routing would be
subject to the same pressures to announce Provider Independent subject to the same pressures to announce Provider Independent
prefixes, resulting in a global IPv6 routing table that exhibited the prefixes, resulting in an IPv6 global routing table that exhibited
same explosive growth as the global IPv4 routing table. During the the same explosive growth as the IPv4 global routing table. During
early 2000s, work began on a protocol that would provide multihoming the early 2000s, work began on a protocol that would provide
for IPv6 sites without requiring sites to advertise Provider multihoming for IPv6 sites without requiring sites to advertise
Independent prefixes into the IPv6 global routing table. Provider Independent prefixes into the IPv6 global routing table.
This protocol, called Shim6, allowed two endpoints to exchange This protocol, called "Shim6", allowed two endpoints to exchange
multiple addresses ("Locators") that all mapped to the same endpoint multiple addresses ("Locators") that all mapped to the same endpoint
("Identity"). After an endpoint learned multiple Locators for the ("Identity"). After an endpoint learned multiple Locators for the
other endpoint, it could send to any of those Locators with the other endpoint, it could send to any of those Locators with the
expectation that those packets would all be delivered to the endpoint expectation that those packets would all be delivered to the endpoint
with the same Identity. Shim6 was an example of an "Identity/Locator with the same Identity. Shim6 was an example of an "Identity/Locator
Split" protocol. Split" protocol.
Shim6, as defined in [RFC5533] and related RFCs, provided a workable Shim6, as defined in [RFC5533] and related RFCs, provided a workable
solution for IPv6 multihoming using Provider Assigned prefixes, solution for IPv6 multihoming using Provider Assigned prefixes,
including capability discovery and negotiation, and allowing end-to- including capability discovery and negotiation, and allowing end-to-
end application communication to continue even in the face of path end application communication to continue even in the face of path
failure, because applications don't see Locator failures, and failure, because applications don't see Locator failures and continue
continue to communicate with the same Identity using a different to communicate with the same Identity using a different Locator.
Locator.
6.6.1. Reasons for Non-deployment 6.6.1. Reasons for Non-deployment
Note that the problem being addressed was "site multihoming", but Note that the problem being addressed was "site multihoming", but
Shim6 was providing "host multihoming". That meant that the decision Shim6 was providing "host multihoming". That meant that the decision
about what path would be used was under host control, not under edge about what path would be used was under host control, not under edge
router control. router control.
Although more work could have been done to provide a better technical Although more work could have been done to provide a better technical
solution, the biggest impediments to Shim6 deployment were solution, the biggest impediments to Shim6 deployment were
operational and business considerations. These impediments were operational and business considerations. These impediments were
discussed at multiple network operator group meetings, including discussed at multiple network operator group meetings, including
[Shim6-35] at [NANOG-35]. [Shim6-35] at [NANOG-35].
The technical issues centered around concerns that Shim6 relied on The technical issues centered around concerns that Shim6 relied on
the host to track all the connections, while also tracking Identity/ the host to track all the connections, while also tracking Identity/
Locator mappings in the kernel, and tracking failures to recognize Locator mappings in the kernel and tracking failures to recognize
that an available path has failed. that an available path has failed.
The operational issues centered around concerns that operators were The operational issues centered around concerns that operators were
performing traffic engineering on traffic aggregates. With Shim6, performing traffic engineering on traffic aggregates. With Shim6,
these operator traffic engineering policies must be pushed down to these operator traffic engineering policies must be pushed down to
individual hosts. individual hosts.
In addition, operators would have no visibility or control over the In addition, operators would have no visibility or control over the
decision of hosts choosing to switch to another path. They expressed decision of hosts choosing to switch to another path. They expressed
concerns that relying on hosts to steer traffic exposed operator concerns that relying on hosts to steer traffic exposed operator
networks to oscillation based on feedback loops, if hosts moved from networks to oscillation based on feedback loops, if hosts moved from
path to path frequently. Given that Shim6 was intended to support path to path frequently. Given that Shim6 was intended to support
multihoming across operators, operators providing only one of the multihoming across operators, operators providing only one of the
paths would have even less visibility as traffic suddenly appeared paths would have even less visibility as traffic suddenly appeared
and disappeared on their networks. and disappeared on their networks.
In addition, firewalls that expected to find a TCP or UDP transport- In addition, firewalls that expected to find a TCP or UDP transport-
level protocol header in the IP payload would see a Shim6 Identity level protocol header in the IP payload would see a Shim6 Identity
header instead, and would not perform transport-protocol-based header instead, and they would not perform transport-protocol-based
firewalling functions because the firewall's normal processing logic firewalling functions because the firewall's normal processing logic
would not look past the Identity header. would not look past the Identity header. The firewall would perform
its default action, which would most likely be to drop packets that
don't match any processing rule.
The business issues centered on reducing or removing the ability to The business issues centered on reducing or removing the ability to
sell BGP multihoming service to their own customers, which is often sell BGP multihoming service to their own customers, which is often
more expensive than two single-homed connectivity services. more expensive than two single-homed connectivity services.
6.6.2. Lessons Learned 6.6.2. Lessons Learned
It is extremely important to take operational concerns into account It is extremely important to take operational concerns into account
when a path-aware protocol is making decisions about path selection when a Path Aware protocol is making decisions about path selection
that may conflict with existing operational practices and business that may conflict with existing operational practices and business
considerations. considerations.
6.6.3. Addendum on MultiPath TCP 6.6.3. Addendum on Multipath TCP
During discussions in the PANRG session at IETF 103 [PANRG-103-Min], During discussions in the PANRG session at IETF 103 [PANRG-103-Min],
Lars Eggert, past Transport Area Director, pointed out that during Lars Eggert, past Transport Area Director, pointed out that during
charter discussions for the Multipath TCP working group [MP-TCP], charter discussions for the Multipath TCP Working Group [MP-TCP],
operators expressed concerns that customers could use Multipath TCP operators expressed concerns that customers could use Multipath TCP
to loadshare TCP connections across operators simultaneously and to load-share TCP connections across operators simultaneously and
compare passive performance measurements across network paths in real compare passive performance measurements across network paths in real
time, changing the balance of power in those business relationships. time, changing the balance of power in those business relationships.
Although the Multipath TCP working group was chartered, this concern Although the Multipath TCP Working Group was chartered, this concern
could have acted as an obstacle to deployment. could have acted as an obstacle to deployment.
Operator objections to Shim6 were focused on technical concerns, but Operator objections to Shim6 were focused on technical concerns, but
this concern could have also been an obstacle to Shim6 deployment if this concern could have also been an obstacle to Shim6 deployment if
the technical concerns had been overcome. the technical concerns had been overcome.
6.7. Next Steps in Signaling (NSIS) 6.7. Next Steps in Signaling (NSIS)
The suggested references for Next Steps in Signaling (NSIS) are: The suggested references for Next Steps in Signaling (NSIS) are:
* the concluded working group charter [NSIS-CHARTER-2001] * the concluded working group charter [NSIS-CHARTER-2001]
* GIST: General Internet Signalling Transport [RFC5971] * "GIST: General Internet Signalling Transport" [RFC5971]
* NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] * "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)" [RFC5973]
* NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service * "NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service
Signaling [RFC5974] Signaling" [RFC5974]
* Authorization for NSIS Signaling Layer Protocols [RFC5981] * "Authorization for NSIS Signaling Layer Protocols" [RFC5981]
The NSIS Working Group worked on signaling techniques for network The NSIS Working Group worked on signaling techniques for network-
layer resources (e.g., QoS resource reservations, Firewall and NAT layer resources (e.g., QoS resource reservations, Firewall and NAT
traversal). traversal).
When RSVP [RFC2205] was used in deployments, a number of questions When RSVP [RFC2205] was used in deployments, a number of questions
came up about its perceived limitations and potential missing came up about its perceived limitations and potential missing
features. The issues noted in the NSIS Working Group charter features. The issues noted in the NSIS Working Group charter
[NSIS-CHARTER-2001] include interworking between domains with [NSIS-CHARTER-2001] include interworking between domains with
different QoS architectures, mobility and roaming for IP interfaces, different QoS architectures, mobility and roaming for IP interfaces,
and complexity. Later, the lack of security in RSVP was also and complexity. Later, the lack of security in RSVP was also
recognized ([RFC4094]). recognized [RFC4094].
The NSIS Working Group was chartered to tackle those issues and The NSIS Working Group was chartered to tackle those issues and
initially focused on QoS signaling as its primary use case. However, initially focused on QoS signaling as its primary use case. However,
over time a new approach evolved that introduced a modular over time a new approach evolved that introduced a modular
architecture using application-specific signaling protocols (the NSIS architecture using two application-specific signaling protocols: a)
Signaling Layer Protocol (NSLP)) on top of a generic signaling the NSIS Signaling Layer Protocol (NSLP) on top of b) a generic
transport protocol (the NSIS Transport Layer Protocol (NTLP)). signaling transport protocol (the NSIS Transport Layer Protocol
(NTLP)).
The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS NTLP is defined in [RFC5971]. Two types of NSLPs are defined: an
Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling NSLP for QoS signaling [RFC5974] and an NSLP for NATs/firewalls
[RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol [RFC5973].
(NSLP) [RFC5973].
6.7.1. Reasons for Non-deployment 6.7.1. Reasons for Non-deployment
The obstacles for deployment can be grouped into implementation- The obstacles for deployment can be grouped into implementation-
related aspects and operational aspects. related aspects and operational aspects.
* Implementation-related aspects: * Implementation-related aspects:
Although NSIS provides benefits with respect to flexibility, Although NSIS provides benefits with respect to flexibility,
mobility, and security compared to other network signaling mobility, and security compared to other network signaling
techniques, hardware vendors were reluctant to deploy this solution, techniques, hardware vendors were reluctant to deploy this
because it would require additional implementation effort and would solution, because it would require additional implementation
result in additional complexity for router implementations. effort and would result in additional complexity for router
implementations.
The NTLP mainly operates as path-coupled signaling protocol, i.e, its NTLP mainly operates as a path-coupled signaling protocol, i.e.,
messages are processed at the intermediate node's control plane that its messages are processed at the control plane of each
are also forwarding the data flows. This requires a mechanism to intermediate node that is also forwarding the data flows. This
intercept signaling packets while they are forwarded in the same requires a mechanism to intercept signaling packets while they are
manner (especially along the same path) as data packets. NSIS uses forwarded in the same manner (especially along the same path) as
the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception data packets. NSIS uses the IPv4 and IPv6 Router Alert Option
of those path-coupled signaling messages, and this technique requires (RAO) to allow for interception of those path-coupled signaling
router implementations to correctly understand and implement the messages, and this technique requires router implementations to
handling of RAOs, e.g., to only process packet with RAOs of interest correctly understand and implement the handling of RAOs, e.g., to
and to leave packets with irrelevant RAOs in the fast forwarding only process packets with RAOs of interest and to leave packets
processing path (a comprehensive discussion of these issues can be with irrelevant RAOs in the fast forwarding processing path (a
found in [RFC6398]). The latter was an issue with some router comprehensive discussion of these issues can be found in
implementations at the time of standardization. [RFC6398]). The latter was an issue with some router
implementations at the time of standardization.
Another reason is that path-coupled signaling protocols that interact Another reason is that path-coupled signaling protocols that
with routers and request manipulation of state at these routers (or interact with routers and request manipulation of state at these
any other network element in general) are under scrutiny: a packet routers (or any other network element in general) are under
(or sequence of packets) out of the mainly untrusted data path is scrutiny: a packet (or sequence of packets) out of the mainly
requesting creation and manipulation of network state. This is seen untrusted data path is requesting creation and manipulation of
as potentially dangerous (e.g., opens up a Denial of Service (DoS) network state. This is seen as potentially dangerous (e.g., opens
threat to a router's control plane) and difficult for an operator to up a DoS threat to a router's control plane) and difficult for an
control. Path-coupled signaling approaches were considered operator to control. Path-coupled signaling approaches were
problematic (see also section 3 of [RFC6398]). There are considered problematic (see also Section 3 of [RFC6398]). There
recommendations on how to secure NSIS nodes and deployments (e.g., are recommendations on how to secure NSIS nodes and deployments
[RFC5981]). (e.g., [RFC5981]).
* Operational Aspects: * Operational Aspects:
NSIS not only required trust between customers and their provider, NSIS not only required trust between customers and their provider,
but also among different providers. Especially, QoS signaling but also among different providers. In particular, QoS signaling
techniques would require some kind of dynamic service level agreement techniques would require some kind of dynamic SLA support that
support that would imply (potentially quite complex) bilateral would imply (potentially quite complex) bilateral negotiations
negotiations between different Internet service providers. This between different Internet Service Providers. This complexity was
complexity was not considered to be justified and increasing the not considered to be justified, and increasing the bandwidth (and
bandwidth (and thus avoiding bottlenecks) was cheaper than actively thus avoiding bottlenecks) was cheaper than actively managing
managing network resource bottlenecks by using path-coupled QoS network resource bottlenecks by using path-coupled QoS signaling
signaling techniques. Furthermore, an end-to-end path typically techniques. Furthermore, an end-to-end path typically involves
involves several provider domains and these providers need to closely several provider domains, and these providers need to closely
cooperate in cases of failures. cooperate in cases of failures.
6.7.2. Lessons Learned 6.7.2. Lessons Learned
One goal of NSIS was to decrease the complexity of the signaling One goal of NSIS was to decrease the complexity of the signaling
protocol, but a path-coupled signaling protocol comes with the protocol, but a path-coupled signaling protocol comes with the
intrinsic complexity of IP-based networks, beyond the complexity of intrinsic complexity of IP-based networks, beyond the complexity of
the signaling protocol itself. Sources of intrinsic complexity the signaling protocol itself. Sources of intrinsic complexity
include: include:
* the presence of asymmetric routes between endpoints and routers * the presence of asymmetric routes between endpoints and routers.
* the lack of security and trust at large in the Internet * the lack of security and trust at large in the Internet
infrastructure infrastructure.
* the presence of different trust boundaries * the presence of different trust boundaries.
* the effects of best-effort networks (e.g., robustness to packet * the effects of best-effort networks (e.g., robustness to packet
loss) loss).
* divergence from the fate sharing principle (e.g., state within the * divergence from the fate-sharing principle (e.g., state within the
network). network).
Any path-coupled signaling protocol has to deal with these realities. Any path-coupled signaling protocol has to deal with these realities.
Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to Operators view the use of IPv4 and IPv6 Router Alert Options (RAOs)
signal routers along the path from end systems with suspicion, to signal routers along the path from end systems with suspicion,
because these end systems are usually not authenticated and heavy use because these end systems are usually not authenticated and heavy use
of RAOs can easily increase the CPU load on routers that are designed of RAOs can easily increase the CPU load on routers that are designed
to process most packets using a hardware "fast path" and diverting to process most packets using a hardware "fast path" and diverting
packets containing RAO to a slower, more capable processor. packets containing RAOs to a slower, more capable processor.
6.8. IPv6 Flow Label 6.8. IPv6 Flow Labels
The suggested references for IPv6 Flow Label are: The suggested reference for IPv6 Flow Labels is:
* IPv6 Flow Label Specification [RFC6437] * "IPv6 Flow Label Specification" [RFC6437]
IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in IPv6 specifies a 20-bit Flow Label field [RFC6437], included in the
the fixed part of the IPv6 header and hence present in every IPv6 fixed part of the IPv6 header and hence present in every IPv6 packet.
packet. An endpoint sets the value in this field to one of a set of An endpoint sets the value in this field to one of a set of
pseudo-randomly assigned values. If a packet is not part of any pseudorandomly assigned values. If a packet is not part of any flow,
flow, the flow label value is set to zero [RFC3697]. A number of the flow label value is set to zero [RFC3697]. A number of Standards
Standards Track and Best Current Practice RFCs (e.g., [RFC8085], Track and Best Current Practice RFCs (e.g., [RFC8085], [RFC6437],
[RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero [RFC6438]) encourage IPv6 endpoints to set a non-zero value in this
value in this field. A multiplexing transport could choose to use field. A multiplexing transport could choose to use multiple flow
multiple flow labels to allow the network to independently forward labels to allow the network to either independently forward its
its subflows, or to use one common value for the traffic aggregate. subflows or use one common value for the traffic aggregate. The flow
The flow label is present in all fragments. IPsec was originally put label is present in all fragments. IPsec was originally put forward
forward as one important use-case for this mechanism and does encrypt as one important use case for this mechanism and does encrypt the
the field [RFC6438]. field [RFC6438].
Once set, the flow label can provide information that can help inform Once set, the flow label can provide information that can help inform
network nodes about subflows present at the transport layer, without network nodes about subflows present at the transport layer, without
needing to interpret the setting of upper layer protocol fields needing to interpret the setting of upper-layer protocol fields
[RFC6294]. This information can also be used to coordinate how [RFC6294]. This information can also be used to coordinate how
aggregates of transport subflows are grouped when queued in the aggregates of transport subflows are grouped when queued in the
network and to select appropriate per-flow forwarding when choosing network and to select appropriate per-flow forwarding when choosing
between alternate paths [RFC6438] (e.g. for Equal Cost Multipath between alternate paths [RFC6438] (e.g., for Equal-Cost Multipath
Routing (ECMP) and Link Aggregation (LAG)). (ECMP) routing and Link Aggregation Groups (LAGs)).
6.8.1. Reasons for Non-deployment 6.8.1. Reasons for Non-deployment
Despite the field being present in every IPv6 packet, the mechanism Despite the field being present in every IPv6 packet, the mechanism
did not receive as much use as originally envisioned. One reason is did not receive as much use as originally envisioned. One reason is
that to be useful it requires engagement by two different that to be useful it requires engagement by two different
stakeholders: stakeholders:
* Endpoint Implementation: * Endpoint Implementation:
For network nodes along a path to utilize the flow label there needs For network nodes along a path to utilize the flow label, there
to be a non-zero value inserted in the field [RFC6437] at the sending needs to be a non-zero value inserted in the field [RFC6437] at
endpoint. There needs to be an incentive for an endpoint to set an the sending endpoint. There needs to be an incentive for an
appropriate non-zero value. The value should appropriately reflect endpoint to set an appropriate non-zero value. The value should
the level of aggregation the traffic expects to be provided by the appropriately reflect the level of aggregation the traffic expects
network. However, this requires the stack to know granularity at to be provided by the network. However, this requires the stack
which flows should be identified (or conversely which flows should to know granularity at which flows should be identified (or,
receive aggregated treatment), i.e., which packets carry the same conversely, which flows should receive aggregated treatment),
flow label. Therefore, setting a non-zero value may result in i.e., which packets carry the same flow label. Therefore, setting
additional choices that need to be made by an application developer. a non-zero value may result in additional choices that need to be
made by an application developer.
Although the standard [RFC3697] forbids any encoding of meaning into Although the original flow label standard [RFC3697] forbids any
the flow label value, the opportunity to use the flow label as a encoding of meaning into the flow label value, the opportunity to
covert channel or to signal other meta-information may have raised use the flow label as a covert channel or to signal other meta-
concerns about setting a non-zero value [RFC6437]. information may have raised concerns about setting a non-zero
value [RFC6437].
Before methods are widely deployed to use this method, there could be Before methods are widely deployed to use this method, there could
no incentive for an endpoint to set the field. be no incentive for an endpoint to set the field.
* Operational support in network nodes: * Operational support in network nodes:
A benefit can only be realized when a network node along the path A benefit can only be realized when a network node along the path
also uses this information to inform its decisions. Network also uses this information to inform its decisions. Network
equipment (routers and/or middleboxes) need to include appropriate equipment (routers and/or middleboxes) need to include appropriate
support so they can utilize the field when making decisions about how support in order to utilize the field when making decisions about
to classify flows, or to inform forwarding choices. Use of any how to classify flows or forward packets. The use of any optional
optional feature in a network node also requires corresponding feature in a network node also requires corresponding updates to
updates to operational procedures, and therefore is normally only operational procedures and therefore is normally only introduced
introduced when the cost can be justified. when the cost can be justified.
A benefit from utilizing the flow label is expected to be increased A benefit from utilizing the flow label is expected to be
quality of experience for applications - but this comes at some increased quality of experience for applications -- but this comes
operational cost to an operator, and requires endpoints to set the at some operational cost to an operator and requires endpoints to
field. set the field.
6.8.2. Lessons Learned 6.8.2. Lessons Learned
The flow label is a general purpose header field for use by the path. The flow label is a general-purpose header field for use by the path.
Multiple uses have been proposed. One candidate use was to reduce Multiple uses have been proposed. One candidate use was to reduce
the complexity of forwarding decisions. However, modern routers can the complexity of forwarding decisions. However, modern routers can
use a "fast path", often taking advantage of hardware to accelerate use a "fast path", often taking advantage of hardware to accelerate
processing. The method can assist in more complex forwarding, such processing. The method can assist in more complex forwarding, such
as ECMP and load balancing. as ECMP routing and load balancing.
Although [RFC6437] recommended that endpoints should by default Although [RFC6437] recommended that endpoints should by default
choose uniformly-distributed labels for their traffic, the choose uniformly distributed labels for their traffic, the
specification permitted an endpoint to choose to set a zero value. specification permitted an endpoint to choose to set a zero value.
This ability of endpoints to choose to set a flow label of zero has This ability of endpoints to choose to set a flow label of zero has
had consequences on deployability: had consequences on deployability:
* Before wide-scale support by endpoints, it would be impossible to * Before wide-scale support by endpoints, it would be impossible to
rely on a non-zero flow label being set. Network nodes therefore rely on a non-zero flow label being set. Network nodes therefore
would need to also employ other techniques to realize equivalent would need to also employ other techniques to realize equivalent
functions. An example of a method is one assuming semantics of functions. An example of a method is one assuming semantics of
the source port field to provide entropy input to a network-layer the source port field to provide entropy input to a network-layer
hash. This use of a 5-tuple to classify a packet represents a hash. This use of a 5-tuple to classify a packet represents a
layering violation [RFC6294]. When other methods have been layering violation [RFC6294]. When other methods have been
deployed, they increase the cost of deploying standards-based deployed, they increase the cost of deploying standards-based
methods, even though they may offer less control to endpoints and methods, even though they may offer less control to endpoints and
result in potential interaction with other uses/interpretation of result in potential interaction with other uses/interpretation of
the field. the field.
* Even though the flow label is specified as an end-to-end field, * Even though the flow label is specified as an end-to-end field,
some network paths have been observed to not transparently forward some network paths have been observed to not transparently forward
the flow label. This could result from non-conformant equipment, the flow label. This could result from non-conformant equipment
or could indicate that some operational networks have chosen to or could indicate that some operational networks have chosen to
re-use the protocol field for other (e.g. internal purposes). reuse the protocol field for other (e.g., internal) purposes.
This results in lack of transparency, and a deployment hurdle to This results in lack of transparency, and a deployment hurdle to
endpoints expecting that they can set a flow label that is endpoints expecting that they can set a flow label that is
utilized by the network. The more recent practice of "greasing" utilized by the network. The more recent practice of "greasing"
[GREASE] would suggest that a different outcome could have been [GREASE] would suggest that a different outcome could have been
achieved if endpoints were always required to set a non-zero achieved if endpoints were always required to set a non-zero
value. value.
* [RFC1809] noted that setting the choice of the flow label value * [RFC1809] noted that setting the choice of the flow label value
can depend on the expectations of the traffic generated by an can depend on the expectations of the traffic generated by an
application, which suggests an API should be presented to control application, which suggests that an API should be presented to
the setting or policy that is used. However, many currently control the setting or policy that is used. However, many
available APIs do not have this support. currently available APIs do not have this support.
A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) A growth in the use of encrypted transports (e.g., QUIC [RFC9000])
seems likely to raise similar issues to those discussed above and seems likely to raise issues similar to those discussed above and
could motivate renewed interest in utilizing the flow label. could motivate renewed interest in utilizing the flow label.
6.9. Explicit Congestion Notification (ECN) 6.9. Explicit Congestion Notification (ECN)
The suggested references for Explicit Congestion Notification (ECN) The suggested references for Explicit Congestion Notification (ECN)
are: are:
* Recommendations on Queue Management and Congestion Avoidance in * "Recommendations on Queue Management and Congestion Avoidance in
the Internet [RFC2309] the Internet" [RFC2309]
* A Proposal to add Explicit Congestion Notification (ECN) to IP * "A Proposal to add Explicit Congestion Notification (ECN) to IP"
[RFC2481] [RFC2481]
* The Addition of Explicit Congestion Notification (ECN) to IP * "The Addition of Explicit Congestion Notification (ECN) to IP"
[RFC3168] [RFC3168]
* Implementation Report on Experiences with Various TCP RFCs * "Implementation Report on Experiences with Various TCP RFCs"
[vista-impl], slides 6 and 7 [vista-impl], slides 6 and 7
* Implementation and Deployment of ECN [SallyFloyd] * "Implementation and Deployment of ECN" (at [SallyFloyd])
In the early 1990s, the large majority of Internet traffic used TCP In the early 1990s, the large majority of Internet traffic used TCP
as its transport protocol, but TCP had no way to detect path as its transport protocol, but TCP had no way to detect path
congestion before the path was so congested that packets were being congestion before the path was so congested that packets were being
dropped, and these congestion events could affect all senders using a dropped. These congestion events could affect all senders using a
path, either by "lockout", where long-lived flows monopolized the path, either by "lockout", where long-lived flows monopolized the
queues along a path, or by "full queues", where queues remain full, queues along a path, or by "full queues", where queues remain full,
or almost full, for a long period of time. or almost full, for a long period of time.
In response to this situation, "Active Queue Management" (AQM) was In response to this situation, "Active Queue Management" (AQM) was
deployed in the network. A number of AQM disciplines have been deployed in the network. A number of AQM disciplines have been
deployed, but one common approach was that routers dropped packets deployed, but one common approach was that routers dropped packets
when a threshold buffer length was reached, so that transport when a threshold buffer length was reached, so that transport
protocols like TCP that were responsive to loss would detect this protocols like TCP that were responsive to loss would detect this
loss and reduce their sending rates. Random Early Detection (RED) loss and reduce their sending rates. Random Early Detection (RED)
was one such proposal in the IETF. As the name suggests, a router was one such proposal in the IETF. As the name suggests, a router
using RED as its AQM discipline that detected time-averaged queue using RED as its AQM discipline that detected time-averaged queue
lengths passing a threshold would choose incoming packets lengths passing a threshold would choose incoming packets
probabilistically to be dropped [RFC2309]. In response to this probabilistically to be dropped [RFC2309].
situation, "Active Queue Management" (AQM) was deployed in the
network. A number of AQM disciplines have been deployed, but one
common approach was that routers dropped packets when a threshold
buffer length was reached, so that transport protocols like TCP that
were responsive to loss would detect this loss and reduce their
sending rates. Random Early Detection (RED) was one such proposal in
the IETF. As the name suggests, a router using RED as its AQM
discipline that detected time-averaged queue lengths passing a
threshold would choose incoming packets probabilistically to be
dropped [RFC2309].
Researchers suggested that providing "explicit congestion Researchers suggested providing "explicit congestion notifications"
notifications" to senders when routers along the path detected their to senders when routers along the path detected that their queues
queues were building, so that some senders would "slow down" as if a were building, giving senders an opportunity to "slow down" as if a
loss had occurred, so that the path queues had time to drain, and the loss had occurred, giving path queues time to drain, while the path
path still had sufficient buffer capacity to accommodate bursty still had sufficient buffer capacity to accommodate bursty arrivals
arrivals of packets from other senders. This was proposed as an of packets from other senders. This was proposed as an experiment in
Experiment in [RFC2481], and standardized in [RFC3168]. [RFC2481] and standardized in [RFC3168].
A key aspect of ECN was the use of IP header fields rather than IP A key aspect of ECN was the use of IP header fields rather than IP
options to carry explicit congestion notifications, since the options to carry explicit congestion notifications, since the
proponents recognized that proponents recognized that
Many routers process the "regular" headers in IP packets more Many routers process the "regular" headers in IP packets more
efficiently than they process the header information in IP efficiently than they process the header information in IP
options. options.
Unlike most of the Path Aware technologies included in this document, Unlike most of the Path Aware technologies included in this document,
the story of ECN continues to the present day, and encountered a the story of ECN continues to the present day and encountered a large
large number of Lessons Learned during that time. The early history number of Lessons Learned during that time. The early history of ECN
of ECN (non-)deployment provides Lessons Learned that were not (non-)deployment provides Lessons Learned that were not captured by
captured by other contributions in Section 6, so that is the emphasis other contributions in Section 6, so that is the emphasis in this
in this section of the document. section of the document.
6.9.1. Reasons for Non-deployment 6.9.1. Reasons for Non-deployment
There are at least three sub-stories - ECN deployment in clients, ECN ECN deployment relied on three factors -- support in client
deployment in routers, and AQM deployment in operational networks. implementations, support in router implementations, and deployment
All three sub-stories mattered. decisions in operational networks.
The proponents of ECN did so much right, anticipating many of the The proponents of ECN did so much right, anticipating many of the
Lessons Learned now recognized in Section 4. They recognized the Lessons Learned now recognized in Section 4. They recognized the
need to support incremental deployment (Section 4.2). They need to support incremental deployment (Section 4.2). They
considered the impact on router throughput (Section 4.8). They even considered the impact on router throughput (Section 4.8). They even
considered trust issues between end nodes and the network, both for considered trust issues between end nodes and the network, for both
non-compliant end nodes (Section 4.10) and non-compliant routers non-compliant end nodes (Section 4.10) and non-compliant routers
(Section 4.9). (Section 4.9).
They were rewarded with ECN being implemented in major operating They were rewarded with ECN being implemented in major operating
systems, both for end nodes and for routers. A number of systems, for both end nodes and routers. A number of implementations
implementations are listed under "Implementation and Deployment of are listed under "Implementation and Deployment of ECN" at
ECN" at [SallyFloyd]. [SallyFloyd].
What they did not anticipate, was routers that would crash, when they What they did not anticipate was routers that would crash when they
saw bits 6 and 7 in the IPv4 TOS octet [RFC0791]/IPv6 Traffic Class saw bits 6 and 7 in the IPv4 Type of Service (TOS) octet [RFC0791] /
field [RFC2460], which [RFC2481] redefined to be "currently unused", IPv6 Traffic Class field [RFC2460], which [RFC2481] redefined to be
being set to a non-zero value. "Currently Unused", being set to a non-zero value.
As described in [vista-impl], As described in [vista-impl] ("IGD" stands for "Intermediate Gateway
Device"),
Intermediate Gateway Device problem #1: one of the most popular | IGD problem #1: one of the most popular versions from one of the
versions from one of the most popular vendors. When a data packet | most popular vendors. When a data packet arrives with either
arrives with either ECT(0) or ECT(1) (indicating successful ECN | ECT(0) or ECT(1) (indicating successful ECN capability
capability negotiation) indicated, router crashed. Cannot be | negotiation) indicated, router crashed. Cannot be recovered at
recovered at TCP layer (sic) | TCP layer [sic]
This implementation, which would be run on a significant percentage This implementation, which would be run on a significant percentage
of Internet end nodes, was shipped with ECN disabled, as was true for of Internet end nodes, was shipped with ECN disabled, as was true for
several of the other implementations listed under "Implementation and several of the other implementations listed under "Implementation and
Deployment of ECN" at [SallyFloyd]. Even if subsequent router Deployment of ECN" at [SallyFloyd]. Even if subsequent router
vendors fixed these implementations, ECN was still disabled on end vendors fixed these implementations, ECN was still disabled on end
nodes, and given the tradeoff between the benefits of enabling ECN nodes, and given the trade-off between the benefits of enabling ECN
(somewhat better behavior during congestion) and the risks of (somewhat better behavior during congestion) and the risks of
enabling ECN (possibly crashing a router somewhere along the path), enabling ECN (possibly crashing a router somewhere along the path),
ECN tended to stay disabled on implementations that supported ECN for ECN tended to stay disabled on implementations that supported ECN for
decades afterwards. decades afterwards.
6.9.2. Lessons Learned 6.9.2. Lessons Learned
Of the contributions included in Section 6, ECN may be unique in Of the contributions included in Section 6, ECN may be unique in
providing these lessons: providing these lessons:
* Even if you do everything right, you may trip over implementation * Even if you do everything right, you may trip over implementation
bugs in devices you know nothing about, that will cause severe bugs in devices you know nothing about, that will cause severe
problems that prevent successful deployment of your path aware problems that prevent successful deployment of your Path Aware
technology. technology.
* After implementations disable your Path Aware technology, it may * After implementations disable your Path Aware technology, it may
take years, or even decades, to convince implementers to re-enable take years, or even decades, to convince implementers to re-enable
it by default. it by default.
These two lessons, taken together, could be summarized as "you get These two lessons, taken together, could be summarized as "you get
one chance to get it right". one chance to get it right."
During discussion of ECN at [PANRG-110], we noted that "you get one During discussion of ECN at [PANRG-110], we noted that "you get one
chance to get it right" isn't quite correct today, because operating chance to get it right" isn't quite correct today, because operating
systems on so many host systems are frequently updated, and transport systems on so many host systems are frequently updated, and transport
protocols like QUIC [I-D.ietf-quic-transport] are being implemented protocols like QUIC [RFC9000] are being implemented in user space and
in user space, and can be updated without touching installed can be updated without touching installed operating systems. Neither
operating systems. Neither of these factors were true in the early of these factors were true in the early 2000s.
2000s.
We think that these restatements of the ECN Lessons Learned are more We think that these restatements of the ECN Lessons Learned are more
useful for current implementers: useful for current implementers:
* Even if you do everything right, you may trip over implementation * Even if you do everything right, you may trip over implementation
bugs in devices you know nothing about, that will cause severe bugs in devices you know nothing about, that will cause severe
problems that prevent successful deployment of your path aware problems that prevent successful deployment of your Path Aware
technology. Testing before deployment isn't enough to ensure technology. Testing before deployment isn't enough to ensure
successful deployment. It is also necessary to "deploy gently", successful deployment. It is also necessary to "deploy gently",
which often means deploying for a small subset of users to gain which often means deploying for a small subset of users to gain
experience, and implementing feedback mechanisms to detect that experience and implementing feedback mechanisms to detect that
user experience is being degraded. user experience is being degraded.
* After implementations disable your Path Aware technology, it may * After implementations disable your Path Aware technology, it may
take years, or even decades, to convince implementers to re-enable take years, or even decades, to convince implementers to re-enable
it by default. This might be based on the difficulty of it by default. This might be based on the difficulty of
distributing implementations that enable it by default, but are distributing implementations that enable it by default, but it is
just as likely to be based on the "bad taste in the mouth" that just as likely to be based on the "bad taste in the mouth" that
implementers have after an unsuccessful deployment attempt that implementers have after an unsuccessful deployment attempt that
degraded user experience. degraded user experience.
With these expansions, the two lessons, taken together, could be more With these expansions, the two lessons, taken together, could be more
helpfully summarized as "plan for failure" - anticipate what your helpfully summarized as "plan for failure" -- anticipate what your
next step will be, if initial deployment is unsuccessful. next step will be, if initial deployment is unsuccessful.
ECN deployment was also hindered by non-deployment of AQM in many ECN deployment was also hindered by non-deployment of AQM in many
devices, because of operator interest in QoS features provided in the devices, because of operator interest in QoS features provided in the
network, rather than using the network to assist end systems in network, rather than using the network to assist end systems in
providing for themselves. But that's another story, and the AQM providing for themselves. But that's another story, and the AQM
Lessons Learned are already covered in other contributions in Lessons Learned are already covered in other contributions in
Section 6. Section 6.
7. Security Considerations 7. Security Considerations
This document describes Path Aware techniques that were not adopted This document describes Path Aware techniques that were not adopted
and widely deployed on the Internet, so it doesn't affect the and widely deployed on the Internet, so it doesn't affect the
security of the Internet. security of the Internet.
If this document meets its goals, we may develop new techniques for If this document meets its goals, we may develop new techniques for
Path Aware Networking that would affect the security of the Internet, Path Aware networking that would affect the security of the Internet,
but security considerations for those techniques will be described in but security considerations for those techniques will be described in
the corresponding RFCs that specify them. the corresponding RFCs that specify them.
8. IANA Considerations 8. IANA Considerations
This document makes no requests of IANA. This document has no IANA actions.
9. Acknowledgments
Initial material for Section 6.1 on ST2 was provided by Gorry
Fairhurst.
Initial material for Section 6.2 on IntServ was provided by Ron
Bonica.
Initial material for Section 6.3 on Quick-Start TCP was provided by
Michael Scharf, who also provided suggestions to improve this section
after it was edited.
Initial material for Section 6.4 on ICMP Source Quench was provided
by Gorry Fairhurst.
Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN)
was provided by Spencer Dawkins.
Section 6.6 on Shim6 builds on initial material describing obstacles
provided by Erik Nordmark, with background added by Spencer Dawkins.
Initial material for Section 6.7 on Next Steps In Signaling (NSIS)
was provided by Roland Bless and Martin Stiemerling.
Initial material for Section 6.8 on IPv6 Flow Labels was provided by
Gorry Fairhurst.
Initial material for Section 6.9 on Explicit Congestion Notification
was provided by Spencer Dawkins.
Our thanks to Adrian Farrel, Bob Briscoe, C.M. Heard, David Black,
Eric Kinnear, Erik Auerswald, Gorry Fairhurst, Jake Holland, Joe
Touch, Joeri de Ruiter, Kireeti Kompella, Mohamed Boucadair, Roland
Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy, who provided
review comments on this document as a "work in process".
Mallory Knodel reviewed this document for the Internet Research
Steering Group, and provided many helpful suggestions.
David Oran also provided helpful comments and text suggestions on
this document during Internet Research Steering Group balloting. In
particular, Section 5 reflects his review.
Benjamin Kaduk and Rob Wilton provided helpful comments during
Internet Engineering Steering Group conflict review.
Special thanks to Adrian Farrel for helping Spencer navigate the
twisty little passages of Flow Specs and Filter Specs in IntServ,
RSVP, MPLS, and BGP. They are all alike, except when they are
different [Colossal-Cave].
10. Informative References 9. Informative References
[Colossal-Cave] [Colossal-Cave]
"Wikipedia Page for Colossal Cave Adventure", January Wikipedia, "Colossal Cave Adventure", June 2021,
2019,
<https://en.wikipedia.org/wiki/Colossal_Cave_Adventure>. <https://en.wikipedia.org/wiki/Colossal_Cave_Adventure>.
[Conviva] "Conviva Precision : Data Sheet", December 2020, [Conviva] "Conviva Precision : Data Sheet", January 2021,
<https://www.conviva.com/datasheets/precision-delivery- <https://www.conviva.com/datasheets/precision-delivery-
intelligence/>. intelligence/>.
[FARRELL-ETM]
Farrell, S., "We're gonna need a bigger threat model",
Work in Progress, Internet-Draft, draft-farrell-etm-03, 6
July 2019,
<https://tools.ietf.org/html/draft-farrell-etm-03>.
[GREASE] Thomson, M., "Long-term Viability of Protocol Extension [GREASE] Thomson, M., "Long-term Viability of Protocol Extension
Mechanisms", July 2019, <https://tools.ietf.org/html/ Mechanisms", Work in Progress, Internet-Draft, draft-iab-
draft-iab-use-it-or-lose-it-00>. use-it-or-lose-it-00, 7 August 2019,
<https://tools.ietf.org/html/draft-iab-use-it-or-lose-it-
00>.
[I-D.arkko-arch-internet-threat-model] [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol",
September 1979,
<https://www.rfc-editor.org/ien/ien119.txt>.
[INTERNET-THREAT-MODEL]
Arkko, J., "Changes in the Internet Threat Model", Work in Arkko, J., "Changes in the Internet Threat Model", Work in
Progress, Internet-Draft, draft-arkko-arch-internet- Progress, Internet-Draft, draft-arkko-arch-internet-
threat-model-01, 8 July 2019, <http://www.ietf.org/ threat-model-01, 8 July 2019,
internet-drafts/draft-arkko-arch-internet-threat-model- <https://tools.ietf.org/html/draft-arkko-arch-internet-
01.txt>. threat-model-01>.
[I-D.farrell-etm]
Farrell, S., "We're gonna need a bigger threat model",
Work in Progress, Internet-Draft, draft-farrell-etm-03, 6
July 2019, <http://www.ietf.org/internet-drafts/draft-
farrell-etm-03.txt>.
[I-D.ietf-quic-transport]
Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed
and Secure Transport", Work in Progress, Internet-Draft,
draft-ietf-quic-transport-34, 14 January 2021,
<http://www.ietf.org/internet-drafts/draft-ietf-quic-
transport-34.txt>.
[I-D.ietf-tsvwg-intserv-multiple-tspec] [INTSERV-MULTIPLE-TSPEC]
Polk, J. and S. Dhesikan, "Integrated Services (IntServ) Polk, J. and S. Dhesikan, "Integrated Services (IntServ)
Extension to Allow Signaling of Multiple Traffic Extension to Allow Signaling of Multiple Traffic
Specifications and Multiple Flow Specifications in Specifications and Multiple Flow Specifications in
RSVPv1", Work in Progress, Internet-Draft, draft-ietf- RSVPv1", Work in Progress, Internet-Draft, draft-ietf-
tsvwg-intserv-multiple-tspec-02, 25 February 2013, tsvwg-intserv-multiple-tspec-02, 25 February 2013,
<http://www.ietf.org/internet-drafts/draft-ietf-tsvwg- <https://tools.ietf.org/html/draft-ietf-tsvwg-intserv-
intserv-multiple-tspec-02.txt>. multiple-tspec-02>.
[I-D.irtf-panrg-path-properties]
Enghardt, T. and C. Krahenbuhl, "A Vocabulary of Path
Properties", Work in Progress, Internet-Draft, draft-irtf-
panrg-path-properties-01, 7 September 2020,
<http://www.ietf.org/internet-drafts/draft-irtf-panrg-
path-properties-01.txt>.
[I-D.irtf-panrg-questions]
Trammell, B., "Current Open Questions in Path Aware
Networking", Work in Progress, Internet-Draft, draft-irtf-
panrg-questions-08, 23 December 2020,
<http://www.ietf.org/internet-drafts/draft-irtf-panrg-
questions-08.txt>.
[IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol",
September 1979,
<https://www.rfc-editor.org/ien/ien119.txt>.
[ITAT] "IAB Workshop on Internet Technology Adoption and [ITAT] "IAB Workshop on Internet Technology Adoption and
Transition (ITAT)", December 2013, Transition (ITAT) 2013", December 2013,
<https://www.iab.org/activities/workshops/itat/>. <https://www.iab.org/activities/workshops/itat/>.
[model-t] "Model-t -- Discussions of changes in Internet deployment [model-t] "Model-t -- Discussions of changes in Internet deployment
patterns and their impact on the Internet threat model", patterns and their impact on the Internet threat model",
n.d., <https://www.iab.org/mailman/listinfo/model-t>. model-t mailing list,
<https://www.iab.org/mailman/listinfo/model-t>.
[MOPS-109-Min] [MOPS-109-Min]
"Media Operations Working Group - IETF-109 Minutes", "Media Operations Working Group - IETF 109 Minutes",
November 2020, November 2020,
<https://datatracker.ietf.org/meeting/109/materials/ <https://datatracker.ietf.org/meeting/109/materials/
minutes-109-mops-00>. minutes-109-mops-00>.
[MP-TCP] "Multipath TCP Working Group Home Page", n.d., [MP-TCP] "Multipath TCP Working Group Home Page",
<https://datatracker.ietf.org/wg/mptcp/about/>. <https://datatracker.ietf.org/wg/mptcp/>.
[NANOG-35] "North American Network Operators Group NANOG-35 Agenda", [NANOG-35] "NANOG 35 Agenda", North American Network Operators' Group
October 2005, (NANOG), October 2005,
<https://www.nanog.org/meetings/nanog35/agenda>. <https://archive.nanog.org/meetings/nanog35/agenda>.
[NSIS-CHARTER-2001] [NSIS-CHARTER-2001]
"Next Steps In Signaling Working Group Charter", March "Next Steps In Signaling Working Group Charter", March
2011, 2011,
<https://datatracker.ietf.org/doc/charter-ietf-nsis/>. <https://datatracker.ietf.org/doc/charter-ietf-nsis/>.
[PANRG] "Path Aware Networking Research Group (Home Page)", n.d., [PANRG] "Path Aware Networking Research Group Home Page",
<https://irtf.org/panrg>. <https://irtf.org/panrg>.
[PANRG-103-Min] [PANRG-103-Min]
"Path Aware Networking Research Group - IETF-103 Minutes", "Path Aware Networking Research Group - IETF 103 Minutes",
November 2018, November 2018,
<https://datatracker.ietf.org/doc/minutes-103-panrg/>. <https://datatracker.ietf.org/doc/minutes-103-panrg/>.
[PANRG-105-Min] [PANRG-105-Min]
"Path Aware Networking Research Group - IETF-105 Minutes", "Path Aware Networking Research Group - IETF 105 Minutes",
July 2019, July 2019,
<https://datatracker.ietf.org/doc/minutes-105-panrg/>. <https://datatracker.ietf.org/doc/minutes-105-panrg/>.
[PANRG-106-Min] [PANRG-106-Min]
"Path Aware Networking Research Group - IETF-106 Minutes", "Path Aware Networking Research Group - IETF 106 Minutes",
November 2019, November 2019,
<https://datatracker.ietf.org/doc/minutes-106-panrg/>. <https://datatracker.ietf.org/doc/minutes-106-panrg/>.
[PANRG-110] [PANRG-110]
"Path Aware Networking Research Group - IETF-110", July "Path Aware Networking Research Group - IETF 110", March
2017, 2021,
<https://datatracker.ietf.org/meeting/110/sessions/panrg>. <https://datatracker.ietf.org/meeting/110/session/panrg>.
[PANRG-99] "Path Aware Networking Research Group - IETF-99", July [PANRG-99] "Path Aware Networking Research Group - IETF 99", July
2017, 2017,
<https://datatracker.ietf.org/meeting/99/sessions/panrg>. <https://datatracker.ietf.org/meeting/99/session/panrg>.
[PANRG-PATH-PROPERTIES]
Enghardt, T. and C. Krähenbühl, "A Vocabulary of Path
Properties", Work in Progress, Internet-Draft, draft-irtf-
panrg-path-properties-02, 22 February 2021,
<https://tools.ietf.org/html/draft-irtf-panrg-path-
properties-02>.
[PANRG-QUESTIONS]
Trammell, B., "Current Open Questions in Path Aware
Networking", Work in Progress, Internet-Draft, draft-irtf-
panrg-questions-09, 16 April 2021,
<https://tools.ietf.org/html/draft-irtf-panrg-questions-
09>.
[PATH-Decade] [PATH-Decade]
Bonaventure, O., "A Decade of Path Awareness", July 2017, Bonaventure, O., "A Decade of Path Awareness", July 2017,
<https://datatracker.ietf.org/doc/slides-99-panrg-a- <https://datatracker.ietf.org/doc/slides-99-panrg-a-
decade-of-path-awareness/>. decade-of-path-awareness/>.
[QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and [QS-SAT] Secchi, R., Sathiaseelan, A., Potortì, F., Gotta, A., and
G. Fairhurst, "Using Quick-Start to enhance TCP-friendly G. Fairhurst, "Using Quick-Start to enhance TCP-friendly
rate control performance in bidirectional satellite rate control performance in bidirectional satellite
networks", 2009, networks", DOI 10.1002/sat.929, May 2009,
<https://dl.acm.org/citation.cfm?id=3160304.3160305>. <https://dl.acm.org/citation.cfm?id=3160304.3160305>.
[QUIC-WG] "QUIC Working Group Home Page", n.d.,
<https://datatracker.ietf.org/wg/quic/about/>.
[RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791,
DOI 10.17487/RFC0791, September 1981, DOI 10.17487/RFC0791, September 1981,
<https://www.rfc-editor.org/info/rfc791>. <https://www.rfc-editor.org/info/rfc791>.
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, DOI 10.17487/RFC0792, September 1981, RFC 792, DOI 10.17487/RFC0792, September 1981,
<https://www.rfc-editor.org/info/rfc792>. <https://www.rfc-editor.org/info/rfc792>.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, [RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981, RFC 793, DOI 10.17487/RFC0793, September 1981,
skipping to change at line 2041 skipping to change at line 1977
DOI 10.17487/RFC8655, October 2019, DOI 10.17487/RFC8655, October 2019,
<https://www.rfc-editor.org/info/rfc8655>. <https://www.rfc-editor.org/info/rfc8655>.
[RFC8793] Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran, [RFC8793] Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran,
D., and C. Tschudin, "Information-Centric Networking D., and C. Tschudin, "Information-Centric Networking
(ICN): Content-Centric Networking (CCNx) and Named Data (ICN): Content-Centric Networking (CCNx) and Named Data
Networking (NDN) Terminology", RFC 8793, Networking (NDN) Terminology", RFC 8793,
DOI 10.17487/RFC8793, June 2020, DOI 10.17487/RFC8793, June 2020,
<https://www.rfc-editor.org/info/rfc8793>. <https://www.rfc-editor.org/info/rfc8793>.
[RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
Multiplexed and Secure Transport", RFC 9000,
DOI 10.17487/RFC9000, May 2021,
<https://www.rfc-editor.org/info/rfc9000>.
[SAAG-105-Min] [SAAG-105-Min]
"Security Area Open Meeting - IETF-105 Minutes", July "Security Area Open Meeting - IETF 105 Minutes", July
2019, <https://datatracker.ietf.org/meeting/105/materials/ 2019, <https://datatracker.ietf.org/meeting/105/materials/
minutes-105-saag-00>. minutes-105-saag-00>.
[SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an
appropriate sending rate over an underutilized network appropriate sending rate over an underutilized network
path", Computer Networking Volume 51, Number 7, May 2007. path", Computer Networks: The International Journal of
Computer and Telecommunications Networking, Volume 51,
Number 7, DOI 10.1016/j.comnet.2006.11.006, May 2007,
<https://dl.acm.org/doi/10.1016/j.comnet.2006.11.006>.
[SallyFloyd] [SallyFloyd]
Floyd, S., "ECN (Explicit Congestion Notification) in TCP/ Floyd, S., "ECN (Explicit Congestion Notification) in TCP/
IP", n.d., <https://www.icir.org/floyd/ecn.html>. IP", June 2009, <https://www.icir.org/floyd/ecn.html>.
[Sch11] Scharf, M., "Fast Startup Internet Congestion Control for [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for
Broadband Interactive Applications", Ph.D. Thesis, Broadband Interactive Applications", Ph.D. Thesis,
University of Stuttgart, April 2011. University of Stuttgart, April 2011.
[Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB
IPv6 Multihoming Panel at NANOG 35", NANOG North American IPv6 Multihoming Panel at NANOG 35", North American
Network Operator Group, October 2005, Network Operators' Group (NANOG), October 2005,
<https://www.youtube.com/watch?v=ji6Y_rYHAQs>. <https://www.youtube.com/watch?v=ji6Y_rYHAQs>.
[TRIGTRAN-55] [TRIGTRAN-55]
"Triggers for Transport BOF at IETF 55", July 2003, "Triggers for Transport BOF at IETF 55", November 2002,
<https://www.ietf.org/proceedings/55/239.htm>. <https://www.ietf.org/proceedings/55/239.htm>.
[TRIGTRAN-56] [TRIGTRAN-56]
"Triggers for Transport BOF at IETF 56", November 2003, "Triggers for Transport BOF at IETF 56", March 2003,
<https://www.ietf.org/proceedings/56/251.htm>. <https://www.ietf.org/proceedings/56/251.htm>.
[vista-impl] [vista-impl]
Sridharan, M., Bansal, D., and D. Thaler, "Implementation Sridharan, M., Bansal, D., and D. Thaler, "Implementation
Report on Experiences with Various TCP RFCs", November Report on Experiences with Various TCP RFCs", March 2007,
2003, <https://www.ietf.org/proceedings/68/slides/tsvarea- <https://www.ietf.org/proceedings/68/slides/tsvarea-3/
3/sld1.htm>. sld1.htm>.
Acknowledgments
Initial material for Section 6.1 on ST2 was provided by Gorry
Fairhurst.
Initial material for Section 6.2 on IntServ was provided by Ron
Bonica.
Initial material for Section 6.3 on Quick-Start TCP was provided by
Michael Scharf, who also provided suggestions to improve this section
after it was edited.
Initial material for Section 6.4 on ICMP Source Quench was provided
by Gorry Fairhurst.
Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN)
was provided by Spencer Dawkins.
Section 6.6 on Shim6 builds on initial material describing obstacles,
which was provided by Erik Nordmark, with background added by Spencer
Dawkins.
Initial material for Section 6.7 on Next Steps in Signaling (NSIS)
was provided by Roland Bless and Martin Stiemerling.
Initial material for Section 6.8 on IPv6 Flow Labels was provided by
Gorry Fairhurst.
Initial material for Section 6.9 on Explicit Congestion Notification
was provided by Spencer Dawkins.
Our thanks to Adrian Farrel, Bob Briscoe, C.M. Heard, David Black,
Eric Kinnear, Erik Auerswald, Gorry Fairhurst, Jake Holland, Joe
Touch, Joeri de Ruiter, Kireeti Kompella, Mohamed Boucadair, Randy
Presuhn, Roland Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy,
who provided review comments on this document as a "work in process".
Mallory Knodel reviewed this document for the Internet Research
Steering Group and provided many helpful suggestions.
David Oran also provided helpful comments and text suggestions on
this document during Internet Research Steering Group balloting. In
particular, Section 5 reflects his review.
Benjamin Kaduk, Martin Duke, and Rob Wilton provided helpful comments
during Internet Engineering Steering Group conflict review.
Special thanks to Adrian Farrel for helping Spencer navigate the
twisty little passages of Flow Specs and Filter Specs in IntServ,
RSVP, MPLS, and BGP. They are all alike, except when they are
different [Colossal-Cave].
Author's Address Author's Address
Spencer Dawkins (editor) Spencer Dawkins (editor)
Tencent America Tencent America
United States of America United States of America
Email: spencerdawkins.ietf@gmail.com Email: spencerdawkins.ietf@gmail.com
 End of changes. 281 change blocks. 
717 lines changed or deleted 713 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/