rfc8626xml2.original.xml   rfc8626.xml 
<?xml version="1.0" encoding="US-ASCII"?> <?xml version='1.0' encoding='utf-8'?>
<!-- This template is for creating an Internet Draft using xml2rfc, <!DOCTYPE rfc SYSTEM "rfc2629-xhtml.ent">
which is available here: http://xml.resource.org. -->
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
]>
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<rfc category="std" ipr="trust200902" docName="draft-ietf-detnet-architecture-13
">
<rfc xmlns:xi="http://www.w3.org/2001/XInclude" number="8626" docName="draft-iet
f-detnet-architecture-13" submissionType="IETF" category="std" consensus="yes" i
pr="trust200902" obsoletes="" updates="" xml:lang="en" sortRefs="true" tocInclud
e="true" symRefs="true" version="3">
<!-- xml2rfc v2v3 conversion 2.31.0 -->
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?> <?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc sortrefs="yes"?>
<?rfc iprnotified-"no" ?>
<?rfc authorship="yes"?>
<?rfc tocappendix="yes"?>
<?rfc strict="yes" ?>
<!-- give errors regarding ID-nits and DTD validation -->
<!-- control the table of contents (ToC) -->
<?rfc toc="yes"?>
<!-- generate a ToC -->
<?rfc tocdepth="4"?>
<!-- the number of levels of subsections in ToC. default: 3 -->
<!-- control references -->
<?rfc symrefs="yes"?>
<!-- use symbolic references tags, i.e, [RFC2119] instead of [1] -->
<?rfc sortrefs="yes" ?>
<!-- sort the reference entries alphabetically -->
<!-- control vertical white space
(using these PIs as follows is recommended by the RFC Editor) -->
<?rfc compact="no" ?>
<front> <front>
<title>Deterministic Networking Architecture</title> <title>Deterministic Networking Architecture</title>
<author initials="N" surname="Finn" fullname="Norman Finn" > <seriesInfo name="RFC" value="8626"/>
<organization> <author initials="N" surname="Finn" fullname="Norman Finn">
<organization>
Huawei Huawei
</organization> </organization>
<address> <address>
<postal> <postal>
<street>3101 Rio Way</street> <street>3101 Rio Way</street>
<city>Spring Valley</city> <city>Spring Valley</city>
<region>California</region> <region>California</region>
<code>91977</code> <code>91977</code>
<country>US</country> <country>United States of America</country>
</postal> </postal>
<phone>+1 925 980 6430</phone> <phone>+1 925 980 6430</phone>
<email>norman.finn@mail01.huawei.com</email> <email>nfinn@nfinnconsulting.com</email>
</address> </address>
</author> </author>
<author initials="P" surname="Thubert" fullname="Pascal Thubert"> <author initials="P" surname="Thubert" fullname="Pascal Thubert">
<organization abbrev="Cisco"> <organization abbrev="Cisco">
Cisco Systems Cisco Systems
</organization> </organization>
<address> <address>
<postal> <postal>
<street>Village d'Entreprises Green Side</street> <street>Village d'Entreprises Green Side</street>
<street>400, Avenue de Roumanille</street> <street>400, Avenue de Roumanille</street>
<street>Batiment T3</street> <street>Batiment T3</street>
<city>Biot - Sophia Antipolis</city> <city>Biot - Sophia Antipolis</city>
<code>06410</code> <code>06410</code>
<country>FRANCE</country> <country>France</country>
</postal> </postal>
<phone>+33 4 97 23 26 34</phone> <phone>+33 4 97 23 26 34</phone>
<email>pthubert@cisco.com</email> <email>pthubert@cisco.com</email>
</address> </address>
</author> </author>
<author fullname="Bal&aacute;zs Varga" initials="B." surname="Varga"> <author fullname="Balázs Varga" initials="B." surname="Varga">
<organization>Ericsson</organization> <organization>Ericsson</organization>
<address> <address>
<postal> <postal>
<street>Magyar tudosok korutja 11</street> <street>Magyar tudosok korutja 11</street>
<city>Budapest</city> <city>Budapest</city>
<country>Hungary</country> <country>Hungary</country>
<code>1117</code> <code>1117</code>
</postal> </postal>
<email>balazs.a.varga@ericsson.com</email> <email>balazs.a.varga@ericsson.com</email>
</address> </address>
</author> </author>
<author fullname="J&aacute;nos Farkas" initials="J." surname="Farkas"> <author fullname="János Farkas" initials="J." surname="Farkas">
<organization>Ericsson</organization> <organization>Ericsson</organization>
<address> <address>
<postal> <postal>
<street>Magyar tudosok korutja 11</street> <street>Magyar tudosok korutja 11</street>
<city>Budapest</city> <city>Budapest</city>
<country>Hungary</country> <country>Hungary</country>
<code>1117</code> <code>1117</code>
</postal> </postal>
<email>janos.farkas@ericsson.com</email> <email>janos.farkas@ericsson.com</email>
</address> </address>
</author> </author>
<date/> <date year="2019" month="September"/>
<area>Internet</area>
<area>Internet</area> <workgroup>DetNet</workgroup>
<keyword>&gt;TSN, Bounded Lantency, Reliable Networking, Available Networkin
<workgroup>DetNet</workgroup> g</keyword>
<abstract>
<abstract> <t>
<t>
This document provides the overall architecture for This document provides the overall architecture for
Deterministic Networking (DetNet), which Deterministic Networking (DetNet), which provides a capability
provides a capability to carry specified unicast or multicast to carry specified unicast or multicast data flows for
data flows for real-time applications with extremely low data los real-time applications with extremely low data loss rates and
s rates and bounded bounded latency within a network domain. Techniques used
latency within a network domain. Techniques used include: 1) res include 1) reserving data-plane resources for individual (or
erving data plane resources for individual aggregated) DetNet flows in some or all of the intermediate
(or aggregated) DetNet flows in some or all of the intermediate n nodes along the path of the flow, 2) providing explicit routes
odes along for DetNet flows that do not immediately change with the
the path of the flow; 2) providing explicit routes for DetNet flo network topology, and 3) distributing data from DetNet flow
ws that do not packets over time and/or space to ensure delivery of each
immediately change with the network topology; and 3) distributing packet's data in spite of the loss of a path. DetNet operates
data from DetNet flow packets at the IP layer and delivers service over lower-layer
over time and/or space to ensure delivery of each packet's data i technologies such as MPLS and Time-Sensitive
n spite of the loss of a path. Networking (TSN) as defined by IEEE 802.1.
DetNet operates at the IP layer and delivers service over lower layer te </t>
chnologies such as MPLS </abstract>
and IEEE 802.1 Time-Sensitive Networking (TSN).
</t>
</abstract>
</front> </front>
<middle> <middle>
<section anchor="introduction" numbered="true" toc="default">
<!-- **************************************************************** --> <name>Introduction</name>
<!-- **************************************************************** -->
<!-- **************************************************************** -->
<!-- **************************************************************** -->
<section anchor='introduction' title="Introduction">
<t> <t>
This document provides the overall architecture for Deterministic This document provides the overall architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of dat Networking (DetNet), which provides a capability for the delivery of
a data flows with extremely low packet loss rates and bounded end-to-end
flows with extremely delivery latency. DetNet is for networks that are under a single
low packet loss rates and bounded end-to-end delivery latency. administrative control or within a closed group of administrative
DetNet is for networks that are under a single administrative con control; these include campus-wide networks and private WANs. DetNet
trol or is not for large groups of domains such as the Internet. </t>
within a closed group of administrative control; these include ca <t>
mpus-wide DetNet operates at the IP layer and delivers service over lower-layer
networks and private WANs. DetNet is not for large groups of doma technologies such as MPLS and IEEE 802.1 Time-Sensitive Networking
ins such (TSN).
as the Internet.
</t><t>
DetNet operates at the IP layer and delivers service over
lower layer technologies such as MPLS and IEEE 802.1 Time-Sensitive Netw
orking (TSN).
DetNet accomplishes these goals by dedicating network resources such as
link bandwidth
and buffer space to DetNet flows and/or
classes of DetNet flows, and by replicating packets along multipl
e paths.
Unused reserved resources are available to non-DetNet
packets as long as all guarantees are fulfilled.
</t><t>
The <xref target="I-D.ietf-detnet-problem-statement">Deterministi
c Networking
Problem Statement</xref> introduces Deterministic Networking, and
<xref target="I-D.ietf-detnet-use-cases">Deterministic Networking
Use Cases</xref> summarizes the need for it.
See <xref target="I-D.ietf-detnet-dp-sol-mpls"/> and
<xref target="I-D.ietf-detnet-dp-sol-ip"/> for specific technique
s
that can be used to identify DetNet flows and assign them to spec
ific paths
through a network.
</t><t>
A goal of DetNet is a converged network in all respects including
the convergence of sensitive non-IP networks
onto a common network infrastructure. The presence
of DetNet flows does not preclude non-DetNet flows, and the benef
its offered
DetNet flows should not, except in extreme cases, prevent existin
g Quality of
Service (QoS) mechanisms from operating in a
normal fashion, subject to the bandwidth required for the DetNet
flows. A
single source-destination pair can trade both DetNet and non-DetN
et flows.
End systems and applications need not instantiate special interfa
ces for DetNet flows.
Networks are not restricted to certain topologies; connectivity i
s not restricted.
Any application that generates a data flow that can be usefully
characterized as having a maximum bandwidth should be able to tak
e advantage
of DetNet, as long as the necessary resources can be reserved. R
eservations
can be made by the application itself, via network management, by
an
application's controller, or by other means, e.g., a dynamic cont
rol plane
(e.g., <xref target="RFC2205"/>). QoS requirements of DetNet flow
s can be met if all
network nodes in a DetNet domain implement DetNet capabilities. D
etNet nodes can
be interconnected with different sub-network technologies
(<xref target="sec_dt_dp"/>), where the nodes of the subnet are n
ot
DetNet aware (<xref target="netref"/>).
</t><t>
Many applications that are intended to be served by Deterministic
Networking require the ability
to synchronize the clocks in end systems to a sub-microsecond acc
uracy. Some
of the queue control techniques defined in <xref target="QueuingM
odels"/> also
require time synchronization among network nodes. The means used
to achieve
time synchronization are not addressed in this document. DetNet
can accommodate
various time synchronization techniques and profiles that are def
ined elsewhere
to address the needs of different market segments.
</t>
</section>
<section title="Terminology"> DetNet provides a reliable and available service by dedicating network
<section title="Terms used in this document"> resources such as link bandwidth and buffer space to DetNet flows and/or
<t> classes of DetNet flows, and by replicating packets along multiple paths.
Unused reserved resources are available to non-DetNet packets as long as all
guarantees are fulfilled. </t>
<t> The <xref target="RFC8557" format="default">"Deterministic
Networking Problem Statement"</xref> introduces DetNet, and <xref target="RFC857
8" format="default">"Deterministic Networking Use Cases"</xref> summarizes the
need for it. See <xref target="I-D.ietf-detnet-data-plane-framework" format="de
fault"/> for specific techniques
that can be used to identify DetNet flows and assign them to specific paths
through a network. </t>
<t> A goal of DetNet is a converged network in all
respects, including the convergence of sensitive non-IP networks onto a common
network infrastructure. The presence of DetNet flows does not preclude
non-DetNet flows, and the benefits offered DetNet flows should not, except in
extreme cases, prevent existing Quality-of-Service (QoS) mechanisms from
operating in a normal fashion, subject to the bandwidth required for the
DetNet flows. A single source-destination pair can trade both DetNet and
non-DetNet flows. End systems and applications need not instantiate special
interfaces for DetNet flows. Networks are not restricted to certain
topologies; connectivity is not restricted. Any application that generates a
data flow that can be usefully characterized as having a maximum bandwidth
should be able to take advantage of DetNet, as long as the necessary resources
can be reserved.
Reservations can be made by the application itself, via network management,
centrally by an application's controller, or by other means, for instance, by
placing on-demand reservation via a distributed Control Plane, e.g.,
leveraging the Resource Reservation Protocol (RSVP) <xref target="RFC2205" forma
t="default"/>.
QoS requirements of DetNet flows can be met if all network
nodes in a DetNet domain implement DetNet capabilities. DetNet nodes can be
interconnected with different sub-network technologies (<xref target="sec_dt_dp"
format="default"/>) where the nodes of the subnet are not DetNet aware
(<xref target="netref" format="default"/>). </t>
<t> Many applications that are intended to be
served by DetNet require the ability to synchronize the clocks in end systems
to a sub-microsecond accuracy. Some of the queue-control techniques defined
in <xref target="QueuingModels" format="default"/> also require time synchroniza
tion among
network nodes. The means used to achieve time synchronization are not
addressed in this document. DetNet can accommodate various
time-synchronization techniques and profiles that are defined elsewhere to
address the needs of different market segments.
</t>
</section>
<section numbered="true" toc="default">
<name>Terminology</name>
<section numbered="true" toc="default">
<name>Terms Used in This Document</name>
<t>
The following terms are used in the context of DetNet in this document: The following terms are used in the context of DetNet in this document:
<list hangIndent="8" style="hanging"> </t>
<t hangText="allocation"><vspace blankLines="0"/> <dl newline="true" spacing="normal" indent="3">
Resources are dedicated to support a DetNet flo <dt>allocation</dt>
w. Depending on an implementation, the resource may be reused by non-DetNet flow <dd>
s when it is not used by the DetNet flow. The dedication of resources
</t> to support a DetNet flow. Depending on an
<t hangText="App-flow"><vspace blankLines="0"/> implementation, the resource may be reused by
non-DetNet flows when it is not used by the DetNet
flow.
</dd>
<dt>App-flow</dt>
<dd>
The payload (data) carried over a DetNet service. The payload (data) carried over a DetNet service.
</t> </dd>
<t hangText="DetNet compound flow and DetNet member flo <dt>DetNet compound flow and DetNet member flow</dt>
w"><vspace blankLines="0"/> <dd> A DetNet compound
A DetNet compound flow is a DetNet flow that has flow is a DetNet flow that has been separated into
been separated into multiple duplicate DetNet member flows for service
multiple duplicate DetNet member flows for servic protection at the DetNet service sub-layer. Member
e protection at the DetNet service sub-layer. flows are merged back into a single DetNet compound
Member flows are merged back into a single DetNet flow such that there are no duplicate packets.
compound flow such that there are no duplicate packets. "Compound" and "member" are strictly relative to
"Compound" and "member" are strictly each other, not absolutes; a DetNet compound flow
relative to each other, not absolutes; a DetNet c comprising multiple DetNet member flows can, in
ompound flow comprising turn, be a member of a higher-order compound.
multiple DetNet member flows can, in turn, be a m </dd>
ember of a higher-order compound. <dt>DetNet destination</dt>
</t> <dd>
<t hangText="DetNet destination"><vspace blankLines="0"
/>
An end system capable of terminating a DetNet flo w. An end system capable of terminating a DetNet flo w.
</t> </dd>
<t hangText="DetNet domain"><vspace blankLines="0"/> <dt>DetNet domain</dt>
The portion of a network that is DetNet aware. <dd>
It includes end The portion of a network that
systems and DetNet nodes. is DetNet aware. It includes end systems and DetNet
</t> nodes.
<t hangText="DetNet edge node"><vspace blankLines="0"/> </dd>
An instance of a DetNet relay node that acts as <dt>DetNet edge node</dt>
a source and/or <dd> An instance
destination at the DetNet service sub-layer. Fo of a DetNet relay node that acts as a source and/or
r example, it can destination at the DetNet service sub-layer. For
include a DetNet service sub-layer proxy functi example, it can include a DetNet service sub-layer
on for DetNet service proxy function for DetNet service protection (e.g.,
protection (e.g., the addition or removal of pa the addition or removal of packet sequencing
cket sequencing information) information) for one or more end systems, it can start
for one or more end systems, or starts or termi or terminate resource allocation at the DetNet
nates resource allocation at forwarding sub-layer, or it can aggregate DetNet servic
the DetNet forwarding sub-layer, or aggregates es
DetNet services into new DetNet flows. into new DetNet flows. It is analogous to a Label
It is analogous to a Label Edge Router (LER) or Edge Router (LER) or a Provider Edge (PE) router.
a Provider Edge (PE) router. </dd>
</t> <dt>DetNet flow</dt>
<t hangText="DetNet flow"><vspace blankLines="0"/> <dd>
A DetNet flow is a sequence of packets which conf A sequence of packets that conforms uniquely
orm uniquely to a flow identifier and to which the DetNet serv
to a flow identifier, and to which the DetNet ser ice is to be
vice is to be
provided. It includes any DetNet headers added to support the provided. It includes any DetNet headers added to support the
DetNet service and forwarding sub-layers. DetNet service and forwarding sub-layers.
</t> </dd>
<t hangText="DetNet forwarding sub-layer"><vspace blank <dt>DetNet forwarding sub-layer</dt>
Lines="0"/> <dd>
DetNet functionality is divided into two sub-la DetNet functionality is divided into two sub-layers.
yers. One of One of
them is the DetNet forwarding sub-layer, which them is the DetNet forwarding sub-layer, which option
optionally ally
provides resource allocation for DetNet flows o ver paths provides resource allocation for DetNet flows o ver paths
provided by the underlying network. provided by the underlying network.
</t> </dd>
<t hangText="DetNet intermediate node"><vspace blankLin <dt>DetNet intermediate node</dt>
es="0"/> <dd>
A DetNet relay node or DetNet transit node. A DetNet relay node or DetNet transit node.
</t> </dd>
<t hangText="DetNet node"><vspace blankLines="0"/> <dt>DetNet node</dt>
A DetNet edge node, a DetNet relay node, or a <dd> A
DetNet transit node. DetNet edge node, a DetNet relay
</t> node, or a DetNet transit node.
<t hangText="DetNet relay node"><vspace blankLines="0"/ </dd>
> <dt>DetNet relay node</dt>
A DetNet node including a service sub-layer funct <dd> A DetNet
ion that interconnects different node that includes a service
DetNet forwarding sub-layer paths to provide serv sub-layer function that interconnects different
ice protection. A DetNet relay node DetNet forwarding sub-layer paths to provide service
participates in the DetNet service protection. A DetNet relay node participates in the
sub-layer. It typically incorporates DetNet forw DetNet service sub-layer. It typically incorporates
arding sub-layer functions as DetNet forwarding sub-layer functions as well, in
well, in which case it is collocated with a trans which case it is collocated with a transit node.
it node. </dd>
</t> <dt>DetNet service sub-layer</dt>
<t hangText="DetNet service sub-layer"><vspace blankLin <dd>
es="0"/> DetNet functionality is divided into two sub-layers. One of
DetNet functionality is divided into two sub-la them is the DetNet service sub-layer, at which a DetNet
yers. One of service (e.g., service protection) is provided.
them is the DetNet service sub-layer, at which </dd>
a DetNet service, <dt>DetNet service proxy</dt>
e.g., service protection is provided. <dd>
</t> A proxy that maps between App-flows and DetNet flows.
<t hangText="DetNet service proxy"><vspace blankLines=" </dd>
0"/> <dt>DetNet source</dt>
Maps between App-flows and DetNet flows. <dd>
</t>
<t hangText="DetNet source"><vspace blankLines="0"/>
An end system capable of originating a DetNet f low. An end system capable of originating a DetNet f low.
</t> </dd>
<t hangText="DetNet system"><vspace blankLines="0"/> <dt>DetNet system</dt>
A DetNet aware end system, transit node, or rel <dd>
ay node. A DetNet-aware end system, transit node, or rel
ay node.
"DetNet" may be omitted in some text. "DetNet" may be omitted in some text.
</t> </dd>
<t hangText="DetNet transit node"><vspace blankLines="0 <dt>DetNet transit node</dt>
"/> <dd>
A DetNet node operating at the DetNet forwardin A DetNet node, operating at the DetNet forwarding sub-
g sub-layer, that utilizes link layer layer,
and/or network layer switching across multiple that utilizes link-layer and/or network-layer
links and/or sub-networks to provide paths for switching across multiple links and/or sub-networks
DetNet service sub-layer functions. to provide paths for DetNet service sub-layer
Typically provides resource allocation over tho functions. It typically provides resource allocation
se paths. over those paths. An MPLS Label Switch Router (LSR) is
An MPLS LSR is an example of a DetNet transit n an example of a
ode. DetNet transit node.
</t> </dd>
<t hangText="DetNet-UNI"><vspace blankLines="0"/> <dt>DetNet-UNI</dt>
User-to-Network Interface with DetNet specific <dd>
functionalities. It is a packet-based reference point and may provide multiple f A User-to-Network Interface (UNI) with DetNet-specific
unctions like encapsulation, status, synchronization, etc. functionalities. It is a packet-based reference
</t> point and may provide multiple functions like
<t hangText="end system"><vspace blankLines="0"/> encapsulation, status, synchronization, etc.
Commonly called a "host" in IETF documents, and </dd>
an "end <dt>end system</dt>
station" is IEEE 802 documents. End systems of <dd>
interest to Commonly called a "host" in the RFC series and an
this document are either sources or destination "end station" in IEEE 802 standards. End systems of
s of DetNet flows. interest to this document are either sources or
And end system may or may not be DetNet forward destinations of DetNet flows, and they may or may
ing sub-layer aware or not be aware of DetNet forwarding sub-layers or
DetNet service sub-layer aware. DetNet service sub-layers.
</t> </dd>
<t hangText="link"><vspace blankLines="0"/> <dt>link</dt>
A connection between two DetNet nodes. It may <dd>
be composed of a A connection between two DetNet nodes. It may be
physical link composed of a physical link or a sub-network
or a sub-network technology that can provide ap technology that can provide appropriate traffic
propriate traffic delivery for DetNet flows.
delivery for DetNet </dd>
flows. <dt>Packet Elimination Function (PEF)</dt>
</t> <dd>
<t hangText="PEF"> A function that eliminates duplicate
A Packet Elimination Function (PEF) eliminates duplicate copi copies of packets to prevent excess packets flooding the
es of packets network or duplicate packets being sent out of the DetNet
to prevent excess packets flooding the network or domain. A PEF can be implemented by a DetNet edge node, a
duplicate packets being DetNet relay node, or an end system.
sent out of the DetNet domain. PEF can be impleme </dd>
nted by a DetNet edge node, a <dt>Packet Replication Function (PRF)</dt>
DetNet relay node, or an end system. <dd>
</t> A function that replicates DetNet flow
packets and forwards them to one or more next hops in the
<t hangText="PRF"> DetNet domain. The number of packet copies sent to the
A Packet Replication Function (PRF) replicates DetNet flow pa next hops is a parameter specific to the DetNet flow at the p
ckets oint
and forwards them to one or more next hops in the of replication. A PRF can be implemented by a DetNet edge
DetNet domain. node, a DetNet relay node, or an end system.
The number of packet copies sent to the next hops </dd>
is a DetNet flow <dt>PREOF</dt>
specific parameter at the point of replication. P <dd>
RF can be A collective name for Packet Replication, Elimination, and Or
implemented by a DetNet edge node, a DetNet relay dering Functions.
node, or an end system. </dd>
</t> <dt>Packet Ordering Function (POF)</dt>
<dd>
<t hangText="PREOF"> A function that reorders packets within
Collective name for Packet Replication, Elimination, and Orde a DetNet flow that are received out of order. This
ring Functions. function can be implemented by a DetNet edge node, a
</t> DetNet relay node, or an end system.
</dd>
<t hangText="POF"> <dt>reservation</dt>
A Packet Ordering Function (POF) re-orders packets within a D <dd>
etNet flow that are received out of order. The set of resources allocated between a source and
This function can be implemented by a DetNet edge node, a Det one or more destinations through DetNet nodes and
Net relay node, or an end system. subnets associated with a DetNet flow in order to provi
</t> de
the provisioned DetNet service.
<t hangText="reservation"><vspace blankLines="0"/> </dd>
The set of resources allocated between a source a </dl>
nd one or more destinations through DetNet nodes </section>
and subnets associated with a DetNet flow, to pro <section numbered="true" toc="default">
vide the provisioned DetNet service. <name>Dictionary of Terms Used by TSN and DetNet</name>
</t> <t>
</list> This section serves as a dictionary for translating the
</t>
</section>
<section title="IEEE 802.1 TSN to DetNet dictionary">
<t>
This section also serves as a dictionary for translatin
g from the
terms used by the Time-Sensitive Networking (TSN) Task Group terms used by the Time-Sensitive Networking (TSN) Task Group
<xref target="IEEE802.1TSNTG"/> of the IEEE 802.1 WG t <xref target="IEEE802.1TSNTG" format="default"/> of the
o those of IEEE 802.1 WG to those of
the DetNet WG. the Deterministic Networking (detnet) WG of the IETF.
<list hangIndent="8" style="hanging"> </t>
<t hangText="Listener"><vspace blankLines="0"/> <dl newline="true" spacing="normal" indent="3">
The IEEE 802.1 term for a destination o <dt>Listener</dt>
f a DetNet flow. <dd>
</t> The term used by IEEE 802.1 for a desti
<t hangText="relay system"><vspace blankLines=" nation of a DetNet flow.
0"/> </dd>
The IEEE 802.1 term for a DetNet interm <dt>Relay system</dt>
ediate node. <dd>
</t> The term used by IEEE
<t hangText="Stream"><vspace blankLines="0"/> 802.1 for a DetNet intermediate node.
The IEEE 802.1 term for a DetNet flow. </dd>
</t> <dt>Stream</dt>
<t hangText="Talker"><vspace blankLines="0"/> <dd>
The IEEE 802.1 term for the source of a The term used by IEEE 802.1 for a DetNe
DetNet flow. t flow.
</t> </dd>
</list> <dt>Talker</dt>
</t> <dd>
</section> The term used by IEEE
</section> 802.1 for the source of a DetNet flow.
</dd>
<section anchor="ProvidingQoS" title="Providing the DetNet Quality of Ser </dl>
vice"> </section>
<section anchor="DefiningGoals" title="Primary goals defining the DetNet Q </section>
oS"> <section anchor="ProvidingQoS" numbered="true" toc="default">
<t> <name>Providing the DetNet Quality of Service</name>
The DetNet Quality of Service can be expressed in terms o <section anchor="DefiningGoals" numbered="true" toc="default">
f: <name>Primary Goals Defining the DetNet QoS</name>
<list style="symbols"> <t>
<t> The DetNet QoS can be expressed in terms of:
Minimum and maximum end-to-end latency from source to des </t>
tination; <ul spacing="normal">
timely delivery, and bounded jitter (packet delay variation) derived <li>
from these constraints. Minimum and maximum end-to-end latency from source to
</t> destination, timely delivery, and bounded jitter
<t> (packet delay variation) derived from these
Packet loss ratio, under various assumptions as to the op constraints.
erational </li>
<li>
Packet loss ratio under various assumptions as to the ope
rational
states of the nodes and links. states of the nodes and links.
</t><t> </li>
<li>
An upper bound on out-of-order packet delivery. It is wor th noting An upper bound on out-of-order packet delivery. It is wor th noting
that some DetNet applications are unable to tolerate any that some DetNet applications are unable to tolerate any
out-of-order delivery. out-of-order delivery.
</t> </li>
</list> </ul>
</t><t> <t>
It is a distinction of DetNet that it is concerned solely with It is a distinction of DetNet that it is concerned solely with
worst-case values for the end-to-end latency, jitter, and worst-case values for the end-to-end latency, jitter, and
misordering. Average, mean, or typical values are of litt le misordering. Average, mean, or typical values are of litt le
interest, because they do not affect the ability of a rea l-time interest, because they do not affect the ability of a rea l-time
system to perform its tasks. In general, a trivial priori ty-based system to perform its tasks. In general, a trivial priori ty-based
queuing scheme will give better average latency to a data flow than queuing scheme will give better average latency to a data flow than
DetNet; however, it may not be a suitable option for DetN et because DetNet; however, it may not be a suitable option for DetN et because
of its worst-case latency. of its worst-case latency.
</t><t> </t>
<t>
Three techniques are used by DetNet to provide these qual ities of service: Three techniques are used by DetNet to provide these qual ities of service:
<list style="symbols"> </t>
<t> <ul spacing="normal">
Resource allocation (<xref target="Zero"/ <li>
>). Resource allocation (<xref target="Zero"
</t><t> format="default"/>)
Service protection (<xref target="srvcpro </li>
t"/>). <li>
</t><t> Service protection (<xref target="srvcpro
Explicit routes (<xref target="pinned"/>) t" format="default"/>)
. </li>
</t> <li>
</list> Explicit routes (<xref target="pinned" fo
</t><t> rmat="default"/>)
</li>
</ul>
<t>
Resource allocation operates by assigning resources, e.g., buffer Resource allocation operates by assigning resources, e.g., buffer
space or link bandwidth, to a DetNet flow (or flow aggregate) along space or link bandwidth, to a DetNet flow (or flow aggregate) along
its path. Resource allocation greatly reduces, or even eliminates its path. Resource allocation greatly reduces, or even eliminates
entirely, packet loss due to output packet contention within the entirely, packet loss due to output packet contention within the
network, but it can only be supplied to a DetNet flow that is limite d network, but it can only be supplied to a DetNet flow that is limite d
at the source to a maximum packet size and transmission rate. at the source to a maximum packet size and transmission rate.
As DetNet flows are assumed to be rate-limited and DetNet is designed As DetNet flows are assumed to be rate limited and DetNet is designed
to provide sufficient allocated resources (including prov isioned to provide sufficient allocated resources (including prov isioned
capacity), the use of transport layer congestion control capacity), the use of transport-layer congestion control
<xref target="RFC2914"/> for App-flows is not required; h <xref target="RFC2914" format="default"/> for App-flows i
owever, s not required; however,
if resources are allocated appropriately, use of congesti on control if resources are allocated appropriately, use of congesti on control
should not impact transmission negatively. should not impact transmission negatively.
</t><t> </t>
Resource allocation addresses two of the DetNet QoS requirements: <t> Resource allocation addresses two of the DetNet QoS requirements: la
latency and packet loss. Given that DetNet nodes have a finite tency
amount of buffer space, resource allocation necessarily results in and packet loss. Given that DetNet nodes have a finite amount of buffer space,
a maximum end-to-end latency. It also addresses contention related resource allocation necessarily results in a maximum end-to-end
packet loss. latency. Resource allocation also addresses contention-related packet loss.
</t><t> </t>
Other important contribution to packet loss are <t> Other important contributions to packet loss are random media errors
random media errors and equipment failures. Service prot and equipment failures. Service protection is the name for the mechanisms
ection is the name for the used by DetNet to address these losses. The mechanisms employed are
mechanisms used by DetNet to address these losses. The m constrained by the need to meet the users' latency requirements. Packet
echanisms employed are replication and elimination (<xref target="Seamless" format="default"/>) and pac
constrained by the requirement to meet the users' latency ket encoding
requirements. (<xref target="PacketEncoding" format="default"/>) are described in this documen
Packet replication and elimination (<xref target="srvcpro t to provide
t"/>) and packet encoding service protection, but other mechanisms may also be found. For instance,
(<xref target="PacketEncoding"/>) are described in this document packet encoding can be used to provide service protection against random media
to provide service protection; others may be found. For i errors, while packet replication and elimination can be used to provide
nstance, packet encoding service protection against equipment failures. This mechanism distributes the
can be used to provide service protection against random media error contents of DetNet flows over multiple paths in time and/or space, so that the
s, packet loss of some of the paths does need not cause the loss of any packets.
replication and elimination can be used to provide service protectio </t>
n against <t> The paths are typically (but not necessarily) explicit routes so tha
equipment failures. This mechanism t
distributes the contents of DetNet flows they do not normally suffer temporary interruptions caused by the convergence
over multiple paths in time and/or space, so that the los of routing or bridging protocols. </t>
s of some of the paths does <t> These three techniques can be applied individually or applied togeth
need not cause the loss of any packets. er; it
</t><t> results that eight combinations, including none (no DetNet), are
The paths are typically (but not necessarily) explicit ro possible. Some combinations, however, are of wider utility than others. This
utes, so that they do not normally separation keeps the protocol stack coherent and maximizes interoperability
suffer temporary interruptions caused by the convergence with existing and developing standards in the IETF and other Standards
of routing or bridging protocols. Development Organizations. The following are examples of typical expected
</t><t> combinations:
These three techniques can be applied independently, givi </t>
ng eight possible combinations, <ul spacing="normal">
including none (no DetNet), although some combinations ar <li>
e of wider utility than others. The combination of explicit routes and service protection is the techniqu
This separation keeps the protocol stack coherent and max e
imizes interoperability with employed by seamless redundancy mechanisms applied on a ring topology,
existing and developing standards in this (IETF) and othe e.g., as described in <xref target="IEC-62439-3" format="default"/>. In t
r his
Standards Development Organizations. Some examples of ty example, explicit routes are achieved by limiting the physical
pical expected combinations: topology of the network to a ring. Sequentialization, replication, and
<list style="symbols"> duplicate elimination are facilitated by packet tags added at the
<t> front or the end of Ethernet frames. <xref target="RFC8227" format="defau
Explicit routes plus service protection a lt"/> provides
re exactly the techniques another example in the context of MPLS. </li>
employed by seamless redundancy mechanism <li> Resource allocation
s applied on a ring topology alone was originally offered by Audio Video Bridging as defined by IEEE 8
as described, e.g., in <xref target="IEC6 02.1 <xref target="IEEE802.1BA" format="default"/>. As long as the network suff
2439-3-2016"/>. In this example, ers no failures,
explicit routes are achieved by limiting packet loss due to output packet contention can be eliminated through
the physical topology of the use of a reservation protocol (e.g., the Multiple Stream Registration
the network to a ring. Sequentialization, Protocol <xref target="IEEE802.1Q" format="default"/>), shapers in every
replication, and bridge,
duplicate elimination are facilitated by and proper dimensioning. </li>
packet tags added at the <li> Using all three together gives
front or the end of Ethernet frames. <xre maximum protection.
f target="RFC8227"/> provides </li>
another example in the context of MPLS. </ul>
</t><t> <t> There are, of course, simpler methods available (and employed today)
Resource allocation alone was originally to
offered by IEEE 802.1 Audio Video bridging achieve levels of latency and packet loss that are satisfactory for many
<xref target="IEEE802.1BA"/>. As long as applications. Prioritization and over-provisioning is one such technique.
the network suffers no failures, However, these methods generally work best in the absence of any significant
packet loss due to output packet contenti amount of noncritical traffic in the network (if, indeed, such traffic is
on can be eliminated through the use of a reservation protocol (e.g., Multiple S supported at all). They may also work only if the critical traffic constitutes o
tream Registration Protocol <xref target="IEEE802.1Q-2018"/>), shapers in every nly a small portion of
bridge, and proper dimensioning. the network's theoretical capacity, if all systems are functioning properly,
</t><t> or if actions by end systems that disrupt the network's
Using all three together gives maximum pr operations are absent. </t>
otection. <t> There are any number of methods in use, defined, or in progress for
</t> accomplishing each of the above techniques. It is expected that the DetNet
</list> architecture defined in this document will assist various vendors, users, and/or
</t><t> "vertical" Standards
There are, of course, simpler methods available (and empl Development Organizations (dedicated to a single industry) in making selections
oyed, today) to achieve among the available means of implementing DetNet networks.
levels of latency and packet loss that are satisfactory f </t>
or many applications. </section>
Prioritization and over-provisioning is one such techniqu <section anchor="Techniques" numbered="true" toc="default">
e. However, these <name>Mechanisms to Achieve DetNet QoS</name>
methods generally work best in the absence of any signifi <section anchor="Zero" numbered="true" toc="default">
cant amount of non-critical <name>Resource Allocation</name>
traffic in the network (if, indeed, such traffic is suppo <section anchor="ZeroL" numbered="true" toc="default">
rted at all), or work only if <name>Eliminate Contention Loss</name>
the critical traffic constitutes only a small portion of <t>
the network's theoretical The primary means by which DetNet achieves its QoS
capacity, or work only if all systems are functioning pro assurances is to reduce, or even completely eliminate,
perly, or in the absence of packet loss due to output packet contention within a
actions by end systems that disrupt the network's operati DetNet node as a cause of packet loss. This can be
ons. achieved only by the provision of sufficient buffer
</t><t> storage at each node through the network to ensure
There are any number of methods in use, defined, or in pr that no packets are dropped due to a lack of buffer
ogress for accomplishing each storage. Note that App-flows are generally not
of the above techniques. It is expected that this DetNet expected to be responsive to implicit <xref target="RFC29
Architecture will assist 14" format="default"/> or explicit congestion notification
various vendors, users, and/or "vertical" <xref target="RFC3168" format="default"/>.
Standards Development Organizations (dedicated to a singl
e industry) to make selections
among the available means of implementing DetNet networks
.
</t>
</section>
<section anchor="Techniques" title="Mechanisms to achieve DetNet QoS">
<section anchor="Zero" title="Resource allocation">
<section anchor="ZeroL" title="Eliminate contention loss">
<t>
The primary means by which DetNet achieves its QoS assura
nces is to reduce, or even completely eliminate packet loss due to output packet
contention
within a DetNet node as a cause of packet loss.
This can be achieved only by the provision of
sufficient buffer storage at each node through the networ
k to ensure that no
packets are dropped due to a lack of buffer storage.
Note that App-flows are generally not expected to be res
ponsive to implicit <xref target="RFC2914"/> or explicit congestion notification
<xref target="RFC3168"/>.
</t><t> </t>
Ensuring adequate buffering requires, in turn, that the s <t> Ensuring adequate buffering requires, in turn, that
ource, and every DetNet the source and every DetNet node along the path to the
node along the path to the destination (or nearly every n destination (or nearly every node; see <xref target="Incomplete"
ode, see format="default"/>) be careful to regulate its output to
<xref target="Incomplete"/>) be careful to regulate its o not exceed the data rate for any DetNet flow, except for brief
utput to not exceed the periods when making up for interfering traffic. Any packet
data rate for any DetNet flow, except for brief periods w sent ahead of its time potentially adds to the number of
hen making up for buffers required by the next-hop DetNet node and may thus
interfering traffic. Any packet sent ahead of its time p exceed the resources allocated for a particular DetNet
otentially adds to the flow. Furthermore, rate limiting (e.g., using traffic policing)
number of buffers required by the next hop DetNet node an and shaping functions (e.g., shaping as defined in <xref target="
d may thus exceed the resources RFC2475" format="default"/>) at the
allocated for a particular DetNet flow. Furthermore, rate ingress of the DetNet domain must be applied. This is needed
limiting, e.g., using traffic for meeting the requirements of DetNet flows as well as for
policing and shaping functions, e.g., <xref target="RFC24 protecting non-DetNet traffic from potentially misbehaving
75"/>, at the ingress of the DetNet traffic sources. Note that large buffers have some
DetNet domain must be applied. This is needed for meeting issues (see, e.g., <xref target="BUFFERBLOAT" format="default"/>)
the requirements of DetNet . </t>
flows as well as for protecting non-DetNet traffic from p <t> The low-level mechanisms described in <xref target="QueuingModel
otentially misbehaving DetNet s" format="default"/>
traffic sources. Note that large buffers have some issues provide the necessary regulation of transmissions by an end system or DetNet
, see, e.g., <xref target="BUFFERBLOAT"/>. node to provide resource allocation. The allocation of the bandwidth and
</t><t> buffers for a DetNet flow requires provisioning. A DetNet node may have other
The low-level mechanisms described in <xref target="Queui resources requiring allocation and/or scheduling that might otherwise be
ngModels"/> provide over-subscribed and trigger the rejection of a reservation.
the necessary </t>
regulation of transmissions by an end system or DetNet no </section>
de to provide <section anchor="Jitterless" numbered="true" toc="default">
resource allocation. The allocation of the bandwidth and <name>Jitter Reduction</name>
buffers for a DetNet flow requires provisioning. <t>
A DetNet node may have other resources requiring allocati
on and/or scheduling,
that might otherwise be over-subscribed and trigger the r
ejection of a reservation.
</t>
</section>
<section anchor="Jitterless" title="Jitter Reduction">
<t>
A core objective of DetNet is to enable the convergence of sensitive non- IP networks A core objective of DetNet is to enable the convergence of sensitive non- IP networks
onto a common network infrastructure. This requires the accurate emulatio n onto a common network infrastructure. This requires the accurate emulatio n
of currently deployed mission-specific networks, which of currently deployed mission-specific networks, which,
for example rely on point-to-point analog (e.g., 4-20mA modulation) and for example, rely on point-to-point analog (e.g., 4-20mA modulation) and
serial-digital cables (or buses) for highly reliable, synchronized and serial-digital cables (or buses) for highly reliable, synchronized, and
jitter-free communications. While the latency of analog transmissions is jitter-free communications. While the latency of analog transmissions is
basically the speed of light, legacy serial links are usually slow (in th e basically the speed of light, legacy serial links are usually slow (in th e
order of Kbps) compared to, say, Gigabit Ethernet, and some latency is us ually order of Kbps) compared to, say, Gigabit Ethernet, and some latency is us ually
acceptable. What is not acceptable is the introduction of excessive jitte r, acceptable. What is not acceptable is the introduction of excessive jitte r,
which may, for instance, affect the stability of control systems. which may, for instance, affect the stability of control systems.
</t> </t>
<t>Applications that are designed to operate on serial links usually do <t>Applications that are designed to operate on serial links usually
do
not provide services to recover the jitter, because jitter simply does no t not provide services to recover the jitter, because jitter simply does no t
exist there. DetNet flows are generally expected to be delivered in-order exist there. DetNet flows are generally expected to be delivered in order ,
and the precise time of reception influences the processes. In order to and the precise time of reception influences the processes. In order to
converge such existing applications, converge such existing applications,
there is a desire to emulate all properties of the serial cable, such there is a desire to emulate all properties of the serial cable, such
as clock transportation, perfect flow isolation and fixed latency. While minimal as clock transportation, perfect flow isolation, and fixed latency. While minimal
jitter (in the form of specifying minimum, as well as maximum, end-to-end latency) jitter (in the form of specifying minimum, as well as maximum, end-to-end latency)
is supported by DetNet, there are practical limitations on packet-based n etworks is supported by DetNet, there are practical limitations on packet-based n etworks
in this regard. In general, users in this regard. In general, users
are encouraged to use a combination of: are encouraged to use a combination of:
<list style="symbols"> </t>
<t> <ul spacing="normal">
<li>
Sub-microsecond time synchronization among all source an d destination Sub-microsecond time synchronization among all source an d destination
end systems, and end systems, and
</t><t> </li>
<li>
Time-of-execution fields in the application packets. Time-of-execution fields in the application packets.
</t> </li>
</list> </ul>
</t><t> <t>
Jitter reduction is provided by the mechanisms described in <xref ta Jitter reduction is provided by the mechanisms described in <xref ta
rget="QueuingModels"/> rget="QueuingModels" format="default"/>
that also provide resource allocation. that also provide resource allocation.
</t> </t>
</section> </section>
</section> </section>
<section anchor="srvcprot" numbered="true" toc="default">
<section anchor="srvcprot" title="Service Protection"> <name>Service Protection</name>
<t> <t>
Service protection aims to mitigate or eliminate packet loss due to Service protection aims to mitigate or eliminate packet loss due
equipment failures, including random media and/or memory faults. to equipment failures, including random media and/or memory
These types of faults. These types of packet loss can be greatly reduced by
packet loss can be greatly reduced by spreading the data over mul spreading the data over multiple disjoint forwarding
tiple paths. Various service protection methods are described in <xref targ
disjoint forwarding paths. Various service protection methods are et="RFC6372" format="default"/>, e.g., 1+1 linear protection. The functional
described in <xref target="RFC6372"/>, e.g., 1+1 linear protectio details of an additional method are described in <xref target="Seamle
n. ss" format="default"/>, which can be implemented as described in
This section describes the functional details of an additional me <xref target="PacketEncoding" format="default"/> or as specified in <
thod xref target="I-D.ietf-detnet-mpls" format="default"/> in order to provide 1+n hi
in <xref target="Seamless"/>, which can be implemented as describ tless
ed in protection. The appropriate service protection mechanism depends
<xref target="PacketEncoding"/> or as specified in on the scenario and the requirements.
<xref target="I-D.ietf-detnet-dp-sol-mpls"/> in order to provide </t>
1+n hitless <section anchor="inorder" numbered="true" toc="default">
protection. The appropriate service protection mechanism depends <name>In-Order Delivery</name>
on the <t>
scenario and the requirements. Out-of-order packet delivery can be a side effect of service
</t> protection. Packets delivered out of order impact the amount of
buffering needed at the destination to properly process the received
<section anchor="inorder" title="In-Order Delivery"> data. Such packets also influence the jitter of a flow. The guarantees
<t> of a DetNet service include a maximum amount of misordering as a
Out-of-order packet delivery can be a side effect of service protection. constraint. Zero misordering would be a valid service constraint to
Packets delivered out-of-order impact the amount of buffering nee reflect that the end system(s) of the flow cannot tolerate any
ded at out-of-order delivery. A DetNet Packet Ordering Function (POF)
the destination to properly process the received data. Such packe (<xref target="Seamless" format="default"/>) can be used to provide in-o
ts rder delivery.
also influence the jitter of a flow. The DetNet service includes </t>
maximum allowed misordering as a constraint. Zero misordering wou </section>
ld be <section anchor="Seamless" numbered="true" toc="default">
a valid service constraint to reflect that the end system(s) of t <name>Packet Replication and Elimination</name>
he <t>
flow cannot tolerate any out-of-order delivery. DetNet Packet Ord
ering
Functionality (POF) (<xref target="Seamless"/>) can be used to pr
ovide
in-order delivery.
</t>
</section>
<section anchor="Seamless" title="Packet Replication and Elimination">
<t>
This section describes a service protection method that sends copies This section describes a service protection method that sends copies
of the same packets over multiple paths. of the same packets over multiple paths. </t>
</t><t> <t> The DetNet service
The DetNet service sub-layer includes the packet replication (PRF), sub-layer includes the PRF, PEF, and POF for use in DetNet edge,
the relay node, and end-system packet processing. These functions can be
packet elimination (PEF), and the packet ordering functionali enabled in a DetNet edge node, relay node, or end system. The
ty (POF) collective name for all three functions is Packet Replication,
for use in DetNet edge, relay node, and end system packet Elimination, and Ordering Functions (PREOF). The packet replication
processing. and elimination service protection method altogether involves four
These functions can be enabled in a DetNet edge node, rel capabilities:
ay </t>
node or end system. The collective name for all three fun <ul spacing="normal">
ctions is <li>
Packet Replication, Elimination, and Ordering Functions ( Sequencing information is provided to the
PREOF).
The packet replication and elimination service protection
method
altogether involves four capabilities:
<list style="symbols">
<t>
Providing sequencing information to the
packets of a DetNet compound flow. This may packets of a DetNet compound flow. This may
be done by adding a sequence number or time stamp be done by adding a sequence number or time
as part of DetNet, or may be stamp as part of DetNet, or it may be inherent
inherent in the packet, e.g., in a higher layer p in the packet, e.g., in a higher-layer
rotocol, or associated to other physical protocol or associated to other physical
properties such as the precise time (and radio ch properties such as the precise time (and radio
annel) of reception of the packet. channel) of reception of the packet. This is
This is typically done once, at or near the sourc typically done once, at or near the source.
e. </li>
</t><t> <li> The PRF
The Packet Replication Function (PRF) replicates replicates these packets into multiple DetNet
these packets member flows and typically sends them along
into multiple DetNet member flows and typically s multiple different paths to the
ends them along destination(s), e.g., over the explicit routes
multiple different paths to the destination(s), e described in <xref target="pinned" format="defaul
.g., over the t"/>. The location
explicit routes of <xref target="pinned"/>. The l within a DetNet node and the mechanism used
ocation within for the PRF are left open for implementations.
a DetNet node, and the mechanism used for the PRF </li>
is left open <li> The PEF
for implementations. eliminates duplicate packets of a DetNet flow
</t><t> based on the sequencing information and a
The Packet Elimination Function (PEF) eliminates history of received packets. The output of the
duplicate packets PEF is always a single packet. This may be
of a DetNet flow based on the sequencing informat done at any DetNet node along the path to save
ion and a history network resources further downstream, in
of received packets. The output of the PEF is alw particular if multiple replication points
ays a single packet. exist. But the most common case is to perform
This may be done at any DetNet node along the pat this operation at the very edge of the DetNet
h to save network network, preferably in or near the
resources further downstream, in particular if mu receiver. The location within a DetNet node
ltiple and the mechanism used for the PEF is left open
Replication points exist. But the most common cas for implementations. </li>
e is to <li> The
perform this operation at the very edge of the De POF uses the sequencing
tNet network, information to reorder a DetNet flow's packets
preferably in or near the receiver. The location that are received out of order.
within a DetNet node, </li>
and mechanism used for the PEF is left open f </ul>
or implementations. <t> The order in which a DetNet node applies PEF, POF, and
</t><t> PRF to a DetNet flow is left open for implementations.
The Packet Ordering Function (POF) uses the sequencin </t>
g information <t> Some service protection mechanisms rely on switching from one fl
to re-order a DetNet flow's packets that are rece ow to
ived out of order. another when a failure of a flow is detected. Contrarily, packet replication
</t> and elimination combines the DetNet member flows sent along multiple different
</list> paths and performs a packet-by-packet selection of which to discard, e.g.,
</t><t> based on sequencing information.
The order in which a DetNet node applies PEF, POF, and PRF to </t>
a DetNet flow is <t>In the simplest case, this amounts to 1) replicating each packet
left open for implementations. in a
</t><t> source that has two interfaces and 2) conveying them through the network along
Some service protection mechanisms rely on switching from one flow separate (Shared Risk Link Group (SRLG) disjoint) paths to the similarly
to another when a failure of a flow is detected. Contrari dual-homed destinations that 3) reorder the packets and 4) discard the
ly, packet duplicates. This ensures that one path
replication and elimination combines the DetNet member fl remains, even if some DetNet intermediate node fails. The sequencing
ows sent information can also be used for loss detection and for reordering. </t>
along multiple different paths, and performs a packet-by- <t>
packet DetNet relay nodes in the network can provide replication and elimination
selection of which to discard, e.g., based on sequencing facilities at various points in the network so that multiple failures can be
information. accommodated. </t>
</t><t> <t> This is shown in <xref target="FigSeamless" format="default"/>,
In the simplest case, this amounts to replicating each pa where the two relay nodes
cket in a source that each replicate (R) the DetNet flow on input, sending the DetNet member flows
has two interfaces, and conveying them through the networ to both the other relay node and to the end system, and eliminate duplicates
k, along separate (E) on the output interface to the right-hand end system. Any one link in the
(Shared Risk Link Group (SRLG) disjoint) paths, to the network can fail, and the DetNet compound flow can still get through.
similarly dual-homed destinations, that discard the extra Furthermore, two links can fail, as long as they are in different segments of
s. This ensures that one the network.
path remains, even if some DetNet intermediate node fails </t>
. <figure anchor="FigSeamless">
The sequencing information can also be used for loss dete <name>Packet Replication and Elimination</name>
ction and for re-ordering. <artwork align="left" name="" type="" alt=""><![CDATA[
</t><t>
DetNet relay nodes in the network can provide
replication and elimination
facilities at various points in the network, so that mult
iple
failures can be accommodated.
</t><t>
This is shown in <xref target="FigSeamless"/>, where the
two relay nodes
each replicate (R) the DetNet flow on input, sending the
DetNet member flows to both the other
relay node and to the end system, and eliminate duplicate
s (E) on the output
interface to the right-hand end system. Any one link in
the network can
fail, and the DetNet compound flow can still get through.
Furthermore, two links can
fail, as long as they are in different segments of the ne
twork.
</t>
<figure align="center" anchor="FigSeamless"
title="Packet replication and elimination">
<artwork align="left"><![CDATA[
> > > > > > > > > relay > > > > > > > > > > > > > > > > > relay > > > > > > > >
> /------------+ R node E +------------\ > > /------------+ R node E +------------\ >
> / v + ^ \ > > / v + ^ \ >
end R + v | ^ + E end end R + v | ^ + E end
system + v | ^ + system system + v | ^ + system
> \ v + ^ / > > \ v + ^ / >
> \------------+ R relay E +-----------/ > > \------------+ R relay E +-----------/ >
> > > > > > > > > node > > > > > > > > > > > > > > > > > node > > > > > > > >]]></artwork>
]]></artwork> </figure>
</figure> <t>Packet replication and elimination does not react to and correct
<t> failures;
Packet replication and elimination does not react to and it is entirely passive. Thus, intermittent failures, mistakenly created
correct failures; it is packet filters, or misrouted data is handled just the same as the equipment
entirely passive. Thus, intermittent failures, mistakenl failures that are handled by typical routing and bridging protocols. </t>
y created packet filters, <t>If member flows that take different-length paths through the netw
or misrouted data is handled just the same as the equipme ork are
nt failures combined, a merge point may require extra buffering to equalize the delays
that are handled by typical routing and bridging protocol over the different paths. This equalization ensures that the resultant
s. compound flow will not exceed its contracted bandwidth even after one of the
</t><t> paths is restored after a failure. The extra buffering can be also used to
If member flows that take different-length paths provide in-order delivery.
through the network are combined, a merge </t>
point may require extra buffering to equalize the delays </section>
over the different paths. This <section anchor="PacketEncoding" numbered="true" toc="default">
equalization ensures that the resultant compound flow wil <name>Packet Encoding for Service Protection</name>
l not exceed its <t>
contracted bandwidth even after one or the other of the p
aths is restored
after a failure. The extra buffering can be also used to
provide in-order delivery.
</t>
</section>
<section anchor="PacketEncoding" title="Packet encoding for service protec
tion">
<t>
There are methods for using multiple paths to provide service prot ection There are methods for using multiple paths to provide service prot ection
that involve encoding the information in a that involve encoding the information in a
packet belonging to a DetNet flow into multiple transmission units , packet belonging to a DetNet flow into multiple transmission units ,
combining information from multiple packets into any given transmi ssion unit. combining information from multiple packets into any given transmi ssion unit.
Such techniques, also known as "network coding", Such techniques, also known as "network coding",
can be used as a DetNet service protection technique. can be used as a DetNet service protection technique.
</t>
</section>
</section>
<section anchor="pinned" numbered="true" toc="default">
<name>Explicit Routes</name>
<t>
In networks controlled by typical dynamic control protocols such as IS-IS or
OSPF, a network topology event in one part of the network can impact, at least
briefly, the delivery of data in parts of the network remote from the failure
or recovery event. Even the use of redundant paths through a network, e.g., as
defined by <xref target="RFC6372" format="default"/>, does not eliminate the cha
nces of packet
loss. Furthermore, out-of-order packet delivery can be a side effect of route
changes. </t>
<t> Many real-time networks rely on physical rings of two-port
devices, with a relatively simple ring control protocol. This supports
redundant paths for service protection with a minimum of wiring. As an
additional benefit, ring topologies can often utilize different topology
management protocols from those used for a mesh network, with a consequent
reduction in the response time to topology changes. Of course, this comes at
some cost in terms of increased hop count, and thus latency, for the typical
path. </t>
<t> In order to get the advantages of low hop count and still ensure a
gainst
even very brief losses of connectivity, DetNet employs explicit routes where
the path taken by a given DetNet flow does not change, at least not
immediately and likely not at all, in response to network topology events.
Service protection (see Sections <xref target="srvcprot" format="counter"/>
and <xref target="PacketEncoding" format="counter"/>) over explicit routes
provides a high likelihood of continuous connectivity. Explicit routes can be
established in various ways, e.g., with RSVP-TE <xref target="RFC3209" format="d
efault"/>, with
Segment Routing (SR) <xref target="RFC8402" format="default"/>, via a SDN approa
ch <xref target="RFC8453" format="default"/>, with IS-IS <xref target="RFC7813"
format="default"/>, etc. Explicit routes
are typically used in MPLS TE (Traffic Engineering) Label Switched Paths
(LSPs). </t>
<t> Out-of-order packet delivery can be a side effect of distributing
a single
flow over multiple paths, especially when there is a change from one path to
another when combining the flow. This is irrespective of the distribution
method used and also applies to service protection over explicit routes. As
described in <xref target="inorder" format="default"/>, out-of-order packets inf
luence the
jitter of a flow and impact the amount of buffering needed to process the
data; therefore, the guarantees of a DetNet service include a maximum amount
of misordering as a constraint. The use of explicit routes helps to provide in-o
rder delivery
because there is no immediate route change with the network topology, but the
changes are plannable as they are between the different explicit routes.
</t> </t>
</section>
</section> </section>
</section> <section anchor="MoreGoals" numbered="true" toc="default">
<name>Secondary Goals for DetNet</name>
<section anchor="pinned" title="Explicit routes"> <t>
<t>
In networks controlled by typical dynamic control protoco
ls such as IS-IS or OSPF,
a network topology event in one part of the network
can impact, at least briefly, the delivery of data in par
ts of the network remote from the
failure or recovery event. Even the use of redundant path
s through a network, e.g.,
as defined by <xref target="RFC6372"/> do not eliminate t
he chances of packet loss. Furthermore,
out-of-order packet delivery can be a side effect of rout
e changes.
</t><t>
Many real-time networks rely on physical rings of two-por
t devices, with
a relatively simple ring control protocol. This supports
redundant paths for
service protection with a minimum
of wiring. As an additional benefit, ring topologies can
often
utilize different topology management protocols than thos
e used for a mesh network, with
a consequent reduction in the response time to topology c
hanges. Of course, this comes
at some cost in terms of increased hop count, and thus la
tency, for the typical path.
</t><t>
In order to get the advantages of low hop count and still
ensure against even very brief
losses
of connectivity, DetNet employs explicit routes, where th
e path taken by a given DetNet flow
does not change, at least immediately, and likely not at
all, in response to network
topology events. Service protection (<xref target="srvcp
rot"/> or
<xref target="PacketEncoding"/>)
over explicit routes provides a high likelihood of contin
uous connectivity.
Explicit routes can be established in various ways, e.g.,
with
RSVP-TE <xref target="RFC3209"/>, with Segment Routing (S
R)
<xref target="RFC8402"/>, via a Software
Defined Networking approach <xref target="RFC8453"/>, wit
h IS-IS
<xref target="RFC7813"/>, etc.
Explicit routes are typically used in MPLS TE LSPs.
</t><t>
Out-of-order packet delivery can be a side effect of dist
ributing a
single flow over multiple paths, especially when there is
a change
from one path to another when combining the flow. This is
irrespective of the distribution method used, and also ap
plies to
service protection over explicit routes. As described in
<xref target="inorder"/>, out-of-order packets influence
the
jitter of a flow and impact the amount of buffering neede
d to
process the data; therefore, DetNet service includes maxi
mum
allowed misordering as a constraint. The use of explicit
routes
helps to provide in-order delivery because there is no im
mediate
route change with the network topology, but the changes a
re plannable
as they are between the different explicit routes.
</t>
</section>
</section>
<section anchor="MoreGoals" title="Secondary goals for DetNet">
<t>
Many applications require DetNet to provide additional services, includi ng coexistence with Many applications require DetNet to provide additional services, includi ng coexistence with
other QoS mechanisms <xref target="Coexistence"/> and protection against other QoS mechanisms (<xref target="Coexistence" format="default"/>) and
misbehaving transmitters protection against misbehaving transmitters
<xref target="FaultMitigation"/>. (<xref target="FaultMitigation" format="default"/>).
</t> </t>
<section anchor="Coexistence" title="Coexistence with normal traffic"> <section anchor="Coexistence" numbered="true" toc="default">
<name>Coexistence with Normal Traffic</name>
<t> <t>
A DetNet network supports the dedication of a high proportion of t A DetNet network supports the dedication of a high proportion of
he the network bandwidth to DetNet flows. But, no matter how much
network bandwidth is dedicated for DetNet flows, it is a goal of DetNet to coexist
to DetNet flows. But, no matter how much is dedicated for DetNet with existing Class-of-Service schemes (e.g., DiffServ). It is
flows, it is also important that non-DetNet traffic not disrupt the DetNet
a goal of DetNet to coexist with existing Class of Service schemes flow, of course (see Sections <xref target="FaultMitigation" forma
(e.g., DiffServ). t="counter"/> and <xref target="SecurityConsiderations" format="counter"/>). Fo
It is also r these reasons:
important that non-DetNet traffic not disrupt the DetNet flow, of
course (see
<xref target="FaultMitigation"/> and <xref target="SecurityConside
rations"/>).
For these reasons:
<list style="symbols">
<t>
Bandwidth (transmission opportunities) not utilized by a D
etNet flow is available
to non-DetNet packets (though not to other DetNet flows).
</t><t>
DetNet flows can be shaped or scheduled, in order to ensur
e that the
highest-priority non-DetNet
packet is also ensured a worst-case latency.
</t><t>
When transmission opportunities for DetNet flows are sched
uled in detail, then
the algorithm constructing the schedule should leave suffi
cient opportunities for
non-DetNet packets to satisfy the needs of the users of th
e network. Detailed
scheduling can also permit the time-shared use of buffer r
esources by different
DetNet flows.
</t>
</list>
</t><t>
Starvation of non-DetNet traffic must be avoided, e.g., by traffic
policing and shaping functions (e.g., <xref target="RFC
2475"/>). Thus, the net
effect of the presence of DetNet flows in a network on
the
non-DetNet flows is primarily a reduction in the availa
ble bandwidth.
</t> </t>
</section> <ul spacing="normal">
<section anchor="FaultMitigation" title="Fault Mitigation"> <li>Bandwidth (transmission opportunities) not utilized by a DetNet
flow is
available to non-DetNet packets (though not to other DetNet flows). </li>
<li> DetNet flows can be shaped or scheduled, in order to ensure tha
t the
highest-priority non-DetNet packet is also ensured a worst-case latency. </li>
<li> When transmission opportunities for DetNet flows are scheduled
in detail,
the algorithm constructing the schedule should leave sufficient
opportunities for non-DetNet packets to satisfy the needs of the users of the
network. Detailed scheduling can also permit the time-shared use of buffer
resources by different DetNet flows.
</li>
</ul>
<t> Starvation of non-DetNet traffic must be avoided, for example, by
traffic policing and shaping functions (e.g., <xref target="RFC2475" f
ormat="default"/>). Thus, the net effect of the presence of DetNet
flows in a network on the non-DetNet flows is primarily a reduction
in the available bandwidth.
</t>
</section>
<section anchor="FaultMitigation" numbered="true" toc="default">
<name>Fault Mitigation</name>
<t>
Robust real-time systems require reducing the number of possible
failures. Filters and policers should be used in a DetNet
network to detect if DetNet packets are received on the wrong
interface, at the wrong time, or in too great a volume.
Furthermore, filters and policers can take actions to discard
the offending packets or flows, or trigger shutting down the
offending flow or the offending interface. </t>
<t> It is also
essential that filters and service remarking be employed at the
network edge to prevent non-DetNet packets from being mistaken
for DetNet packets and thus impinging on the resources
allocated to DetNet packets. In particular, sending DetNet
traffic into networks that have not been provisioned in advance
to handle that DetNet traffic has to be treated as a fault. The
use of egress traffic filters, or equivalent mechanisms, to
prevent this from happening are strongly recommended at the
edges of DetNet networks and DetNet supporting networks. In
this context, the term 'provisioned' has a broad meaning, e.g.,
provisioning could be performed via an administrative decision
that the downstream network has the available capacity to carry
the DetNet traffic that is being sent into it. </t>
<t> <t>
Robust real-time systems require reducing the number of
possible failures. Filters and policers should be used in a DetNet
network to
detect if DetNet packets are received
on the wrong interface, or at the wrong time, or in too great a vo
lume.
Furthermore, filters and policers can take
actions to discard the offending packets or flows, or trigger
shutting down the offending flow or the offending interface.
</t><t>
It is also essential that filters and service remarking be employe
d at the network edge
to prevent non-DetNet
packets from being mistaken for DetNet packets, and thus impinging
on the resources
allocated to DetNet packets. In particular, sending DetNet traffic
into networks that have not been provisioned in advance
to handle
that DetNet traffic has to be treated as a fault. The
use of
egress traffic filters, or equivalent mechanisms, to pr
event this
from happening are strongly recommended at the edges of
a DetNet
networks and DetNet supporting networks. In this conte
xt, the
term 'provisioned' has a broad meaning, e.g., provision
ing could be
performed via an administrative decision that the downs
tream network
has the available capacity to carry the DetNet traffic
that is being
sent into it.
</t><t>
Note that the sending of App-flows that do not use tran Note that the sending of App-flows that do not use
sport layer transport-layer congestion control per <xref target="RF
congestion control per <xref target="RFC2914"/> into a C2914" format="default"/> into a network that is not
network that provisioned to handle such traffic has to be treated
is not provisioned to handle such DetNet traffic has to as a fault and prevented. PRF-generated DetNet
be treated member flows also need to be treated as not using
as a fault and prevented. PRF generated DetNet member f transport-layer congestion control even if the
lows also original App-flow supports transport-layer
need to be treated as not using transport layer congest congestion control because PREOF can remove
ion control congestion indications at the PEF and thereby hide
even if the original App-flow supports transport layer such indications (e.g., drops, ECN markings,
congestion increased latency) from end systems.
control because PREOF can remove congestion indications
at the PEF
and thereby hide such indications (e.g., drops, ECN mar
kings,
increased latency) from end systems.
</t><t>
The mechanisms to support these requirements are both data plane
and implementation specific. Data plane specific solut
ions will
be specified in the relevant data plane solution docume
nt. There
also exist techniques, at present and/or in various sta
ges of
standardization, that can support these fault mitigation tasks tha
t deliver a high probability that misbehaving
systems will have zero impact on well-behaved DetNet flows, except
of course, for
the receiving interface(s) immediately downstream of the misbehavi
ng device.
Examples of such techniques include traffic policing and shaping f
unctions (e.g.,
<xref target="RFC2475"/>) and separating flows into per-flow rate-
limited queues,
and potentially apply active queue management <xref tar
get="RFC7567"/>.
</t> </t>
<t> The mechanisms to support these requirements are both Data
Plane and implementation specific. Solutions that are data-plane
specific will be specified in the relevant data-plane solution
document. There also exist techniques, at present and/or in various
stages of standardization, that can support these fault-mitigation
tasks that deliver a high probability that misbehaving systems will
have zero impact on well-behaved DetNet flows with the exception, of
course, of the receiving interface(s) immediately downstream from
the misbehaving device. Examples of such techniques include traffic
policing and shaping functions (e.g., those described in <xref target=
"RFC2475" format="default"/>),
separating flows into per-flow rate-limited queues, and potentially
applying active queue management <xref target="RFC7567" format="defaul
t"/>.
</t>
</section>
</section> </section>
</section>
</section> </section>
<section anchor="arch" numbered="true" toc="default">
<section anchor="arch" title="DetNet Architecture"> <name>DetNet Architecture</name>
<section anchor="Stacks" title="DetNet stack model"> <section anchor="Stacks" numbered="true" toc="default">
<t> <name>DetNet Stack Model</name>
DetNet functionality (<xref target="ProvidingQoS"/>) is impleme <t>
nted DetNet functionality (<xref target="ProvidingQoS" format="defau
lt"/>) is implemented
in two adjacent sub-layers in the protocol stack: the DetNet se rvice in two adjacent sub-layers in the protocol stack: the DetNet se rvice
sub-layer and the DetNet forwarding sub-layer. The DetNet servi ce sub-layer sub-layer and the DetNet forwarding sub-layer. The DetNet servi ce sub-layer
provides DetNet service, e.g., service protection, to higher l ayers provides DetNet service, e.g., service protection, to higher l ayers
in the protocol stack and applications. The DetNet forwarding s ub-layer in the protocol stack and applications. The DetNet forwarding s ub-layer
supports DetNet service in the underlying network, e.g., by supports DetNet service in the underlying network, e.g., by
providing explicit routes and resource allocation to DetNet flo ws. providing explicit routes and resource allocation to DetNet flo ws.
</t> </t>
<section anchor="StackModel" title="Representative Protocol Stack Model" <section anchor="StackModel" numbered="true" toc="default">
> <name>Representative Protocol Stack Model</name>
<t> <t>
<xref target="ProtStack1"/> illustrates a conceptual DetNet data <xref target="ProtStack1" format="default"/> illustrates a conce
plane layering model. ptual DetNet data-plane layering model.
One may compare it to that in <xref target="IEEE802.1CB"/>, Anne One may compare it to that in <xref target="IEEE802.1CB" format=
x C. "default"/>, Annex C.
</t> </t>
<figure align="center" anchor="ProtStack1" <figure anchor="ProtStack1">
title="DetNet data plane protocol stack"> <name>DetNet Data-Plane Protocol Stack</name>
<artwork align="center"><![CDATA[ <artwork align="center" name="" type="" alt=""><![CDATA[
| packets going | ^ packets coming ^ | packets going | ^ packets coming ^
v down the stack v | up the stack | v down the stack v | up the stack |
+-----------------------+ +-----------------------+ +-----------------------+ +-----------------------+
| Source | | Destination | | Source | | Destination |
+-----------------------+ +-----------------------+ +-----------------------+ +-----------------------+
| Service sub-layer: | | Service sub-layer: | | Service sub-layer: | | Service sub-layer: |
| Packet sequencing | | Duplicate elimination | | Packet sequencing | | Duplicate elimination |
| Flow replication | | Flow merging | | Flow replication | | Flow merging |
| Packet encoding | | Packet decoding | | Packet encoding | | Packet decoding |
+-----------------------+ +-----------------------+ +-----------------------+ +-----------------------+
| Forwarding sub-layer: | | Forwarding sub-layer: | | Forwarding sub-layer: | | Forwarding sub-layer: |
| Resource allocation | | Resource allocation | | Resource allocation | | Resource allocation |
| Explicit routes | | Explicit routes | | Explicit routes | | Explicit routes |
+-----------------------+ +-----------------------+ +-----------------------+ +-----------------------+
| Lower layers | | Lower layers | | Lower layers | | Lower layers |
+-----------------------+ +-----------------------+ +-----------------------+ +-----------------------+
v ^ v ^
\_________________________/ \_________________________/
]]></artwork> ]]></artwork>
</figure> </figure>
<t> <t>
Not all sub-layers are required for any given application, or ev en for any Not all sub-layers are required for any given application, or ev en for any
given network. The functionality shown in <xref target="ProtStac given network. The functionality shown in <xref target="ProtStac
k1"/> is: k1" format="default"/> is:
<list hangIndent="8" style="hanging"> </t>
<t hangText="Application"><vspace blankLines="0"/> <dl newline="true" spacing="normal" indent="3">
<dt>Application</dt>
<dd>
Shown as "source" and "destination" in the diagram. Shown as "source" and "destination" in the diagram.
</t> </dd>
<t hangText="Packet sequencing"><vspace blankLines="0"/> <dt>Packet sequencing</dt>
As part of DetNet service protection, supplies the seque <dd>
nce number for As part of the DetNet service sub-layer, the packet
packet replication and elimination (<xref target="srvcpr sequencing function supplies the sequence number for
ot"/>), thus peers packet replication and elimination for DetNet service
with Duplicate elimination. This sub-layer is not neede protection (<xref target="Seamless" format="default"/>); thu
d if a higher layer protocol s, its peer is
is expected to perform any packet sequencing and duplica duplicate elimination. This sub-layer is not needed if a
te elimination higher-layer protocol is expected to perform any packet
required by the DetNet flow replication. sequencing and duplicate elimination required by the
</t> DetNet flow replication.
<t hangText="Duplicate elimination"><vspace blankLines="0"/> </dd>
As part of the DetNet service sub-layer, based on the se <dt>Duplicate elimination</dt>
quenced number <dd>
supplied by its peer, packet sequencing, As part of the DetNet service sub-layer, based on the se
Duplicate elimination discards any duplicate packets gen quence number
erated by DetNet supplied by its peer (packet sequencing),
duplicate elimination discards any duplicate packets gen
erated by DetNet
flow replication. It can operate on member flows, compo und flows, or both. flow replication. It can operate on member flows, compo und flows, or both.
The replication may also be inferred from other The replication may also be inferred from other
information such as the precise time of reception in a s cheduled network. information such as the precise time of reception in a s cheduled network.
The duplicate elimination sub-layer may also perform res equencing of packets The duplicate elimination sub-layer may also perform res equencing of packets
to restore packet order in a to restore packet order in a
flow that was disrupted by the loss of packets on one or another of flow that was disrupted by the loss of packets on one or another of
the multiple paths taken. the multiple paths taken.
</t> </dd>
<t hangText="Flow replication"><vspace blankLines="0"/> <dt>Flow replication</dt>
As part of DetNet service protection, packets that belon <dd> As
g to a part of DetNet service protection, packets that belong to
DetNet compound flow are replicated into two or more Det a DetNet compound flow are replicated into two or more
Net member flows. DetNet member flows. This function is separate from
This function is separate from packet sequencing. Flow packet sequencing. Flow replication can be an explicit
replication replication and remarking of packets or can be performed
can be an explicit replication and remarking of packets, by, for example, techniques similar to ordinary multicast
or can be performed by, replication, albeit with resource allocation implications.
for example, techniques similar to ordinary multicast re Its peer is DetNet flow merging.
plication, albeit with </dd>
resource allocation implications. <dt>Flow merging</dt>
Peers with DetNet flow merging. <dd> As
</t> part of the DetNet service sub-layer, the flow merging
<t hangText="Flow merging"><vspace blankLines="0"/> function combines DetNet member flows together for packets
As part of DetNet service protection, merges coming up the stack belonging to a specific DetNet
DetNet member flows together for packets coming up the s compound flow. DetNet flow merging, together with packet
tack belonging to a sequencing, duplicate elimination, and DetNet flow
specific DetNet compound flow. replication perform packet replication and elimination
Peers with DetNet flow replication. (<xref target="srvcprot" format="default"/>). Its peer is De
DetNet flow merging, together with packet sequencing, du tNet flow
plicate elimination, replication.
and DetNet flow replication perform </dd>
packet replication and elimination (<xref target="srvcpr <dt>Packet encoding</dt>
ot"/>). <dd> As
</t> part of DetNet service protection, as an alternative to
<t hangText="Packet encoding"><vspace blankLines="0"/> packet sequencing and flow replication, packet encoding
As part of DetNet service protection, as an alternative combines the information in multiple DetNet packets,
to packet sequencing perhaps from different DetNet compound flows, and
and flow replication, packet encoding combines the infor transmits that information in packets on different DetNet
mation in member flows. Its peer is packet decoding.
multiple DetNet packets, perhaps from different DetNet c </dd>
ompound flows, and transmits that <dt>Packet decoding</dt>
information in packets on different DetNet member Flows. <dd> As
Peers with Packet decoding. part of DetNet service protection, as an alternative to
</t> flow merging and duplicate elimination, packet decoding
<t hangText="Packet decoding"><vspace blankLines="0"/> takes packets from different DetNet member flows and
As part of DetNet service protection, as an alternative computes from those packets the original DetNet packets
to flow merging and from the compound flows input to packet encoding. Its
duplicate elimination, packet decoding takes packets fro peer is packet encoding.
m different DetNet member </dd>
flows, and computes from those packets the original DetN <dt>Resource allocation</dt>
et packets from the <dd>
compound flows input to packet encoding. Peers with Pac The DetNet forwarding sub-layer provides resource
ket encoding. allocation. See <xref target="QueuingModels" format="default
</t> "/>. The
<t hangText="Resource allocation"><vspace blankLines="0"/> actual queuing and shaping mechanisms are typically
The DetNet forwarding sub-layer provides resource alloca provided by the underlying subnet. These can be closely
tion. See <xref target="QueuingModels"/>. associated with the means of providing paths for DetNet
The actual queuing and shaping mechanisms are typically flows. The path and the resource allocation are conflated
provided by underlying subnet. in this figure.
These can be closely associated with the means of provid </dd>
ing paths <dt>Explicit routes</dt>
for DetNet flows. <dd>
The path and the resource allocation are conflated in th Explicit routes are arrangements of fixed paths operated at
is figure. the DetNet forwarding sub-layer that are determined in advance
</t> to avoid the impact of network convergence on DetNet flows.
<t hangText="Explicit routes"><vspace bla </dd>
nkLines="0"/> </dl>
The DetNet forwarding sub-layer provides mechanisms to e <t>
nsure that fixed paths are provided Operations, Administration, and Maintenance (OAM) leverages
for DetNet flows. These explicit paths avoid the impact in-band and out-of-band signaling that validates whether the
of network convergence. service is effectively obtained within QoS constraints. OAM is
</t> not shown in <xref target="ProtStack1" format="default"/>; it may re
side in any
</list> number of the layers. OAM can involve specific tagging added in
</t> the packets for tracing implementation or network configuration
<t> errors; traceability enables finding whether a packet is a
Operations, Administration, and Maintenance (OAM) leverages in-band replica, which DetNet relay node performed the replication, and
and which segment was intended for the replica. Active and hybrid OAM
out-of-band signaling that validates whether the service methods require additional bandwidth to perform fault management
is effectively and performance monitoring of the DetNet domain. OAM may, for
obtained within QoS constraints. OAM is not shown in instance, generate special test probes or add OAM information into
<xref target="ProtStack1"/>; it may reside in any number the data packet.
of the layers. </t>
OAM can involve specific tagging added in the packets for <t>
tracing implementation The packet replication and elimination functions may be
or network configuration errors; traceability enables to performed either at the source and destination ends of a
find whether a DetNet compound flow or in a DetNet relay node.
packet is a replica, which DetNet relay node performed th </t>
e replication, and which
segment was intended for the replica. Active and hybrid O
AM methods require
additional bandwidth to perform fault management and perf
ormance monitoring
of the DetNet domain. OAM may, for instance, generate spe
cial test probes or
add OAM information into the data packet.
</t>
<t>
The packet sequencing and replication elimination functions at t
he
source and destination ends of a DetNet compound flow may be per
formed either
in the end system or in a DetNet relay node.
</t>
</section> </section>
<section title="DetNet Data Plane Overview" anchor="sec_dt_dp"> <section anchor="sec_dt_dp" numbered="true" toc="default">
<name>DetNet Data-Plane Overview</name>
<t> <t>
A "Deterministic Network" will be composed of DetNet enabled end A "Deterministic Network" will be composed of DetNet-enabled end
systems, DetNet edge nodes, and DetNet relay nodes, which collective systems, DetNet edge nodes, and DetNet relay nodes, which
ly deliver DetNet collectively deliver DetNet services. DetNet relay and edge nodes
services. DetNet relay and edge nodes are interconnected via DetNet are interconnected via DetNet transit nodes (e.g., LSRs), which
transit nodes support DetNet but are not DetNet service aware. All DetNet nodes
(e.g., LSRs) which support DetNet, but are not DetNet service are connected to sub-networks, where a point-to-point link is also
aware. All DetNet nodes are connected to considered a simple sub-network. These sub-networks provide
sub-networks, where a point-to-point link is also considered as a DetNet-compatible service for support of DetNet traffic. Examples
simple sub-network. These of sub-network technologies include MPLS TE, TSN as defined by
sub-networks will provide DetNet compatible service for support of D IEEE 802.1, and OTN (Optical Transport Network). Of course,
etNet multilayer DetNet systems may also be possible, where one DetNet
traffic. Examples of sub-network technologies include MPLS TE, IEEE appears as a sub-network and provides service to a higher-layer
802.1 TSN and DetNet system. A simple DetNet concept network is shown in <xref tar
OTN. Of course, multi-layer DetNet systems may also be possible, wh get="fig_detnet" format="default"/>.
ere
one DetNet appears as a sub-network, and provides service to, a high Note that in this and following figures, "Forwarding" and "Fwd" refer to the
er layer DetNet forwarding sub-layer, and "Service" and "Svc" refer to the DetNet
DetNet system. A simple DetNet concept network is shown in <xref tar service sub-layer; both of these sub-layers are described in detail in <xref tar
get="fig_detnet"/>. get="StackModel" format="default"/>.
Note that in this and following figures "Forwarding" and
"Fwd" refer to the DetNet
forwarding sub-layer, "Service" and "Svc" refer to the De
tNet service sub-layer,
which are described in detail in <xref target="Stacks"/>.
</t> </t>
<figure anchor="fig_detnet" align="center" <figure anchor="fig_detnet">
title="A Simple DetNet Enabled Network"> <name>A Simple DetNet-Enabled Network</name>
<artwork align="center"><![CDATA[ <artwork align="center" name="" type="" alt=""><![CDATA[
TSN Edge Transit Relay DetNet TSN Edge Transit Relay DetNet
End System Node Node Node End System End System Node Node Node End System
+----------+ +.........+ +----------+ +----------+ +.........+ +----------+
| Appl. |<--:Svc Proxy:-- End to End Service -------->| Appl. | | Appl. |<--:Svc Proxy:-- End-to-End Service -------->| Appl. |
+----------+ +---------+ +---------+ +----------+ +----------+ +---------+ +---------+ +----------+
| TSN | |TSN| |Svc|<- DetNet flow --: Service :-->| Service | | TSN | |TSN| |Svc|<- DetNet flow --: Service :-->| Service |
+----------+ +---+ +---+ +--------+ +---------+ +----------+ +----------+ +---+ +---+ +--------+ +---------+ +----------+
|Forwarding| |Fwd| |Fwd| | Fwd | |Fwd| |Fwd| |Forwarding| |Forwarding| |Fwd| |Fwd| | Fwd | |Fwd| |Fwd| |Forwarding|
+-------.--+ +-.-+ +-.-+ +--.----.+ +-.-+ +-.-+ +---.------+ +-------.--+ +-.-+ +-.-+ +--.----.+ +-.-+ +-.-+ +---.------+
: Link : / ,-----. \ : Link : / ,-----. \ : Link : / ,-----. \ : Link : / ,-----. \
+........+ +-[ Sub ]-+ +.......+ +-[ Sub ]-+ +........+ +-[ Sub- ]-+ +.......+ +-[ Sub- ]-+
[Network] [Network] [network] [network]
`-----' `-----' `-----' `-----'
]]></artwork> ]]></artwork>
</figure> </figure>
<t> <t>
DetNet data plane is divided into two sub-layers: the DetNet DetNet Data Plane is divided into two sub-layers: the DetNet
service sub-layer and the DetNet forwarding sub-layer. Th service sub-layer and the DetNet forwarding sub-layer. This helps
is to explore and evaluate various combinations of the data-plane
helps to explore and evaluate various combinations of the solutions available. Some of them are illustrated in <xref target="f
data ig_adaptation" format="default"/>. This separation of DetNet sub-layers,
plane solutions available. Some of them are illustrated i while helpful, should not be considered a formal requirement.
n For example, some technologies may violate these strict sub-layers
<xref target="fig_adaptation"/>. This separation of DetN and still be able to deliver a DetNet service.
et
sub-layers, while helpful, should not be considered as fo
rmal
requirement. For example, some technologies may violate
these
strict sub-layers and still be able to deliver a DetNet s
ervice.
</t> </t>
<figure anchor="fig_adaptation">
<figure anchor="fig_adaptation" align="center" <name>DetNet Adaptation to Data Plane</name>
title="DetNet adaptation to data plane"> <artwork align="center" name="" type="" alt=""><![CDATA[
<artwork align="center"><![CDATA[
. .
. .
+-----------------------------+ +-----------------------------+
| DetNet Service sub-layer | PW, UDP, GRE | DetNet Service sub-layer | PW, UDP, GRE
+-----------------------------+ +-----------------------------+
| DetNet Forwarding sub-layer | IPv6, IPv4, MPLS TE LSPs, MPLS SR | DetNet Forwarding sub-layer | IPv6, IPv4, MPLS TE LSPs, MPLS SR
+-----------------------------+ +-----------------------------+
. .
. .
]]></artwork> ]]></artwork>
</figure> </figure>
<t> <t>
In some networking scenarios, the end system initially provides a De In some networking scenarios, the end system initially provides a
tNet DetNet flow encapsulation, which contains all information needed
flow encapsulation, which contains all information needed by DetNet by DetNet nodes (e.g., DetNet flow based on the Real-time Transport
nodes Protocol (RTP) <xref target="RFC3550" format="default"/> that is car
(e.g., Real-time Transport Protocol (RTP) <xref target="RFC3550"/> b ried over a
ased native UDP/IP network or pseudowire (PW)). In other scenarios, the
DetNet flow carried over a native UDP/IP network or PseudoWire). In encapsulation formats might differ significantly.
other scenarios, the encapsulation formats might differ significantl
y.
</t> </t>
<t> <t>
There are many valid options to create a data plane solution for Det Net There are many valid options to create a data-plane solution for Det Net
traffic by selecting a technology approach for the DetNet service su b-layer and traffic by selecting a technology approach for the DetNet service su b-layer and
also selecting a technology approach for the DetNet forwarding sub-l ayer. There are also selecting a technology approach for the DetNet forwarding sub-l ayer. There are
a large number of valid combinations. a large number of valid combinations.
</t> </t>
<t> <t>
One of the most fundamental differences between different potential One of the most fundamental differences between different
data plane options is the basic headers used by DetNet potential data-plane options is the basic headers used by DetNet
nodes. For example, the basic service can be delivered based on an nodes. For example, the basic service can be delivered based on
MPLS label an MPLS label or an IP header. This decision impacts the basic
or an IP header. This decision impacts the basic forwardi forwarding logic for the DetNet service sub-layer. Note that in
ng logic for the DetNet both cases, IP addresses are used to address DetNet nodes. The
service sub-layer. Note that in both cases, IP addresses selected DetNet forwarding sub-layer technology also needs to be
are used to address DetNet nodes. mapped to the subnet technology used to interconnect DetNet
The selected DetNet forwarding sub-layer technology also needs to nodes. For example, DetNet flows will need to be mapped to TSN
be mapped to the sub-net Streams.
technology used to interconnect DetNet nodes. For example
, DetNet flows will need to
be mapped to TSN Streams.
</t> </t>
</section> </section>
<section anchor="netref" title="Network reference model"> <section anchor="netref" numbered="true" toc="default">
<name>Network Reference Model</name>
<t> <xref target="fig_DetNetservice"/> shows another view of the <t> <xref target="fig_DetNetservice" format="default"/> shows another
DetNet service related reference points and main componen view of the
ts. DetNet service-related reference points and main componen
</t> ts.
</t>
<figure anchor="fig_DetNetservice" align="center" <figure anchor="fig_DetNetservice">
title="DetNet Service Reference Model (multi-domain)"> <name>DetNet Service Reference Model (Multidomain)</name>
<artwork><![CDATA[ <artwork name="" type="" align="left" alt=""><![CDATA[
DetNet DetNet DetNet DetNet
end system end system End System End System
_ _ _ _
/ \ +----DetNet-UNI (U) / \ / \ +----DetNet-UNI (U) / \
/App\ | /App\ /App\ | /App\
/-----\ | /-----\ /-----\ | /-----\
| NIC | v ________ | NIC | | NIC | v ________ | NIC |
+--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+
| / \__/ \ | | | / \__/ \ | |
| / +----+ +----+ \_____ | | | / +----+ +----+ \_____ | |
| / | | | | \_______ | | | / | | | | \_______ | |
+------U PE +----+ P +----+ \ _ v | +------U PE +----+ P +----+ \ _ v |
| | | | | | | ___/ \ | | | | | | | | ___/ \ |
| +--+-+ +----+ | +----+ | / \_ | | +--+-+ +----+ | +----+ | / \_ |
\ | | | | | / \ | \ | | | | | / \ |
\ | +----+ +--+-+ +--+PE |------ U-----+ \ | +----+ +--+-+ +--+PE |------ U-----+
\ | | | | | | | | | \_ _/ \ | | | | | | | | | \_ _/
\ +---+ P +----+ P +--+ +----+ | \____/ \ +---+ P +----+ P +--+ +----+ | \____/
\___ | | | | / \___ | | | | /
\ +----+__ +----+ DetNet-1 DetNet-2 \ +----+__ +----+ DetNet-1 DetNet-2
| \_____/ \___________/ | | \_____/ \___________/ |
| | | |
| | End-to-End service | | | | | | End-to-End Service | | | |
<-------------------------------------------------------------> <------------------------------------------------------------->
| | DetNet service | | | | | | DetNet Service | | | |
| <------------------------------------------------> | | <------------------------------------------------> |
| | | | | | | | | | | |
]]></artwork> ]]></artwork>
</figure> </figure>
<t>DetNet User-to-Network Interfaces (DetNet-UNIs) ("U" in <xref targe
<t>DetNet User Network Interfaces (DetNet-UNIs) ("U" in <xref target t="fig_DetNetservice" format="default"/>) are assumed in this document to be
="fig_DetNetservice"/>) are assumed in this document to be packet-based referenc packet-based reference points and provide connectivity over the
e points and provide connectivity over the packet network. A DetNet-UNI may prov packet network. A DetNet-UNI may provide multiple functions. For
ide multiple functions, e.g., it may add networking technology specific encapsul example, it may:
ation to the DetNet flows if necessary; it may provide status of the availabilit </t>
y of the resources associated with a reservation; it may provide a synchronizati <ul spacing="normal">
on service for the end system; it may carry enough signaling to place the reserv <li>add encapsulation specific to networking technology to the DetNe
ation in a network without a controller, or if the controller only deals with th t flows if necessary,
e network but not the end systems. Internal reference points of end systems (bet </li>
ween the application and the NIC) are more challenging from control perspective <li>provide status of the availability of the resources associated w
and they may have extra requirements (e.g., in-order delivery is expected in end ith a reservation,
system internal reference points, whereas it is considered optional over the De </li>
tNet-UNI).</t> <li>provide a synchronization service for the end system, or
</li>
<li>carry enough signaling to place the reservation in a network wit
hout a
controller or in a network where the controller only deals with the network
but not the end systems.
</li>
</ul>
<t>
Internal reference points of end systems (between the application
and the Network Interface Card (NIC)) are more challenging from the
control perspective, and they may have extra requirements (e.g.,
in-order delivery is expected in end system internal reference
points, whereas it is considered optional over the DetNet-UNI).</t>
</section> </section>
</section> </section>
<section anchor="netrefsys" title="DetNet systems"> <section anchor="netrefsys" numbered="true" toc="default">
<section anchor="es" title="End system"> <name>DetNet Systems</name>
<t> <section anchor="es" numbered="true" toc="default">
The traffic characteristics of an App-flow can be CBR (constant b <name>End System</name>
it rate) or VBR (variable bit rate) and can have Layer-1 or Layer-2 or Layer-3 e <t>
ncapsulation (e.g., TDM (time-division multiplexing), Ethernet, IP). These chara The traffic characteristics of an App-flow can be CBR
cteristics are considered as input for resource reservation and might be simplif (constant bit rate) or VBR (variable bit rate) and can have
ied to ensure determinism during packet forwarding (e.g., making reservations fo Layer 1, Layer 2, or Layer 3 encapsulation (e.g., TDM
r the peak rate of VBR traffic, etc.). (time-division multiplexing) Ethernet, IP). These
</t> characteristics are considered as input for resource
reservation and might be simplified to ensure determinism
<t> during packet forwarding (e.g., making reservations for the
An end system may or may not be DetNet forwarding sub-layer aware peak rate of VBR traffic, etc.).
or DetNet service sub-layer aware. That is, an end system may or may not contai </t>
n DetNet specific functionality. End systems with DetNet functionalities may hav <t>
e the same or different forwarding sub-layer as the connected DetNet domain. Cat An end system may or may not be aware of the DetNet forwarding su
egorization of end systems are shown in <xref target="fig_endsys2"/>. b-layer
</t> or DetNet service sub-layer. That is, an end
system may or may not contain DetNet-specific
<figure anchor="fig_endsys2" align="center" functionality. End systems with DetNet functionalities may
title="Categorization of end systems"> have the same or different forwarding sub-layer as the
<artwork><![CDATA[ connected DetNet domain. Categorization of end systems are
shown in <xref target="fig_endsys2" format="default"/>.
</t>
<figure anchor="fig_endsys2">
<name>Categorization of End Systems</name>
<artwork name="" type="" align="left" alt=""><![CDATA[
End system End system
| |
| |
| DetNet aware ? | DetNet aware ?
/ \ / \
+------< >------+ +------< >------+
NO | \ / | YES NO | \ / | YES
| v | | v |
DetNet unaware | DetNet-unaware |
End system | End system |
| Service/Forwarding | Service/Forwarding
| sub-layer | sub-layer
/ \ aware ? / \ aware ?
+--------< >-------------+ +--------< >-------------+
f-aware | \ / | s-aware f-aware | \ / | s-aware
| v | | v |
| | both | | | both |
| | | | | |
DetNet f-aware | DetNet s-aware DetNet f-aware | DetNet s-aware
End system | End system End system | End system
v v
DetNet sf-aware DetNet sf-aware
End system End system
]]></artwork> ]]></artwork>
</figure> </figure>
<t>
Note some known use case examples for end systems:
<list style="symbols">
<t> DetNet unaware: The classic case requiring service pr
oxies.</t>
<t> DetNet f-aware: A DetNet forwarding sub-layer aware s
ystem. It knows about some TSN functions (e.g., reservation), but not about serv
ice protection. </t>
<t> DetNet s-aware: A DetNet service sub-layer aware syst
em. It supplies sequence numbers, but doesn't know about resource allocation. </
t>
<t> DetNet sf-aware: A full functioning DetNet end system
, it has DetNet functionalities and usually the same forwarding paradigm as the
connected DetNet domain. It can be treated as an integral part of the DetNet dom
ain. </t>
</list>
</t>
</section>
<section anchor="ertn" title="DetNet edge, relay, and transit nodes">
<t> <t>
As shown in <xref target="fig_detnet"/>, DetNet edge nodes providing The following are some known use case examples for end systems:
proxy </t>
service and DetNet relay nodes providing the DetNet service sub-laye <ul spacing="normal">
r are <li> DetNet unaware: The classic case requiring service proxies.</li
DetNet-aware, and DetNet transit nodes need only be aware of the Det >
Net <li> DetNet f-aware: A system that is aware of the DetNet forwarding
forwarding sub-layer. sub-layer. It knows about some TSN
</t><t> functions (e.g., reservation) but not about service
In general, if a DetNet flow passes through one or more DetNet-unawa protection. </li>
re <li> DetNet s-aware: A system that is aware of the DetNet service
network nodes between two DetNet nodes providing the DetNet forwardi sub-layer. It supplies sequence numbers but
ng sub-layer doesn't know about resource allocation. </li>
for that flow, there is a potential for disruption or failure of the <li> DetNet sf-aware: A fully functioning DetNet end
DetNet QoS. A network administrator needs to ensure that the DetNet system. It has DetNet functionalities and usually the
-unaware same forwarding paradigm as the connected DetNet
network nodes are configured to minimize the chances of packet loss domain. It can be treated as an integral part of the
and DetNet domain. </li>
delay, and provision enough extra buffer space in the DetNet transit </ul>
node </section>
following the DetNet-unaware network nodes to absorb the induced lat <section anchor="ertn" numbered="true" toc="default">
ency <name>DetNet Edge, Relay, and Transit Nodes</name>
variations. <t>
As shown in <xref target="fig_detnet" format="default"/>, DetNet edg
e nodes
providing proxy service and DetNet relay nodes providing the
DetNet service sub-layer are DetNet aware, and DetNet transit
nodes need only be aware of the DetNet forwarding sub-layer.
</t>
<t> In general, if a DetNet flow passes through one or more
DetNet-unaware network nodes between two DetNet nodes providing
the DetNet forwarding sub-layer for that flow, there is a
potential for disruption or failure of the DetNet QoS. A network
administrator needs to 1) ensure that the DetNet-unaware network
nodes are configured to minimize the chances of packet loss and
delay and 2) provision enough extra buffer space in the DetNet
transit node following the DetNet-unaware network nodes to absorb
the induced latency variations.
</t> </t>
</section> </section>
</section> </section>
<section anchor="DetNetFlows" title="DetNet flows"> <section anchor="DetNetFlows" numbered="true" toc="default">
<name>DetNet Flows</name>
<section anchor="DetNetFlowsTypes" title="DetNet flow types"> <section anchor="DetNetFlowsTypes" numbered="true" toc="default">
<t> <name>DetNet Flow Types</name>
A DetNet flow can have different formats while its p <t>
ackets are A DetNet flow can have different formats while its packets are forwarded
forwarded between the peer end systems depending between the peer end systems depending on the type of the end
on the type systems. Corresponding to the end system types, the following possible
of the end systems. Corresponding to the end sys types/formats of a DetNet flow are distinguished in this document. The
tem types, different flow types have different requirements to DetNet nodes.
the following possible types / formats of a DetN </t>
et flow are <ul spacing="normal">
distinguished in this document. The different fl <li> App-flow: The payload (data) carried over a DetNet
ow types have flow between DetNet-unaware end systems. An App-flow does
different requirements to DetNet nodes. not contain any DetNet-related attributes and does not
<list style="symbols"> imply any specific requirement on DetNet nodes.</li>
<t> App-flow: the payload (data) carried over a DetNet flow <li> DetNet-f-flow: The specific format of a DetNet flow. It
between DetNet unaware end systems. An only requires the resource allocation features provided by
app-flow does not the DetNet forwarding sub-layer. </li>
contain any DetNet related attributes a <li> DetNet-s-flow: The specific format of a DetNet flow. It
nd does not imply any only requires the service protection feature ensured by
specific requirement on DetNet nodes.</ the DetNet service sub-layer. </li>
t> <li> DetNet-sf-flow: The specific format of a DetNet
<t> DetNet-f-flow: specific format of a DetNet flow. It only flow. It requires both the DetNet service sub-layer and
requires the resource allocation features provided by the DetNet forwarding sub the DetNet forwarding sub-layer functions during
-layer. </t> forwarding. </li>
<t> DetNet-s-flow: specific format of a DetNet flow. It onl </ul>
y requires the service protection feature ensured by the DetNet service sub-laye </section>
r. </t> <section anchor="FlowLimits" numbered="true" toc="default">
<t> DetNet-sf-flow: specific format of a DetNet flow. It re <name>Source Transmission Behavior</name>
quires both DetNet service <t>
sub-layer and DetNet forwarding sub-layer functions du
ring forwarding. </t>
</list>
</t>
</section>
<section anchor="FlowLimits" title="Source transmission behavior">
<t>
For the purposes of resource allocation, For the purposes of resource allocation,
DetNet flows can be synchronous or asynchronous. DetNet flows can be synchronous or asynchronous.
In synchronous DetNet flows, at least the DetNet nodes (and po ssibly In synchronous DetNet flows, at least the DetNet nodes (and po ssibly
the end systems) are closely time the end systems) are closely time
synchronized, typically to better than 1 microsecond. By tran smitting synchronized, typically to better than 1 microsecond. By tran smitting
packets from different DetNet flows or classes of DetNet flows at different times, packets from different DetNet flows or classes of DetNet flows at different times,
using repeating schedules synchronized among the DetNet nodes, resources using repeating schedules synchronized among the DetNet nodes, resources
such as buffers and link bandwidth can be shared over the time domain such as buffers and link bandwidth can be shared over the time domain
among different DetNet flows. There is a tradeoff among techn iques for among different DetNet flows. There is a trade-off among tech niques for
synchronous DetNet flows between the burden of fine-grained sc heduling and the synchronous DetNet flows between the burden of fine-grained sc heduling and the
benefit of reducing the required resources, especially buffer space. benefit of reducing the required resources, especially buffer space.
</t><t> </t>
<t>
In contrast, asynchronous DetNet flows are not coordinated wit h a fine-grained In contrast, asynchronous DetNet flows are not coordinated wit h a fine-grained
schedule, so relay and end systems must assume worst-case inte rference schedule, so relay and end systems must assume worst-case inte rference
among DetNet flows contending for buffer resources. among DetNet flows contending for buffer resources.
Asynchronous DetNet flows are characterized by: Asynchronous DetNet flows are characterized by:
<list style="symbols"> </t>
<t> <ul spacing="normal">
<li>
A maximum packet size; A maximum packet size;
</t><t> </li>
<li>
An observation interval; and An observation interval; and
</t><t> </li>
<li>
A maximum number of transmissions during that observat ion interval. A maximum number of transmissions during that observat ion interval.
</t> </li>
</list> </ul>
</t><t> <t> These parameters, together with knowledge of the
These parameters, together with knowledge of the protocol stac protocol stack used (and thus the size of the various headers
k used (and thus the added to a packet), provide the bandwidth that is needed for the
size of the various headers added to a packet), provide the ba DetNet flow. </t>
ndwidth that is needed for the DetNet flow. <t> The source is required not to exceed these
</t><t> limits in order to obtain DetNet service. If the source
The source is required not to exceed these limits in order to transmits less data than this limit allows, then the unused resour
obtain DetNet service. If the source ce,
transmits less data than this limit allows, the unused resourc such as link bandwidth, can be made available by the DetNet
e such as link system to non-DetNet packets as long as all guarantees are
bandwidth can be made available by the DetNet system to non-De fulfilled. However, making those resources available to DetNet
tNet packets as long as all guarantees are fulfilled. However, packets in other DetNet flows would serve no purpose. Those
making those resources available to DetNet packets in other De other DetNet flows have their own dedicated resources, on the
tNet flows would serve assumption that all DetNet flows can use all of their resources
no purpose. Those other DetNet flows have their own dedicated over a long period of time.
resources, on the
assumption that all DetNet flows can use all of their resource
s over a long
period of time.
</t><t> </t>
There is no expectation in DetNet for App-flows to be responsi <t>
ve to congestion control <xref target="RFC2914"/> or explicit congestion notific There is no expectation in DetNet for App-flows to be responsive to
ation <xref target="RFC3168"/>. The assumption congestion control <xref target="RFC2914" format="default"/> or explicit co
is that a DetNet flow, to be useful, must be delivered in its ngestion
entirety. That notification <xref target="RFC3168" format="default"/>. The assumption is t
is, while any useful application is written to expect a certai hat a DetNet
n number of lost flow, to be useful, must be delivered in its entirety. That is, while
packets, the real-time applications of interest to DetNet dema any useful application is written to expect a certain number of lost
nd that the loss of packets, the real-time applications of interest to DetNet demand that the
data due to the network is a rare event. loss of data due to the network is a rare event. </t>
</t><t> <t> Although DetNet strives to minimize the changes required of an
Although DetNet strives to minimize the changes required of an application to allow it to shift from a special-purpose digital network to an
application to Internet Protocol network, one fundamental shift in the behavior of network
allow it to shift from a special-purpose digital network to an applications is impossible to avoid: the reservation of resources before the
Internet Protocol application starts. In the first place, a network cannot deliver finite
network, one fundamental shift in latency and practically zero packet loss to an arbitrarily high offered load.
the behavior of network applications is impossible to avoid: t Secondly, achieving practically zero packet loss for DetNet flows means that
he reservation DetNet nodes have to dedicate buffer resources to specific DetNet flows or to
of resources before the application starts. classes of DetNet flows. The requirements of each reservation have to be
In the first place, a network cannot deliver finite latency an translated into the parameters that control each DetNet system's queuing,
d practically zero shaping, and scheduling functions, and they have to be delivered to the DetNet
packet loss to an arbitrarily high offered load. Secondly, ac nodes and end
hieving systems. </t>
practically zero packet loss for DetNet flows <t> All nodes in a DetNet domain are expected to support the data beha
means that DetNet nodes have to dedicate buffer resources to s vior
pecific required to deliver a particular DetNet service. If a node itself is not
DetNet flows or to classes of DetNet flows. The requirements DetNet service aware, the DetNet nodes that are adjacent to them must ensure
of each reservation have to be that the node that is non-DetNet aware is provisioned to appropriately support
translated into the parameters that control each DetNet system the DetNet service. For example, a TSN node (as defined by IEEE 802.1) may be us
's ed to
queuing, shaping, and scheduling functions and delivered to th interconnect DetNet-aware nodes, and these DetNet nodes can map DetNet flows
e DetNet nodes and end systems. to 802.1 TSN flows. As another example, an MPLS-TE or MPLS-TP (Transport Profile
</t><t> ) domain may be used to
All nodes in a DetNet domain are expected to suppor interconnect DetNet-aware nodes, and these DetNet nodes can map DetNet flows
t the data to TE LSPs, which can provide the QoS requirements of the DetNet service.
behavior required to deliver a particular DetNe </t>
t service. If </section>
a node itself is not DetNet service aware, the <section anchor="Incomplete" numbered="true" toc="default">
DetNet nodes <name>Incomplete Networks</name>
that are adjacent to such non-DetNet aware node <t>
s must ensure The presence in the network of intermediate nodes or subnets
that the non-DetNet aware node is provisioned t that are not fully capable of offering DetNet services
o appropriately complicates the ability of the intermediate nodes and/or
support the DetNet service. For example, an IEE controller to allocate resources, as extra buffering must be
E 802.1 TSN allocated at points downstream from the non-DetNet
node may be used to interconnect DetNet aware n intermediate node for a DetNet flow. This extra buffering
odes, and these may increase latency and/or jitter.
DetNet nodes can map DetNet flows to 802.1 TSN </t>
flows. Another </section>
example, an MPLS-TE or TP domain may be used to
interconnect
DetNet aware nodes, and these DetNet nodes can
map DetNet
flows to TE LSPs which can provide the QoS requ
irements of the
DetNet service.
</t>
</section>
<section anchor="Incomplete" title="Incomplete Networks">
<t>
The presence in the network of intermediate nodes or subnets t
hat are not fully capable of offering
DetNet services complicates the ability of the intermediate no
des and/or controller to
allocate resources,
as extra buffering must be allocated at points
downstream from the non-DetNet intermediate node
for a DetNet flow. This extra buffering may inc
rease latency and/or jitter.
</t>
</section>
</section> </section>
<section anchor="te" title="Traffic Engineering for DetNet"> <section anchor="te" numbered="true" toc="default">
<name>Traffic Engineering for DetNet</name>
<t> <t>
<xref target="TEAS">Traffic Engineering Architecture and Signaling (TEA <xref target="TEAS" format="default">Traffic Engineering Architecture a
S) nd Signaling (TEAS)
</xref> defines traffic-engineering architectures for generic applicabi </xref> defines traffic-engineering architectures for generic applicab
lity ility
across packet and non-packet networks. across packet and nonpacket networks.
From a TEAS perspective, Traffic Engineering (TE) refers to techniques From a TEAS perspective, Traffic Engineering (TE) refers to techniques
that enable operators to control how specific traffic flows are treated that enable operators to control how specific traffic flows are treated
within their networks. within their networks.
</t> </t>
<t> <t>
Because if its very nature of establishing explicit optimized paths, Because of its very nature of establishing explicit optimized paths,
Deterministic Networking can be seen as a new, specialized branch of DetNet can be seen as a new, specialized branch of
Traffic Engineering, and inherits its architecture with a separation TE, and it inherits its architecture with a separation
into planes. into planes.
</t><t> </t>
The Deterministic Networking architecture is thus composed <t>
of three planes, a (User) Application Plane, a Controller Plane, and a The DetNet architecture is thus composed
Network Plane, which echoes that of Figure 1 of of three planes: a (User) Application Plane, a Controller Plane, and a
<xref target="RFC7426"> Software-Defined Networking (SDN): Network Plane. This echoes the composition of Figure 1 of
Layers and Architecture Terminology</xref>, and the Controllers <xref target="RFC7426" format="default">"Software-Defined Networking (S
identified in <xref target="RFC8453"/> and <xref target="RFC7149 DN):
"/>. Layers and Architecture Terminology"</xref> and the controllers
</t> identified in <xref target="RFC8453" format="default"/> and <xre
f target="RFC7149" format="default"/>.
<section anchor="appplane" title="The Application Plane"> </t>
<t> <section anchor="appplane" numbered="true" toc="default">
Per <xref target="RFC7426"/>, <name>The Application Plane</name>
<t>
Per <xref target="RFC7426" format="default"/>,
the Application Plane includes both applications and services. In parti cular, the Application Plane includes both applications and services. In parti cular,
the Application Plane incorporates the User Agent, a specialized applic ation the Application Plane incorporates the User Agent, a specialized applic ation
that interacts with the end user / operator and performs requests for that interacts with the end user and operator and performs requests for
Deterministic Networking services via an abstract Flow Management Entit DetNet services via an abstract Flow Management Entity
y, (FME), which may or may not be collocated with (one of) the end systems
(FME) which may or may not be collocated with (one of) the end systems. .
</t> </t>
<t>At the Application Plane, a management interface enables the n <t>At the Application Plane, a management interface enables
egotiation of flows between end the negotiation of flows between end systems. An abstraction
systems. An abstraction of the flow called a Traffic Spec of the flow called a Traffic Specification (TSpec) provides
ification (TSpec) provides the the representation. This abstraction is used to place a
representation. This abstraction is used to place a reser reservation over the (Northbound) Service Interface and within
vation over the (Northbound) Service the Application Plane. It is associated with an abstraction
Interface and within the Application plane. of location, such as IP addresses and DNS names, to identify
It is associated with an abstraction of location, such as IP addresses the end systems and possibly specify DetNet nodes.
and DNS </t>
names, to identify the end systems and possibly specify D </section>
etNet nodes. <section anchor="ctrlplane" numbered="true" toc="default">
</t> <name>The Controller Plane</name>
</section> <t>
<section anchor="ctrlplane" title="The Controller Plane"> The Controller Plane corresponds to the aggregation of the Control
<t> and Management Planes in <xref target="RFC7426" format="default"/>, tho
The Controller Plane corresponds to the aggregation of the Control and ugh Common
Management Planes in <xref target="RFC7426"/>, though Control and Measurement Plane (CCAMP) (as defined by the CCAMP
Common Control and Measurement Plane (CCAMP) <xref target="CCAMP"/> Working Group <xref target="CCAMP" format="default"/>) makes an
makes an additional distinction between management and measurement. additional distinction between management and measurement. When the
When the logical separation of the Control, Measurement and other logical separation of the Control, Measurement, and other Management
Management entities is not relevant, the term Controller Plane is used entities is not relevant, the term "Controller Plane" is used for
for simplicity to represent them all, and the term Controller Plane Fun simplicity to represent them all, and the term "Controller Plane
ction (CPF) refers to Function (CPF)" refers to any device operating in that plane, whether
any device operating in that plane, whether is it a Path Computation it is a Path Computation Element (PCE) <xref target="RFC4655" format="d
Element (PCE) <xref target="RFC4655"/>, or a Network Management entity efault"/>, a
(NME), or a distributed control plane. Network Management Entity (NME), or a distributed control protocol.
The CPF is a core
element of a controller, in charge of computing Deterministic paths The CPF is a core element of a controller, in charge of computing
to be applied in the Network Plane. deterministic paths to be applied in the Network Plane.
</t> </t>
<t> <t>
A (Northbound) Service Interface enables applications in the Applicatio n A (Northbound) Service Interface enables applications in the Applicatio n
Plane to communicate with the entities in the Controller Plane as Plane to communicate with the entities in the Controller Plane as
illustrated in <xref target="NorthSouth"/>. illustrated in <xref target="NorthSouth" format="default"/>.
</t> </t>
<t> <t>
One or more CPF(s) collaborate to implement the requests from the FME One or more CPFs collaborate to implement the requests from the FME
as Per-Flow Per-Hop Behaviors installed in the DetNet nod as per-flow, per-hop behaviors installed in the DetNet nodes for each
es for individual flow. The CPFs place each flow along a deterministic
each individual flow. The CPFs arrangement of DetNet nodes so as to respect per-flow constraints such
place each flow along a deterministic sequence of DetNet nodes so as as security and latency, and to optimize the overall result for metrics
to respect per-flow constraints such as security and such as an abstract aggregated cost. The deterministic arrangement can
latency, and optimize the overall result for metrics such typically be more complex than a direct arrangement and include
as an redundant paths with one or more packet replication and elimination
abstract aggregated cost. The deterministic sequence can typically be points. Scaling to larger networks is discussed in <xref target="Scalin
more complex than a direct sequence and include redundant paths, with g" format="default"/>.
one or more packet replication and elimination points. Scaling to large </t>
r </section>
networks is discussed in <xref target="Scaling"/>. <section anchor="netplane" numbered="true" toc="default">
</t> <name>The Network Plane</name>
</section> <t>
<section anchor="netplane" title="The Network Plane"><t>
The Network Plane represents the network devices and protocols as a The Network Plane represents the network devices and protocols as a whole,
whole, regardless of the Layer at which the network devices operate. regardless of the layer at which the network devices operate. It includes the
It includes Forwarding Plane (data plane), Application, and Data Plane and Operational Plane (e.g., OAM) aspects.
Operational Plane (e.g., OAM) aspects. </t>
</t> <t>The Network Plane comprises the Network Interface Cards (NICs) in t
<t> he
The network Plane comprises the Network Interface Cards (NIC) in the end systems, which are typically IP hosts, and DetNet nodes, which
end systems, which are typically IP hosts, are typically IP routers and MPLS switches.
and DetNet nodes, which are typically IP routers and MPLS switches. </t>
Network-to-Network Interfaces such as used for Traffic Engineering <t>
path reservation in <xref target="RFC5921"/>,
as well as User-to-Network Interfaces (UNI) such as provided by
the Local Management Interface (LMI) between network and end systems,
are both part of the Network Plane, both in the control plane and
the data plane.
</t>
<t>
A Southbound (Network) Interface enables the entities in the Controller A Southbound (Network) Interface enables the entities in the Controller
Plane to communicate with devices in the Network Plane as illustrated Plane to communicate with devices in the Network Plane as illustrated
in <xref target="NorthSouth"/>. This interface leverages and ext ends in <xref target="NorthSouth" format="default"/>. This interface leverages and extends
TEAS to describe the physical topology and resources in the Netw ork TEAS to describe the physical topology and resources in the Netw ork
Plane. Plane.
</t> </t>
<figure align="center" anchor="NorthSouth" <figure anchor="NorthSouth">
title="Northbound and Southbound interfaces"> <name>Northbound and Southbound Interfaces</name>
<artwork align="left"><![CDATA[ <artwork align="left" name="" type="" alt=""><![CDATA[
End End End End
System System System System
-+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
CPF CPF CPF CPF CPF CPF CPF CPF
-+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
DetNet DetNet DetNet DetNet DetNet DetNet DetNet DetNet
Node Node Node Node Node Node Node Node
NIC NIC NIC NIC
DetNet DetNet DetNet DetNet DetNet DetNet DetNet DetNet
Node Node Node Node Node Node Node Node
]]></artwork> ]]></artwork>
</figure> </figure>
<t> <t>
The DetNet nodes (and possibly the end systems NIC) expos The DetNet nodes (and possibly the end systems' NICs) expose their capabilities
e their capabilities and physical and physical resources to the controller (the CPF) and update the CPFs with
resources to the controller (the CPF), and update the CPF their dynamic perception of the topology across the Southbound Interface. In
s with their dynamic perception of the return, the CPFs set the per-flow paths up, providing a Flow Characterization
topology, across the Southbound Interface. In return, the that is more tightly coupled to the DetNet node operation than a TSpec.
CPFs set the per-flow </t>
paths up, providing a Flow Characterization that is more <t> At the Network Plane, DetNet nodes may exchange information regard
tightly coupled to the DetNet node ing
Operation than a TSpec. the state of the paths, between adjacent DetNet nodes and possibly with the
</t><t> end systems, and forward packets within constraints associated to each flow,
At the Network plane, DetNet nodes may exchange informati or, when unable to do so, perform a last-resort operation such as drop or
on regarding the state of the paths, declassify. </t>
between adjacent DetNet nodes and possibly with the end s <t> This document focuses on the Southbound interface and the
ystems, and forward packets within operation of the Network Plane.
constraints associated to each flow, or, when unable to d </t>
o so, perform a last resort </section>
operation such as drop or declassify.
</t><t>
This document focuses on the Southbound interface and the
operation of the Network Plane.
</t>
</section> </section>
</section> <section anchor="QueuingModels" numbered="true" toc="default">
<section anchor="QueuingModels" title="Queuing, Shaping, Scheduling, an <name>Queuing, Shaping, Scheduling, and Preemption</name>
d Preemption"> <t> DetNet achieves bounded delivery latency by reserving bandwidth and
<t> buffer
DetNet achieves bounded delivery latency resources at each DetNet node along the path of the DetNet flow. The
by reserving bandwidth and buffer resources at each DetNet node reservation itself is not sufficient, however. Implementors and users of a
along number of proprietary and standard real-time networks have found that
the path of the DetNet flow. standards for specific data-plane techniques are required to enable these
The reservation itself is not sufficient, however. Implementor assurances to be made in a multivendor network. The fundamental reason is
s and users of a that latency variation in one DetNet system results in the need for extra
number of buffer space in the next-hop DetNet system(s), which in turn increases the
proprietary and standard real-time networks have found that sta worst-case per-hop latency. </t>
ndards for <t> Standard queuing and transmission-selection algorithms allow TE
specific data plane techniques are required to enable these ass (<xref target="te" format="default"/>) to compute the latency contribution of ea
urances to be ch
made in a multi-vendor DetNet node to the end-to-end latency, to compute the amount of buffer space
network. The fundamental reason is that latency variation in o required in each DetNet node for each incremental DetNet flow, and most
ne DetNet system results importantly, to translate from a flow specification to a set of values for the
in the need for extra buffer space in the next-hop DetNet syste managed objects that control each relay or end system. For example, the IEEE
m(s), which in turn, 802.1 WG has specified (and is specifying) a set of queuing, shaping, and
increases the worst-case per-hop latency. scheduling algorithms that enable each DetNet node, and/or a central
</t><t> controller, to compute these values. These algorithms include:
Standard queuing and transmission selection algorithms allow tr </t>
affic engineering <ul spacing="normal">
<xref target="te"/> to compute the latency contribution <li>
of each DetNet node to the end-to-end latency, to compute the a A credit-based shaper <xref target="IEEE802.1Qav" format="default"/> (incorpor
mount of buffer space ated to <xref target="IEEE802.1Q" format="default"/>). </li>
required in each DetNet node for each incremental DetNet flow, <li> Time-gated queues governed by a rotating time schedule based on
and most importantly, to synchronized time <xref target="IEEE802.1Qbv" format="default"/> (incorporated t
translate from a flow specification to a set of values for the o <xref target="IEEE802.1Q" format="default"/>).
managed objects that </li>
control each relay or end system. For example, the IEEE 802.1 <li> Synchronized double (or triple) buffers driven by synchronized ti
WG has specified (and is me ticks.
specifying) a set of queuing, shaping, and scheduling algorithm <xref target="IEEE802.1Qch" format="default"/> (incorporated to <xref target="IE
s EE802.1Q" format="default"/>). </li>
that enable each DetNet node, and/or a central controller, to <li> Preemption of an Ethernet packet in transmission by a packet with
compute these values. These algorithms include: a more
<list style="symbols"> stringent latency requirement, followed by the resumption of the preempted
<t> packet <xref target="IEEE802.1Qbu" format="default"/> (incorporated to <xref tar
A credit-based shaper <xref target="IEEE802.1Qav"/> (su get="IEEE802.1Q" format="default"/>) <xref target="IEEE802.3br" format="default"
perseded by <xref target="IEEE802.1Q-2018"/>). /> (incorporated to <xref target="IEEE802.3" format="default"/>).
</t><t> </li>
Time-gated queues governed by a rotating time schedule </ul>
based on synchronized time <t> While these techniques are currently embedded in
<xref target="IEEE802.1Qbv"/> (superseded by <xref targ Ethernet <xref target="IEEE802.3" format="default"/> and bridging
et="IEEE802.1Q-2018"/>). standards, we can note that they are all, except perhaps for
</t><t> packet preemption, equally applicable to media other than
Synchronized double (or triple) buffers driven by synch Ethernet and to routers as well as bridges. Other media may
ronized time ticks. have their own methods (see, e.g., <xref target="I-D.ietf-6tisch-
<xref target="IEEE802.1Qch"/> (superseded by <xref targ architecture" format="default"/> and <xref target="RFC7554" format="default"/>).
et="IEEE802.1Q-2018"/>). Further techniques are defined by the IETF (e.g., <xref target="R
</t><t> FC8289" format="default"/> and <xref target="RFC8033" format="default"/>). DetN
Pre-emption of an Ethernet packet in transmission by a et may
packet with a more stringent include such definitions in the future or may define how
latency requirement, followed by the resumption of the these techniques can be used by DetNet nodes.
preempted packet </t>
<xref target="IEEE802.1Qbu"/> (superseded by <xref targ </section>
et="IEEE802.1Q-2018"/>), <section anchor="ServInst" numbered="true" toc="default">
<xref target="IEEE802.3br"/> (superseded by <xref targe <name>Service Instance</name>
t="IEEE802.3-2018"/>). <t> A service instance represents all the functions required
</t> on a DetNet node to allow the end-to-end service between the
</list> UNIs.
</t><t> </t>
While these techniques are currently embedded in Ethernet <xre <t> The DetNet network general reference model is shown in <xref target=
f target="IEEE802.3-2018"/> and bridging standards, "fig_DetNetrefmodel" format="default"/> for a DetNet service scenario (i.e., bet
we can note that they are all, except perhaps for packet preemp ween two
tion, equally applicable DetNet-UNIs). In this figure, end systems ("A" and "B") are connected directly
to other media than Ethernet, and to routers as well as bridges to the edge nodes of an IP/MPLS network ("PE1" and "PE2"). End systems
. Other media may have its participating in DetNet communication may require connectivity before setting
own methods, see, e.g., <xref target="I-D.ietf-6tisch-architect up an App-flow that requires the DetNet service. Such a connectivity-related
ure"/>, <xref target="RFC7554"/>. service instance and the one dedicated for DetNet service share the same
Further techniques are defined by the IETF, e.g., <xref target= access. Packets belonging to a DetNet flow are selected by a filter configured
"RFC8289"/> and <xref target="RFC8033"/>. on the access ("F1" and "F2"). As a result, data-flow-specific access
DetNet may include such ("access-A + F1" and "access-B + F2") is terminated in the flow-specific
definitions in the future, or may define how these techniques c service instance ("SI-1" and "SI-2"). A tunnel is used to provide connectivity
an be used by DetNet nodes. between the service instances.
</t> </t>
</section> <t> The tunnel is exclusively used for the packets of the DetNet flow be
tween "SI-1" and "SI-2". The service instances are configured to implement DetNe
<section anchor="ServInst" title="Service instance"> t functions and a flow-specific DetNet forwarding. The service instance and the
<t> A Service instance represents all the functions required on a tunnel may or may not be shared by multiple DetNet flows. Sharing the service in
DetNet node to allow the end-to-end service between the UNIs. stance by multiple DetNet flows requires properly populated forwarding tables of
</t> the service instance.
</t>
<t> The DetNet network general reference model is shown in <xref <figure anchor="fig_DetNetrefmodel">
target="fig_DetNetrefmodel"/> for a DetNet service scenario (i.e., between two D <name>DetNet Network General Reference Model</name>
etNet-UNIs). In this figure, end systems ("A" and "B") are connected directly to <artwork name="" type="" align="left" alt=""><![CDATA[
the edge nodes of an IP/MPLS network ("PE1" and "PE2"). End systems participati
ng in DetNet communication may require connectivity before setting up an App-flo
w that requires the DetNet service. Such a connectivity related service instance
and the one dedicated for DetNet service share the same access. Packets belongi
ng to a DetNet flow are selected by a filter configured on the access ("F1" and
"F2"). As a result, data flow specific access ("access-A + F1" and "access-B + F
2") are terminated in the flow specific service instance ("SI-1" and "SI-2"). A
tunnel is used to provide connectivity between the service instances.
</t>
<t> The tunnel is exclusively used for the packets of the DetNet
flow between "SI-1" and "SI-2". The service instances are configured to implemen
t DetNet functions and a flow specific DetNet forwarding. The service instance a
nd the tunnel may or may not be shared by multiple DetNet flows. Sharing the ser
vice instance by multiple DetNet flows requires properly populated forwarding ta
bles of the service instance.
</t>
<figure anchor="fig_DetNetrefmodel" align="center"
title="DetNet network general reference model">
<artwork><![CDATA[
access-A access-B access-A access-B
<-----> <-------- tunnel ----------> <-----> <-----> <-------- tunnel ----------> <----->
+---------+ ___ _ +---------+ +---------+ ___ _ +---------+
End system | +----+ | / \/ \_ | +----+ | End system End system | +----+ | / \/ \_ | +----+ | End system
"A" -------F1+ | | / \ | | +F2----- "B" "A" -------F1+ | | / \ | | +F2----- "B"
| | +========+ IP/MPLS +=======+ | | | | +========+ IP/MPLS +=======+ | |
| |SI-1| | \__ Net._/ | |SI-2| | | |SI-1| | \__ Net._/ | |SI-2| |
| +----+ | \____/ | +----+ | | +----+ | \____/ | +----+ |
|PE1 | | PE2| |PE1 | | PE2|
+---------+ +---------+ +---------+ +---------+]]></artwork>
</figure>
]]></artwork> <t> The tunnel between the service instances may have some special
</figure> characteristics. For example, in case of a DetNet L3 service, there are
differences in the usage of the PW for DetNet traffic compared to the network
<t> The tunnel between the service instances may have some specia model described in <xref target="RFC6658" format="default"/>. In the DetNet scen
l characteristics. For example, in case of a DetNet L3 service, there are differ ario, the PW is
ences in the usage of the PW for DetNet traffic compared to the network model de likely to be used exclusively by the DetNet flow, whereas <xref target="RFC6658"
scribed in <xref target="RFC6658"/>. In the DetNet scenario, the PW is likely to format="default"/> states:
be used exclusively by the DetNet flow, whereas <xref target="RFC6658"/> states </t>
: "The packet PW appears as a single point-to-point link to the client layer. N <blockquote>
etwork-layer adjacency formation and maintenance between the client equipment wi The packet PW appears as a single point-to-point link to the client
ll follow the normal practice needed to support the required relationship in the layer. Network-layer adjacency formation and maintenance between the
client layer ... This packet PseudoWire is used to transport all of the require client equipments will follow the normal practice needed to support
d Layer-2 and Layer-3 protocols between LSR1 and LSR2". Further details are netw the required relationship in the client layer.
ork technology specific and can be found in <xref target="I-D.ietf-detnet-dp-sol &br;...
-mpls"/> and <xref target="I-D.ietf-detnet-dp-sol-ip"/>. &br;
</t> This packet pseudowire is used to transport all of the required layer 2 and
layer 3 protocols between LSR1 and LSR2.
</section> </blockquote>
<section anchor="FlowIDatTechBord" title="Flow identification at techno
logy borders">
<t>This section discusses what needs to be done at technology border
s including
Ethernet as one of the technologies. Flow identification for MPL
S and IP data
planes are described in <xref target="I-D.ietf-detnet-dp-sol-mpl
s"/> and
<xref target="I-D.ietf-detnet-dp-sol-ip"/>, respectively.
</t>
<section anchor="Relayering" title="Exporting flow identification"> <t>
<t> Further details are network technology specific and can be found in <xref target
A DetNet node may need to map specific flows to lower layer flo ="I-D.ietf-detnet-data-plane-framework" format="default"/>.
ws (or Streams) in order to provide specific </t>
queuing and shaping services for specific flows. For example: </section>
<list style="symbols"> <section anchor="FlowIDatTechBord" numbered="true" toc="default">
<t> <name>Flow Identification at Technology Borders</name>
<t>This section discusses what needs to be done at technology
borders including Ethernet as one of the technologies. Flow
identification for MPLS and IP Data Planes are described in <xref ta
rget="I-D.ietf-detnet-mpls" format="default"/> and <xref target="I-D.ietf-detnet
-ip" format="default"/>,
respectively.
</t>
<section anchor="Relayering" numbered="true" toc="default">
<name>Exporting Flow Identification</name>
<t>
A DetNet node may need to map specific flows to lower-layer
flows (or Streams) in order to provide specific queuing and
shaping services for specific flows. For example:
</t>
<ul spacing="normal">
<li>
A non-IP, strictly L2 source end system X may be sending multiple flows A non-IP, strictly L2 source end system X may be sending multiple flows
to the same L2 destination end system Y. Those f lows may include to the same L2 destination end system Y. Those f lows may include
DetNet flows with different QoS requirements, and may include non-DetNet DetNet flows with different QoS requirements and may include non-DetNet
flows. flows.
</t><t> </li>
<li>
A router may be sending any number of flows to an other router. A router may be sending any number of flows to an other router.
Again, those flows may include Again, those flows may include
DetNet flows with different QoS requirements, and may include non-DetNet DetNet flows with different QoS requirements and may include non-DetNet
flows. flows.
</t><t> </li>
<li>
Two routers may be separated by bridges. For the se bridges to perform Two routers may be separated by bridges. For the se bridges to perform
any required per-flow queuing and shaping, they m ust be able to identify any required per-flow queuing and shaping, they m ust be able to identify
the individual flows. the individual flows.
</t><t> </li>
<li>
A Label Edge Router (LER) may have a Label Switch ed A Label Edge Router (LER) may have a Label Switch ed
Path (LSP) set up for handling traffic Path (LSP) set up for handling traffic
destined for a particular IP address carrying onl y non-DetNet flows. If destined for a particular IP address carrying onl y non-DetNet flows. If
a DetNet flow to that same address is requested, a separate LSP may be a DetNet flow to that same address is requested, a separate LSP may be
needed, in order that all of the Label Switch Rou needed in order for all of the Label Switch Route
ters (LSRs) along the rs (LSRs) along the
path to the destination give that flow special qu path to the destination to give that flow special
euing and shaping. queuing and shaping.
</t> </li>
</list> </ul>
</t><t> <t> The need for a lower-layer node to be aware of
The need for a lower-layer node to be aware of individual highe individual higher-layer flows is not unique to DetNet. But,
r-layer given the endless complexity of layering and relayering over
flows is not unique to DetNet. But, given the endless complexi tunnels that is available to network designers, DetNet needs
ty of layering to provide a model for flow identification that is better than
and relayering over tunnels that is available to network design packet inspection. That is not to say that packet inspection
ers, DetNet to Layer 4 or Layer 5 addresses will not be used or the
needs to provide a model for flow identification that is capability standardized; however, there are alternatives. </t>
better than packet inspection. That is not to say that packet <t>
inspection A DetNet relay node can connect DetNet flows on different
to Layer-4 or Layer-5 addresses paths using different flow identification methods. For
will not be used, or the capability standardized; but, there ar example:
e alternatives. </t>
</t><t> <ul spacing="normal">
A DetNet relay node can connect DetNet flows on different paths <li>
using different
flow identification methods. For example:
<list style="symbols">
<t>
A single unicast DetNet flow passing from router A thro ugh a bridged network A single unicast DetNet flow passing from router A thro ugh a bridged network
to router B may be assigned a TSN Stream identifier tha t is to router B may be assigned a TSN Stream identifier tha t is
unique within that bridged network. The bridges can th en identify the unique within that bridged network. The bridges can th en identify the
flow without accessing higher-layer headers. Of course , the receiving router flow without accessing higher-layer headers. Of course , the receiving router
must recognize and accept that TSN Stream. must recognize and accept that TSN Stream.
</t><t> </li>
<li>
A DetNet flow passing from LSR A to LSR B may be assign ed a different A DetNet flow passing from LSR A to LSR B may be assign ed a different
label than that used for other flows to the same IP des tination. label than that used for other flows to the same IP des tination.
</t> </li>
</list> </ul>
</t><t> <t>
In any of the above cases, it is possible that an existing DetN et flow can In any of the above cases, it is possible that an existing DetN et flow can
be an aggregate carrying multiple other DetNet flows. (Not to be an aggregate carrying multiple other DetNet flows (not to be
be confused confused
with DetNet compound vs. member flows.) Of course, this requir with DetNet compound vs. member flows). Of course, this requir
es that the es that the
aggregate DetNet flow be provisioned properly to carry the aggr egated flows. aggregate DetNet flow be provisioned properly to carry the aggr egated flows.
</t><t> </t>
<t>
Thus, rather than packet inspection, there is the option to exp ort Thus, rather than packet inspection, there is the option to exp ort
higher-layer information to the lower layer. The requirement t o support higher-layer information to the lower layer. The requirement t o support
one or the other method for flow identification (or both) is a one or the other method for flow identification (or both) is a
complexity that is part of DetNet control models. complexity that is part of DetNet control models.
</t> </t>
</section> </section>
<section anchor="FlowAttrMapping" numbered="true" toc="default">
<section anchor="FlowAttrMapping" title="Flow attribute mapping between <name>Flow Attribute Mapping between Layers</name>
layers"> <t>Forwarding of packets of DetNet flows over multiple
<t>Forwarding of packets of DetNet flows over multiple technology technology domains may require that lower layers are aware of
domains may require that lower layers are aware of specific flows of higher lay specific flows of higher layers. Such an "exporting of flow
ers. Such an "exporting of flow identification" is needed each time when the for identification" is needed each time when the forwarding
warding paradigm is changed on the forwarding path (e.g., two LSRs are interconn paradigm is changed on the forwarding path (e.g., two LSRs are
ected by a L2 bridged domain, etc.). The three representative forwarding methods interconnected by an L2 bridged domain, etc.). The three
considered for deterministic networking are: representative forwarding methods considered for DetNet
<list style="symbols"> are:
<t>IP routing</t> </t>
<t>MPLS label switching</t> <ul spacing="normal">
<t>Ethernet bridging</t> <li>IP routing</li>
</list> <li>MPLS label switching</li>
A packet with corresponding Flow-IDs is illustrated in <xref targ <li>Ethernet bridging</li>
et="fig_FlowIDStack"/>, </ul>
<t>
A packet with corresponding Flow-IDs is illustrated in <xref targ
et="fig_FlowIDStack" format="default"/>,
which also indicates where each Flow-ID can be added or removed. which also indicates where each Flow-ID can be added or removed.
</t> </t>
<figure anchor="fig_FlowIDStack">
<figure anchor="fig_FlowIDStack" align="center" <name>Packet with Multiple Flow-IDs</name>
title="Packet with multiple Flow-IDs"> <artwork name="" type="" align="left" alt=""><![CDATA[
<artwork><![CDATA[
add/remove add/remove add/remove add/remove
Eth Flow-ID IP Flow-ID Eth Flow-ID IP Flow-ID
| | | |
v v v v
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| | | | | | | | | |
| Eth | MPLS | IP | Application data | | Eth | MPLS | IP | Application data |
| | | | | | | | | |
+-----------------------------------------------------------+ +-----------------------------------------------------------+
^ ^
| |
add/remove add/remove
MPLS Flow-ID MPLS Flow-ID
]]></artwork> ]]></artwork>
</figure> </figure>
<t> The additional (domain-specific) Flow-ID can be:
<t> The additional (domain specific) Flow-ID can be </t>
<list style="symbols"> <ul spacing="normal">
<t>created by a domain specific function or </t> <li>created by a domain-specific function or </li>
<t>derived from the Flow-ID added to the App-flow. </t> <li>derived from the Flow-ID added to the App-flow. </li>
</list> </ul>
<t>
The Flow-ID must be unique inside a given domain. Note that the F low-ID added to the App-flow is still present in the packet, but some nodes may lack the function to recognize it; that's why the additional Flow-ID is added. The Flow-ID must be unique inside a given domain. Note that the F low-ID added to the App-flow is still present in the packet, but some nodes may lack the function to recognize it; that's why the additional Flow-ID is added.
</t> </t>
</section> </section>
<section anchor="FlowIDMapEx" numbered="true" toc="default">
<section anchor="FlowIDMapEx" title="Flow-ID mapping examples"> <name>Flow-ID Mapping Examples</name>
<t> IP nodes and MPLS nodes are assumed to be configured to push <t> IP nodes and MPLS nodes are assumed to be configured to push such
such an additional (domain specific) Flow-ID when sending traffic to an Ethernet an additional (domain-specific) Flow-ID when sending traffic to an Ethernet swit
switch (as shown in the examples below). ch (as shown in the examples below).
</t> </t>
<t> <xref target="fig_ippushflowid" format="default"/> shows a scenari
<t> <xref target="fig_ippushflowid"/> shows a scenario where an I o where an IP end system ("IP-A") is connected via two Ethernet switches ("ETH-n
P end system ("IP-A") is connected via two Ethernet switches ("ETH-n") to an IP ") to an IP router ("IP-1").
router ("IP-1"). </t>
</t> <figure anchor="fig_ippushflowid">
<name>IP Nodes Interconnected by an Ethernet Domain</name>
<figure anchor="fig_ippushflowid" align="center" <artwork name="" type="" align="left" alt=""><![CDATA[
title="IP nodes interconnected by an Ethernet domain">
<artwork><![CDATA[
IP domain IP domain
<----------------------------------------------- <-----------------------------------------------
+======+ +======+ +======+ +======+
|L3-ID | |L3-ID | |L3-ID | |L3-ID |
+======+ /\ +-----+ +======+ +======+ /\ +-----+ +======+
/ \ Forward as | | / \ Forward as | |
/IP-A\ per ETH-ID |IP-1 | Recognize /IP-A\ per ETH-ID |IP-1 | Recognize
Push ------> +-+----+ | +---+-+ <----- ETH-ID Push ------> +-+----+ | +---+-+ <----- ETH-ID
ETH-ID | +----+-----+ | ETH-ID | +----+-----+ |
skipping to change at line 1592 skipping to change at line 1759
.L3-ID . +-----+ +-----+ |L3-ID | .L3-ID . +-----+ +-----+ |L3-ID |
+======+ +......+ +======+ +======+ +......+ +======+
|ETH-ID| .L3-ID . |ETH-ID| |ETH-ID| .L3-ID . |ETH-ID|
+======+ +======+ +------+ +======+ +======+ +------+
|ETH-ID| |ETH-ID|
+======+ +======+
Ethernet domain Ethernet domain
<----------------> <---------------->
]]></artwork> ]]></artwork>
</figure> </figure>
<t> End system "IP-A" uses the original App-flow-specific ID
<t> End system "IP-A" uses the original App-flow specific ID ("L3 ("L3-ID"), but as it is connected to an Ethernet
-ID"), but as it is connected to an Ethernet domain it has to push an Ethernet-d domain, it has to push an Ethernet-domain-specific
omain specific flow-ID ("ETH-ID") before sending the packet to "ETH-1" node. Eth Flow-ID ("ETH-ID") before sending the packet to
ernet switch "ETH-1" can recognize the data flow based on the "ETH-ID" and it do "ETH-1". Ethernet switch "ETH-1" can recognize the data flow
es forwarding toward "ETH-2". "ETH-2" switches the packet toward the IP router. based on the "ETH-ID", and it does forwarding toward
"IP-1" must be configured to receive the Ethernet Flow-ID specific multicast flo "ETH-2". "ETH-2" switches the packet toward the IP
w, but (as it is an L3 node) it decodes the data flow ID based on the "L3-ID" fi router. "IP-1" must be configured to receive the Ethernet
elds of the received packet. Flow-ID-specific multicast
</t> flow, but (as it is an L3
node) it decodes the data flow ID based on the "L3-ID" fields
<t> <xref target="fig_mplspushflowid"/> shows a scenario where MP of the received packet.
LS domain nodes ("PE-n" and "P-m") are connected via two Ethernet switches ("ETH </t>
-n"). <t> <xref target="fig_mplspushflowid" format="default"/> shows a scena
</t> rio where MPLS domain nodes ("PE-n" and "P-m") are connected via two Ethernet sw
itches ("ETH-n").
<figure anchor="fig_mplspushflowid" align="center" </t>
title="MPLS nodes interconnected by an Ethernet domain"> <figure anchor="fig_mplspushflowid">
<artwork><![CDATA[ <name>MPLS Nodes Interconnected by an Ethernet Domain</name>
<artwork name="" type="" align="left" alt=""><![CDATA[
MPLS domain MPLS domain
<-----------------------------------------------> <----------------------------------------------->
+=======+ +=======+ +=======+ +=======+
|MPLS-ID| |MPLS-ID| |MPLS-ID| |MPLS-ID|
+=======+ +-----+ +-----+ +=======+ +-----+ +=======+ +-----+ +-----+ +=======+ +-----+
| | Forward as | | | | | | Forward as | | | |
|PE-1 | per ETH-ID | P-2 +-----------+ PE-2| |PE-1 | per ETH-ID | P-2 +-----------+ PE-2|
Push -----> +-+---+ | +---+-+ +-----+ Push -----> +-+---+ | +---+-+ +-----+
ETH-ID | +-----+----+ | \ Recognize ETH-ID | +-----+----+ | \ Recognize
skipping to change at line 1627 skipping to change at line 1802
.MPLS-ID. +-----+ +-----+ |MPLS-ID| .MPLS-ID. +-----+ +-----+ |MPLS-ID|
+=======+ +=======+ +=======+ +=======+
|ETH-ID | +.......+ |ETH-ID | |ETH-ID | +.......+ |ETH-ID |
+=======+ .MPLS-ID. +-------+ +=======+ .MPLS-ID. +-------+
+=======+ +=======+
|ETH-ID | |ETH-ID |
+=======+ +=======+
Ethernet domain Ethernet domain
<----------------> <---------------->
]]></artwork> ]]></artwork>
</figure> </figure>
<t> "PE-1" uses the MPLS-specific ID ("MPLS-ID"), but as it
<t> "PE-1" uses the MPLS specific ID ("MPLS-ID"), but as it is co is connected to an Ethernet domain, it has to push an
nnected to an Ethernet domain it has to push an Ethernet-domain specific flow-ID Ethernet-domain-specific Flow-ID
("ETH-ID") before sending the packet to "ETH-1". Ethernet switch "ETH-1" can re ("ETH-ID") before sending the
cognize the data flow based on the "ETH-ID" and it does forwarding toward "ETH-2 packet to "ETH-1". Ethernet switch "ETH-1" can recognize the
". "ETH-2" switches the packet toward the MPLS node ("P-2"). "P-2" must be confi data flow based on the "ETH-ID", and it does forwarding toward
gured to receive the Ethernet Flow-ID specific multicast flow, but (as it is an "ETH-2". "ETH-2" switches the packet toward the MPLS node
MPLS node) it decodes the data flow ID based on the "MPLS-ID" fields of the rece ("P-2"). "P-2" must be configured to receive the Ethernet
ived packet. Flow-ID-specific multicast flow, but (as it is an MPLS node)
</t> it decodes the data flow ID based on the "MPLS-ID" fields of
<t>One can appreciate from the above example that, when the means used f the received packet.
or DetNet flow identification </t>
<t>One can appreciate from the above example that, when the means used
for DetNet flow identification
is altered or exported, the means for encoding the sequence number i nformation must similarly is altered or exported, the means for encoding the sequence number i nformation must similarly
be altered or exported. be altered or exported.
</t> </t>
</section>
</section> </section>
</section> <section anchor="Advertising" numbered="true" toc="default">
<section anchor="Advertising" title="Advertising resources, capabilitie <name>Advertising Resources, Capabilities, and Adjacencies</name>
s and adjacencies"> <t>
<t>
Provisioning of DetNet requires knowledge about: Provisioning of DetNet requires knowledge about:
<list style="symbols"><t> </t>
<ul spacing="normal">
<li>
Details of the DetNet system's capabilities that are requ ired in order to Details of the DetNet system's capabilities that are requ ired in order to
accurately allocate that DetNet system's resources, as we ll as other DetNet systems' accurately allocate that DetNet system's resources, as we ll as other DetNet systems'
resources. This includes, for example, which specific qu euing and resources. This includes, for example, which specific qu euing and
shaping algorithms are implemented (<xref target="Queuing Models"/>), shaping algorithms are implemented (<xref target="Queuing Models" format="default"/>),
the number of buffers dedicated for DetNet allocation, an d the worst-case the number of buffers dedicated for DetNet allocation, an d the worst-case
forwarding delay and misordering. forwarding delay and misordering.
</t><t> </li>
<li>
The actual state of a DetNet node's DetNet resources. The actual state of a DetNet node's DetNet resources.
</t><t> </li>
The identity of the DetNet system's neighbors, and the ch <li>
aracteristics of the The identity of the DetNet system's neighbors and the cha
racteristics of the
link(s) between the DetNet systems, including the latency of link(s) between the DetNet systems, including the latency of
the links (in nanoseconds). the links (in nanoseconds).
</t> </li>
</list> </ul>
</t> </section>
</section> <section anchor="Scaling" numbered="true" toc="default">
<name>Scaling to Larger Networks</name>
<section anchor="Scaling" title="Scaling to larger networks"> <t>
<t>
Reservations for individual DetNet flows require considerable s tate information in Reservations for individual DetNet flows require considerable s tate information in
each DetNet node, especially when adequate fault mitigation each DetNet node, especially when adequate fault mitigation
(<xref target="FaultMitigation"/>) is required. The DetNet dat a plane, in order to (<xref target="FaultMitigation" format="default"/>) is required . The DetNet Data Plane, in order to
support larger numbers of DetNet flows, must support the aggreg ation of DetNet flows. support larger numbers of DetNet flows, must support the aggreg ation of DetNet flows.
Such aggregated flows can be viewed by the DetNet nodes' data p lane Such aggregated flows can be viewed by the DetNet nodes' Data P lane
largely as individual DetNet flows. Without such aggregation, the per-relay largely as individual DetNet flows. Without such aggregation, the per-relay
system may limit the scale of DetNet networks. Example techniqu es that may be used system may limit the scale of DetNet networks. Example techniqu es that may be used
include MPLS hierarchy and IP DiffServ Code Points (DSCPs). include MPLS hierarchy and IP DiffServ Code Points (DSCPs).
</t> </t>
</section> </section>
<section anchor="Compatibility" title="Compatibility with Layer-2"> <section anchor="Compatibility" numbered="true" toc="default">
<t> <name>Compatibility with Layer 2</name>
Standards providing similar <t>
capabilities for bridged networks (only) have been and are bein Standards providing similar capabilities for bridged networks (only) have
g generated in the been and are being generated in the IEEE 802 LAN/MAN Standards Committee.
IEEE 802 LAN/MAN Standards Committee. The present architecture The present architecture describes an abstract model that can be applicable
describes an abstract model that can be applicable both at Laye both at Layer 2 and Layer 3, and over links not defined by IEEE 802. </t>
r-2 <t> DetNet-enabled end systems and DetNet nodes can be interconnected by
and Layer-3, and over links not defined by IEEE 802. sub-networks, i.e., Layer 2 technologies. These sub-networks will provide
</t><t> DetNet compatible service for support of DetNet traffic. Examples of
DetNet enabled end systems and DetNet nodes can be interc sub-network technologies include MPLS TE, TSN as defined by IEEE 802.1, and a po
onnected by int-to-point OTN
sub-networks, i.e., Layer-2 technologies. link. Of course, multilayer DetNet systems may be possible too, where one
These sub-networks will DetNet appears as a sub-network and provides service to a higher-layer DetNet
provide DetNet compatible service for support of DetNet t system.
raffic. </t>
Examples of sub-network technologies include MPLS TE, 802 </section>
.1 TSN, and a point-to-point OTN link.
Of course, multi-layer DetNet systems may be possible too
, where one
DetNet appears as a sub-network, and provides service to,
a higher layer
DetNet system.
</t>
</section>
</section> </section>
<section anchor="SecurityConsiderations" numbered="true" toc="default">
<section anchor="SecurityConsiderations" title="Security Considerations"> <name>Security Considerations</name>
<t>
<t> Security considerations for DetNet are described in detail
Security considerations for DetNet are described in detail in in <xref target="I-D.ietf-detnet-security" format="default"/>.
<xref target="I-D.ietf-detnet-security"/>. This section conside This section considers
rs exclusively security considerations that are specific to the
exclusively security considerations which are specific to the D DetNet architecture. </t>
etNet <t> Security aspects that are
architecture. unique to DetNet are those whose aim is to provide the
</t><t> specific QoS aspects of DetNet, which are
Security aspects which are unique to DetNet are those whose aim primarily to deliver data flows with extremely low packet
is to loss rates and bounded end-to-end delivery latency. A DetNet
provide the specific quality of service aspects of DetNet, whic may be implemented using MPLS and/or IP (including both v4
h are and v6) technologies and thus inherits the security
primarily to deliver data flows with extremely low packet loss properties of those technologies at both the Data Plane and
rates the Controller Plane. </t>
and bounded end-to-end delivery latency. A DetNet may be implem <t> Security considerations for
ented DetNet are constrained (compared to, for example, the open
using MPLS and/or IP (including both v4 and v6) technologies, a Internet) because DetNet is defined to operate only within a
nd single administrative domain (see <xref target="introduction"/>
thus inherits the security properties of those technologies at ). The primary
both considerations are to secure the request and control of
the data plane and the control plane. DetNet resources, maintain confidentiality of data
</t><t> traversing the DetNet, and provide the availability of the
Security considerations for DetNet are constrained (compared to DetNet QoS. </t>
, for <t> To secure the request
example, the open Internet) because DetNet is defined to operat and control of DetNet resources, authentication and
e only authorization can be used for each device connected to a
within a single administrative domain (see Section 1). The prim DetNet domain, most importantly to network controller
ary devices. Control of a DetNet network may be centralized or
considerations are to secure the request and control of DetNet distributed (within a single administrative domain). In the
resources, maintain confidentiality of data traversing the DetN case of centralized control, precedent for security
et, considerations as defined for Abstraction and Control of
and provide the availability of the DetNet quality of service. Traffic Engineered Networks (ACTN) can be found in <xref
</t><t> target="RFC8453" sectionFormat ="of" section="9"/>. In the case
To secure the request and control of DetNet resources, authenti of distributed
cation control protocols, DetNet security is expected to be
and authorization can be used for each device connected to a provided by the security properties of the protocols in
DetNet domain, most importantly to network controller devices. use. In any case, the result is that manipulation of
Control administratively configurable parameters is limited to
of a DetNet network may be centralized or distributed (within a authorized entities. </t>
single <t> To maintain confidentiality of
administrative domain). In the case of centralized control, pre data traversing the DetNet, application flows can be
cedent protected through whatever means is provided by the
for security considerations as defined for Abstraction and Cont underlying technology. For example, encryption may be used,
rol of such as that provided by IPsec <xref target="RFC4301" format="d
Traffic Engineered Networks (ACTN) can be found in efault"/>, for
<xref target="RFC8453"/>, Section 9. In the case of distributed IP flows and by MACSec <xref target="IEEE802.1AE" format="defau
control protocols, DetNet security is expected to be provided b lt"/> for
y the Ethernet (Layer 2) flows. </t>
security properties of the protocols in use. In any case, the r <t> DetNet flows are
esult identified on a per-flow basis, which may provide attackers
is that manipulation of administratively configurable parameter with additional information about the data flows (when
s is compared to networks that do not include per-flow
limited to authorized entities. identification). This is an inherent property of DetNet
</t><t> that has security implications that should be considered
To maintain confidentiality of data traversing the DetNet, when determining if DetNet is a suitable technology for any
application flows can be protected through whatever means is given use case. </t>
provided by the underlying technology. For example, encryption <t> To provide uninterrupted
may be availability of the DetNet QoS, provisions
used, such as that provided by IPSec <xref target="RFC4301"/> f can be made against DoS attacks and delay attacks. To
or IP protect against DoS attacks, excess traffic due to malicious
flows and by MACSec <xref target="IEEE802.1AE-2018"/> for Ether or malfunctioning devices can be prevented or mitigated, for
net example, through the use of traffic admission control
(Layer-2) flows. applied at the input of a DetNet domain as described in
</t><t> <xref target="Zero" format="default"/> and through the fault-mi
DetNet flows are identified on a per-flow basis, which may provide tigation
attackers with additional information about the data flows (when methods described in <xref target="FaultMitigation" format="def
compared to networks that do not include per-flow identification). ault"/>. To
This is an inherent property of DetNet which has security prevent DetNet packets from being delayed by an entity
implications that should be considered when determining if DetNet is external to a DetNet domain, DetNet technology definition
a suitable technology for any given use case. can allow for the mitigation of man-in-the-middle attacks,
</t><t> for example, through use of authentication and authorization
To provide uninterrupted availability of the DetNet quality of of devices within the DetNet domain. </t>
service, provisions can be made against DOS attacks and delay <t> Because DetNet
attacks. To protect against DOS attacks, excess traffic due to mechanisms or applications that rely on DetNet can make
malicious or malfunctioning devices can be prevented or mitigat heavy use of methods that require precise time
ed, synchronization, the accuracy, availability, and integrity
for example through the use of traffic admission control applie of time synchronization is of critical importance. Extensive
d at discussion of this topic can be found in <xref target="RFC7384"
the input of a DetNet domain, as described in <xref target="Zer format="default"/>. </t>
o"/>, <t> DetNet use cases are known to
and through the fault mitigation methods described in have widely divergent security requirements. The intent of
<xref target="FaultMitigation"/>. To prevent DetNet packets fro this section is to provide a baseline for security
m considerations that are common to all DetNet designs and
being delayed by an entity external to a DetNet domain, DetNet implementations, without burdening individual designs with
technology definition can allow for the mitigation of specifics of security infrastructure that may not be
Man-In-The-Middle attacks, for example through use of germane to the given use case. Designers and implementors of
authentication and authorization of devices within the DetNet d DetNet systems are expected to take use-case-specific considera
omain. tions
</t><t> into account in their DetNet designs
Because DetNet mechanisms or applications that rely on DetNet can and implementations.
make heavy use of methods that require precise time synchroniza </t>
tion, </section>
the accuracy, availability, and integrity of time synchronizati <section numbered="true" toc="default">
on is <name>Privacy Considerations</name>
of critical importance. Extensive discussion of this topic can <t>
be DetNet provides a QoS, and the generic
found in <xref target="RFC7384"/>.
</t><t>
DetNet use cases are known to have widely divergent security
requirements. The intent of this section is to provide a baseli
ne for
security considerations which are common to all DetNet designs
and
implementations, without burdening individual designs with spec
ifics
of security infrastructure which may not be germane to the give
n use
case. Designers and implementers of DetNet systems are expected
to
take use case specific considerations into account in their Det
Net
designs and implementations.
</t>
</section>
<section title="Privacy Considerations">
<t>
DetNet provides a Quality of Service (QoS), and the generic
considerations for such mechanisms apply. In particular, such mar kings considerations for such mechanisms apply. In particular, such mar kings
allow for an attacker to correlate flows or to select particular types allow for an attacker to correlate flows or to select particular types
of flow for more detailed inspection. of flow for more detailed inspection.
</t><t> </t>
<t>
However, the requirement for every (or almost every) node along t he path of However, the requirement for every (or almost every) node along t he path of
a DetNet flow to identify DetNet flows may present an additional attack a DetNet flow to identify DetNet flows may present an additional attack
surface for privacy, should the DetNet paradigm be found useful i n broader surface for privacy should the DetNet paradigm be found useful in broader
environments. environments.
</t> </t>
</section> </section>
<section numbered="true" toc="default">
<name>IANA Considerations</name>
<t>This document has no IANA actions.
</t>
</section>
</middle>
<back>
<section title="IANA Considerations"> <!-- I-D.ietf-detnet-security-05: I-D Exists -->
<t>This document does not require an action from IANA. <displayreference target="I-D.ietf-detnet-security" to="DETNET-SECURITY"/>
</t>
</section>
<section title="Acknowledgements"> <!-- draft-ietf-detnet-ip-01: I-D exists -->
<t>The authors wish to thank Lou Berger, David Black, Stewart Bryant, <displayreference target="I-D.ietf-detnet-ip" to="DETNET-IP"/>
Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel Kiessling,
Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu Shah,
Wilfried Steiner, George Swallow, Michael Johas Teener, Pat Thaler,
Thomas Watteyne, Patrick Wetterwald, Karl Weber, Anca Zamfir,
for their various contributions to this work.
</t>
</section>
</middle> <!-- draft-ietf-detnet-data-plane-framework-02: I-D exists-->
<displayreference target="I-D.ietf-detnet-data-plane-framework" to="DETNET-FRAME
WORK"/>
<back> <!-- draft-ietf-detnet-mpls-01: I-D exists-->
<references title='Informative References'> <displayreference target="I-D.ietf-detnet-mpls" to="DETNET-MPLS"/>
<?rfc include='reference.I-D.ietf-6tisch-architecture'?> <!-- I-D.ietf-6tisch-architecture-26: I-D exists-->
<?rfc include='reference.I-D.ietf-detnet-problem-statement'?> <displayreference target="I-D.ietf-6tisch-architecture" to="TSCH-ARCH"/>
<?rfc include='reference.I-D.ietf-detnet-use-cases'?>
<?rfc include='reference.I-D.ietf-detnet-dp-sol-mpls'?>
<?rfc include='reference.I-D.ietf-detnet-dp-sol-ip'?>
<?rfc include='reference.I-D.ietf-detnet-security'?>
<?rfc include='reference.RFC.2205'?>
<?rfc include='reference.RFC.2475'?>
<?rfc include='reference.RFC.2914'?>
<?rfc include='reference.RFC.3168'?>
<?rfc include='reference.RFC.3209'?>
<?rfc include='reference.RFC.3550'?>
<?rfc include='reference.RFC.4301'?>
<?rfc include='reference.RFC.4655'?>
<?rfc include='reference.RFC.5921'?>
<?rfc include='reference.RFC.6372'?>
<?rfc include='reference.RFC.6658'?>
<?rfc include='reference.RFC.7149'?>
<?rfc include='reference.RFC.7384'?>
<?rfc include='reference.RFC.7426'?>
<?rfc include='reference.RFC.7554'?>
<?rfc include='reference.RFC.7567'?>
<?rfc include='reference.RFC.7813'?>
<?rfc include='reference.RFC.8033'?>
<?rfc include='reference.RFC.8227'?>
<?rfc include='reference.RFC.8289'?>
<?rfc include='reference.RFC.8402'?>
<?rfc include='reference.RFC.8453'?>
<reference anchor="IEC62439-3-2016" <references>
target="https://webstore.iec.ch/publication/24447"> <name>Informative References</name>
<front>
<title>IEC 62439-3:2016 Industrial communication networks - Hig
h
availability automation networks - Part 3: Parallel Redundan
cy
Protocol (PRP) and High-availability Seamless Redundancy (HS
R)
</title>
<author>
<organization>International Electrotechnical Commission (
IEC)
TC 65/SC 65C - Industrial networks</organization>
</author>
<date year="2016" />
</front>
</reference>
<reference anchor="IEEE802.1AE-2018" <xi:include href="https://www.rfc-editor.org/refs/bibxml3/reference.I-D.ietf-det
target="https://ieeexplore.ieee.org/document/8585421"> net-security.xml"/>
<front>
<title>IEEE Std 802.1AE-2018 MAC Security (MACsec)</title>
<author>
<organization>IEEE Standards Association</organization>
</author>
<date year="2018" />
</front>
</reference>
<reference anchor="IEEE802.1CB" <xi:include href="https://www.rfc-editor.org/refs/bibxml3/reference.I-D.ietf-det
target="https://ieeexplore.ieee.org/document/8091139/"> net-ip.xml"/>
<front>
<title>IEEE Std 802.1CB-2017 Frame Replication and Elimination
for Reliability</title>
<author>
<organization>IEEE Standards Association</organization>
</author>
<date year="2017" />
</front>
</reference>
<reference anchor="IEEE802.1Q-2018" <xi:include href="https://www.rfc-editor.org/refs/bibxml3/reference.I-D.ietf-det
target="https://ieeexplore.ieee.org/document/8403927"> net-data-plane-framework.xml"/>
<front>
<title>IEEE Std 802.1Q-2018 Bridges and Bridged Networks</title>
<author>
<organization>IEEE Standards Association</organization>
</author>
<date year="2018" />
</front>
</reference>
<reference anchor="IEEE802.1Qav" <xi:include href="https://www.rfc-editor.org/refs/bibxml3/reference.I-D.ietf-det
target="https://ieeexplore.ieee.org/document/5375704/"> net-mpls.xml"/>
<front>
<title>IEEE Std 802.1Qav-2009 Bridges and Bridged Networks - Amend
ment 12:
Forwarding and Queuing Enhancements for Time-Sensitive
Streams</title>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2009" />
</front>
</reference>
<reference anchor="IEEE802.1Qbu" <xi:include href="https://www.rfc-editor.org/refs/bibxml3/reference.I-D.ietf-6ti
target="https://ieeexplore.ieee.org/document/7553415/"> sch-architecture.xml"/>
<front>
<title>IEEE Std 802.1Qbu-2016 Bridges and Bridged Networks - Amend
ment 26: Frame Preemption</title>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2016" />
</front>
</reference>
<reference anchor="IEEE802.1Qbv" <!-- draft-ietf-detnet-use-cases now RFC 8578-->
target="https://ieeexplore.ieee.org/document/7572858/">
<front>
<title>IEEE Std 802.1Qbv-2015 Bridges and Bridged Networks - Amend
ment 25: Enhancements for Scheduled Traffic</title>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2015" />
</front>
</reference>
<reference anchor="IEEE802.1BA" <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8578.
target="https://ieeexplore.ieee.org/document/6032690/"> xml"/>
<front>
<title>IEEE Std 802.1BA-2011 Audio Video Bridging (AVB) Systems
</title>
<author>
<organization>IEEE Standards Association</organization>
</author>
<date year="2011" />
</front>
</reference>
<reference anchor="IEEE802.1Qch" <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8557.
target="https://ieeexplore.ieee.org/document/7961303/"> xml"/>
<front>
<title>IEEE Std 802.1Qch-2017 Bridges and Bridged Netwo
rks - Amendment 29: Cyclic Queuing and Forwarding</title>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2017" />
</front>
</reference>
<reference anchor="IEEE802.3-2018" <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2205.
target="https://ieeexplore.ieee.org/document/8457469"> xml"/>
<front>
<title>IEEE Std 802.3-2018 Standard for Ethernet</title
>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2018" />
</front>
</reference>
<reference anchor="IEEE802.3br" <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2475.
target="http://ieeexplore.ieee.org/document/7900321/"> xml"/>
<front>
<title>IEEE Std 802.3br-2016 Standard for Ethernet Amen
dment 5:
Specification and Management Parameters for Interspersing Expr
ess Traffic</title>
<author>
<organization>IEEE Standards Association</organ
ization>
</author>
<date year="2016" />
</front>
</reference>
<reference anchor="IEEE802.1TSNTG" target="http://www.ieee802.org/1/tsn <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2914.
"> xml"/>
<front>
<title>IEEE 802.1 Time-Sensitive Networking Task Group</title> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3168.
<author> xml"/>
<organization>IEEE Standards Association</organization>
</author>
<date></date>
</front>
</reference>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3209.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3550.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4301.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4655.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6372.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6658.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7149.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7384.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7426.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7554.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7567.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7813.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8033.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8227.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8289.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8402.
xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8453.
xml"/>
<reference anchor="IEC-62439-3" target="https://webstore.iec.ch/publicatio
n/24447">
<front>
<title>Industrial communication networks - High
availability automation networks - Part 3: Parallel Redundan
cy
Protocol (PRP) and High-availability Seamless Redundancy (HS
R)
</title>
<seriesInfo name="TC 65 /" value="SC 65C"/>
<seriesInfo name="IEC" value="62439-3:2016"/>
<author>
<organization>IEC</organization>
</author>
<date month="March" year="2016"/>
</front>
</reference>
<reference anchor="IEEE802.1AE" target="https://ieeexplore.ieee.org/docume
nt/8585421">
<front>
<title>IEEE Standard for Local and metropolitan area
networks-Media Access Control (MAC) Security</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2018.8585421"/>
<seriesInfo name="IEEE" value="802.1AE-2018"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1CB" target="https://ieeexplore.ieee.org/docume
nt/8091139">
<front>
<title>IEEE Standard for Local and metropolitan area
networks--Frame Replication and Elimination for Reliability</ti
tle>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2017.8091139"/>
<seriesInfo name="IEEE" value="802.1CB-2017"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1Q" target="https://ieeexplore.ieee.org/documen
t/8403927">
<front>
<title>IEEE Standard for Local and Metropolitan Area
Network--Bridges and Bridged Networks</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2018.8403927"/>
<seriesInfo name="IEEE" value="802.1Q-2018"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1Qav" target="https://ieeexplore.ieee.org/docum
ent/5375704">
<front>
<title>IEEE Standard for Local and Metropolitan Area Networks -
Virtual Bridged Local Area Networks Amendment 12: Forwarding and
Queuing Enhancements for Time-Sensitive Streams</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2009.5375704"/>
<seriesInfo name="IEEE" value="802.1Qav-2009"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1Qbu" target="https://ieeexplore.ieee.org/docum
ent/7553415">
<front>
<title>IEEE Standard for Local and metropolitan area networks --
Bridges and Bridged Networks -- Amendment 26: Frame Preemption</tit
le>
<seriesInfo name="DOI" value=" 10.1109/IEEESTD.2016.7553415"/>
<seriesInfo name="IEEE" value="802.1Qbu-2016"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1Qbv" target="https://ieeexplore.ieee.org/docum
ent/7440741">
<front>
<title>IEEE Standard for Local and metropolitan area networks --
Bridges and Bridged Networks - Amendment 25: Enhancements for
Scheduled Traffic</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2016.7572858"/>
<seriesInfo name="IEEE" value="802.1Qbv-2015"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1BA" target="https://ieeexplore.ieee.org/docume
nt/6032690">
<front>
<title>IEEE Standard for Local and metropolitan area
networks--Audio Video Bridging (AVB) Systems</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2011.6032690"/>
<seriesInfo name="IEEE" value="802.1BA-2011"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1Qch" target="https://ieeexplore.ieee.org/docum
ent/7961303">
<front>
<title>IEEE Standard for Local and metropolitan area
networks--Bridges and Bridged Networks--Amendment
29: Cyclic Queuing and Forwarding</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2017.7961303"/>
<seriesInfo name="IEEE" value="802.1Qch-2017"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.3" target="https://ieeexplore.ieee.org/document
/8457469">
<front>
<title>IEEE Standard for Ethernet</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2018.8457469"/>
<seriesInfo name="IEEE" value="802.3-2018"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.3br" target="https://ieeexplore.ieee.org/docume
nt/7900321">
<front>
<title>IEEE Standard for Ethernet Amendment 5:
Specification and Management Parameters for
Interspersing Express Traffic</title>
<seriesInfo name="DOI" value="10.1109/IEEESTD.2016.7900321"/>
<seriesInfo name="IEEE" value="802.3br"/>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="IEEE802.1TSNTG" target="https://1.ieee802.org/tsn/">
<front>
<title>Time-Sensitive Networking (TSN) Task Group</title>
<author>
<organization>IEEE</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="TEAS" target="https://datatracker.ietf.org/doc/charter- ietf-teas/"> <reference anchor="TEAS" target="https://datatracker.ietf.org/doc/charter- ietf-teas/">
<front> <front>
<title>Traffic Engineering Architecture and Signaling Working Group< <title>Traffic Engineering Architecture and Signaling (teas)</title>
/title> <author>
<author> <organization>IETF</organization>
<organization>IETF</organization> </author>
</author> <date/>
<date></date> </front>
</front>
</reference> </reference>
<reference anchor="CCAMP" target="https://datatracker.ietf.org/wg/ccamp/ch
<reference anchor="CCAMP" target="https://datatracker.ietf.org/doc/char arter/">
ter-ietf-ccamp/"> <front>
<front> <title>Common Control and Measurement Plane (ccamp)</title>
<title>Common Control and Measurement Plane Working Group</title> <author>
<author> <organization>IETF</organization>
<organization>IETF</organization> </author>
</author> <date/>
<date></date> </front>
</front>
</reference> </reference>
<reference anchor="BUFFERBLOAT"> <reference anchor="BUFFERBLOAT">
<front> <front>
<title>Bufferbloat: Dark Buffers in the Internet</title> <title>Bufferbloat: Dark Buffers in the Internet</title>
<seriesInfo name="DOI" value="10.1145/2063176.2063196"/>
<author fullname="J. Gettys" initials="J." surname="Gettys"></author> <seriesInfo name="Communications of the ACM," value="Volume 55, Issue
1"/>
<author fullname="K. Nichols" initials="K." surname="Nichols"></author <author initials="J." surname="Gettys" fullname="Jim Gettys">
> <organization/>
</author>
<date month="January" year="2012" /> <author initials="K." surname="Nichols" fullname="Kathleen Nichols">
<organization/>
</author>
<date month="January" year="2012"/>
</front> </front>
<format octets="Communications of the ACM, Volume 55 Issue 1, Pages 57-6
5 " target="https://dl.acm.org/citation.cfm?id=2063176.2063196" type="PDF" />
</reference> </reference>
</references>
</references> <section numbered="false" toc="default">
<name>Acknowledgements</name>
<t>The authors wish to thank Lou Berger, David Black, Stewart
Bryant, Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel
Kiessling, Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu
Shah, Wilfried Steiner, George Swallow, Michael Johas Teener, Pat
Thaler, Thomas Watteyne, Patrick Wetterwald, Karl Weber, and Anca
Zamfir for their various contributions to this work.
</t>
</section>
</back> </back>
</rfc> </rfc>
 End of changes. 151 change blocks. 
2253 lines changed or deleted 2043 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/