rfc9318xml2.original.xml   rfc9318.xml 
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?> <!DOCTYPE rfc [
<!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.3.32 --> <!ENTITY nbsp "&#160;">
<!ENTITY zwsp "&#8203;">
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [ <!ENTITY nbhy "&#8209;">
<!ENTITY wj "&#8288;">
]> ]>
<?rfc toc="yes"?> <!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.3.32 -->
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes"?>
<?rfc docmapping="yes"?>
<rfc ipr="trust200902" docName="draft-iab-mnqeu-report-04" category="info"> <rfc xmlns:xi="http://www.w3.org/2001/XInclude" ipr="trust200902" docName="draft -iab-mnqeu-report-04" number="9318" submissionType="IAB" category="info" consens us="true" obsoletes="" updates="" xml:lang="en" tocInclude="true" sortRefs="true " symRefs="true" version="3">
<front> <!-- [rfced]
<title abbrev="title">IAB workshop report: Measuring Network Quality for End b) The boilerplate that follows the Abstract appears to be missing.
-Users</title> -->
<!-- Wes: sounds fine -->
<!-- xml2rfc v2v3 conversion 3.14.1 -->
<front>
<title abbrev="Measuring Network Quality for End-Users">IAB Workshop Report:
Measuring Network Quality for End-Users</title>
<seriesInfo name="RFC" value="9318"/>
<author initials="W." surname="Hardaker" fullname="Wes Hardaker"> <author initials="W." surname="Hardaker" fullname="Wes Hardaker">
<organization>USC/ISI</organization>
<address> <address>
<email>ietf@hardakers.net</email> <email>ietf@hardakers.net</email>
</address> </address>
</author> </author>
<author initials="O." surname="Shapira" fullname="Omer Shapira"> <author initials="O." surname="Shapira" fullname="Omer Shapira">
<organization>Apple</organization>
<address> <address>
<email>omer_shapira@apple.com</email> <email>omer_shapira@apple.com</email>
</address> </address>
</author> </author>
<date year="2022" month="September"/>
<date year="2022" month="August" day="10"/> <keyword>QoE</keyword>
<keyword>QoS</keyword>
<keyword>Quality of Service</keyword>
<keyword>Quality of Experience</keyword>
<keyword>Measurement</keyword>
<keyword>End User</keyword>
<abstract> <abstract>
<t>The Measuring Network Quality for End-Users workshop was held
<t>The Measuring Network Quality for End-Users workshop was held virtually by the Internet Architecture Board (IAB) on September 14-16, 2021.
virtually by the Internet Architecture Board (IAB) from September 14-16, 2021.
This report summarizes the workshop, the topics discussed, and some This report summarizes the workshop, the topics discussed, and some
preliminary conclusions drawn at the end of the workshop.</t> preliminary conclusions drawn at the end of the workshop.</t>
<t>Note that this document is a report on the proceedings of the
workshop. The views and positions documented in this report are
those of the workshop participants and do not necessarily reflect IAB
views and positions. </t>
</abstract> </abstract>
</front> </front>
<middle> <middle>
<section anchor="introduction" numbered="true" toc="default">
<section anchor="introduction" title="Introduction"> <name>Introduction</name>
<t>The Internet Architecture Board (IAB) holds occasional workshops design
<t>The Internet Architecture Board (IAB) holds occasional workshops designed to ed to
consider long-term issues and strategies for the Internet, and to suggest consider long-term issues and strategies for the Internet, and to suggest
future directions for the Internet architecture. This long-term planning future directions for the Internet architecture. This long-term planning
function of the IAB is complementary to the ongoing engineering efforts function of the IAB is complementary to the ongoing engineering efforts
performed by working groups of the Internet Engineering Task Force (IETF).</t> performed by working groups of the Internet Engineering Task Force (IETF).
</t>
<t>The Measuring Network Quality for End-Users workshop <xref target="WORKSHOP"/ <t>The Measuring Network Quality for End-Users workshop <xref target="WORK
> was held SHOP" format="default"/> was held
virtually by the Internet Architecture Board (IAB) in September 14-16, 2021. virtually by the Internet Architecture Board (IAB) on September 14-16, 2021.
This report summarizes the workshop, the topics discussed, and some preliminary This report summarizes the workshop, the topics discussed, and some preliminary
conclusions drawn at the end of the workshop.</t> conclusions drawn at the end of the workshop.</t>
<section anchor="problem-space" numbered="true" toc="default">
<section anchor="problem-space" title="Problem space"> <name>Problem Space</name>
<t>The Internet in 2021 is quite different from what it was 10 years ago
<t>The Internet in 2021 is quite different from what it was 10 years ago. Today, . Today, it
it is a crucial part of everyone's daily life. People use the Internet for their
is a crucial part of everyone’s daily life. People use the Internet for their
social life, for their daily jobs, for routine shopping, and for keeping up social life, for their daily jobs, for routine shopping, and for keeping up
with major events. An increasing number of people can access a Gigabit with major events. An increasing number of people can access a gigabit
connection, which would be hard to imagine a decade ago. And, thanks to connection, which would be hard to imagine a decade ago. Additionally, thanks to
improvements in security, people trust the Internet for financial improvements in security, people trust the Internet for financial
banking transactions, purchasing goods and everyday bill payments.</t> banking transactions, purchasing goods, and everyday bill payments.</t>
<t>At the same time, some aspects of the end-user experience have not
<t>At the same time, some aspects of end-user experience have not
improved as much. Many users have typical connection latencies that improved as much. Many users have typical connection latencies that
remain at decade-old levels. Despite significant reliability remain at decade-old levels. Despite significant reliability
improvements in data center environments, end users also still often see improvements in data center environments, end users also still often see
interruptions in service. Despite algorithmic advances in the field of interruptions in service. Despite algorithmic advances in the field of
control theory, one still finds that the queuing delays in the control theory, one still finds that the queuing delays in the
last-mile equipment exceeds the accumulated transit delays. Transport last-mile equipment exceeds the accumulated transit delays. Transport
improvements, such as QUIC, Multipath TCP, and TCP Fast Open are still improvements, such as QUIC, Multipath TCP, and TCP Fast Open, are still
not fully supported in some networks. not fully supported in some networks. Likewise, various advances in
<!-- I'm not sure there is agreement about this next sentence - Wes -->
Likewise, various advances in
the security and privacy of user data are not widely supported, such the security and privacy of user data are not widely supported, such
as encrypted DNS to the local resolver.</t> as encrypted DNS to the local resolver.</t>
<t>Some of the major factors behind this lack of progress is the popular
<t>Some of the major factors behind this lack of progress is the popular perception that throughput is often the sole measure of the quality of
perception that throughput is the often sole measure of the quality of Internet connectivity. With such a narrow focus, the Measuring Network
Internet connectivity. With such narrow focus, the Measuring Network Quality for End-Users workshop aimed to discuss various topics:</t>
Quality for End-Users workshop aimed to discuss various questions:</t> <ul spacing="normal">
<li>What is user latency under typical working conditions?</li>
<t><list style="symbols"> <li>How reliable is connectivity across longer time periods?</li>
<t>What is user latency under typical working conditions?</t> <li>Do networks allow the use of a broad range of protocols?</li>
<t>How reliable is connectivity across longer time periods?</t> <li>What services can be run by network clients?</li>
<t>Do networks allow the use of a broad range of protocols?</t> <li>What kind of IPv4, NAT, or IPv6 connectivity is offered, and are t
<t>What services can be run by network clients?</t> here
<t>What kind of IPv4, NAT, or IPv6 connectivity is offered, and are there firewalls?</li>
firewalls?</t> <li>What security mechanisms are available for local services, such as
<t>What security mechanisms are available for local services, such as DNS?</t> DNS?</li>
<t>To what degree are the privacy, confidentiality, integrity, and authenticit <li>To what degree are the privacy, confidentiality, integrity, and au
y thenticity
of user communications guarded?</t> of user communications guarded?</li>
<t>Improving these aspects of network quality will likely depend on <li>Improving these aspects of network quality will likely depend on
measurement and exposing metrics in a meaningful way to all involved measuring and exposing metrics in a meaningful way to all involved
parties, including to end users. Such measurement and exposure of parties, including to end users. Such measurement and exposure of
the right metrics will allow service providers and network operators the right metrics will allow service providers and network operators
to concentrate focus on their users’ experience and will to concentrate focus on their users' experience and will
simultaneously empower users to choose the Internet service simultaneously empower users to choose the Internet Service
providers that can deliver the best experience based on their needs.</t> Providers (ISPs) that can deliver the best experience based on their needs.</li>
<t>What are the fundamental properties of a network that contributes to <li>What are the fundamental properties of a network that contributes
a good user experience?</t> to
<t>What metrics quantify these properties, and how can we collect such metrics a good user experience?</li>
in a <li>What metrics quantify these properties, and how can we collect suc
practical way?</t> h metrics in a
<t>What are the best practices for interpreting those metrics, and incorporati practical way?</li>
ng <li>What are the best practices for interpreting those metrics and inc
those in a decision making process?</t> orporating
<t>What are the best ways to communicate these properties to service providers them in a decision-making process?</li>
and network operators?</t> <li>What are the best ways to communicate these properties to service
<t>How can these metrics be displayed to users in a meaningful way?</t> providers
</list></t> and network operators?</li>
<li>How can these metrics be displayed to users in a meaningful way?</
</section> li>
</section> </ul>
<section anchor="workshop-agenda" title="Workshop Agenda"> </section>
</section>
<t>The Measuring Network Quality for End-Users workshop was divided into the <section anchor="workshop-agenda" numbered="true" toc="default">
following main topic areas, further discussion in <xref target="discussions"/>:< <name>Workshop Agenda</name>
/t> <t>The Measuring Network Quality for End-Users workshop was divided into t
he
<t><list style="symbols"> following main topic areas; see further discussion in Sections <xref target="dis
<t>Introduction overviews and a keynote by Vint Cerf</t> cussions" format="counter"/> and <xref target="conclusions" format="counter"/>:<
<t>Metrics considerations</t> /t>
<t>Cross-layer considerations</t> <ul spacing="normal">
<t>Synthesis</t> <li>Introduction overviews and a keynote by Vint Cerf</li>
<t>Group conclusions</t> <li>Metrics considerations</li>
</list></t> <li>Cross-layer considerations</li>
<li>Synthesis</li>
</section> <li>Group conclusions</li>
<section anchor="positionpapers" title="Position Papers"> </ul>
</section>
<t>The following position papers were received for consideration by the <section anchor="positionpapers" numbered="true" toc="default">
workshop attendees. The workshop’s web-page <xref target="WORKSHOP"/> contains <name>Position Papers</name>
archives of the papers, presentations and recorded videos.</t> <t>The following position papers were received for consideration by the
workshop attendees. The workshop's web page <xref target="WORKSHOP" format="def
<t><list style="symbols"> ault"/> contains
<t>Ahmed Aldabbagh. “Regulatory perspective on measuring network quality for e archives of the papers, presentations, and recorded videos.</t>
nd users” <xref target="Aldabbagh2021"/></t> <ul spacing="normal">
<t>Al Morton. “Dream-Pipe or Pipe-Dream: What Do Users Want (and how can we as <li>Ahmed Aldabbagh. "Regulatory perspective on measuring network qualit
sure it)?” <xref target="Morton2021"/></t> y for end users" <xref target="Aldabbagh2021" format="default"/></li>
<t>Alexander Kozlov . “The 2021 National Internet Segment Reliability Research <li>Al Morton. "Dream-Pipe or Pipe-Dream: What Do Users Want (and how ca
”</t> n we assure it)?" <xref target="I-D.morton-ippm-pipe-dream" format="default"/></
<t>Anna Brunstrom. “Measuring newtork quality - the MONROE experience”</t> li>
<t>Bob Briscoe, Greg White, Vidhi Goel and Koen De Schepper. “A single common <li>Alexander Kozlov. "The 2021 National Internet Segment Reliability Re
metric to characterize varying packet delay” <xref target="Briscoe2021"/></t> search"</li>
<t>Brandon Schlinker. “Internet’s performance from Facebook’s edge” <xref targ <li>Anna Brunstrom. "Measuring network quality - the MONROE experience"<
et="Schlinker2019"/></t> /li>
<t>Christoph Paasch, Kristen McIntyre, Randall Meyer, Stuart Cheshire, Omer Sh <li>Bob Briscoe, Greg White, Vidhi Goel, and Koen De Schepper. "A Single
apira. “An end-user approach to the Internet Score” <xref target="McIntyre2021"/ Common Metric to Characterize Varying Packet Delay" <xref target="Briscoe2021"
></t> format="default"/></li>
<t>Christoph Paasch, Randall Meyer, Stuart Cheshire, Omer Shapira. “Responsive <li>Brandon Schlinker. "Internet Performance from Facebook's Edge" <xref
ness under Working Conditions” <xref target="Paasch2021"/></t> target="Schlinker2019" format="default"/></li>
<t>Dave Reed, Levi Perigo. “Measuring ISP Performance in Broadband America: a <li>Christoph Paasch, Kristen McIntyre, Randall Meyer, Stuart Cheshire,
Study of Latency Under Load” <xref target="Reed2021"/></t> and Omer Shapira. "An end-user approach to the Internet Score" <xref target="McI
<t>Eve M. Schooler, Rick Taylor. “Non-traditional Network Metrics”</t> ntyre2021" format="default"/></li>
<t>Gino Dion. “Focusing on latency, not throughput, to provide better internet <li>Christoph Paasch, Randall Meyer, Stuart Cheshire, and Omer Shapira.
experience and network quality” <xref target="Dion2021"/></t> "Responsiveness under Working Conditions" <xref target="I-D.cpaasch-ippm-respons
<t>Gregory Mirsky, Xiao Min, Gyan Mishra, Liuyan Han. “Error Performance Measu iveness" format="default"/></li>
rement in Packet-Switched Networks” <xref target="Mirsky2021"/></t> <li>Dave Reed and Levi Perigo. "Measuring ISP Performance in Broadband A
<t>Jana Iyengar. “The Internet Exists In Its Use” <xref target="Iyengar2021"/> merica: A Study of Latency Under Load" <xref target="Reed2021" format="default"/
</t> ></li>
<t>Jari Arkko, Mirja Kuehlewind. “Observability is needed to improve network q <li>Eve M.&nbsp;Schooler and Rick Taylor. "Non-traditional Network Metri
uality” <xref target="Arkko2021"/></t> cs"</li>
<t>Joachim Fabini. “Objective and subjective network quality” <xref target="Fa <li>Gino Dion. "Focusing on latency, not throughput, to provide better i
bini2021"/></t> nternet experience and network quality" <xref target="Dion2021" format="default
<t>Jonathan Foulkes. “Metrics helpful in assessing Internet Quality” <xref tar "/></li>
get="Foulkes2021"/></t> <li>Gregory Mirsky, Xiao Min, Gyan Mishra, and Liuyan Han. "The error pe
<t>Kalevi Kilkki, Benajamin Finley. “In Search of Lost QoS” <xref target="Kilk rformance metric in a packet-switched network" <xref target="Mirsky2021" format=
ki2021"/></t> "default"/></li>
<t>Karthik Sundaresan, Greg White, Steve Glennon . “Latency Measurement: What <li>Jana Iyengar. "The Internet Exists In Its Use" <xref target="Iyengar
is latency and how do we measure it?”</t> 2021" format="default"/></li>
<t>Keith Winstein. “Five Observations on Measuring Network Quality for Users o <li>Jari Arkko and Mirja Kuehlewind. "Observability is needed to improve
f Real-Time Media Applications”</t> network quality" <xref target="Arkko2021" format="default"/></li>
<t>Ken Kerpez, Jinous Shafiei, John Cioffi, Pete Chow, Djamel Bousaber. “State <li>Joachim Fabini. "Network Quality from an End User Perspective" <xref
of Wi-Fi Reporting” <xref target="Kerpez2021"/></t> target="Fabini2021" format="default"/></li>
<t>Kenjiro Cho. “Access Network Quality as Fitness for Purpose”</t> <li>Jonathan Foulkes. "Metrics helpful in assessing Internet Quality" <x
<t>Koen De Schepper, Olivier Tilmans, Gino Dion. “Challenges and opportunities ref target="Foulkes2021" format="default"/></li>
of hardware support for Low Queuing Latency without Packet Loss” <xref target=" <li>Kalevi Kilkki and Benajamin Finley. "In Search of Lost QoS" <xref ta
DeSchepper2021"/></t> rget="Kilkki2021" format="default"/></li>
<t>Kyle MacMillian, Nick Feamster. “Beyond Speed Test: Measuring Latency Under <li>Karthik Sundaresan, Greg White, and Steve Glennon. "Latency Measurem
Load Across Different Speed Tiers” <xref target="MacMillian2021"/></t> ent: What is latency and how do we measure it?"</li>
<t>Lucas Pardue, Sreeni Tellakula. “Lower layer performance not indicative of <li>Keith Winstein. "Five Observations on Measuring Network Quality for
upper layer success” <xref target="Pardue2021"/></t> Users of Real-Time Media Applications"</li>
<t>Matt Mathis. “Preliminary Longitudinal Study of Internet Responsiveness” <x <li>Ken Kerpez, Jinous Shafiei, John Cioffi, Pete Chow, and Djamel Bousa
ref target="Mathis2021"/></t> ber. "Wi-Fi and Broadband Data" <xref target="Kerpez2021" format="default"/></li
<t>Michael Welzl. “A Case for Long-Term Statistics” <xref target="Welzl2021"/> >
</t> <li>Kenjiro Cho. "Access Network Quality as Fitness for Purpose"</li>
<t>Mikhail Liubogoshchev. “Cross-layer Cooperation for Better Network Service” <li>Koen De Schepper, Olivier Tilmans, and Gino Dion. "Challenges and op
<xref target="Liubogoshchev2021"/></t> portunities of hardware support for Low Queuing Latency without Packet Loss" <xr
<t>Mingrui Zhang, Vidhi Goel, Lisong Xu. “User-Perceived Latency to measure CC ef target="DeSchepper2021" format="default"/></li>
As” <xref target="Zhang2021"/></t> <li>Kyle MacMillian and Nick Feamster. "Beyond Speed Test: Measuring Lat
<t>Neil Davies, Peter Thompson. “Measuring Network Impact on Application Outco ency Under Load Across Different Speed Tiers" <xref target="MacMillian2021" for
mes using Quality Attenuation” <xref target="Davies2021"/></t> mat="default"/></li>
<t>Olivier Bonaventure, Francois Michel. “Packet delivery time as a tie-breake <li>Lucas Pardue and Sreeni Tellakula. "Lower-layer performance not indi
r for assessing Wi-Fi access points” <xref target="Michel2021"/></t> cative of upper-layer success" <xref target="Pardue2021" format="default"/></li>
<t>Pedro Casas. “10 Years of Internet-QoE Measurements. Video, Cloud, <li>Matt Mathis. "Preliminary Longitudinal Study of Internet Responsiven
Conferencing, Web and Apps. What do we need from the Network Side?” <xref target ess" <xref target="Mathis2021" format="default"/></li>
="Casas2021"/></t> <li>Michael Welzl. "A Case for Long-Term Statistics" <xref target="Welzl
<t>Praveen Balasubramanian. “Transport Layer Statistics for Network Quality” < 2021" format="default"/></li>
xref target="Balasubramanian2021"/></t> <li>Mikhail Liubogoshchev. "Cross-layer Cooperation for Better Network S
<t>Rajat Ghai. “Measuring &amp; Improving QoE on the Xfinity Wi-Fi Network” <x ervice" <xref target="Liubogoshchev2021" format="default"/></li>
ref target="Ghai2021"/></t> <li>Mingrui Zhang, Vidhi Goel, and Lisong Xu. "User-Perceived Latency to
<t>Robin Marx, Joris Herbots. “Merge Those Metrics: Towards Holistic (Protocol Measure CCAs" <xref target="Zhang2021" format="default"/></li>
) Logging” <xref target="Marx2021"/></t> <li>Neil Davies and Peter Thompson. "Measuring Network Impact on Applica
<t>Sandor Laki, Szilveszter Nadas, Balazs Varga, Luis M. tion Outcomes Using Quality Attenuation" <xref target="Davies2021" format="defau
Contreras. “Incentive-Based Traffic Management and QoS Measurements” <xref targe lt"/></li>
t="Laki2021"/></t> <li>Olivier Bonaventure and Francois Michel. "Packet delivery time as a
<t>Satadal Sengupta, Hyojoon Kim, Jennifer Rexford. “Fine-Grained RTT Monitori tie-breaker for assessing Wi-Fi access points" <xref target="Michel2021" format=
ng Inside the Network” <xref target="Sengupta2021"/></t> "default"/></li>
<t>Stuart Cheshire. “The Internet is a Shared Network” <xref target="Cheshire2 <li>Pedro Casas. "10 Years of Internet-QoE Measurements. Video, Cloud,
021"/></t> Conferencing, Web and Apps. What do we Need from the Network Side?" <xref target
<t>Toerless Eckert, Alex Clemm. “network-quality-eckert-clemm-00.4”</t> ="Casas2021" format="default"/></li>
<t>Vijay Sivaraman, Sharat Madanapalli, Himal Kumar. “Measuring Network Experi <li>Praveen Balasubramanian. "Transport Layer Statistics for Network Qua
ence Meaningfully, Accurately, and Scalably” <xref target="Sivaraman2021"/></t> lity" <xref target="Balasubramanian2021" format="default"/></li>
<t>Yaakov (J) Stein. “The Futility of QoS” <xref target="Stein2021"/></t> <li>Rajat Ghai. "Using TCP Connect Latency for measuring CX and Network
</list></t> Optimization" <xref target="Ghai2021" format="default"/></li>
<li>Robin Marx and Joris Herbots. "Merge Those Metrics: Towards Holistic
</section> (Protocol) Logging" <xref target="Marx2021" format="default"/></li>
<section anchor="discussions" title="Workshop Topics and Discussion"> <li>Sandor Laki, Szilveszter Nadas, Balazs Varga, and Luis M.
Contreras. "Incentive-Based Traffic Management and QoS Measurements" <xref targe
<t>The agenda for the three day workshop was broken into four separate t="Laki2021" format="default"/></li>
<li>Satadal Sengupta, Hyojoon Kim, and Jennifer Rexford. "Fine-Grained R
TT Monitoring Inside the Network" <xref target="Sengupta2021" format="default"/>
</li>
<li>Stuart Cheshire. "The Internet is a Shared Network" <xref target="Ch
eshire2021" format="default"/></li>
<li>Toerless Eckert and Alex Clemm. "network-quality-eckert-clemm-00.4"<
/li>
<li>Vijay Sivaraman, Sharat Madanapalli, and Himal Kumar. "Measuring Net
work Experience Meaningfully, Accurately, and Scalably" <xref target="Sivaraman2
021" format="default"/></li>
<li>Yaakov (J) Stein. "The Futility of QoS" <xref target="Stein2021" for
mat="default"/></li>
</ul>
</section>
<section anchor="discussions" numbered="true" toc="default">
<name>Workshop Topics and Discussion</name>
<t>The agenda for the three-day workshop was broken into four separate
sections that each played a role in framing the discussions. The sections that each played a role in framing the discussions. The
workshop started with a series of Introduction and problem space workshop started with a series of introduction and problem space
presentations {introduction-section}, followed by metrics considerations presentations (<xref target="introduction-section"/>), followed by metrics consi
<xref target="discussion-metrics"/>, cross layer considerations derations
<xref target="discussions-cross-layer"/> and a synthesis discussion <xref target (<xref target="discussion-metrics" format="default"/>), cross-layer consideratio
="synthesis"/>. ns
(<xref target="discussions-cross-layer" format="default"/>), and a synthesis dis
cussion (<xref target="synthesis" format="default"/>).
After the four subsections concluded, a follow-on discussion was held After the four subsections concluded, a follow-on discussion was held
to draw conclusions that could be agreed upon by workshop participants to draw conclusions that could be agreed upon by workshop participants
(<xref target="conclusions"/>).</t> (<xref target="conclusions" format="default"/>).</t>
<section anchor="introduction-section" numbered="true" toc="default">
<section anchor="introduction-section" title="Introduction and overviews"> <name>Introduction and Overviews</name>
<t>The workshop started with a broad focus on the state of user Quality
<t>The workshop started with a broad focus on the state of user Quality of Service (QoS) and Quality of Experience (QoE) on the Internet today.
of Service (QoS) and quality of experience (QoE) on the Internet today.
The goal of the introductory talks was to set the stage for the The goal of the introductory talks was to set the stage for the
workshop by describing both the problem space and the current workshop by describing both the problem space and the current
solutions in place and their limitations.</t> solutions in place and their limitations.</t>
<t>The introduction presentations provided views of existing QoS and QoE
<t>The introduction presentations provided views of existing QoS and QoE
measurements and their effectiveness. Also discussed was the measurements and their effectiveness. Also discussed was the
interaction between multiple users within the network, as well as the interaction between multiple users within the network, as well as the
interaction between multiple layers of the OSI stack. Vint Cerf interaction between multiple layers of the OSI stack. Vint Cerf
provided a key note describing the history and importance of the provided a keynote describing the history and importance of the
topic.</t> topic.</t>
<section anchor="dicsucssion-intro-keynote" numbered="true" toc="default
<section anchor="dicsucssion-intro-keynote" title="Key points from the keynote b ">
y Vint Cerf"> <name>Key Points from the Keynote by Vint Cerf</name>
<t>We may be operating in a networking space with dramatically differe
<t>We may be operating in a networking space with dramatically different nt
parameters compared to 30 years ago. This differentiation justifies parameters compared to 30 years ago. This differentiation justifies
re-considering not only the importance of one metric over the other, reconsidering not only the importance of one metric over the other
but also re-considering the entire metaphor.</t> but also reconsidering the entire metaphor.</t>
<t>It is time for the experts to look at not only adjusting TCP but
<t>It is time for the experts to look at not only at adjusting TCP, but also exploring other protocols, such as QUIC has done lately. It's
also at exploring other protocols, such as QUIC has done lately. It’s
important that we feel free to consider alternatives to TCP. TCP is important that we feel free to consider alternatives to TCP. TCP is
not a teddy bear, and one should not be afraid to replace it with a not a teddy bear, and one should not be afraid to replace it with a
transport later with better properties that better benefits its users.</t> transport layer with better properties that better benefit its users.</t>
<t>A suggestion: we should consider exercises to identify desirable
<t>A suggestion: we should consider exercises to identify desirable
properties. As we are looking at the parametric spaces, one can properties. As we are looking at the parametric spaces, one can
identify “desirable properties”, as opposed to “fundamental identify "desirable properties", as opposed to "fundamental
properties”, for example a low-latency property. An example coming properties", for example, a low-latency property. An example coming
from ARPA: you want to know where the missile is now, not where it from the Advanced Research Projects Agency (ARPA): you want to know where the mi
ssile is now, not where it
was. Understanding drives particular parameter creation and selection was. Understanding drives particular parameter creation and selection
in the design space.</t> in the design space.</t>
<t>When parameter values are changed in extreme, such as connectivenes
<t>When parameter values are changed in extreme, such as connectiveness, s,
alternative designs will emerge. One case study of note is the alternative designs will emerge. One case study of note is the
interplanetary protocol, where “ping” is no long indicative of interplanetary protocol, where "ping" is no longer indicative of
anything useful. While we look at responsiveness, we should not ignore anything useful. While we look at responsiveness, we should not ignore
connectivity.</t> connectivity.</t>
<t>Unfortunately, maintaining backward compatibility is painful. The w
<t>Unfortunately, maintaining backward compatibility is painful. The work ork
on designing IPv6 so as to transition from IPv4 could have been done on designing IPv6 so as to transition from IPv4 could have been done
better if the backward compatibility was considered. This is too late better if the backward compatibility was considered.
for IPv6, but this problem space is not too late for the future laying It is too late for IPv6, but it is not too late to consider this issue for poten
problems.</t> tial future problems.</t>
<t>IPv6 is still not implemented fully everywhere. It's been a long r
<t>IPv6 is still not implemented fully everywhere. It’s been a long road oad
to deployment since starting work in 1996, and we are still not to deployment since starting work in 1996, and we are still not
there. In 1996, the thinking was that it was quite easy to implement there. In 1996, the thinking was that it was quite easy to implement
IPv6, but that failed to hold true. In 1996 the dot-com boom began, IPv6, but that failed to hold true. In 1996, the dot-com boom began,
with lots of money was spent quickly, and the moment was not caught in where a lot of money was spent quickly, and the moment was not caught in
time while the market expanded exponentially. This should serve as a time while the market expanded exponentially. This should serve as a
cautionary tale.</t> cautionary tale.</t>
<t>One last point: consider performance across multiple hops in the
<t>One last point: consider performance across multiple hops in the Internet. We've not seen many end-to-end metrics, as successfully
Internet. We’ve not seen many end-to-end metrics, as successfully
developing end-to-end measurements across different network and developing end-to-end measurements across different network and
business boundaries is quite hard to achieve. A good question to ask business boundaries is quite hard to achieve. A good question to ask
when developing new protocols is “will the new protocol work across when developing new protocols is "will the new protocol work across
multiple network hops?”</t> multiple network hops?"</t>
<t>Multi-hop networks are being gradually replaced by humongous, flat
<t>Multi-hop networks are being gradually replaced by humongous, flat
networks with sufficient connectivity between operators so that networks with sufficient connectivity between operators so that
systems become 1 hop or 2 hop at most away from each other systems become 1 hop, or 2 hops at most, away from each other
(e.g. Google, Facebook, Amazon). The fundamental architecture of the (e.g., Google, Facebook, and Amazon). The fundamental architecture of the
Internet is changing.</t> Internet is changing.</t>
</section>
</section> <section anchor="discussion-introductions" numbered="true" toc="default"
<section anchor="discussion-introductions" title="Introductory talks"> >
<name>Introductory Talks</name>
<t>The Internet is a shared network, built on the IP protocols using <t>The Internet is a shared network built on IP protocols using
packet-switching to interconnect multiple autonomous networks. The packet switching to interconnect multiple autonomous networks. The
Internet’s departure from circuit-switching technologies allowed it to Internet's departure from circuit-switching technologies allowed it to
scale beyond any other known network design. On the other hand, the scale beyond any other known network design. On the other hand, the
lack of in-network regulation made it difficult to ensure the best lack of in-network regulation made it difficult to ensure the best
experience for every user.</t> experience for every user.</t>
<t>As Internet use cases continue to expand, it becomes increasingly m
<t>As Internet use cases continue to expand, it becomes increasingly more ore
difficult to predict which network characteristics correlate with difficult to predict which network characteristics correlate with
better user experiences. Different application classes, e.g., video better user experiences. Different application classes, e.g., video
streaming and teleconferencing, can affect user experience in complex streaming and teleconferencing, can affect user experience in ways that are comp
and difficult to measure ways. Internet utilization shifts rapidly lex
during the course of each day, week and year, which further and difficult to measure. Internet utilization shifts rapidly
during the course of each day, week, and year, which further
complicates identifying key metrics capable of predicting a good user complicates identifying key metrics capable of predicting a good user
experience.</t> experience.</t>
<t>QoS initiatives attempted to overcome these
<t>Quality of Service (QoS) initiatives attempted to overcome these
difficulties by strictly prioritizing different types of difficulties by strictly prioritizing different types of
traffic. However, QoS metrics do not always correlate with user traffic. However, QoS metrics do not always correlate with user
experience. The utility of the QoS metric is further limited by the experience. The utility of the QoS metric is further limited by the
difficulties in building solutions with the desired QoS difficulties in building solutions with the desired QoS
characteristics.</t> characteristics.</t>
<t>QoE initiatives attempted to integrate the
<t>Quality of Experience (QoE) initiatives attempted to integrate the psychological aspects of how quality is perceived and create
psychological aspects of how quality is perceived, and created
statistical models designed to optimize the user experience. Despite statistical models designed to optimize the user experience. Despite
these high modeling efforts, the QoE approach proved beneficial in these high modeling efforts, the QoE approach proved beneficial in
certain application classes. Unfortunately, generalizing the models certain application classes. Unfortunately, generalizing the models
proved to be difficult, and the question of how different applications proved to be difficult, and the question of how different applications
affect each other when sharing the same network remains an open problem.</t> affect each other when sharing the same network remains an open problem.</t>
<t>The industry's focus on giving the end user more throughput/bandwid
<t>The industry’s focus on giving the end-user more throughput/bandwidth th
led to remarkable advances. In many places around the world, a home led to remarkable advances. In many places around the world, a home
user enjoys gigabit speeds to their Internet Service Provider. This user enjoys gigabit speeds to their ISP. This
is so remarkable that it would have been brushed off as science is so remarkable that it would have been brushed off as science
fiction a decade ago. However, the focus on increased capacity came at fiction a decade ago. However, the focus on increased capacity came at
the expense of neglecting another important core metric: latency. As the expense of neglecting another important core metric: latency. As
a result, end-users whose experience is negatively affected by high a result, end users whose experience is negatively affected by high
latency were advised to upgrade their equipment to get more latency were advised to upgrade their equipment to get more
throughput instead. <xref target="MacMillian2021"/> showed that sometimes such a n throughput instead. <xref target="MacMillian2021" format="default"/> showed that sometimes such an
upgrade can lead to latency improvements, due to the economical upgrade can lead to latency improvements, due to the economical
reasons of overselling the “value-priced” data plans.</t> reasons of overselling the "value-priced" data plans.</t>
<t>As the industry continued to give end users more throughput, while
<t>As the industry continued to give end users more throughput, while
mostly neglecting latency concerns, application designs started to mostly neglecting latency concerns, application designs started to
employ various latency and short service disruption hiding techniques. employ various latency and short service disruption hiding techniques.
For example, a user’s experience of web browser performance is closely For example, a user's web browser performance experience is closely
tired to the content in the browser’s local cache. While such tied to the content in the browser's local cache. While such
techniques can clearly improve the user experience when using stale techniques can clearly improve the user experience when using stale
data is possible, this development further decouples user experience data is possible, this development further decouples user experience
from core metrics.</t> from core metrics.</t>
<t>In the most recent 10 years, efforts by Dave Taht and the bufferblo
<t>In the most recent 10 years, efforts by Dave Taht and the bufferbloat at
society had led to significant progress updating queuing algorithms to society have led to significant progress in updating queuing algorithms to
reduce latencies under load compared to simipler FIFO reduce latencies under load compared to simpler FIFO
queues. Unfortunately, the home router industry has yet to implement queues. Unfortunately, the home router industry has yet to implement
these algorithms, mostly due to marketing and cost concerns. Most home these algorithms, mostly due to marketing and cost concerns. Most home
router manufacturers depend on System on a Chip (SoC) acceleration to router manufacturers depend on System on a Chip (SoC) acceleration to
create products with a desired throughput. SoC manufacturers opt for create products with a desired throughput. SoC manufacturers opt for
simpler algorithms and aggressive aggregation, reasoning that a simpler algorithms and aggressive aggregation, reasoning that a
higher-throughput chip will have guaranteed demand. Because consumers higher-throughput chip will have guaranteed demand. Because consumers
are offered choices primarily among different high throughput devices, are offered choices primarily among different high-throughput devices,
the perception that a higher throughput leads to higher a quality of the perception that a higher throughput leads to higher a QoS continues to stren
service continues to strengthen.</t> gthen.</t>
<t>The home router is not the only place that can benefit from clearer
<t>The home router is not the only place that can benefit from clearer indications of acceptable performance for users.
indications of acceptable performance for users. Since users perceive Since users perceive the Internet via the lens of applications, it
the Internet via the lens of applications, its important to appeal to is important that we call upon application vendors to adopt solutions
the application vendors that they should adopt solutions that stress that stress lower latencies. Unfortunately, while bandwidth is straightforward
lower latencies. Unfortunately, while bandwidth is straightforward to to
measure, responsiveness is trickier. Many applications have found a measure, responsiveness is trickier. Many applications have found a
set of metrics which are helpful to their realm, but do not generalize set of metrics that are helpful to their realm but do not generalize
well and cannot become universally applicable. Furthermore, due to the well and cannot become universally applicable. Furthermore, due to the
highly competitive application space, vendors may have economic highly competitive application space, vendors may have economic
reasons to avoid sharing their most useful metrics.</t> reasons to avoid sharing their most useful metrics.</t>
<!-- incomplete: suggest we leave it out
Finally, the Internet infrastructure that connects the applictions to the users
is yet another place where better measurements may help driving towards the bett
er Internet.
</section> </section>
<section anchor="discussion-introductions-summary" title="Introductory talks - k <section anchor="discussion-introductions-summary" numbered="true" toc="
ey points"> default">
<name>Introductory Talks - Key Points</name>
<t><list style="numbers"> <ol spacing="normal" type="1">
<t>Measuring bandwidth is necessary, but is not alone sufficient.</t> <li>Measuring bandwidth is necessary but is not alone sufficient.</li
<t>In many cases, Internet users don’t need more bandwidth, but rather >
need “better bandwidth” – i.e., they need other connectivity improvements.</t> <li>In many cases, Internet users don't need more bandwidth but rath
<t>Users perceive the quality of their Internet connection based er
need "better bandwidth", i.e., they need other connectivity improvem
ents.</li>
<li>Users perceive the quality of their Internet connection based
on the applications they use, which are affected by a combination on the applications they use, which are affected by a combination
of factors. There’s little value in exposing a typical user to the of factors. There's little value in exposing a typical user to the
entire spectrum of possible reasons for the poor performance entire spectrum of possible reasons for the poor performance
perceived in their application-centric view.</t> perceived in their application-centric view.</li>
<t>Many factors affecting user experience are outside the users’ <li>Many factors affecting user experience are outside the users'
sphere of control. It’s unclear whether exposing users to these sphere of control. It's unclear whether exposing users to these
other factors will help users understand the state of their network other factors will help them understand the state of their network
performance. In general, users prefer simple, categorical performance. In general, users prefer simple, categorical
choices (e.g. “good”, “better”, and “best” options).</t> choices (e.g., "good", "better", and "best" options).</li>
<t>The Internet content market is highly competitive, and many <li>The Internet content market is highly competitive, and many
applications develop their own “secret sauce.”</t> applications develop their own "secret sauce".</li>
</list></t> </ol>
</section>
</section> </section>
</section> <section anchor="discussion-metrics" numbered="true" toc="default">
<section anchor="discussion-metrics" title="Metrics considerations"> <name>Metrics Considerations</name>
<t>In the second agenda section, the workshop continued its discussion
<t>In the second agenda section, the workshop continued its discussion
about metrics that can be used instead of or in addition to available about metrics that can be used instead of or in addition to available
bandwidth. Several workshop attendees presented deep-dive studies on bandwidth. Several workshop attendees presented deep-dive studies on
measurement methodology.</t> measurement methodology.</t>
<section anchor="common-performance-metrics" numbered="true" toc="defaul
<section anchor="common-performance-metrics" title="Common performance metrics"> t">
<name>Common Performance Metrics</name>
<t>Losing Internet access entirely is, of course, the worst user <t>Losing Internet access entirely is, of course, the worst user
experience. Unfortunately, unless rebooting the home router restores experience. Unfortunately, unless rebooting the home router restores
connectivity, there is little a user can do other than contacting connectivity, there is little a user can do other than contacting
their service provider. Nevertheless, there is value in the systematic their service provider. Nevertheless, there is value in the systematic
collection of availability metrics on the client side: these can help collection of availability metrics on the client side; these can help
the user’s ISP localize and resolve issues faster, while enabling the user's ISP localize and resolve issues faster while enabling
users to better choose between ISPs. One can measure availability users to better choose between ISPs. One can measure availability
directly by simply attempting connections from the client-side to directly by simply attempting connections from the client side to
distant locations of interest. For example, Ookla’s (<xref target="Speedtest"/>) distant locations of interest. For example, Ookla's <xref target="Speedtest" for
mat="default"/>
uses a large number of Android devices to measure network and cellular uses a large number of Android devices to measure network and cellular
availability around the globe. Ookla collects hundreds of millions of availability around the globe. Ookla collects hundreds of millions of
data points per day, and uses these for accurate availability data points per day and uses these for accurate availability
reporting. An alternative approach is to derive availability from the reporting. An alternative approach is to derive availability from the
failure rates of other tests. For example, <xref target="FCC_MBA"/> failure rates of other tests. For example, <xref target="FCC_MBA" format="defaul
<xref target="FCC_MBA_methodology"/> uses thousands of off-the shelf routers, t"/> and
called “Whiteboxes”, with measurement software developed by <xref target="FCC_MBA_methodology" format="default"/> use thousands
<xref target="SamKnows"/>. These Whiteboxes perform an array of network tests an of off-the-shelf routers, with measurement software developed by
d <xref target="SamKnows" format="default"/>. These routers perform an array of ne
report availability based whether test connections were successful or twork tests and
report availability based on whether test connections were successful or
not.</t> not.</t>
<t>Measuring available capacity can be helpful to end users, but it is
<t>Measuring available capacity can be helpful to end-users, but it is
even more valuable for service providers and application even more valuable for service providers and application
developers. High-definition video streaming requires significantly developers. High-definition video streaming requires significantly
more capacity than any other type of traffic. At the time of the more capacity than any other type of traffic. At the time of the
workshop, video traffic constituted 90% of overall Internet traffic workshop, video traffic constituted 90% of overall Internet traffic
and contributed to 95% of the revenues from monetization (via and contributed to 95% of the revenues from monetization (via
subscriptions, fees, or ads). As a result, video streaming services, subscriptions, fees, or ads). As a result, video streaming services,
such as Netflix, need to continuously cope with rapid changes in such as Netflix, need to continuously cope with rapid changes in
available capacity. The ability to measure available capacity in available capacity. The ability to measure available capacity in
real-time leverages the different adaptive bitrate (ABR) compression real time leverages the different adaptive bitrate (ABR) compression
algorithms to ensure the best possible user experience. Measuring algorithms to ensure the best possible user experience. Measuring
aggregated capacity demand allows Internet Service Provider’s to be aggregated capacity demand allows ISPs to be
ready for traffic spikes. For example, during the end-of-year holiday ready for traffic spikes. For example, during the end-of-year holiday
season, the global demand for capacity has been shown to be 5-7 times season, the global demand for capacity has been shown to be 5-7 times
higher than during other seasons. For end-users, knowledge of their higher than during other seasons. For end users, knowledge of their
capacity needs can help them select the best data plan given their capacity needs can help them select the best data plan given their
intended usage. In many cases, however, end-users have more than intended usage. In many cases, however, end users have more than
enough capacity and adding more bandwidth will not improve their enough capacity, and adding more bandwidth will not improve their
experience – after a point it is no longer the limiting factor in experience -- after a point, it is no longer the limiting factor in
user experience. Finally, the ability to differentiate between the user experience. Finally, the ability to differentiate between the
“throughput” and the “goodput” can be helpful in identifying when the "throughput" and the "goodput" can be helpful in identifying when the
network is saturated.</t> network is saturated.</t>
<t>In measuring network quality, latency is defined as the time it tak
<t>In measuring network quality, latency is defined as the time it takes es
a packet to traverse a network path from one end to the other. At the a packet to traverse a network path from one end to the other. At the
time of this report, users in many places worldwide can enjoy Internet time of this report, users in many places worldwide can enjoy Internet
access that has adequately high capacity and availability for their access that has adequately high capacity and availability for their
current needs. For these users, latency improvements rather than current needs. For these users, latency improvements, rather than
bandwidth improvements can lead to the most significant improvements bandwidth improvements, can lead to the most significant improvements
in quality of experience. The established latency metric is a in QoE. The established latency metric is a
round-trip time (RTT), commonly measured in milliseconds. However, round-trip time (RTT), commonly measured in milliseconds. However,
users often find RTT values unintuitive since, unlike other users often find RTT values unintuitive since, unlike other
performance metrics, high RTT values indicate poor latency and users performance metrics, high RTT values indicate poor latency and users
typically understand higher scores to be better. To address this, typically understand higher scores to be better. To address this,
<xref target="Paasch2021"/> and <xref target="Mathis2021"/> presented an inverse <xref target="I-D.cpaasch-ippm-responsiveness" format="default"/> and <xref targ
metric, called et="Mathis2021" format="default"/> present an inverse metric, called
“Round-trips per minute” (RPM).</t> "Round-trips Per Minute" (RPM).</t>
<t>There is an important distinction between "idle latency" and "laten
<t>There is an important distinction between “idle latency” and “latency cy
under working conditions.” The former is measured when the network is under working conditions". The former is measured when the network is
underused and reflects a best-case scenario. The latter is measured underused and reflects a best-case scenario. The latter is measured
when the network is under a typical workload. Until recently, typical when the network is under a typical workload. Until recently, typical
tools reported a network’s idle latency, which can be misleading. For tools reported a network's idle latency, which can be misleading. For
example, data presented at the workshop shows that idle latencies can example, data presented at the workshop shows that idle latencies can
be up to 25 times lower than the latency under typical working be up to 25 times lower than the latency under typical working
loads. Because of this, it is essential to make a clear distinction loads. Because of this, it is essential to make a clear distinction
between the two when presenting latency to end-users.</t> between the two when presenting latency to end users.</t>
<t>Data shows that rapid changes in capacity affect
<t>Data shows that rapid changes in capacity affect latency. <xref target="Foulkes2021" format="default"/> attempts to quantify how
latency. <xref target="Foulkes2021"/> attempts to quantify how often a rapid cha often a rapid change
nge in capacity can cause network connectivity to become "unstable" (i.e.,
in capacity can cause network connectivity to become “unstable” (i.e.,
having high latency with very little throughput). Such changes in having high latency with very little throughput). Such changes in
capacity can be caused by infrastructure failures, but are much more capacity can be caused by infrastructure failures but are much more
often caused by in-network phenomena, like changing traffic often caused by in-network phenomena, like changing traffic
engineering policies or rapid changes in cross-traffic.</t> engineering policies or rapid changes in cross-traffic.</t>
<t>Data presented at the workshop shows that 36% of measured lines hav
<t>Data presented at the workshop shows that 36% of measured lines have e
capacity metrics that vary by more than 10% throughout the day and capacity metrics that vary by more than 10% throughout the day and
across multiple days. These differences are caused by many variables, across multiple days. These differences are caused by many variables,
including local connectivity methods (WiFi vs. Ethernet), competing including local connectivity methods (Wi-Fi vs. Ethernet), competing
LAN traffic, device load/configuration, time of day and local LAN traffic, device load/configuration, time of day, and local
loop/backhaul capacity. These factor variations make measuring loop/backhaul capacity. These factor variations make measuring
capacity using only an end-user device or other end-network capacity using only an end-user device or other end-network
measurement difficult. A network router seeing aggregated traffic from measurement difficult. A network router seeing aggregated traffic from
multiple devices provides a better vantage point for capacity multiple devices provides a better vantage point for capacity
measurements. Such a test can account for the totality of local measurements. Such a test can account for the totality of local
traffic and perform an independent capacity test. However, various traffic and perform an independent capacity test. However, various
factors might still limit the accuracy of such a test. Accurate factors might still limit the accuracy of such a test. Accurate
capacity measurement requires multiple samples.</t> capacity measurement requires multiple samples.</t>
<t>As users perceive the Internet through the lens of applications, it
<t>As users perceive the Internet through the lens of applications, it
may be difficult to correlate changes in capacity and latency with the may be difficult to correlate changes in capacity and latency with the
quality of the end-user experience. For example, web browsers rely on quality of the end-user experience. For example, web browsers rely on
cached page versions to shorten page load times and mitigate cached page versions to shorten page load times and mitigate
connectivity losses. In addition, social networking applications often connectivity losses. In addition, social networking applications often
rely on pre-fetching their “feed” items. These techniques make the rely on prefetching their "feed" items. These techniques make the
core in-network metrics less indicative of the users’ experience and core in-network metrics less indicative of the users' experience and
necessitates collecting data in-application.</t> necessitates collecting data from the end-user applications themselves.</t>
<t>It is helpful to distinguish between applications that operate on a
<t>It is helpful to distinguish between applications that operate on a "fixed latency budget" from those that have more tolerance to latency
“fixed latency budget” from those that have more tolerance to latency
variance. Cloud gaming serves as an example application that requires variance. Cloud gaming serves as an example application that requires
a “fixed latency budget”, as a sudden latency spike can decide the a "fixed latency budget", as a sudden latency spike can decide the
“win/lose” ratio for a player. Companies that compete in the lucrative "win/lose" ratio for a player. Companies that compete in the lucrative
cloud gaming market make significant infrastructure investments, such cloud gaming market make significant infrastructure investments, such
as buiding entire datacenters closer to their users. These data as building entire data centers closer to their users. These data
centers highlight the economic benefits that lower numbers of latency centers highlight the economic benefit that lower numbers of latency
spikes outweighs the associated deployment costs. On the other hand, spikes outweigh the associated deployment costs. On the other hand,
applications that are more tolerant to latency spikes can continue to applications that are more tolerant to latency spikes can continue to
operate reasonably well through short spikes. Yet even those operate reasonably well through short spikes. Yet, even those
applications can benefit from consistently low latency depending on applications can benefit from consistently low latency depending on
usage shifts. For example, Video-on-Demand (VOD) apps can work usage shifts. For example, Video-on-Demand (VOD) apps can work
reasonably well when the video is consumed linearly, but once the user reasonably well when the video is consumed linearly, but once the user
tries to “switch a channel”, or to “skip ahead”, the user experience tries to "switch a channel" or to "skip ahead", the user experience
suffers unless the latency is sufficiently low.</t> suffers unless the latency is sufficiently low.</t>
<t>Finally, as applications continue to evolve, in-application metrics
<t>Finally, as applications continue to evolve, in-application metrics
are gaining in importance. For example, VOD applications can assess are gaining in importance. For example, VOD applications can assess
the quality of experience by application-specific metrics such as the QoE by application-specific metrics, such as
whether the video player is able to use the highest possible whether the video player is able to use the highest possible
resolution, identify when the video is smooth or freezing, or other resolution, identifying when the video is smooth or freezing, or other
similar metrics. Application developers can then effectively use these similar metrics. Application developers can then effectively use these
metrics to prioritize future work. All popular video platforms metrics to prioritize future work. All popular video platforms
(Youtube, Instagram, Netflix, and others) have developed frameworks to (YouTube, Instagram, Netflix, and others) have developed frameworks to
collect and analyze VOD metrics at scale. One example is the Scuba collect and analyze VOD metrics at scale. One example is the Scuba
framework used by Meta <xref target="Scuba"/>.</t> framework used by Meta <xref target="Scuba" format="default"/>.</t>
<t>Unfortunately, in-application metrics can be challenging to use
<t>Unfortunately, the in-application metrics can be challenging to use for comparative research purposes. First, different applications
for comparative research purposes. Firstly, different applications
often use different metrics to measure the same phenomena. For often use different metrics to measure the same phenomena. For
example, application A may measure the smoothness of video via “mean example, application A may measure the smoothness of video via "mean
time to re-buffer”, while application B may rely on the “probability time to rebuffer", while application B may rely on the "probability
of re-buffering per second” for the same purpose. A different of rebuffering per second" for the same purpose. A different
challenge with in-application metrics is VOD is a significant source challenge with in-application metrics is that VOD is a significant source
of revenue for companies such as YouTube, Facebook, and Netflix, of revenue for companies, such as YouTube, Facebook, and Netflix,
placing a proprietary incentive against exchanging the in-application placing a proprietary incentive against exchanging the in-application
data. A final concern centers on the privacy issues resulting from data. A final concern centers on the privacy issues resulting from
in-application metrics that accurately describe the activities and in-application metrics that accurately describe the activities and
preferences of an individual end-user.</t> preferences of an individual end user.</t>
</section>
</section> <section anchor="availability-metrics" numbered="true" toc="default">
<section anchor="availability-metrics" title="Availability metrics"> <name>Availability Metrics</name>
<t>Availability is simply defined as whether or not a packet can be se
<t>Availability is simply defined as whether or not a packet can be sent nt
and then received by its intended recipient. Availability is naively and then received by its intended recipient. Availability is naively
thought to be the simplest to measure, but is more complex when thought to be the simplest to measure, but it is more complex when
considering that continual, instantaneous measurements would be needed considering that continual, instantaneous measurements would be needed
to detect the smallest of outages. Also difficult is determining the to detect the smallest of outages. Also difficult is determining the
root cause of infallibility: was the user’s line down, something in root cause of infallibility: was the user's line down, was something in
the middle of the network or was it the service with which the user the middle of the network, or was it the service with which the user
was attempting to communicate.</t> was attempting to communicate?</t>
</section>
</section> <section anchor="capacity-metrics" numbered="true" toc="default">
<section anchor="capacity-metrics" title="Capacity metrics"> <name>Capacity Metrics</name>
<t>If the network capacity does not meet user demands, the network qua
<t>If the network capacity does not meet the user demands, the network quality lity
will be impacted. Once the capacity meets the demands, increasing capacity will be impacted. Once the capacity meets the demands, increasing capacity
won’t lead to further quality improvements.</t> won't lead to further quality improvements.</t>
<t>The actual network connection capacity is determined by the equipme
<t>The actual network connection capacity is determined by the equipment and the nt and the
lines along the network path, and it varies throughout the day and across lines along the network path, and it varies throughout the day and across
multiple days. Studies involving DSL lines in North America indicate that over multiple days. Studies involving DSL lines in North America indicate that over
30% of the DSL lines have capacity metrics that vary by more than 10% 30% of the DSL lines have capacity metrics that vary by more than 10%
throughout the day and accross multiple days.</t> throughout the day and across multiple days.</t>
<t>Some factors that affect the actual capacity are:</t>
<t>Some factors that affect the actual capacity are:</t> <ol spacing="normal" type="1">
<li>Presence of a competing traffic, either in the LAN or in the WAN
<t><list style="numbers">
<t>Presence of a competing traffic, either in the LAN or in the WAN
environments. In the LAN setting, the competing traffic reflects the environments. In the LAN setting, the competing traffic reflects the
multiple devices that share the Internet connection. In the WAN setting the multiple devices that share the Internet connection. In the WAN setting, the
competing traffic often originates from the unrelated network flows that competing traffic often originates from the unrelated network flows that
happen to share the same network path.</t> happen to share the same network path.</li>
<t>Capabilities of the equipment along the path of the network connection, <li>Capabilities of the equipment along the path of the network conn
ection,
including the data transfer rate and the amount of memory used for including the data transfer rate and the amount of memory used for
buffering.</t> buffering.</li>
<t>Active traffic management measures, such as traffic shapers and policers <li>Active traffic management measures, such as traffic shapers and
that are often used by the network providers.</t> policers
</list></t> that are often used by the network providers.</li>
</ol>
<t>There are other factors that can negatively affect the actual line capacities <t>There are other factors that can negatively affect the actual line
.</t> capacities.</t>
<t>The user demands of the traffic follow the usage patterns and prefe
<t>The user demands of the traffic follow the usage patterns and preferences of rences of
the particular users. For example, large data transfers can use any available the particular users. For example, large data transfers can use any available
capacity, while the media streaming applicaitons require limited capacity to capacity, while the media streaming applications require limited capacity to
function correclty. Video-conferencing applications typically need less function correctly. Videoconferencing applications typically need less
capacity than high-definition video streaming.</t> capacity than high-definition video streaming.</t>
</section>
</section> <section anchor="latency-metrics" numbered="true" toc="default">
<section anchor="latency-metrics" title="Latency metrics"> <name>Latency Metrics</name>
<t>End-to-end latency is the time that a particular packet takes to tr
<t>End-to-end latency is the time that a particular packet takes to traverse the averse the
network path from the user to their destination and back. The end-to-end network path from the user to their destination and back. The end-to-end
latency comprises several components:</t> latency comprises several components:</t>
<ol spacing="normal" type="1">
<t><list style="numbers"> <li>The propagation delay, which reflects the path distance and the
<t>The propagation delay, which reflects the path distance and the individual individual
link technologies (e.g. fibre vs satellite). The propagation doesn’t depend link technologies (e.g., fiber vs. satellite). The propagation doesn't depend
on the utilization of the network, to the extent that the network path on the utilization of the network, to the extent that the network path
remains constant.</t> remains constant.</li>
<t>The buffering delay, which reflects the time segments spend in the memory o <li>The buffering delay, which reflects the time segments spent in t
f he memory of
the network equipment that connect the individual network links, as well as the network equipment that connect the individual network links, as well as
in the memory of the transmitting endpoint. The buffering delay depends on in the memory of the transmitting endpoint. The buffering delay depends on
the network utilization, as well as on the algorithms that govern the queued seg the network utilization, as well as on the algorithms that govern the queued seg
ments.</t> ments.</li>
<t>The transport protocol delays, which reflects the time spent in <li>The transport protocol delays, which reflect the time spent in
retransmission and reassembly, as well as the time spent when the transport retransmission and reassembly, as well as the time spent when the transport
is “head-of-line blocked.”</t> is "head-of-line blocked".</li>
<t>Some of the workshop sumbissions have explicitly called out the application <li>Some of the workshop submissions that have explicitly called out
delay, which reflects the inefficiencies in the application layer.</t> the application
</list></t> delay, which reflects the inefficiencies in the application layer.</li>
</ol>
<t>Traditionally, end-to-end latency is measured when the network is <t>Typically, end-to-end latency is measured when the network is
idle. Results of such measurements reflect mostly the propagation idle. Results of such measurements mostly reflect the propagation
delay, but not other kinds of delay. This report uses the term “idle delay but not other kinds of delay. This report uses the term "idle
latency” to refer to results achieved under idle network conditions.</t> latency" to refer to results achieved under idle network conditions.</t>
<t>Alternatively, if the latency is measured when the network is under
<t>Alternatively, if the latency is measured when the network is under
its typical working conditions, the results reflect multiple types of its typical working conditions, the results reflect multiple types of
delays. This report uses the term “working latency” to refer to such delays. This report uses the term "working latency" to refer to such
results. Other sources use the term “latency under load” (LUL) as a results. Other sources use the term "latency under load" (LUL) as a
synonym.</t> synonym.</t>
<t>Data presented at the workshop reveals a substantial difference
<t>Data presented at the workshop reveals a substantial difference
between the idle latency and the working latency. Depending on the between the idle latency and the working latency. Depending on the
traffic direciton and the technology type, the working latency is traffic direction and the technology type, the working latency is
between 6 to 25 times higher than the idle latency:</t> between 6 to 25 times higher than the idle latency:</t>
<table align="center">
<texttable> <thead>
<ttcol align='left'>Direction</ttcol> <tr>
<ttcol align='left'>Technology type</ttcol> <th align="left">Direction</th>
<ttcol align='left'>Working latency</ttcol> <th align="left">Technology Type</th>
<ttcol align='left'>Idle latency</ttcol> <th align="left">Working Latency</th>
<ttcol align='left'>Working - Idle difference</ttcol> <th align="left">Idle Latency</th>
<ttcol align='left'>Working / Idle ratio</ttcol> <th align="left">Working - Idle Difference</th>
<c>Downstream</c> <th align="left">Working / Idle Ratio</th>
<c>FTTH</c> </tr>
<c>148</c> </thead>
<c>10</c> <tbody>
<c>138</c> <tr>
<c>15</c> <td align="left">Downstream</td>
<c>Dowstream</c> <td align="left">FTTH</td>
<c>Cable</c> <td align="left">148</td>
<c>103</c> <td align="left">10</td>
<c>13</c> <td align="left">138</td>
<c>90</c> <td align="left">15</td>
<c>8</c> </tr>
<c>Downstream</c> <tr>
<c>DSL</c> <td align="left">Downstream</td>
<c>194</c> <td align="left">Cable</td>
<c>10</c> <td align="left">103</td>
<c>184</c> <td align="left">13</td>
<c>19</c> <td align="left">90</td>
<c>Upstream</c> <td align="left">8</td>
<c>FTTH</c> </tr>
<c>207</c> <tr>
<c>12</c> <td align="left">Downstream</td>
<c>195</c> <td align="left">DSL</td>
<c>17</c> <td align="left">194</td>
<c>Upstream</c> <td align="left">10</td>
<c>Cable</c> <td align="left">184</td>
<c>176</c> <td align="left">19</td>
<c>27</c> </tr>
<c>149</c> <tr>
<c>6</c> <td align="left">Upstream</td>
<c>Upstream</c> <td align="left">FTTH</td>
<c>DSL</c> <td align="left">207</td>
<c>686</c> <td align="left">12</td>
<c>27</c> <td align="left">195</td>
<c>659</c> <td align="left">17</td>
<c>25</c> </tr>
</texttable> <tr>
<td align="left">Upstream</td>
<t>While historically the tooling available for measuring latency focused <td align="left">Cable</td>
<td align="left">176</td>
<td align="left">27</td>
<td align="left">149</td>
<td align="left">6</td>
</tr>
<tr>
<td align="left">Upstream</td>
<td align="left">DSL</td>
<td align="left">686</td>
<td align="left">27</td>
<td align="left">659</td>
<td align="left">25</td>
</tr>
</tbody>
</table>
<t>While historically the tooling available for measuring latency focu
sed
on measuring the idle latency, there is a trend in the industry to on measuring the idle latency, there is a trend in the industry to
start measuring the working latency as well, start measuring the working latency as well,
e.g. Apple’s <xref target="NetworkQuality"/>.</t> e.g., Apple's <xref target="NetworkQuality" format="default"/>.</t>
<!-- missing the RPM and goodput text from PR#8 - maybe here? -->
</section> </section>
<section anchor="measurement-case-studies" title="Measurement case studies"> <section anchor="measurement-case-studies" numbered="true" toc="default"
>
<t>The participants have proposed several concrete methodologies for <name>Measurement Case Studies</name>
measuring the onetwork quality for the end users.</t> <t>The participants have proposed several concrete methodologies for
measuring the network quality for the end users.</t>
<t><xref target="Paasch2021"/> introduced a methodology for measuring working la <t><xref target="I-D.cpaasch-ippm-responsiveness" format="default"/> i
tency ntroduced a methodology for measuring working latency
from the end-user vantage point. The suggested method incrementally from the end-user vantage point. The suggested method incrementally
adds network flows between the user device and a server endpoint until adds network flows between the user device and a server endpoint until
a bottleneck capacity is reached. From these measurements, a round a bottleneck capacity is reached. From these measurements, a round-trip
trip latency is measured and reported to the end-user. The authors latency is measured and reported to the end user. The authors
chose to report results with the RPM metric. The methodology had been chose to report results with the RPM metric. The methodology had been
implemented in Apple Monterey OS.</t> implemented in Apple's macOS Monterey.</t>
<t><xref target="Mathis2021" format="default"/> applied the RPM metric
<t><xref target="Mathis2021"/> have applied the RPM metric to the results of mor to the results of more than
e than 4 billion download tests that M-Lab performed from 2010-2021. During
4 billion download tests that M-Lab performed in 2010-2021. During
this time frame, the M-Lab measurement platform underwent several this time frame, the M-Lab measurement platform underwent several
upgrades which allowed the research team to compare the effect of upgrades that allowed the research team to compare the effect of
different TCP congestion control algorithms (CCAs) on the measured different TCP congestion control algorithms (CCAs) on the measured
end-to-end latency. The study showed that the use Cubic CCA leads to end-to-end latency. The study showed that the use of cubic CCA leads to
increased working latency, which is attributed to its use of larger increased working latency, which is attributed to its use of larger
queues.</t> queues.</t>
<t><xref target="Schlinker2019" format="default"/> presented a large-s
<t><xref target="Schlinker2019"/> presented a large-scale study that aimed to cale study that aimed to
establish a correlation between goodput and quality of experience on a establish a correlation between goodput and QoE on a
large social network. The authors performed the measurements at large social network. The authors performed the measurements at
multiple data centers from which video segments of set sizes were multiple data centers from which video segments of set sizes were
streamed to a large number of end users. The authors used the goodput streamed to a large number of end users. The authors used the goodput
and throughput metrics to determine whether particular paths were and throughput metrics to determine whether particular paths were
congested.</t> congested.</t>
<t><xref target="Reed2021" format="default"/> presented the analysis o
<t><xref target="Reed2021"/> presented the analysis of working latency measureme f working latency measurements collected as part of the Measuring Broadband Amer
nts ica (MBA)
collected as part of the FCC’s “Measuring Broadband America” (MBA) program by the Federal Communication Commission (FCC). The FCC does not include
program. The FCC does not include working latency in its yearly report, working latency in its yearly report
but does offer it in the raw data files. The authors used a but does offer it in the raw data files. The authors used a
subset of the raw data to identify important differences in the subset of the raw data to identify important differences in the
working latencies across different ISPs.</t> working latencies across different ISPs.</t>
<t><xref target="MacMillian2021" format="default"/> presented analysis
<t><xref target="MacMillian2021"/> presented analysis of working latency across of working latency across
multiple service tiers. They found that, unsurprisingly, “premium” multiple service tiers. They found that, unsurprisingly, "premium"
tier users experienced lower working latency compared to a “value” tier users experienced lower working latency compared to a "value"
tier. The data demonstrated that working latency varies significantly tier. The data demonstrated that working latency varies significantly
within each tier; one possible explanation is the difference in within each tier; one possible explanation is the difference in
equipment deployed in the homes.</t> equipment deployed in the homes.</t>
<t>These studies have stressed the importance of measurement of workin
<t>These studies have stressed the importance of measurement of working g
latency. At the time of this report, many home router manufacturers latency. At the time of this report, many home router manufacturers
rely on hardware-accelerated routing which used FIFO queues. Focusing rely on hardware-accelerated routing that uses FIFO queues. Focusing
on measuring the working latency measurements on these devices, and on measuring the working latency measurements on these devices and
making the consumer aware of the effect of chosing one manufacturer making the consumer aware of the effect of choosing one manufacturer
vs. another, can help improving the home router situation. The ideal vs. another can help improve the home router situation. The ideal
test would be able to identify the working latency, and to pinpoint to test would be able to identify the working latency and pinpoint
the source of delay (home router, ISP, server side, or some network the source of the delay (home router, ISP, server side, or some network
node in between).</t> node in between).</t>
<t>Another source of high working latency comes from network routers
<t>Another source of high working latency comes from network routers exposed to cross-traffic. As <xref target="Schlinker2019" format="default"/> ind
exposed to cross-traffic. As <xref target="Schlinker2019"/> indicated, these can icated, these can
become saturated during the peak hours of the day. Systematic testing become saturated during the peak hours of the day. Systematic testing
of the working latency in routers under load can help improve both our of the working latency in routers under load can help improve both our
understanding of latency and the impact of deployed infrastructure.</t> understanding of latency and the impact of deployed infrastructure.</t>
</section>
</section> <section anchor="discussions-metrics-key-points" numbered="true" toc="de
<section anchor="discussions-metrics-key-points" title="Metrics Key Points"> fault">
<name>Metrics Key Points</name>
<t>The metrics for network quality can be roughly grouped into:</t> <t>The metrics for network quality can be roughly grouped into the fol
lowing:</t>
<t><list style="numbers"> <ol spacing="normal" type="1">
<t>Availability metrics, which indicate whether the user can access <li>Availability metrics, which indicate whether the user can access
the network at all.</t> the network at all.</li>
<t>Capacity metrics, which indicate whether the actual line capacity is <li>Capacity metrics, which indicate whether the actual line capacit
sufficient to meet the user’s demands.</t> y is
<t>Latency metrics, indicating if the user gets the data in a timely fashion.< sufficient to meet the user's demands.</li>
/t> <li>Latency metrics, which indicate if the user gets the data in a t
<t>Higher-order metrics, which include both the network metrics, such as imely fashion.</li>
inter-packet arrival time, and the applicaiton metrics, such as the mean <li>Higher-order metrics, which include both the network metrics, su
time between rebuffering for video streaming.</t> ch as
</list></t> inter-packet arrival time, and the application metrics, such as the mean
time between rebuffering for video streaming.</li>
<t>The availabiltiy metrics can be seen as derivative of either the capacity (ze </ol>
ro <t>The availability metrics can be seen as a derivative of either the
capacity (zero
capacity leading to zero availability) or the latency (infinite latency capacity leading to zero availability) or the latency (infinite latency
leading to zero availability).</t> leading to zero availability).</t>
<t>Key points from the presentations and discussions included the foll
<t>Key points from the presentations and discussions included:</t> owing:</t>
<ol spacing="normal" type="1">
<t><list style="numbers"> <li>Availability and capacity are "hygienic factors" -- unless an
<t>Availability and capacity are “hygienic factors” - unless an application is capable of using extra capacity, end users will see
application is capable of using extra capacity, end-users will see little benefit from using over-provisioned lines.</li>
little benefit from using overprovisioned lines.</t> <li>Working latency has a stronger correlation with the user experie
<t>Working latency has stronger correlation with user experience nce
than latency under an idle network load. Working latency can than latency under an idle network load. Working latency can
exceed the idle latency by order of magnitude.</t> exceed the idle latency by order of magnitude.</li>
<t>The RPM metric is a stable metric, with positive values being <li>The RPM metric is a stable metric, with positive values being
better, that may be more effective when communicating latency to better, that may be more effective when communicating latency to
end-users.</t> end users.</li>
<t>The relationship between throughput and goodput can be effective in <li>The relationship between throughput and goodput can be effective
finding the saturation points, both in client-side <xref target="Paasch2021"/> in
and server-side <xref target="Schlinker2019"/> settings.</t> finding the saturation points, both in client-side <xref target="I-D.cpaasch-ipp
<t>Working latency depends on algorithm choice for addressing endpoint m-responsiveness" format="default"/>
congestion control and router queuing.</t> and server-side <xref target="Schlinker2019" format="default"/> settings.</li>
</list></t> <li>Working latency depends on the algorithm choice for addressing e
ndpoint
<t>Finally, it was commonly agreed to that the best metrics are those congestion control and router queuing.</li>
</ol>
<t>Finally, it was commonly agreed to that the best metrics are those
that are actionable.</t> that are actionable.</t>
</section>
</section> </section>
</section> <section anchor="discussions-cross-layer" numbered="true" toc="default">
<section anchor="discussions-cross-layer" title="Cross-layer Considerations"> <name>Cross-Layer Considerations</name>
<t>In the cross-layer segment of the workshop, participants presented
<t>In the Cross-layer segment of the workshop, participants presented
material on and discussed how to accurately measure exactly where material on and discussed how to accurately measure exactly where
problems occur. Discussion centered especially on the differences problems occur. Discussion centered especially on the differences
between physically wired and wireless connections and the difficulties between physically wired and wireless connections and the difficulties
of accurately determining problem spots when multiple different types of accurately determining problem spots when multiple different types
of network segments are responsible for the quality. As an example, of network segments are responsible for the quality. As an example,
<xref target="Kerpez2021"/> showed that limited bandwidth of 2.4Ghz wifi is the <xref target="Kerpez2021" format="default"/> showed that a limited bandwidth of
most frequently the bottleneck. In comparison, the wider bandwidth of 2.4 Ghz Wi-Fi bottlenecks the most frequently. In comparison, the wider bandwidt
the 5Ghz WiFi have only been the bottleneck in 20% of observations.</t> h of
the 5 Ghz Wi-Fi has only bottlenecked in 20% of observations.</t>
<t>The participants agreed that no single component of a network <t>The participants agreed that no single component of a network
connection has all the data required to measure the effects of the connection has all the data required to measure the effects of the
network performance on the quality of the end user experience.</t> network performance on the quality of the end-user experience.</t>
<ul spacing="normal">
<t><list style="symbols"> <li>Applications that are running on the end-user devices have the bes
<t>Applications that are running on the end-user devices have the best t
insight into their respective performance, but have limited insight into their respective performance but have limited
visibility into the behavior of the network itself, and are unable visibility into the behavior of the network itself and are unable
to act based on their limited perspective.</t> to act based on their limited perspective.</li>
<t>Internet service providers have good insight into QoS <li>ISPs have good insight into QoS
considerations, but are not able to infer the effect of the QoS considerations but are not able to infer the effect of the QoS
metrics on the quality of end user experiences.</t> metrics on the quality of end-user experiences.</li>
<t>Content providers have good insight into the aggregated behavior of <li>Content providers have good insight into the aggregated behavior o
the end users, but lack the insight on what aspects of network f
performance are leading indicators of user behavior.</t> the end users but lack the insight on what aspects of network
</list></t> performance are leading indicators of user behavior.</li>
</ul>
<t>The workshop had identified the need for a standard and extensible way <t>The workshop had identified the need for a standard and extensible wa
y
to exchange network performance characteristics. Such an exchange to exchange network performance characteristics. Such an exchange
standard should address (at least) the following:</t> standard should address (at least) the following:</t>
<ul spacing="normal">
<t><list style="symbols"> <li>A scalable way to capture the performance of multiple (potentially
<t>A scalable way to capture the performance of multiple (potentially thousands of) endpoints.</li>
thousands of) endpoints.</t> <li>The data exchange format should prevent data manipulation so that
<t>The data exchange format should prevent data manipulation, so that the different participants won't be able to game the mechanisms.</li>
the different participants won’t be able to game the mechanisms.</t> <li>Preservation of end-user privacy. In particular, federated learnin
<t>Preservation of end-user privacy. In particular, federated learning g
approaches should be preferred so no centralized entity has the approaches should be preferred so that no centralized entity has the
access to the whole picture.</t> access to the whole picture.</li>
<t>A transparent model for giving the different actors on a network <li>A transparent model for giving the different actors on a network
connection an incentive to share the performance data they collect.</t> connection an incentive to share the performance data they collect.</li>
<t>An accompanying set of tools to analyze the data is needed as well.</t> <li>An accompanying set of tools to analyze the data.</li>
</list></t> </ul>
<section anchor="separation-of-concerns" numbered="true" toc="default">
<section anchor="separation-of-concerns" title="Separation of Concerns"> <name>Separation of Concerns</name>
<t>Commonly, there's a tight coupling between collecting performance
<t>Commonly, there’s a tight coupling between collecting performance metrics, interpreting those metrics, and acting upon the
metrics, interpreting those metrics, and and acting upon the interpretation. Unfortunately, such a model is not the best for
interpretation. Unfortunately, such model is not the best for successfully exchanging cross-layer data, as:</t>
successfully exchanging cross-layer data as:</t> <ul spacing="normal">
<li>actors that are able to collect particular performance metrics
<t><list style="symbols"> (e.g., the TCP RTT) do not necessarily have the context necessary for
<t>Actors that are able to collect particular performance metrics a meaningful interpretation,</li>
(e.g. the TCP RTT) do not necessarily have the context necessary for <li>the actors that have the context and the computational/storage
a meaningful interpretation.</t>
<t>The actors that have the context and the computational/storage
capacity to interpret metrics do not necessarily have the ability to capacity to interpret metrics do not necessarily have the ability to
control the behavior of network / application.</t> control the behavior of the network/application, and</li>
<t>The actors that can control the behavior of networks and/or <li>the actors that can control the behavior of networks and/or
applications typically do not have access to complete measurement applications typically do not have access to complete measurement
data.</t> data.</li>
</list></t> </ul>
<t>The participants agreed that it is important to separate the above
<t>The participants agreed that it is important to separate the above
three aspects, so that:</t> three aspects, so that:</t>
<ul spacing="normal">
<t><list style="symbols"> <li>the different actors that have the data, but not the ability to
<t>The different actors that have the data but not the ability to interpret and/or act upon it, should publish their measured data and</li>
interpret and/or act upon it should publish their measured data.</t> <li>the actors that have the expertise in interpreting and synthesiz
<t>The actors that have the expertise in interpreting and synthesizing ing
performance data should publish the results of their interpretations.</t> performance data should publish the results of their interpretations.</li>
</list></t> </ul>
</section>
</section> <section anchor="security-and-privacy-considerations" numbered="true" to
<section anchor="security-and-privacy-considerations" title="Security and Privac c="default">
y Considerations"> <name>Security and Privacy Considerations</name>
<t>Preserving the privacy of Internet end users is a difficult
<t>Preserving the privacy of Internet end users is a difficult
requirement to meet when addressing this problem space. There is an requirement to meet when addressing this problem space. There is an
intrinsic trade-off between collecting more data about user intrinsic trade-off between collecting more data about user
activities, and infringing their privacy while doing so. activities and infringing on their privacy while doing so.
Participants agreed that observability across multiple layers is Participants agreed that observability across multiple layers is
necessary for an accurate measurement of the network quality, but necessary for an accurate measurement of the network quality, but
doing so in a way that minimizes privacy leakage is an open question.</t> doing so in a way that minimizes privacy leakage is an open question.</t>
</section>
</section> <section anchor="metric-measurement-considerations" numbered="true" toc=
<section anchor="metric-measurement-considerations" title="Metric Measurement Co "default">
nsiderations"> <name>Metric Measurement Considerations</name>
<ul spacing="normal">
<t><list style="symbols"> <li>
<t>The following TCP protocol metrics have been found to be effective <t>The following TCP protocol metrics have been found to be effect
ive
and are available for passive measurement: and are available for passive measurement:
<list style="symbols"> </t>
<t>TCP connection latency measured using SACK/ACK timing, as well as <ul spacing="normal">
<li>TCP connection latency measured using selective acknowledgme
nt (SACK) or acknowledgment (ACK) timing, as well as
the timing between TCP retransmission events, are good proxies for the timing between TCP retransmission events, are good proxies for
end-to-end RTT measurements.</t> end-to-end RTT measurements.</li>
<t>On the Linux platform, the tcp_info structure is the de-facto <li>On the Linux platform, the tcp_info structure is the de fact
o
standard for an application to inspect the performance of standard for an application to inspect the performance of
kernel-space networking. However, there is no equivalent kernel-space networking. However, there is no equivalent
de-facto standard for the user-space networking.</t> de facto standard for user-space networking.</li>
</list></t> </ul>
<t>The QUIC and MASQUE protocols make passive performance measurements </li>
<li>
<t>The QUIC and MASQUE protocols make passive performance measurem
ents
more challenging. more challenging.
<list style="symbols"> </t>
<t>An approach that uses federated measurement / hierarchical <ul spacing="normal">
aggregation may be more valuable for these protocols.</t> <li>An approach that uses federated measurement/hierarchical
<t>The QLOG format seems to be the most mature candidate for such aggregation may be more valuable for these protocols.</li>
an exchange.</t> <li>The QLOG format seems to be the most mature candidate for su
</list></t> ch
</list></t> an exchange.</li>
</ul>
</section> </li>
<section anchor="discussions-cross-observability" title="Towards Improving Futur </ul>
e Cross-layer Observability"> </section>
<section anchor="discussions-cross-observability" numbered="true" toc="d
<t>The ownership of the Internet is spread across multiple administrative efault">
<name>Towards Improving Future Cross-Layer Observability</name>
<t>The ownership of the Internet is spread across multiple administrat
ive
domains, making measurement of end-to-end performance data domains, making measurement of end-to-end performance data
difficult. Furthermore, the immense scale of the Internet makes difficult. Furthermore, the immense scale of the Internet makes
aggregation and analysis of this difficult. <xref target="Marx2021"/> presented a aggregation and analysis of this difficult. <xref target="Marx2021" format="defa ult"/> presented a
simple logging format that could potentially be used to collect and simple logging format that could potentially be used to collect and
aggregate data from different layers.</t> aggregate data from different layers.</t>
<t>Another aspect of the cross-layer collaboration hampering measureme
<t>Another aspect of cross-layer collaboration hampering measurement is nt is
that the majority of current algorithms do not explicitly provide that the majority of current algorithms do not explicitly provide
performance data that can be used in cross-layer analysis. The IETF performance data that can be used in cross-layer analysis. The IETF
community could be more diligent in identifying each protocol’s key community could be more diligent in identifying each protocol's key
performance indicators, and exposing them as part of the protocol performance indicators and exposing them as part of the protocol
specification.</t> specification.</t>
<t>Despite all these challenges, it should still be possible to perfor
<t>Despite all these challenges, it should still be possible to perform m
limited-scope studies in order to have a better understanding of how limited-scope studies in order to have a better understanding of how
user quality is affected by the interaction of the different user quality is affected by the interaction of the different
components that constitute the Internet. Furthermore, recent components that constitute the Internet. Furthermore, recent
development of federated learning algorithms suggests that it might be development of federated learning algorithms suggests that it might be
possible to perform cross-layer performance measurements while possible to perform cross-layer performance measurements while
preserving user privacy.</t> preserving user privacy.</t>
</section>
</section> <section anchor="discussions-cross-layer-hw-tp" numbered="true" toc="def
<section anchor="discussions-cross-layer-hw-tp" title="Efficient Collaboration B ault">
etween Hardware and Transport Protocols"> <name>Efficient Collaboration between Hardware and Transport Protocols
</name>
<t>With the advent of the low latency, low loss and scalable throughput <t>With the advent of the low latency, low loss, and scalable throughp
ut
(L4S) congestion notification and control, there is an even higher (L4S) congestion notification and control, there is an even higher
need for the transport protocols and the underlying hardware to work need for the transport protocols and the underlying hardware to work
in unison.</t> in unison.</t>
<t>At the time of the workshop, the typical home router uses a single
<t>At the time of the workshop, the typical home router uses a single FIFO queue that is large enough to allow amortizing the lower-layer header
FIFO queue, large enough to allow amortizing the lower-layer header
overhead across multiple transport PDUs. These designs worked well overhead across multiple transport PDUs. These designs worked well
with the Cubic congestion control algorithm, yet the newer generation with the cubic congestion control algorithm, yet the newer generation
of CCAs can operate on much smaller queues. To fully support latencies of algorithms can operate on much smaller queues. To fully support latencies
less than 1ms, the home router needs to work efficiently on sequential less than 1 ms, the home router needs to work efficiently on sequential
transmissions of just a few segments vs. being optimized for large transmissions of just a few segments vs. being optimized for large
packet bursts.</t> packet bursts.</t>
<t>Another design trait common in home routers is the use of packet
<t>Another design trait common in home routers is the use of packet
aggregation to further amortize the overhead added by the lower-layer aggregation to further amortize the overhead added by the lower-layer
headers. Specifically, multiple IP datagrams are combined into a headers. Specifically, multiple IP datagrams are combined into a
single, large tranfer frame. However, this aggregation can add up to single, large transfer frame. However, this aggregation can add up to
10ms to the packet sojourn delay.</t> 10 ms to the packet sojourn delay.</t>
<t>Following the famous "you can't improve what you don't measure" ada
<t>Following the famous “you can’t improve what you don’t measure” adage, ge,
it is important to expose these aggregation delays in a way that would it is important to expose these aggregation delays in a way that would
allow identifying the source of the bottlenecks, and making hardware allow identifying the source of the bottlenecks and making hardware
more suitable for the next generation transport protocols.</t> more suitable for the next generation of transport protocols.</t>
</section>
</section> <section anchor="cross-layer-keypoints" numbered="true" toc="default">
<section anchor="cross-layer-keypoints" title="Cross-Layer Key Points"> <name>Cross-Layer Key Points</name>
<ul spacing="normal">
<t><list style="symbols"> <li>Significant differences exist in the characteristics of metrics
<t>Significant differences exist in the characteristics of metrics to to be measured and the required optimizations needed in wireless vs. wired
measured and required optimizations needed in wireless vs wired networks.</li>
networks.</t> <li>Identification of an issue's root cause is hampered by the
<t>Identification of an issue’s root-cause is hampered by the challenges in measuring multi-segment network paths.</li>
challenges in measuring multi-segment network paths.</t> <li>No single component of a network connection has all the data
<t>No single component of a network connection has all the data
required to measure the effects of the complete network performance required to measure the effects of the complete network performance
on the quality of the end user experience.</t> on the quality of the end-user experience.</li>
<t>Actionable results require both proper collection and interpretation.</t> <li>Actionable results require both proper collection and interpreta
<t>Coordination among network providers is important to successful tion.</li>
improve measurement of end user experiences.</t> <li>Coordination among network providers is important to successfull
<t>Simultaneously providing accurate measurements while preserving y
end-user privacy is challenging.</t> improve the measurement of end-user experiences.</li>
<t>Passive measurements from protocol implementations may provide <li>Simultaneously providing accurate measurements while preserving
beneficial data.</t> end-user privacy is challenging.</li>
</list></t> <li>Passive measurements from protocol implementations may provide
beneficial data.</li>
</section> </ul>
</section> </section>
<section anchor="synthesis" title="Synthesis"> </section>
<section anchor="synthesis" numbered="true" toc="default">
<t>Finally, in the Synthesis section of the workshop, the presentations <name>Synthesis</name>
<t>Finally, in the synthesis section of the workshop, the presentations
and discussions concentrated on the next steps likely needed to make and discussions concentrated on the next steps likely needed to make
forward progress. Of particular concern is how to bring forward forward progress. Of particular concern is how to bring forward
measurements that can make sense to end users trying to select measurements that can make sense to end users trying to select
between various networking subscription options.</t> between various networking subscription options.</t>
<section anchor="measurement-and-metrics-considerations" numbered="true"
<section anchor="measurement-and-metrics-considerations" title="Measurement and toc="default">
Metrics Considerations"> <name>Measurement and Metrics Considerations</name>
<t>One important consideration is how decisions can be made and what a
<t>One important consideration is how decisions can be made and actions ctions
taken based on collected metrics. Measurements must be integrated can be taken based on collected metrics. Measurements must be integrated
with applications in order to get true application views of with applications in order to get true application views of
congestion, as measurements over different infrastructure or via other congestion, as measurements over different infrastructure or via other
applications may return incorrect results. Congestion itself can be a applications may return incorrect results. Congestion itself can be a
temporary problem, and mitigation strategies may need to be different temporary problem, and mitigation strategies may need to be different
depending on whether it is expected to be a short-term or long-term depending on whether it is expected to be a short-term or long-term
phenomenon. A significant challenge exists in measuring short-term phenomenon. A significant challenge exists in measuring short-term
problems, driving the need for continuous measurements to ensure problems, driving the need for continuous measurements to ensure
capture of critical moments and long-term trends. For short-term critical moments and long-term trends are captured. For short-term
problems, workshop participants debated whether an issue that goes problems, workshop participants debated whether an issue that goes
away is indeed a problem or is a sign that a network is properly away is indeed a problem or is a sign that a network is properly
adapting and self-recovering.</t> adapting and self-recovering.</t>
<t>Important consideration must be taken when constructing metrics in
<t>Important consideration must be taken when constructing metrics in order to understand the results. Measurements can also be affected by
order to understand the results. Measurements can also affected by individual packet characteristics -- differently sized packets typically have a
individual packet characteristics – different sized packets have a linear relationship with their delay. With this in mind,
typically linear relationship with their delay. With this in mind,
measurements can be divided into a delay based on geographical measurements can be divided into a delay based on geographical
distances, a packet-size serialization delay and a variable (noise) distances, a packet-size serialization delay, and a variable (noise)
delay. Each of these three sub-component delays can be different and delay. Each of these three sub-component delays can be different and
individually measured across each segment in a multi-hop path. individually measured across each segment in a multi-hop path.
Variable delay can also be significantly impacted by external factors, Variable delay can also be significantly impacted by external factors,
such as bufferbloat, routing changes, network load sharing, and other such as bufferbloat, routing changes, network load sharing, and other
local or remote changes in performance. Network measurements, local or remote changes in performance. Network measurements,
especially load-specific tests, must also be run long enough to ensure especially load-specific tests, must also be run long enough to ensure
capture of any problems associated with buffering, queuing, etc. that any problems associated with buffering, queuing, etc. are captured.
Measurement technologies should also distinguish between upsteam and Measurement technologies should also distinguish between upstream and
downstream measurements, as well as measure the difference between downstream measurements, as well as measure the difference between
end-to-end paths and sub-path measurements.</t> end-to-end paths and sub-path measurements.</t>
</section>
</section> <section anchor="end-user-metrics-presentation" numbered="true" toc="def
<section anchor="end-user-metrics-presentation" title="End-User metrics presenta ault">
tion"> <name>End-User Metrics Presentation</name>
<t>Determining end-user needs requires informative measurements and
<t>Determining end-user needs requires informative measurements and
metrics. How do we provide the users with the service they need or metrics. How do we provide the users with the service they need or
want? Is it possible for users to even voice their desires want? Is it possible for users to even voice their desires
effectively? Only high-level, simplistic answers like “reliability”, effectively? Only high-level, simplistic answers like "reliability",
“capacity”, and “service bundling” are typical answers given in "capacity", and "service bundling" are typical answers given in
end-user surveys. Technical requirements that operators can consume, end-user surveys. Technical requirements that operators can consume,
like “low-latency” and “congestion avoidance”,are not terms known to like "low-latency" and "congestion avoidance", are not terms known to
and used by end-users.</t> and used by end users.</t>
<t>Example metrics useful to end users might include the number of use
<t>Example metrics useful to end users might include the number of users rs
supported by a service, and the number of applications or streams that supported by a service and the number of applications or streams that
a network can support. An example solution to combat netwokring a network can support. An example solution to combat networking
issues include incentive-based traffic management strategies (e.g. an issues include incentive-based traffic management strategies (e.g., an
application requesting lower latency may also mean accepting lower application requesting lower latency may also mean accepting lower
bandwidth). User perceived latency must be considered, not just bandwidth). User-perceived latency must be considered, not just
network latency – users experience in-application to in-server network latency -- user experience in-application to in-server
latency, and network to network measurements may only be studying the latency and network-to-network measurements may only be studying the
lowest level latency. Thus, picking the right protocol to use in a lowest-level latency. Thus, picking the right protocol to use in a
measurement is critical in order to match user experience (for measurement is critical in order to match user experience (for
example, users do not transmit data over ICMP even though it is a example, users do not transmit data over ICMP, even though it is a
common measurement tool).</t> common measurement tool).</t>
<t>In-application measurements should consider how to measure differen
<t>In-application measurements should consider how to measure different t
types of applications, such as video streaming, file sharing, types of applications, such as video streaming, file sharing,
multi-user gaming, and real-time voice communications. It may be that multi-user gaming, and real-time voice communications. It may be that
asking users for what tradeoffs they are willing to accept would be a asking users for what trade-offs they are willing to accept would be a
helpful approach: would they rather have a network with low latency, helpful approach: would they rather have a network with low latency
or a network with higher bandwidth. Gamers may make different or a network with higher bandwidth? Gamers may make different
decisions than home office users or content producers, for example.</t> decisions than home office users or content producers, for example.</t>
<t>Furthermore, how can users make these trade-offs in a fair manner t
<t>Furthermore, how can users make these trade-offs in a fair manner that hat
does not impact other users? There is a tension between solutions in does not impact other users? There is a tension between solutions in
this space vs the cost associated with solving these solutions, and this space vs. the cost associated with solving these problems, as well as
which customers are willing to front these improvement costs.</t> which customers are willing to front these improvement costs.</t>
<t>Challenges in providing higher-priority traffic to users centers
<t>Challenges in providing higher-priority traffic to users centers
around the ability for networks to be willing to listen to client around the ability for networks to be willing to listen to client
requests for higher incentives, even though commercial interests may requests for higher incentives, even though commercial interests may
not flow to them without a cost incentive. Shared mediums in general not flow to them without a cost incentive. Shared mediums in general
are subject to oversubscribing such that the number of users a network are subject to oversubscribing, such that the number of users a network
can support is either accurate on an underutilized network, or may can support is either accurate on an underutilized network or may
assume an average bandwidth or other usage metric that fails to be assume an average bandwidth or other usage metric that fails to be
accurate during utilization spikes. Individual metrics are also accurate during utilization spikes. Individual metrics are also
affected by in-home devices from cheap routers to microwaves and from affected by in-home devices from cheap routers to microwaves and by
(multi-)user behaviors during tests. Thus, a single metric alone or a (multi-)user behaviors during tests. Thus, a single metric alone or a
single reading without context may not be useful in assisting a user single reading without context may not be useful in assisting a user
or operator to determine where the problem source actually is.</t> or operator to determine where the problem source actually is.</t>
<t>User comprehension of a network remains a challenging problem.
<t>User comprehension of a network remains a challenging problem.
Multiple workshop participants argued for a single number (potentially Multiple workshop participants argued for a single number (potentially
calculated with weighted aggregation formula), or a small number of calculated with a weighted aggregation formula) or a small number of
measurements per expected usage (a “gaming” score vs a “content measurements per expected usage (e.g., a "gaming" score vs. a "content
producer” score). Many agreed that some users may instead prefer to producer" score). Many agreed that some users may instead prefer to
consume simplified or color-coded ratings (good/better/best, consume simplified or color-coded ratings (e.g., good/better/best,
red/yellow/green, or bronze/gold/platinum).</t> red/yellow/green, or bronze/gold/platinum).</t>
</section>
<section anchor="synthesis-key-points" numbered="true" toc="default">
<name>Synthesis Key Points</name>
<ul spacing="normal">
<li>
<t>Some proposed metrics:</t>
<ul spacing="normal">
<li>Round-trips Per Minute (RPM)</li>
<li>users per network</li>
<li>latency</li>
<li>99% latency and bandwidth</li>
</ul>
</li>
<li>Median and mean measurements are distractions from the real prob
lems.</li>
<li>Shared network usage greatly affects quality.</li>
<li>Long measurements are needed to capture all facets of potential
network bottlenecks.</li>
<li>Better-funded research in all these areas is needed for progress
.</li>
<li>End users will best understand a simplified score or ranking sys
tem.</li>
</ul>
</section>
</section>
</section>
<section anchor="conclusions" numbered="true" toc="default">
<name>Conclusions</name>
</section> <t>During the final hour of the three-day workshop, statements that the gr
<section anchor="synthesis-key-points" title="Synthesis Key Points"> oup deemed to be summary statements were gathered. Later, any statements that we
re in contention were discarded (listed further below for completeness).
<t><list style="symbols"> For this document, the authors took the original list
<t>Some proposed metrics:
<list style="symbols">
<t>Round-trips Per Minute (RPMs)</t>
<t>Users per network</t>
<t>Latency</t>
<t>99% latency and bandwidth</t>
</list></t>
<t>Median and mean measurements are distractions from the real problems.</t>
<t>Shared network usage greatly affect quality.</t>
<t>Long measurements are needed to capture all facets of potential
network bottlenecks.</t>
<t>Better funded research in all these areas is needed for progress.</t>
<t>End-users will best understand a simplified score or ranking system.</t>
</list></t>
</section>
</section>
</section>
<section anchor="conclusions" title="Conclusions">
<t>During the final hour of the workshop we gathered statements that the
group thought were summary statements from the 3 day event. We later
discarded any that were in contention (listed further below for
completeness). For this document, the editor took the original list
and divided it into rough categories, applied some suggested edits and divided it into rough categories, applied some suggested edits
discussed on the mailing list and further edited for clarity and to discussed on the mailing list, and further edited for clarity and to
provide context.</t> provide context.</t>
<section anchor="general-statements" numbered="true" toc="default">
<section anchor="general-statements" title="General statements"> <name>General Statements</name>
<ol spacing="normal" type="1">
<t><list style="numbers"> <li>Bandwidth is necessary but not alone sufficient.</li>
<t>Bandwidth is necessary but not alone sufficient.</t> <li>In many cases, Internet users don't need more bandwidth but rather
<t>In many cases, Internet users don’t need more bandwidth, but rather need "better bandwidth", i.e., they need other improvements to
need “better bandwidth” – i.e., they need other improvements to their connectivity.</li>
their connectivity.</t> <li>We need both active and passive measurements -- passive measuremen
<t>We need both active and passive measurements – passive measurements ts
can provide historical debugging.</t> can provide historical debugging.</li>
<t>We need passive measurements to be continuous and archivable and <li>We need passive measurements to be continuous, archivable, and
queriable – include reliability/connectivity measurements.</t> queriable, including reliability/connectivity measurements.</li>
<t>A really meaningful metric for users is whether their application <li>A really meaningful metric for users is whether their application
will work properly or fail because of a lack of a network with will work properly or fail because of a lack of a network with
sufficient characteristics.</t> sufficient characteristics.</li>
<t>A useful metric for goodness must actually incentive goodness – <li>A useful metric for goodness must actually incentivize goodness --
good metrics should be actionable to help drive industries toward good metrics should be actionable to help drive industries towards
improvement.</t> improvement.</li>
<t>A lower latency Internet, however achieved would benefit all end <li>A lower-latency Internet, however achieved, would benefit all end
users.</t> users.</li>
</list></t> </ol>
</section>
</section> <section anchor="specific-statements-about-detailed-protocolstechniques" n
<section anchor="specific-statements-about-detailed-protocolstechniques" title=" umbered="true" toc="default">
Specific statements about detailed protocols/techniques"> <name>Specific Statements about Detailed Protocols/Techniques</name>
<ol spacing="normal" type="1">
<t><list style="numbers"> <li>Round-trips Per Minute (RPM) is a useful, consumable metric.</li>
<t>Round trips Per Minute (RPM) is a useful, consumable metric.</t> <li>We need a usable tool that fills the current gap between network
<t>We need a usable tool that fills the current gap between network reachability, latency, and speed tests.</li>
reachability, latency, and speed tests.</t> <li>End users that want to be involved in QoS decisions should be able
<t>End-users that want to be involved in QoS decisions should be able to voice their needs and desires.</li>
to voice their needs and desires.</t> <li>Applications are needed that can perform and report good quality
<t>Applications are needed that can perform and report good quality
measurements in order to identify insufficient points in measurements in order to identify insufficient points in
network access.</t> network access.</li>
<t>Research done by regulators indicate that users/consumers prefer <li>Research done by regulators indicate that users/consumers prefer
a simple metric per application, which frequently resolves to a simple metric per application, which frequently resolves to
whether the application will work properly or not.</t> whether the application will work properly or not.</li>
<t>New measurements and QoS or QoE techniques should not rely only or <li>New measurements and QoS or QoE techniques should not rely only or
depend on reading TCP headers.</t> depend on reading TCP headers.</li>
<t>It is clear from developers of interactive applications and from <li>It is clear from developers of interactive applications and from
network operators that lower latency is a strong factor in user network operators that lower latency is a strong factor in user
QoE. However, metrics are lacking to support this statement QoE. However, metrics are lacking to support this statement
directly.</t> directly.</li>
</list></t> </ol>
</section>
</section> <section anchor="problem-statements-and-concerns" numbered="true" toc="def
<section anchor="problem-statements-and-concerns" title="Problem statements and ault">
concerns"> <name>Problem Statements and Concerns</name>
<ol spacing="normal" type="1">
<t><list style="numbers"> <li>Latency mean and medians are distractions from better measurements.
<t>Latency mean and medians are distractions from better measurements.</t> </li>
<t>It is frustrating to only measure network services without <li>It is frustrating to only measure network services without
simultaneously improving those services.</t> simultaneously improving those services.</li>
<t>Stakeholder incentives aren’t aligned for easy wins in this space. <li>Stakeholder incentives aren't aligned for easy wins in this space.
Incentives are needed to motivate improvements in public network Incentives are needed to motivate improvements in public network
access. Measurements may be one step toward driving competitive access. Measurements may be one step towards driving competitive
market incentive.</t> market incentives.</li>
<t>For future-proof networking, it is important to measure the <li>For future-proof networking, it is important to measure the
ecological impact of material and energy usage.</t> ecological impact of material and energy usage.</li>
<t>We do not have incontrovertible evidence that any one metric <li>We do not have incontrovertible evidence that any one metric
(e.g., latency or speed) is more important than others to persuade (e.g., latency or speed) is more important than others to persuade
device vendors to concentrate on any one optimization.</t> device vendors to concentrate on any one optimization.</li>
</list></t> </ol>
</section>
<section anchor="no-consensus-reached-statements" numbered="true" toc="def
ault">
<name>No-Consensus-Reached Statements</name>
<t>Additional statements were discussed and recorded that did not have c
onsensus of the
group at the time, but they are listed here for completeness:</t>
<ol spacing="normal" type="1">
<li>We do not have incontrovertible evidence that bufferbloat is a
prevalent problem.</li>
<li>
<t>The measurement needs to support reporting localization in order
to
find problems. Specifically:</t>
<ul spacing="normal">
<li>Detecting a problem is not sufficient if you can't find the lo
cation.</li>
<li>Need more than just English -- different localization concerns
.</li>
</ul>
</li>
<li>Stakeholder incentives aren't aligned for easy wins in this space.
</li>
</ol>
</section>
</section>
<section anchor="follow-on-work" numbered="true" toc="default">
<name>Follow-On Work</name>
<t>There was discussion during the workshop about where future work
should be performed. The group agreed that some work could be done
more immediately within existing IETF working groups (e.g., IPPM,
DetNet, and RAW), while other longer-term research may be needed in
IRTF groups.</t>
</section>
<section anchor="iana-considerations" numbered="true" toc="default">
<name>IANA Considerations</name>
<t>This document has no IANA actions.</t>
</section>
<section anchor="security-considerations" numbered="true" toc="default">
<name>Security Considerations</name>
<t>A few security-relevant topics were discussed at the workshop,
including but not limited to:</t>
<ul spacing="normal">
<li>what prioritization techniques can work without invading the privacy
of the communicating parties and</li>
<li>how oversubscribed networks can essentially be viewed as a DDoS
attack.</li>
</ul>
</section>
</middle>
<back>
<displayreference target="I-D.morton-ippm-pipe-dream" to="Morton2021"/>
<displayreference target="I-D.cpaasch-ippm-responsiveness" to="Paasch2021"/>
</section> <references>
<section anchor="no-consensus-reached-statements" title="No-consensus reached st
atements">
<t>Additional statements were recorded that did not have consensus of the <name>Informative References</name>
group at the time, but we list them here for completeness about the fact
they were discussed:</t>
<t><list style="numbers"> <reference anchor="FCC_MBA" target="https://www.fcc.gov/general/measuring-
<t>We do not have incontrovertible evidence that buffer bloat is a broadband-america">
prevalent problem.</t> <front>
<t>The measurement needs to support reporting localization in order to <title>Measuring Broadband America</title>
find problems. Specifically: <author>
<list style="symbols"> <organization>FCC</organization>
<t>Detecting a problem is not sufficient if you can’t find the location.</ </author>
t> </front>
<t>Need more than just English – different localization concerns.</t> </reference>
</list></t>
<t>Stakeholder incentives aren’t aligned for easy wins in this space.</t>
</list></t>
</section> <reference anchor="FCC_MBA_methodology" target="https://www.fcc.gov/genera
</section> l/measuring-broadband-america-open-methodology">
<section anchor="follow-on-work" title="Follow-on work"> <front>
<title>Measuring Broadband America - Open Methodology</title>
<author>
<organization>FCC</organization>
</author>
</front>
</reference>
<t>There was discussion during the workshop about where future work <reference anchor="Scuba" target="https://research.facebook.com/publicatio
should be performed. The group agreed that some work could be done ns/scuba-diving-into-data-at-facebook/">
more immediately within existing IETF working groups (e.g. IPPM, <front>
DetNet and RAW), while other longer-term research may be needed in <title>Scuba: Diving into Data at Facebook</title>
IRTF groups.</t> <author initials="" surname="Abraham, L. et al."/>
</front>
</reference>
</section> <reference anchor="WORKSHOP" target="https://www.iab.org/activities/worksh
<section anchor="security-considerations" title="Security considerations"> ops/network-quality">
<front>
<title>IAB Workshop: Measuring Network Quality for End-Users, 2021</ti
tle>
<author>
<organization>IAB</organization>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<t>A few security relevant topics were discussed at the workshop, <reference anchor="Cheshire2021" target="https://www.iab.org/wp-content/IA
including but not limited to:</t> B-uploads/2021/09/draft-cheshire-internet-is-shared-00b.pdf">
<front>
<title>The Internet is a Shared Network</title>
<author initials="S." surname="Cheshire">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
<t><list style="symbols"> <reference anchor="Iyengar2021" target="https://www.iab.org/wp-content/IAB
<t>What prioritization techniques can work without invading the privacy -uploads/2021/09/The-Internet-Exists-In-Its-Use.pdf">
of the communicating parties.</t> <front>
<t>How oversubscribed networks can essentially be viewed as a DDoS <title>The Internet Exists In Its Use</title>
attack.</t> <author initials="J." surname="Iyengar">
</list></t> <organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
</section> <reference anchor="Stein2021" target="https://www.iab.org/wp-content/IAB-u
ploads/2021/09/QoS-futility.pdf">
<front>
<title>The Futility of QoS</title>
<author initials="Y." surname="Stein">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
</middle> <reference anchor="Casas2021" target="https://www.iab.org/wp-content/IAB-u
ploads/2021/09/net_quality_internet_qoe_CASAS.pdf">
<front>
<title>10 Years of Internet-QoE Measurements Video, Cloud, Conferencin
g, Web and Apps. What do we need from the Network Side?</title>
<author initials="P." surname="Casas">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
<back> <reference anchor="Pardue2021" target="https://www.iab.org/wp-content/IAB-
uploads/2021/09/Lower-layer-performance-is-not-indicative-of-upper-layer-success
-20210906-00-1.pdf">
<front>
<title>Lower-layer performance is not indicative of upper-layer succes
s</title>
<author initials="L." surname="Pardue">
<organization/>
</author>
<author initials="S." surname="Tellakula">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<references title='Informative References'> <reference anchor="Aldabbagh2021" target="https://www.iab.org/wp-content/I
AB-uploads/2021/09/2021-09-07-Aldabbagh-Ofcom-presentationt-to-IAB-1v00-1.pdf">
<front>
<title>Regulatory perspective on measuring network quality for end-use
rs</title>
<author initials="A." surname="Aldabbagh">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<reference anchor="FCC_MBA" > <reference anchor="Welzl2021" target="https://www.iab.org/wp-content/IAB-u
<front> ploads/2021/09/iab-longtermstats_cameraready.docx-1.pdf">
<title>Measuring Broadband America</title> <front>
<author > <title>A Case for Long-Term Statistics</title>
<organization></organization> <author initials="M." surname="Welzl">
</author> <organization/>
<date year="n.d."/> </author>
</front> <date year="2021" month="February"/>
<seriesInfo name="https://www.fcc.gov/general/measuring-broadband-america" val </front>
ue=""/> </reference>
</reference>
<reference anchor="FCC_MBA_methodology" >
<front>
<title>Measuring Broadband America - Open Methodology</title>
<author >
<organization></organization>
</author>
<date year="n.d."/>
</front>
<seriesInfo name="https://www.fcc.gov/general/measuring-broadband-america-open
-methodology" value=""/>
</reference>
<reference anchor="Scuba" target="https://research.facebook.com/publications/scu
ba-diving-into-data-at-facebook/">
<front>
<title>Facebook Scuba</title>
<author >
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="WORKSHOP" >
<front>
<title>IAB Workshop: Measuring Network Quality for End-Users, 2021</title>
<author initials="." surname="IAB" fullname="IAB">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<reference anchor="Cheshire2021" >
<front>
<title>The Internet is a Shared Network</title>
<author initials="S." surname="Cheshire">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/draft-che
shire-internet-is-shared-00b.pdf" value=""/>
</reference>
<reference anchor="Iyengar2021" >
<front>
<title>The Internet Exists In Its Use</title>
<author initials="J." surname="Iyengar">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/The-Inter
net-Exists-In-Its-Use.pdf" value=""/>
</reference>
<reference anchor="Stein2021" >
<front>
<title>The Futility of QoS</title>
<author initials="J." surname="Stein">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/QoS-futil
ity.pdf" value=""/>
</reference>
<reference anchor="Casas2021" >
<front>
<title>10 Years of Internet-QoE Measurements. Video, Cloud, Conferencing, We
b and Apps. What do we need from the Network Side?</title>
<author initials="P." surname="Casas">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/net_quali
ty_internet_qoe_CASAS.pdf" value=""/>
</reference>
<reference anchor="Pardue2021" >
<front>
<title>Lower-layer performance is not indicative of upper-layer success</tit
le>
<author initials="L." surname="Pardue">
<organization></organization>
</author>
<author initials="S." surname="Tellakula">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Lower-lay
er-performance-is-not-indicative-of-upper-layer-success-20210906-00-1.pdf" value
=""/>
</reference>
<reference anchor="Aldabbagh2021" >
<front>
<title>Regulatory perspective on measuring network quality for end users</ti
tle>
<author initials="A." surname="Aldabbagh">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/2021-09-0
7-Aldabbagh-Ofcom-presentationt-to-IAB-1v00-1.pdf" value=""/>
</reference>
<reference anchor="Welzl2021" >
<front>
<title>A Case for Long-Term Statistics</title>
<author initials="M." surname="Welzl">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/iab-longt
ermstats_cameraready.docx-1.pdf" value=""/>
</reference>
<reference anchor="Fabini2021" >
<front>
<title>Network Quality from an End User Perspective</title>
<author initials="J." surname="Fabini">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Fabini-IA
B-NetworkQuality.txt" value=""/>
</reference>
<reference anchor="Mathis2021" >
<front>
<title>Preliminary Longitudinal Study of Internet Responsiveness</title>
<author initials="M." surname="Mathis">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Prelimina
ry-Longitudinal-Study-of-Internet-Responsiveness-1.pdf" value=""/>
</reference>
<reference anchor="Schlinker2019" >
<front>
<title>Internet's performance from Facebook's edge</title>
<author initials="B." surname="Schlinker">
<organization></organization>
</author>
<author initials="I." surname="Cunha">
<organization></organization>
</author>
<author initials="Y." surname="Chiu">
<organization></organization>
</author>
<author initials="S." surname="Sundaresan">
<organization></organization>
</author>
<author initials="E." surname="Katz-Basset">
<organization></organization>
</author>
<date year="2019" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Internet-
Performance-from-Facebooks-Edge.pdf" value=""/>
</reference>
<reference anchor="Foulkes2021" >
<front>
<title>Metrics helpful in assessing Internet Quality</title>
<author initials="J." surname="Foulkes">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/IAB_Metri
cs_helpful_in_assessing_Internet_Quality.pdf" value=""/>
</reference>
<reference anchor="Sivaraman2021" >
<front>
<title>Measuring Network Experience Meaningfully, Accurately, and Scalably</
title>
<author initials="V." surname="Sivaraman">
<organization></organization>
</author>
<author initials="S." surname="Madanapalli">
<organization></organization>
</author>
<author initials="H." surname="Kumar">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/CanopusPo
sitionPaperCameraReady.pdf" value=""/>
</reference>
<reference anchor="Reed2021" >
<front>
<title>Measuring IKSP Performance in Broadband America: A Study of Latency U
nder Load</title>
<author initials="D.P." surname="Reed">
<organization></organization>
</author>
<author initials="L." surname="Perigo">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Camera_Re
ady_-Measuring-ISP-Performance-in-Broadband-America.pdf" value=""/>
</reference>
<reference anchor="MacMillian2021" >
<front>
<title>Beyond Speed Test: Measuring Latency Under Load Across Different Spee
d Tiers</title>
<author initials="K." surname="MacMillian">
<organization></organization>
</author>
<author initials="N." surname="Feamster">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/2021_nqw_
lul.pdf" value=""/>
</reference>
<reference anchor="Mirsky2021" >
<front>
<title>The Error Performance Metric in a Packet-Switched Network</title>
<author initials="G." surname="Mirsky">
<organization></organization>
</author>
<author initials="X." surname="Min">
<organization></organization>
</author>
<author initials="G." surname="Mishra">
<organization></organization>
</author>
<author initials="L." surname="Han">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/IAB-worsh
op-Error-performance-measurement-in-packet-switched-networks.pdf" value=""/>
</reference>
<reference anchor="Dion2021" >
<front>
<title>Focusing on latency, not throughput, to provide a better internet exp
erience and network quality</title>
<author initials="G." surname="Dion">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Nokia-IAB
-Measuring-Network-Quality-Improving-and-focusing-on-latency-.pdf" value=""/>
</reference>
<reference anchor="Balasubramanian2021" >
<front>
<title>Transport Layer Statistics for Network Quality</title>
<author initials="P." surname="Balasubramanian">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/transport
statsquality.pdf" value=""/>
</reference>
<reference anchor="Arkko2021" >
<front>
<title>Observability is needed to improve network quality</title>
<author initials="J." surname="Arkko">
<organization></organization>
</author>
<author initials="M." surname="Kühlewind">
<organization></organization>
</author>
<date year="2021" month="August"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/iab-posit
ion-paper-observability.pdf" value=""/>
</reference>
<reference anchor="Marx2021" >
<front>
<title>Merge Those Metrics: Towards Holistic (Protocol) Logging</title>
<author initials="R." surname="Marx">
<organization></organization>
</author>
<author initials="J." surname="Herbots">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/MergeThos
eMetrics_Marx_Jul2021.pdf" value=""/>
</reference>
<reference anchor="Ghai2021" >
<front>
<title>Using TCP Connect Latency for Measuring CX and Network Optimization</
title>
<author initials="R." surname="Ghai">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/xfinity-w
ifi-ietf-iab-v2-1.pdf" value=""/>
</reference>
<reference anchor="DeSchepper2021" >
<front>
<title>Challenges and opportunities of hardware support for Low Queuing Late
ncy without Packet Loss</title>
<author initials="K." surname="De Schepper">
<organization></organization>
</author>
<author initials="O." surname="Tilmans">
<organization></organization>
</author>
<author initials="G." surname="Dion">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Nokia-IAB
-Measuring-Network-Quality-Low-Latency-measurement-workshop-20210802.pdf" value=
""/>
</reference>
<reference anchor="Michel2021" >
<front>
<title>Packet delivery time as a tie-breaker for assessing Wi-Fi access poin
ts</title>
<author initials="F." surname="Michel">
<organization></organization>
</author>
<author initials="O." surname="Bonaventure">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/camera_re
ady_Packet_delivery_time_as_a_tie_breaker_for_assessing_Wi_Fi_access_points.pdf"
value=""/>
</reference>
<reference anchor="Kerpez2021" >
<front>
<title>Wi-Fi and Broadband Data</title>
<author initials="J." surname="Shafiei">
<organization></organization>
</author>
<author initials="K." surname="Kerpez">
<organization></organization>
</author>
<author initials="J." surname="Cioffi">
<organization></organization>
</author>
<author initials="P." surname="Chow">
<organization></organization>
</author>
<author initials="D." surname="Bousaber">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Wi-Fi-Rep
ort-ASSIA.pdf" value=""/>
</reference>
<reference anchor="Liubogoshchev2021" >
<front>
<title>Cross-layer cooperation for Better Network Service</title>
<author initials="M." surname="Liubogoshchev">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Cross-lay
er-Cooperation-for-Better-Network-Service-2.pdf" value=""/>
</reference>
<reference anchor="Laki2021" >
<front>
<title>Incentive-Based Traffic Management and QoS Measurements</title>
<author initials="S." surname="Nadas">
<organization></organization>
</author>
<author initials="B." surname="Varga">
<organization></organization>
</author>
<author initials="L.M." surname="Contreras">
<organization></organization>
</author>
<author initials="S." surname="Laki">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/11/CamRdy-IA
B_user_meas_WS_Nadas_et_al_IncentiveBasedTMwQoS.pdf" value=""/>
</reference>
<reference anchor="Sengupta2021" >
<front>
<title>Fine-Grained RTT Monitoring Inside the Network</title>
<author initials="S." surname="Sengupta">
<organization></organization>
</author>
<author initials="H." surname="Kim">
<organization></organization>
</author>
<author initials="J." surname="Rexford">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Camera_Re
ady__Fine-Grained_RTT_Monitoring_Inside_the_Network.pdf" value=""/>
</reference>
<reference anchor="Morton2021" >
<front>
<title>Dream-Pipe or Pipe-Dream: What Do Users Want (and how can we assure i
t)?</title>
<author initials="A." surname="Morton">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/draft-mor
ton-ippm-pipe-dream-01.pdf" value=""/>
</reference>
<reference anchor="Kilkki2021" >
<front>
<title>In Search of Lost QoS</title>
<author initials="K." surname="Kilkki">
<organization></organization>
</author>
<author initials="B." surname="Finley">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Kilkki-In
-Search-of-Lost-QoS.pdf" value=""/>
</reference>
<reference anchor="Davies2021" >
<front>
<title>Measuring Network Impact on Application Outcomes using Quality Attenu
ation</title>
<author initials="N." surname="Davies">
<organization></organization>
</author>
<author initials="P." surname="Thompson">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/PNSol-et-
al-Submission-to-Measuring-Network-Quality-for-End-Users-1.pdf" value=""/>
</reference>
<reference anchor="Zhang2021" >
<front>
<title>User-Perceived Latency to measure CCAs</title>
<author initials="M." surname="Zhang">
<organization></organization>
</author>
<author initials="V." surname="Goel">
<organization></organization>
</author>
<author initials="L." surname="Xu">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/User_Perc
eived_Latency-1.pdf" value=""/>
</reference>
<reference anchor="Paasch2021" >
<front>
<title>Responsiveness under Working Conditions</title>
<author initials="C." surname="Paasch">
<organization></organization>
</author>
<author initials="R." surname="Meyer">
<organization></organization>
</author>
<author initials="S." surname="Cheshire">
<organization></organization>
</author>
<author initials="O." surname="Shapira">
<organization></organization>
</author>
<date year="2021" month="February"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/draft-cpa
asch-ippm-responsiveness-1-1.pdf" value=""/>
</reference>
<reference anchor="Briscoe2021" >
<front>
<title>A Single Common Metric to Characterize Varying Packet Delay</title>
<author initials="B." surname="Briscoe">
<organization></organization>
</author>
<author initials="G." surname="White">
<organization></organization>
</author>
<author initials="V." surname="Goel">
<organization></organization>
</author>
<author initials="K." surname="De Schepper">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/single-de
lay-metric-1.pdf" value=""/>
</reference>
<reference anchor="McIntyre2021" >
<front>
<title>An end-user approach to an Internet Score</title>
<author initials="C." surname="Paasch">
<organization></organization>
</author>
<author initials="K." surname="McIntyre">
<organization></organization>
</author>
<author initials="O." surname="Shapira">
<organization></organization>
</author>
<author initials="R." surname="Meyer">
<organization></organization>
</author>
<author initials="S." surname="Cheshire">
<organization></organization>
</author>
<date year="2021" month="September"/>
</front>
<seriesInfo name="https://www.iab.org/wp-content/IAB-uploads/2021/09/Internet-
Score-2.pdf" value=""/>
</reference>
<reference anchor="Speedtest" target="https://www.speedtest.net">
<front>
<title>Speedtest by Ookla</title>
<author >
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="NetworkQuality" >
<front>
<title>Apple Network Quality</title>
<author >
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="SamKnows" target="https://www.samknows.com/">
<front>
<title>SamKnows</title>
<author >
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
</references> <reference anchor="Fabini2021" target="https://www.iab.org/wp-content/IAB-
uploads/2021/09/Fabini-IAB-NetworkQuality.txt">
<front>
<title>Network Quality from an End User Perspective</title>
<author initials="J." surname="Fabini">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<section anchor="participants-list" title="Participants List"> <reference anchor="Mathis2021" target="https://www.iab.org/wp-content/IAB-
uploads/2021/09/Preliminary-Longitudinal-Study-of-Internet-Responsiveness-1.pdf"
>
<front>
<title>Preliminary Longitudinal Study of Internet Responsiveness</titl
e>
<author initials="M." surname="Mathis">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
<t>The following is a list of participants who attended the workshop over a remo <reference anchor="Schlinker2019" target="https://www.iab.org/wp-content/I
te connection:</t> AB-uploads/2021/09/Internet-Performance-from-Facebooks-Edge.pdf">
<front>
<title>Internet Performance from Facebook's Edge</title>
<author initials="B." surname="Schlinker">
<organization/>
</author>
<author initials="I." surname="Cunha">
<organization/>
</author>
<author initials="Y." surname="Chiu">
<organization/>
</author>
<author initials="S." surname="Sundaresan">
<organization/>
</author>
<author initials="E." surname="Katz-Basset">
<organization/>
</author>
<date year="2019" month="February"/>
</front>
</reference>
<figure><artwork><![CDATA[ <reference anchor="Foulkes2021" target="https://www.iab.org/wp-content/IAB
Ahmed Aldabbagh -uploads/2021/09/IAB_Metrics_helpful_in_assessing_Internet_Quality.pdf">
Jari Arkko <front>
Praveen Balasubramanian <title>Metrics helpful in assessing Internet Quality</title>
Olivier Bonaventure <author initials="J." surname="Foulkes">
Djamel Bousaber <organization/>
Bob Briscoe </author>
Rich Brown <date year="2021" month="September"/>
Anna Brunstrom </front>
Pedro Casas </reference>
Vint Cerf
Stuart Cheshire
Kenjiro Cho
Steve Christianson
John Cioffi
Alexander Clemm
Luis M. Contreras
Sam Crawford
Neil Davies
Gino Dion
Toerless Eckert
Lars Eggert
Joachim Fabini
Gorry Fairhurst
Nick Feamster
Mat Ford
Jonathan Foulkes
Jim Gettys
Rajat Ghai
Vidhi Goel
Wes Hardaker
Joris Herbots
Geoff Huston
Toke Høiland-Jørgensen
Jana Iyengar
Cullen Jennings
Ken Kerpez
Evgeny Khorov
Kalevi Kilkki
Joon Kim
Zhenbin Li
Mikhail Liubogoshchev
Jason Livingood
Kyle MacMillan
Sharat Madanapalli
Vesna Manojlovic
Robin Marx
Matt Mathis
Jared Mauch
Kristen McIntyre
Randall Meyer
François Michel
Greg Mirsky
Cindy Morgan
Al Morton
Szilveszter Nadas
Kathleen Nichols
Lai Yi Ohlsen
Christoph Paasch
Lucas Pardue
Tommy Pauly
Levi Perigo
David Reed
Alvaro Retana
Roberto
Koen De Schepper
David Schinazi
Brandon Schlinker
Eve Schooler
Satadal Sengupta
Jinous Shafiei
Shapelez
Omer Shapira
Dan Siemon
Vijay Sivaraman
Karthik Sundaresan
Dave Taht
Rick Taylor
Bjørn Ivar Teigen
Nicolas Tessares
Peter Thompson
Balazs Varga
Bren Tully Walsh
Michael Welzl
Greg White
Russ White
Keith Winstein
Lisong Xu
Jiankang Yao
Gavin Young
Mingrui Zhang
]]></artwork></figure>
</section> <reference anchor="Sivaraman2021" target="https://www.iab.org/wp-content/I
<section anchor="iab-members-at-the-time-of-approval" title="IAB Members at the AB-uploads/2021/09/CanopusPositionPaperCameraReady.pdf">
Time of Approval"> <front>
<title>Measuring Network Experience Meaningfully, Accurately, and Scal
ably</title>
<author initials="V." surname="Sivaraman">
<organization/>
</author>
<author initials="S." surname="Madanapalli">
<organization/>
</author>
<author initials="H." surname="Kumar">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<t>Internet Architecture Board members at the time this document was <reference anchor="Reed2021" target="https://www.iab.org/wp-content/IAB-up
approved for publication were:</t> loads/2021/09/Camera_Ready_-Measuring-ISP-Performance-in-Broadband-America.pdf">
<front>
<title>Measuring ISP Performance in Broadband America: A Study of Late
ncy Under Load</title>
<author initials="D.P." surname="Reed">
<organization/>
</author>
<author initials="L." surname="Perigo">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<figure><artwork><![CDATA[ <reference anchor="MacMillian2021" target="https://www.iab.org/wp-content/
Jari Arkko IAB-uploads/2021/09/2021_nqw_lul.pdf">
Deborah Brungard <front>
Ben Campbell <title>Beyond Speed Test: Measuring Latency Under Load Across Differen
Lars Eggert t Speed Tiers</title>
Wes Hardaker <author initials="K." surname="MacMillian">
Cullen Jennings <organization/>
Mirja Kühlewind </author>
Zhenbin Li <author initials="N." surname="Feamster">
Jared Mauch <organization/>
Tommy Pauly </author>
Colin Perkins <date year="2021" month="February"/>
David Schinazi </front>
Russ White </reference>
Jiankang Yao
]]></artwork></figure>
</section> <reference anchor="Mirsky2021" target="https://www.iab.org/wp-content/IAB-
<section anchor="acknowledgements" title="Acknowledgements"> uploads/2021/09/IAB-worshop-Error-performance-measurement-in-packet-switched-net
works.pdf">
<front>
<title>The error performance metric in a packet-switched network</titl
e>
<author initials="G." surname="Mirsky">
<organization/>
</author>
<author initials="X." surname="Min">
<organization/>
</author>
<author initials="G." surname="Mishra">
<organization/>
</author>
<author initials="L." surname="Han">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<t>The authors would like to thank the workshop participants, the members <reference anchor="Dion2021" target="https://www.iab.org/wp-content/IAB-up
of the IAB, and the program committee for creating and participating loads/2021/09/Nokia-IAB-Measuring-Network-Quality-Improving-and-focusing-on-late
in many interesting discussions.</t> ncy-.pdf">
<front>
<title>Focusing on latency, not throughput, to provide a better intern
et experience and network quality</title>
<author initials="G." surname="Dion">
<organization/>
</author>
<author initials="K." surname="De Schepper">
<organization/>
</author>
<author initials="O." surname="Tilmans">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
<section anchor="draft-contributors" title="Draft contributors"> <reference anchor="Balasubramanian2021" target="https://www.iab.org/wp-con
tent/IAB-uploads/2021/09/transportstatsquality.pdf">
<front>
<title>Transport Layer Statistics for Network Quality</title>
<author initials="P." surname="Balasubramanian">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<t>Thank you to the people that contributed edits to this draft:</t> <reference anchor="Arkko2021" target="https://www.iab.org/wp-content/IAB-u
ploads/2021/09/iab-position-paper-observability.pdf">
<front>
<title>Observability is needed to improve network quality</title>
<author initials="J." surname="Arkko">
<organization/>
</author>
<author initials="M." surname="Kühlewind">
<organization/>
</author>
<date year="2021" month="August"/>
</front>
</reference>
<figure><artwork><![CDATA[ <reference anchor="Marx2021" target="https://www.iab.org/wp-content/IAB-up
Erik Auerswald loads/2021/09/MergeThoseMetrics_Marx_Jul2021.pdf">
Simon Leinen <front>
Brian Trammell <title>Merge Those Metrics: Towards Holistic (Protocol) Logging</title
]]></artwork></figure> >
<author initials="R." surname="Marx">
<organization/>
</author>
<author initials="J." surname="Herbots">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
</section> <reference anchor="Ghai2021" target="https://www.iab.org/wp-content/IAB-up
<section anchor="workshop-chairs" title="Workshop Chairs"> loads/2021/09/xfinity-wifi-ietf-iab-v2-1.pdf">
<front>
<title>Using TCP Connect Latency for measuring CX and Network Optimiza
tion</title>
<author initials="R." surname="Ghai">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<t>The workshop chairs consisted of:</t> <reference anchor="DeSchepper2021" target="https://www.iab.org/wp-content/
IAB-uploads/2021/09/Nokia-IAB-Measuring-Network-Quality-Low-Latency-measurement-
workshop-20210802.pdf">
<front>
<title>Challenges and opportunities of hardware support for Low Queuin
g Latency without Packet Loss</title>
<author initials="K." surname="De Schepper">
<organization/>
</author>
<author initials="O." surname="Tilmans">
<organization/>
</author>
<author initials="G." surname="Dion">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<figure><artwork><![CDATA[ <reference anchor="Michel2021" target="https://www.iab.org/wp-content/IAB-
Wes Hardaker uploads/2021/09/camera_ready_Packet_delivery_time_as_a_tie_breaker_for_assessing
Evgeny Khorov _Wi_Fi_access_points.pdf">
Omer Shapira <front>
]]></artwork></figure> <title>Packet delivery time as a tie-breaker for assessing Wi-Fi acces
s points</title>
<author initials="F." surname="Michel">
<organization/>
</author>
<author initials="O." surname="Bonaventure">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
</section> <reference anchor="Kerpez2021" target="https://www.iab.org/wp-content/IAB-
<section anchor="program-committee" title="Program Committee"> uploads/2021/09/Wi-Fi-Report-ASSIA.pdf">
<front>
<title>Wi-Fi and Broadband Data</title>
<author initials="J." surname="Shafiei">
<organization/>
</author>
<author initials="K." surname="Kerpez">
<organization/>
</author>
<author initials="J." surname="Cioffi">
<organization/>
</author>
<author initials="P." surname="Chow">
<organization/>
</author>
<author initials="D." surname="Bousaber">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<t>The program committee consisted of:</t> <reference anchor="Liubogoshchev2021" target="https://www.iab.org/wp-conte
nt/IAB-uploads/2021/09/Cross-layer-Cooperation-for-Better-Network-Service-2.pdf"
>
<front>
<title>Cross-layer Cooperation for Better Network Service</title>
<author initials="M." surname="Liubogoshchev">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<figure><artwork><![CDATA[ <reference anchor="Laki2021" target="https://www.iab.org/wp-content/IAB-up
Jari Arkko loads/2021/11/CamRdy-IAB_user_meas_WS_Nadas_et_al_IncentiveBasedTMwQoS.pdf">
Olivier Bonaventure <front>
Vint Cerf <title>Incentive-Based Traffic Management and QoS Measurements</title>
Stuart Cheshire <author initials="S." surname="Nadas">
Sam Crowford <organization/>
Nick Feamster </author>
Jim Gettys <author initials="B." surname="Varga">
Toke Hoiland-Jorgensen <organization/>
Geoff Huston </author>
Cullen Jennings <author initials="L.M." surname="Contreras">
Katarzyna Kosek-Szott <organization/>
Mirja Kuehlewind </author>
Jason Livingood <author initials="S." surname="Laki">
Matt Mathis <organization/>
Randall Meyer </author>
Kathleen Nichols <date year="2021" month="February"/>
Christoph Paasch </front>
Tommy Pauly </reference>
Greg White
Keith Winstein
]]></artwork></figure>
</section> <reference anchor="Sengupta2021" target="https://www.iab.org/wp-content/IA
</section> B-uploads/2021/09/Camera_Ready__Fine-Grained_RTT_Monitoring_Inside_the_Network.p
<section anchor="github-version-of-this-document" title="Github Version of this df">
document"> <front>
<title>Fine-Grained RTT Monitoring Inside the Network</title>
<author initials="S." surname="Sengupta">
<organization/>
</author>
<author initials="H." surname="Kim">
<organization/>
</author>
<author initials="J." surname="Rexford">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
<t>While this document is under development, it can be viewed and tracked <xi:include href="https://datatracker.ietf.org/doc/bibxml3/reference.I-D.m
here:</t> orton-ippm-pipe-dream.xml"/>
<t>https://github.com/intarchboard/network-quality-workshop-report</t> <reference anchor="Kilkki2021" target="https://www.iab.org/wp-content/IAB-
uploads/2021/09/Kilkki-In-Search-of-Lost-QoS.pdf">
<front>
<title>In Search of Lost QoS</title>
<author initials="K." surname="Kilkki">
<organization/>
</author>
<author initials="B." surname="Finley">
<organization/>
</author>
<date year="2021" month="February"/>
</front>
</reference>
</section> <reference anchor="Davies2021" target="https://www.iab.org/wp-content/IAB-
uploads/2021/09/PNSol-et-al-Submission-to-Measuring-Network-Quality-for-End-User
s-1.pdf">
<front>
<title>Measuring Network Impact on Application Outcomes Using Quality
Attenuation</title>
<author initials="N." surname="Davies">
<organization/>
</author>
<author initials="P." surname="Thompson">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
</back> <reference anchor="Zhang2021" target="https://www.iab.org/wp-content/IAB-u
ploads/2021/09/User_Perceived_Latency-1.pdf">
<front>
<title>User-Perceived Latency to Measure CCAs</title>
<author initials="M." surname="Zhang">
<organization/>
</author>
<author initials="V." surname="Goel">
<organization/>
</author>
<author initials="L." surname="Xu">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<!-- ##markdown-source: <xi:include href="https://datatracker.ietf.org/doc/bibxml3/reference.I-D.cpaas
H4sIADEb9GIAA8W925IbV7Il+B5fEZ1lp0WaIUBSJalKnLFWJ5MXpXhLMVNi ch-ippm-responsiveness.xml"/>
VbeNpW0AG0AwA7FRcckkqEOz+od+OmYzz/ML/dBv9Sf1JePL3fclAiAlsdg2
x+yUmEBgx7749uty96Iosq7sKns/Pzo9fpDfuOaqXbtt3tita7r7+XNr2r4p
61X+wnb4Nv+xN1XZ7fKla/JH9aL4qbVNe5SZ2ayx1/dzHi1buHltNjTqojHL
rijNrNjUf7N9IeMWd7/K5qazK9fs7udlvXRZuW3ox03fdl/evfvt3S+zrO1M
vbg0latpnJ1ts215P//vnZtP8pbGaOyypX/tNvIPeuPGbLc00/8ry0zfrV1z
P8vzgv4/pze09/PX0/x70yzMlW34Q5nga9sOP3bN6n7+0/nJndPzU/7AbkxZ
0Sxtt/yva32ynda2Gw7/cpqfr822bEwy+suNbQYf8+jH2y3tUTK2o8cuW3ns
vxp8O527TZbVrtmYrry297MMuxT+yvPHJyeXzx8c3+dh/BHG03rQOLOY0Qbm
xzR2OTdH/CCdVWlbDHU/X3fdtr1/587Nzc10OZ9PV+76zsrWtjHVnY0fqJj5
gQojA2Xx5ZcbS/u8cJVb7X7zRPIif7m1NVFW+O1nm1vhaOQimRUmez7vZ2Y4
vcdmbmfOXcl38vrONCvbxVc3trWmma+nS30YR3Jn288qelFXurq90+LXxaK8
xmTKunPFwnSmMF3hf3MHE3j98tXT8+9fng3ngOv2Wq/bb75nk/zLu1/ekwlH
Isf/FfpfpUYaPXwgpOg/oSnSX+d229nNzDYyIqZ5srbtumws/h5O9WJt89O6
sw0RfV62uQFJN3bh53qUDPzYzpreNDsZ9sA8ZX7n0/C+jx4+cY4p3Zk7N9ti
7mgKdXeHFlL024qOvr2Dl9y5++0dYTNzHRFnwZMtyrZoearF3buz6XaxpJed
7my9Ms2vLPPR27LtWvo7P6X/0OanizzuV8SofmWJP0z9qz7HCmlyhZ9cIZOj
vwuaHChDl3be2bI+vLDHfVcyPbll/qM7/5TV8OifYy30/mKp89GZn5jWtPsz
v3c3/yvdwhazDqv/0T3S+2I39Ip2mv9cLqyb5CeV6xf0H1cvbWPrOV2nCTH4
Wc7sZ7ulJ1+vTUeyIr+xeW2JgpeN2+Qd7Y+/duc01He/e3fOprKCz7E7tMTL
v8nlv/SEfPk3Zy9Pjs+Pz3W/zkgQ9Qeu6jN3Y5uiMjuSO1vbsMio5xa3tnZ0
eesFc69rix3tt9vwcNvP57ZtP+EqP5vqbPZu+IWtKnPVV+ZzbEuysiJZGa44
rayIKyvcskhWVujKCgxEmsU3xAqKe7qNx9WCFBezWu/v5Cu7opl3pKFgI9ut
ncu21XkQQERCQjR/S3i1JVrrRSc6wG5/ZS+Pp3FKn2PT8N/i7rfF3T8VYdzi
5ZIkWbGFgKs7FmVdQbILv793nWzOa1u9q/Y35hikbnmtzxwJvgvbbIg30EBt
V84/hYKeT+Vdn2PFUDVJZVzRvdmQBtm1l3MoByQDzGI3JS3xbVjfYzMr63J/
gXsSGDzC1JDCEARNfhbp4RNWS5xU3vw5lisj8dnptHXW0+4tFNTnpluXBxjr
WWOrclPWmC5Osez6Bf1V0UH2i13KbvNXtt2StkOLrUcM4jexRjpcmcTnWG4y
7SKddsHTxs0PUmI47XDo5/N1Vdakwn959963I41Mf/pFO2CdfPxeY6Tv7GL1
oWO/9+2H9+HBNL57+M0pCY++Xpvhp3+FhlT2e0z1vK8XRM2tqYdfPZrmT033
rnhg2pZtk395s8NWniXsFrtR+N1oi0e0Gf46ub66sgdojTR9Us7bfG2r7bKv
aLo55ti2YKGBzJRuP4Vr4kbJyz/Lso8fXOqUL3XKJIcvw5Qv/ZQv/VVTyiqv
TWNolw7twFi7f/R2i1mCvujLmr6i11S7SX48n/cNrR7/htpyPjeVmVW7T2A0
P0/jnPbI6LlZmNpsTVWVw+++JzrqN59HaT0xtdv27ZlrS4iZM0OLPmF2/IrZ
sWzcK1LEPrZnp0/Pz/KzVJmp981K4kaRdz2jbarnu/ynemEhpcziE7bv4RRq
HSa3r+/QK1fu8+wQduOSt+OyCGsuTs/PBteurIuw5EKXrNv33Myfl3SMhwjv
gd05ENEWuu6FbQcenf1dIuprXNvmD8sla9Cd/2U50md+4xY+nSazG371gu6s
NSSi7WehNPz3sv7bzWXVV35fyqa92h22hh41jWsGNCU3nnkTqbPzK2J75zdl
Ryblv2LmPpnqNIYf/wUf1wcebdeN2aO2781nsbrwGS0D/oaC1z9QozfRoAKx
bWULWt2CQjXdVvf2Id3m/Z197OY9M3VSkishrgmbHd26cf1qve27Sd65fNu4
azKzaKdntiMCyL2Zk9vIFnG3R/r179Y9aE8x08+xey/cVWlYz4qXVOmiUDlQ
nG54ZfQNbulSd6NwdaG7Uej2PSCe3vYz5swH7+1FY+oWnlK6pLDOonrNevdI
Q/0EwiTONprE59ikzk+bNe+/DcTjcXN15fZX+nJG77wmLZZVbZipxHDozhGZ
lLyd9l+mAlIP+OV7eunTf/yvdWVvyHT8XJbHViUd3R8YoC5dXGDXzdtD0q5Z
2fxi7VrPimiKF+6GDOs2/95VfPj5rbPGdW7uqtvErlcrIq5POPpXU57D3h59
b5uZ6z6LCsWr4cV4RQpvvPyhZ3NSN+LJ2hwwvn5iBnJxcgY3Tk1GVhBTIPwo
vE7+whzC34SX247sgndszX7anmA6n2Ptb5dkkBEzuCmXZYGgAUc/rr8MFshD
S3aAhYNif/Una1LIbL2yLa/ObXGZehqPZgTFBvEHogmbtz1/pVb4DXEC26cy
nfj22vWdijJ65JNcOyS/H9rcT3f43cspqQUV8Y72/1eeS4svdNEDGeYjWeL2
+fPdL4NWQKs54NXQjVqQfXltaWOInEgGwddNe1/MGouwD293tF1el8XjMjfs
Xsq3jqTYp2zy46lOam9/H7jakPna9Z/HSS6OkEv2hFzKei/9ei+xXrJxLg39
y17qei9pvYnh87q8fFxeynovZb26q09ts7Xv9ndVd4hIOSrsD01nPtHKO1+b
ZWnLPSKV1+89flK65XL0NDy1a3ezp+o/cH1rZp9HGeVVF68k1Hl8fn56rNv0
rOxnbuXaNR339YHbD+Vb/bFz5+jKMT9jqnsgmlJwU5NcKeef4n4iuTeYxmex
YuLEi5M4cVKBmkImHu6tTrzw9/GZuTogBU5JA6zZmfvAtDBAGkMnOSfBVZsV
X3Amqh/d+SAS8GmBqBdkCo+Y2INp/rNpVvu6OO0eyaWuoRW2ewNhLf/ydt67
B6Pw1WIHxncJT/Il+Nrl6/NLnugl3VpTXYYd4g26eH5De+EdESRA+m1nDijo
ZW2LJ42h/yzyVxcX+XNHssWJlV230MmTWMin7aZ/+75Xodzs3dBX9i3RyGdR
vwaW9GW60kta6WVc6aWs9JJWeqkr9bKB7ushu+YhccNNcVZubQ6rkf5b8Ef3
JZz00LFbuM1fG6LLWyBM4jD53NQIMxH7JPLMy+72d58YEZBpfY49kjDphscr
yu12U2yxmAWv765XUJ6W1dXhS0mzRkicXSyO9O5RIPG3axXyir07R6dW2d3n
WKm8ABFSmTJcw5hyEa/JQ3NdHnZXjp11ZNcZUkSJEwO5oeH//GXfzd2G9DKx
eX204JjYXd3vKaK/9bxfTHVie2KL9OnNtv08hHD24txVBRn4cJ33s01JAp5I
onMf0bTAzAMQIWiz/21t6tUhNZ54/plt5pZY1CKopWTVqZKWn5wcf1KIjDgw
v3P46c+kwLuxCkX8+i/959gurOYyrObS65v3QjDWtPODIcQ0BJH37GcD5oMN
GFcv2Fb8FKl1MtWX7pt2djfW1MdYi1TFTLFJnweDseV5CXdpRjGYsGMPmrKd
uwPx6+P8nDanIvJwm42rvVuO6IZMo4ZuIc3xnYVw3mETVWt/aEn3+BRiIp6j
U9kzYl6vy87+BiI7YCP9ixvZ8g4UCywKYCbagLBxz+endbc7BNI5rhF6LqAw
5Ga7JXWbGDXtG0mhEGM5n7vGfso+fYDe4N/VCX2Usn4ngX6uuBUvVzVN6EXw
ZXdwgsvr/c6Fz/PZLn/prio1T/YgYZhD6x9WBOAw6joamYF+h7x152bztHY3
7Xgm+vHH3m82V3iE0Wh5nmVFUeRm1na4HFkG5/ZvxJJFsOeN4eDcIrsum44e
rXbYiS6FQx2TDKX7MIctSraSaRb5Ldr72xIdjWR076vi3jeCKZvSZMpWkaR5
2282Bne35YH9uyf8V+e2cG0u6Cb2ZG0uJPTVknTNtkmgmg59XvWQVC2ApTd1
bjr+PTAXpJOkA09lZzblYlHZLPsDltK4RT8Hy5V9+vXFrV21aHM3nxu81FRh
dHo/EeiqZkdlNnesUTY5gAcFkAc5CdRenTg4ms6u4MHB/qfbKuukS9r2qxXR
VLbseQ4Lugk80f2f5CaZ7DTPeY/je7eVqRFNpJFqHsHvC+CG9CRRDZEkjCVs
KL0Z39GvHciFNHfSly2Tjl0u4cfNNEZACyWSuFHZtWpcv23D0AEzl/z+wrRX
+WNHIpP28tHF49vTTyTOX37x+Mn37/8lSi3r/410mid0mv1OOv3DH/Kzxs3o
WPKWFE07ok6aN6aJ0/tbTyuj1/voHF++G9ggZcd7c+9uvmO8nFk50hjdwuwm
9F3GoM15089LIuKtoXXSNCwcP662//z7f9A8TUmbWZVLi/CmA98iUTLcWqXF
sslaxyPh8Un8WAd542atfEpk0hFF5FjoltF42C98c2UtPsj7bQZXZb4xb4Cd
uhY8H8mysp6TVcKadd3zkdGMtzIxWFbqdTP5k3JlZrTEubiLadMntCUlCb8b
11dEtpa9phJQMCBQ+tHCzg0CUNil43qB0zX1VYvLrFEH9idg71s7J3rtaB/1
7QxS39+YJZ19jV3JZjQU5s3RECP3mH7dE2HKglbOLYQ38BHQIeWzssLB7ATQ
mGXH8gbi9pZdkROhMsO4I755QdgnMbO1QbjEdX4RC3gwN/18PUXcod4JME0e
63ZEy3SGcd80Zlcy6Zsua4BOZ9qV7SqIG+YVTbmiIyKlp92CGsEGyyUNRfSI
O6DRjr19BDQ6h88CU67p/rqav5tExFxuqpZ4YYe9cEuaC22+zTg42PRb4Yd8
IuxCmoYpmGpFln233pCmaBbXiGfyg9jBZUnsgkYDfZAAqPCha+g0HeiSX0Un
t5Al8y/+pq501r/8OFllyHjclHT+lm7hll1Q9u2clAFhFESP/QaAQYgEHHzZ
6QjTPETzBpsyAeZyjRP68afTk0n+vK+6cmvoMlycnMlNQRjkMb1YEPPs98eM
M8RUGS/i4wD0UmwMSCREarP/8z+RADz9YsMhWLa5aKINY0HNqrHqRpshTAB8
Fv3yLT2HEwItFZwaURT/JXtWXtmbsiUavCbO6Po23eSMyVTvCE9625TXZs44
DKZPPnjMHdO4ITmZTls2IaNNoJc2uy1W8vDFuZdMlQOJkhnhKropdC/OsURl
ocIzlnTDHNHOzK5LCFOWh2QWML9oHC2U2EQph7R1WzqiBkJtbpmg/LH7+LR/
UsnP0Xl7g1Xf6rGeRFLh/vs7dI0wX/4aDI3PlqRB425yjgSLBNkTf9mviD9T
biQaqkInnAGRactX4j5pOuKIKlvZcY02q8HpL7oX3vNgeH5HP/ye5ifXtrKi
IMSlEFEzHATqBQZCSAS8hrgXfvrQBWKjK1jRQFghpAZtlck5OyMn0l9ZPQsO
XfJPebp6j1vm58Smm76GNPfh3nlV4pqEx69KkZ6nZ9dfTfIXxxcTeOPor2+G
ky7BHiEgVTwbT/ikVC9JsbqhuaaTUNLdWGLPddluWv6FuSZZxpuCgxE69BOO
N5dIFSNdOJHCC4t75d/ob8IE81sS4dddyWc9YcDDSqQKT5FsLnw7B+PMw80h
ZW3T1z7dJF/1JMfsAi8MSAO8px3IhTEm+QYsrqIrTNduYbesgsCHlATLRBS9
RfCaRtwoXI+hMJsATiP9ghVG2j366hoXEn5bKBMltqSEzrPgKbnI0oFXpK06
+DK5VDQG9qopV+suvJsnLTSlm+5BI007QIZIrIHuP4ZxbCHQO6Bwy63L+YpD
N+HpkKozxpjgVfTjtiTu3Zna0t2inbKbLfDmKpcw8tq5sT6kU8MuhMkxPwFB
a2SNfzGDcZm8d8ZBjTA1QB6IXytFeupZAujJmnqF8enHPgpswvLlbZBs5azv
LCswZMKzhpGPtINA8n6TiUKI5pY7paH4DiHKxIdN17ZCKL6Vs4z0wUuHisMM
xuy+Gy+CV66PqAHE8pzU5U7IF7uqQ8p7iZBcQ9LB4AGmDjzC1Eh6SAm1mlg/
szKaMrTAw2+9gfRmmvC3yO6tlG2vMYFhBw+RmGeX2BQZyG/FDEp5u0UAjHm1
UM2BC/QdLFGf9ZUfr+iemH/BaEfq2YJlv8jLbOlwZ/gWQ3VjewW7YqCQ9w3Y
oBck2EZ65Jdf4t/t+/csTFJTOSd1hfbH3si9M6S570iQW3Dqn+m9+QkZiPQb
D/L1trDwLPpiGNQcfXm+q7GTJf79BEZlauNjrzx0NGfsaJv/8gcPsWGETfte
di+u23+dy/dEvUQTZE2LHxp7OZiEmo9ZlLdw3y+sbdm2jnbaFxhqVmwNSbOB
SYrbR3vdZmyYX9tgFssEJnmaaCGbSNNx4OQ5Ts9B3S/y4zXkfEjSmH6OBBSa
6CDB5f17vKjSgNL0Mwa26E0xeKavsW8N6x9P3bvKXedTQV6yKfuCN4NYRvRM
2hULh1fRgkDiAcdujjBcXZv8AWkILZHmZppGaWp706V7UIii9fLFq5ePEvaH
UR64mff3Toje7Eq8vBMkkK1Ldu7yAT11pPwlXt0pHNPilmV2wmfgHdPz1DF9
rY7pbYCTmB12J3F48/Y8IL1oQcOElIBpzD9ge/yDGQj8Lecg0LCDbAYe+GRN
b6J7v1aH7SR/ig+QcKue2kn+it4NMc6u2Alg0/AIeBfsZJC1PP2wZ3kgC8W1
DDpIPNQfmNHvfP9vDaPQy2Mkhl/9EIYuENyT/Jm9LhW3PaCe09+ELTf5R8Hl
9GYPYuf3PqLXPud8D0cWBC3xVUkGyYXZVQ4H/QKhtsbIvOkWeI6vPBSU+qSs
HeOopr8fVjsG1f4aqpZm7/G8PHtcDHAdQS5P8r+UxgGvTFdmR1dfUMoToEjw
5/cGczwEqI4qX1l/CFPNpxaR2vz+HwzddU2dnf5qWi79Psno1QGaUiCfEyzi
jcmf9laRntNPgpyCk3r8qrwCl6DcaCYXD/pGGTS7BPvw54GhYuKZjlUb+J98
Esv096bMYMiYfMNjPjUVCF7C4JP8ga3NG7OhYSTEPv1QNJ+GiuF/HYnUhvIq
yTsass7zztIqn1S2rok+aWB/PxICuB/sU2+aelkiqbjexC6770D8Ty0s6Nck
VJFxjBuAjdRjEzHKocGPaUwiuGhpr6ypiguYrs/tojRpCL+Vl9WKH5vkP9Ct
I5tBYWb0t1vXiiKbEHGT1gPs2CR/SHtJosJjxmiGAGizmaugQAF/AaCLHQ3o
ONlRW78pG0Q0wYqOxZE5XgNpd4/LjhkelnPWk0rcshAbCyfil2RplHTjFY85
GXCP/62QUrCOAZxVFrgjORkTP8hYB/vz6R7T35WW8hvyUsBBBjkwPIln/Zz2
UDKTiUrJLq/LmJE81VTpfD9V+gN50vkwT5qFjc/B5hc+J9VRUx2n/0qCJS/H
Z23KyCXpGERvnCQ7/fUMXCioPndXB7ham7IaAv+mQ8DhyW8DHNLYeyhGfUe9
avpSwBmpRgVJ0dI087/0098MDKHXBGQJD//C0vwFGSNXsQmAmOlnxeyApAMy
iF/tr1cCx53kj0mBmzviaILdnX4+BDELRA9S5gmc2QXYBQoMTD+tJgKZtJ+h
KgLNLNRpkIk1tCHEjUapJNPfncACDXk/J4bf8YoEV8fo/ME5/+fEB4bFizMl
/4ug73VjPZCRhvfZBjKmIwHMWRDg8aSe+uSH6b+QicH3VnI7+CXn0PAbBoUS
/3lXVmQbvuMrBRjnhDftXSs4U7ojPWhpKgcl+NLpp+NgcUtNIsTPTUcvrQI4
c5J/v3NvHG3a03JDe0DCuyTq8HjM6e+GisIWSWCn8tKhXj9W5Q4VkgGFJWVo
eJgLZ5sK9+MR3a+GVF2YlkTXdgM7UNWrQtWrwvJDxRxfF3fvTr+CyPy5fGN2
MRl2wq81XZoESztSbmiHOPf1IEf5/Sm72JQ0KZiX81djrsgcvvXDbSmoMj1c
nwW/9dVc6HcDv9GFhKEZTB+dOb/8IfXkiF/EsIcpgAjIYLA2R7xx4EaaNe7K
1uJFWrqehJzdYn9s1noYAjsaLQw/9XKZvEFwhK7RsoFiyZ7oxLeEsFfqV2k7
w2EqjvYahfgoC4vOJokfpbHwoffklzJ5utDZvZ+o/0dACpvDnqjU0aWoqvY9
/VRDHIf8UwPfWDGPwvL9e3WGtd5/lXrVfvklfPz+/TQ7XnbqBpa97WdhV8XZ
teBIha6hoAGSsQLgARGgxtwMMDDq/dUoN8f0FqSsiFcr7Dx76Ofl1hBjyG79
8ksywvv3twV/sHcI0ev3yx8O7rnQ14eOV2I/qQceT4iGzH4EZfwZ/a2aRX6L
yP42vzyG2FLTlb5/dNuPFrhIB4zDlCezcqbyzrcwZxiynamuWt5Kdvd2fj4r
629GpNMZoiTtvClnoGmSCWsN5SREKagd+pTuPbTRrHVVH8LTdEHiIyXpjaQG
Kv0qBibd0JF7UG14uAax+bwDkDgs5c6V2T/KknhKm7zKknrMVicUSZSSaV0E
q8gGrDWgLrgE+ApuIME3HHwW0Ae8pnSMGkBXBjuBKnNjEZb5DaPwLQme0Jfn
p9ju+dU0T3zGYansU87ZqZxsPX64huOoEWuRTHNSJ1hDl2Ez9m4z/f6BbKqd
6k9ReTnkqWYuOSclXvgAn0ShDxJNv0ZUeYfbpMowTYTd+LoL+FtIgAl9AdbO
wQ8E17xxkoF5bqCiCt6KJRxR3h+H4Jw1cw39TSn66ZueznoJ9HdjC8+N2Mnp
oMtWgnUabgWADOqNdD7g5ODpn2QzMtcYUzEaTbBIHUlZ/NJs1w6x9VOJfkNt
9QKDr1/H96ZCvTziN2EmiLcseMKSsDnJ6XUZv85wtKsSbYHnEsO/Q9gDmZ+0
C1hCxRJ0mp92X7SZX2EnPI700qUlA2gJ8SVhPgHcmQpsgA01niRNY8qoibJl
kATp3naxwIGaRoSzEzgSeCYeAN8kAVbyATVW7i7AVMzHspDQzNNr5GP1r6VR
JExSP57R5VuWwLx0rcZAs+zYI/zokO9jOTqFsBD7lkyispVVSKh4yZyobBCE
zuLL6Fq37H5vLJ8JtliRK0p3oAQm0lZQLnNTZ2HIf/79/w6jJkv459//H77h
cA+0Qq70ZBKCzEbPcqThrQGakHYZgsu7dvTBHcO4/CN0DxiViLt5/Ors+H6+
cz1xJJywy4FozW8YnMLADqQjVFq07EZ8nfJt2WU3UIzZN8AFQhmp0/D5i6AD
wCMPNzAHiixItdZWIr4yZW6C4pTdomN6vbZ18ttrUzGIk14MbMBKUDb2bQfO
G+k4oA+Y7U6yhCj1BRrQpl+RbTHNX/KhtJBB6gtgPlUmjBU4TssYTX9xJroF
R1s2NnhvGJkxdFZkpt6Bd69Ae6SgMpCd9vLGhhs8ROVPEnJk18eqdo3NBpCW
LPsJ1Ue7vlY1F9FFRL1YRhJjh3UkvK4ro191S0/wFLyqkEG54R1hMwLQDXAL
JnoFTbH7AVQCmIcqN4xXm0HAgFNk3r8twuUDr78xUQO0C2W32GHn+C5nSwWP
MNsS2NBQyGvJPP+DwBMVo0siDhStv8El5/XQrwRVxnvpwbawqRmvxYA/PkiS
hacS6uGVGTlL6E2s6xErcowGRNxpbkXBwq6xLUJkeO/bb78RlqbMILw262T8
U/+QqP6loBJFC4iQUcGUkj6xUwe4zDhL94aeXhqiIuYLAEUDBBnfIFfJkdVF
5zZz+B+7IjNLoJ2VE2DKhs5OzqXdYl304vmVt5n42jteL57A5s1ND0gIIGYQ
STdMxoL7aq6kVggCjAImqQVcAwHCJ60EDY+xOGEyGo7DLaIP4ra/ZKkDfAIU
h/uRGafeQIVABc2Gwd+KCfRKKMrX0UkK9hKYRQAU6h1HzTpXICAbEQ6tdyEy
QWQL4CndVuDXyeOpfidTiLBfH1Kg5ZOIJ/qAdTxz7KCHOApIYQ98RayC3kMc
WZAhHjvG37VX2Q34XjKT2t5EgY3hjph/iToYvxJalOllYYf87LBT3x1lGeMa
C+jWES3W4D4LktwsBMetwpdNuHW/ASYdoLkl3b0s/PBGoHXwgQAcNgR+eS00
ADbAXBjJ2u7aju4oPQHvX34Pc0PE+8tcQv5EekQGBggn5j1s6LLekt2y0xXS
ftyqgtdPg7Bk9G/MO1ffFt6WQnVSfL7XVFOfB0sSWroqrqf7hsrAki9SY8Gb
9UMfilS4jYr6rC+rLhhKZ8lJstczG9T2UcAWCx3dzUjsdGVc7TaIiQRYKZv1
Sam8BXwFvFjeunnZzIn20tHtfF2jGjNI06idXoKxZi1pzqAEjgbgxoiuCHWg
DnQkAgNSM6q2RNmC2gYwV7CeZe2LFBEpMXJCgEIL1udweaAbdIJO82hYRgpl
iZG5FBR6I1BpaG5t3G1AGyG3WbIQM+5ZGRU+BJi90lebgNeJsDcQp4P3k8lH
ErtTmHrAOwYcgfhJ546MS5Y8oHov9UagrnaaBEVM4umeV+xlnuSg34kgTbK2
A7qDNUbwXGhDA68wA+vZhhy/ByxPckfeZvjtYD3ec3/DaOe4XfBoSUkWotFy
SaysMdtyAa7XBzuERHwjkFG+dZyxQNeYuRvbSx7Or/iljKfBcK42aMoYDEZk
8P2YLSu4jDvlzeZFR1xccuZ0yD9Gh8PQGwFncqnmBZBBGwYo05JhaTEvYSBY
PF7QODGwFtPoKmhvJaDp5TvWU8NJdbste79gYOCXUwDLQHcTNvP9MhaOJYqp
GMs2JIi9ZTAnSryI2Nw4GFiFR4CxR0I4LW7QYPJ0zuAfrFdHtwa/0CvMYDY0
cDYi2OE+Pho7bj64lYKFVXRetm138zVzC0AKE1wrosXeMVQyPkZCSKI/sJpv
F2gaIHEG+vHGLWw1yNQi0cBlgqxHKqckHvIJMgH3rcvVWsZI8qEmuq2PIhJG
My3E7uO0GFJZ5mQBcfLE/pWE9TJQprWwvdCIKEKYeKYD07RnNl64qC8FKa67
szjECNpML3SUaTmLe0gN/0JOM4ncE9o93EqQpLXXioPvakEmf7P7oo3uvVV5
Hb0KChQC10sAKneAp7kpF8TIVI3Ea5orvqU+nYAVStacWBWAngClxudLVewm
XSMrUI6ufuPoWqwkASjnzMxWgUllk2LL5EqfKcBTs+aQFNUO5hHU4pHVMWv6
FpgVt1yy/jZnesmWpfpLB+lE4SKLx1e3SCUCDQLWBJx3jnI8dBMy72ephQvW
dsVGKjNpOa/oDQHISi/0fY+lgEMgM7DqmDr8CdCl5RhWysMhxld8B+G/YbpQ
hYtoPfMGPEMm6UxKdQT0W2hp1vsYQ/oLfbWynci3NIMCsA1DJtd+WB5qOaQ/
bzSyVaDZt2pI15l/EcRQRUOw20knNcycWYjo5a2bQ0XBhc+wwwwOWTJ/Jmu/
8nR5xLZ8QeyYVMwjyUmBjd2KiO8Syg7Cnd+/glkdM5RGZD0RsySD/ljt0rPz
82ZEegNQRsoKvFvAu81JFyKGSBZfSO9IgTK0a01Am8Ohq9lQdGyLoGGVYAfT
7HH0y+C2YNaoWRyJgDbnxs7gob9p94vEzyuiGRLQXalOSxHRnGztU6r0p1+0
mhYxJ85ivZ+B03nihPgs53SYTRXO8BDzFZ4kUfkW9lnGRwRGT8ZFOcNq2EZX
K4UJMCCaiQZ6WnA7HlX8TcmlYTO9VibbdgwMpoF82uTEs3ncCQYPXph1Fxju
rAeDnVUOJoUjLkC3eG2QEydJvEkmXMg86rcLcSP7tLKQrMZgfdrlfm6TzDtB
NyKhfeA7bkls0Qqb/PHp45cZxjogSNhlDqUEeZeMAFSKhpN1x8GSxMDXzJEw
nUmuVKyXS8xsryzOsV2emFGThv5kVqzvIgrqkYpFemDTxkST/JxNr5zZ5Mm6
3Oa3zt3JbYY8VB5ughRqlt/YNpg6rY8jeXUj3rhpTr8fvY2kOhT3rOXFNekW
c5huxUfB2Dz8c2UkS1TYhXAIOLMzcEHbFAkvm2PGbPuyNEAGDp0uwmwLEhtA
Ez4g1s92ATGefoPsAcN2HycgIWuE8x6I7yCrGFwXlm0iqlnLSN5I9M0pRiwW
xnlqJpcppj8Ao2Spp1+ZNEPN8wzP0iQCRlZAvULGkcr0Ac2o14vTwiuVxDGr
Rb3bauvhXpMS6h2QyntxtttOHMwpitk1ISeIXVrCUb0ilw3CetelkQRAq2Mm
Gs2EXetJiMDha0uciAgJP0o57bUFCiOmd+68a8gsQDZRxRWZ1IFSskqhYXop
9y6a+KKCSiNOv8YghYmeuxGviw/STUYOV/ZCEi+6KqGKcFpuujqhtCUrPiZD
tBK+M58XxZYQKMwjQ4O6Q9RcbcRhp2ZD0CttJoE7XGNTS9iDjZe+BlSpZf+L
zoFObZo/FsYKYZfKWr4g1Y4ZE3EGQbsme81+00nYckTSeDFeRAcBjRO7duUi
VULLRliyuK0Tfs1prEgKwuVG0RINpbBL22J8okci3uwx8HWeDybp88vG0On0
4pHxGVM12xWRWpQIXBBOfE5gml4Lk5sgTng1xgduOl4unQqHI8StIqAh8TPw
D4LHMENi7YccQAXbshrO/Ig3qJBiBbv3WXZvmiAoB4RJCyWiM0h5nkl+q5iU
HAgLnrRp9mXUvtnHMRn4PRqO0v3z7//RCUKM9aDwHhmauPlaSrvwI0c+IOaf
OspxkFM7nchF5Kdkb4cJnImqN83+OFUsr2cUavuYxM5Ntf0kn53T7DAf9YUN
rhlPoUdWc7xUqUpsQOazsubHeZClzzVmU7uxUIDKrqusBIokNqRJlCak3bJK
oveHRtGwK9u1Tb9hD4VqOLm/Hj7SsHVuoKDh98HuVW2sbNJlFZz9SNY+4APT
7CvlLz5HWtanwaGB+sVCq+8CrouP/Au8sN0yxdM8NYNeQrTEO5j940LwEYal
h5RJcY1g5/gBPwkRp7go8mQfYnlDnEinuZGSJi1L9zvBtKr8beIFSWMBYRMl
AL4s7i/IhgH92Eti8ecewRF0NPE0eiRG9RG8gUfsJKBjuD3NvhanSkpbrAhr
BILu0j5HlKFwj/DaAcmp7qorg4/zqLWk+ZDcMaQFTo8Yi3M4k27IBjx+KWiz
LXjswuO9Wl8Io0sxOtGugfyMo2VSA8DLmETWY2cX3qBjs6phOMRCUleEj2ue
dBbuOWpAXuNkkix2n1Xn4S6sQdktmudJGJTRYHUKbMmTPn7qLNeiYKlWobPO
smdumByhoFq5bzA+EA1fqrsx7IwInKEfbSTs+5rRhw0c/11ApiQqE62ICNu2
g5gpv0GKLSiTMJrUjdxgp3eC0z44g5CvZSaUMc5IneYvsKH0ZcUB2zBy4DtM
AqxpA46Sab6u+ob0iMQx6E9ZWaIk2ecgtPtyYXmCuJ6Z5wN02ZElxcYefGeS
wsglGXydo6UBst8rRrYmgsByAitQUaBJ1D5QQ6O2PhpeBy9yOt1MKiFJmR++
2TvvPNRSBnUolORhP7KkQliZoxFa1hIx/aCicrgDRbzygb3Mxb9oubd++SUU
BXv//jbWgUBLhXpcSTma43rRQItRnT11hSchupwsnYrLTgwOIvFtrSo3AywA
b/e51sRa6OsGHi3of/CiyNzFMFblYMv1NTSGyrOUM2SUucJShxva+MQUhmek
UIXg0OQoeQ6o0PXwx2GPM0SDscyGvfBgDELOtF/taE9/+UW7hb5/n4V/p51D
37/3M0c6TS0LJgOqYKImSlzqRSOjCEgrqBacezRzb21LrFsKCCWMo3XLjlNa
lOGyQKeX+/Jq798zX6d9iuN4pgK3p2kas0trKfC6ONyqVaIGuyLZ/F4OciG5
lDDZnRZjvsRDAU0ijhYVtlhsIvEOMgNO1Pzg2FNFDvInQ8UkUcbAC0K9isNF
ExJh5OPObI59T0KsWFiGzrPNhGBRHoNFDXx+DRx10cNR7TJ+bZgws7IYw0OE
g6W4D3BoRSOO5WtYNBbXkjfqsyz5SJj2kBLf3v03789DymjEfcqzmfgmfAEE
9pV8+/W/+fhHg/1hBgXKBQKh8xGpW2RiZgDjzptyq3bl0jJmiu7OgsQ/gFbR
rzrelFAMJPM4oBe2W1bl24kotgJTg8CVihJz2mwhVQ6CKaKIS+jsH7/oHZ7A
Er5ygFJoANh+Be9sxZJ3pdXLkojAwmz5ls9KqY5x6/jBq9ust7BvBFpA6pga
x0ijkroXNQlknHnvSurlFj+JxH3bD/vkv1AxkXF1etGAlRrabclJkQOukgQQ
cS/csoADD9iUkvghmc3QpCeBu5IuohPhCgB+cvCLsYcffulaIy1fF39iKm2z
4GyByO4TMKMMjxIBjyXh3l9MRK4rJGcHBTYLL+MqH0G64tuNAtLiLge/NPud
VcFnRBgjXYg/ruyenbb2EYfo+mezW33Vps5sDW9RXDefyIJ9x0NDTpRzhS55
X22Zqkew4Qwj2lUICSPyYDTFn3KEEcOL0g8a3aObgbmekHqKiY26AvjFUXR7
HQWnLKvy/MmIZZJalIaH2cGMUTxTh9PGdCwiF+IW/mBdhUkMQ0CLX3JSjOCg
haEBzmDQf874zH9BtHEUIqnXwuW9mBfB/LZ1cLEzXXkmmUUmGQoTeiunHEbI
OCyGolq8eo6IhUuWqQrMCj1I3SyIkbNWK27HIT0MBH2o9KcQd61Rw/QuKoZS
/KH4jHoChPYSZ0T6SBrjCe741IOePg245sGkAOGTdHOgcHKMzs8nRr1NxppW
QX9v5bRuvbq4uD3RSg5AaAhzZZOaNS2xptoYzFNNVqqCoV4cp0QpPrSv6Rr0
4g5jsB7bDMS0FEJ0wFyZyAEkg6gPVY3+NPrDr87UnVDtUntZGVSLCIcyUFW1
UfwRV7wRAiDjJxtWR+Chh5mmiW1mELEU4pUZw56G6pUdvQqbKeonCUMSvEe0
qWfPtcin1pirEx/tgpMYhtkCR+Wi8qGPnVzoI/0rkzDIfsmy6ZEArlCSlJ3V
4ez8/c7j/ZZR2IYVm2UpmrVhZlsIBHdO1kpTOqGkynTdcNzswLgapDGD0moI
2cB27MpKw0rM2+SJrHPAX8lV5pQHHe0LYFjiLnhnlLKyTdnijrC2Tjcvi+KP
5UQ8rm5o6UOceZRnHL2UcFwGu34LavnyaxF0ufi7Wc51a/vx+nEZF1uOgQ9l
UxOVA0RvAsWUENIVuJ84ihISyBKuntM+yOHpetLoaar2Em2hdUy6uLEmlXA0
9nRlIUQ+qpDgjUi+NKEOFkAUcsXNYOgsHZkDmrzyANxKnZd8Cdm5foRiNVDV
6G6w1zMjqYzF8dUPwXZohAw4UzdBlHG3tXhaoimOzQOeCLsrR25uNdDUVoAt
tOHaXYjVyxLTnwbk3JbOAShcM+GScQGsGNTttETwlpQtJipUet07Cc6W89q/
Ht1votg/fvNvEvLQi10B38oaTVz+wFeFmjuc+Of1nfweGQ26j1LdUpIeYcCN
8bwLqdDJIs1rHnMP+g9bxCIXoXmcJ/HSWOxOQ+ApCYht2+a3XpePy/yahn8E
OUB7LEJny1HV7NnxC7+tE3UhcNz3DlcLXPWNhim9MqArkDfSLXTbO8C+r01f
De0G2P+idvGMxQrlqxgUnLiTvrIMHCtJnR+dD40hGi++8K7Y1NQOsCRgiwOC
SPxirWWIb2IVeJUeGlAEDHv3iRqrbewLeU0XE1l6omimqvsgA07viVHbW6oD
k5jqYuKr64L2IPvnp8IZp9H0JzHMoWsGFwfTlj1FAdijEI3MO7Q3XLpQ4Pes
+IpGy+4XqYPaxulNQ7pwSs1xQ4OtHbanZZ6vQJV+PxQSTWKh+I/GTTNNbxvg
NyOw8CArrRdDbgUNdRh/OVSMeGSwJZgTyEEiNwd2xrWAuKgatA0fg2PAC6fi
rOROqJxizzopAivevvTOVU6gdafRO42SyQzHS3L3Bu545oOZzgW8qVhaj1lm
P+zR0gIsVAI77m9WAm3hK4XNYIBJwkU9d2K/8bCUSIytjKoyZRKnQ6Yo44sr
DyQSHExdJFMPuXqJd0ik66onLTgoWKN4F7FKgcZzETmTHS3Lt4nGPOvJbiUz
Sp18Uu2SLYdgS6KYFauxEZiVMY/h8+ZSE/kqukdwYqwHhkSxJFgsAlypneym
w7OZSAWNtl/QlQxfsktAq2zONWSVHd2U9R3Al45yZp3iBJV89QZtuzZbU4ds
PeHDwXle9fOGDymbp6vQQA+f9MA4GUpb6Mptl1R1RkHjWS/oLI344Ryl/LWi
rHxUsAywCJVC9GDmn+T4EvOXFOsWkwx5LaK+iV+ab70/G3GdILB3Y2kQjXe3
fC8kBBOyjIDvaQ+B7LN9KmJ1IiGILgXq6TvnGthQlHzmKU9inCiPINnEnmkp
xk19PX9Ffo84QGinhlPYh6EgUIYSd4gToGSsn4qwcpFvGftOFIc+4k1cKgVt
eh+Kj+jWzy8f3gaxystY6I3nHcwC8QtK5WQggERhAd5NdC/H90WvPYkdrTp6
JAkS0I5pm2tbHbHjkb+5IkPVrEnxP5ocwsplLePQWh+ZSlV2ODVCUF+2g7hF
cLXgMg32Ms1j4Jq+kxGvCQE2HPpKU/7KOslBHm/my4f53nlJxZtsFLlPK+IO
QDAFwuO4a4GVqo81C172sPNyv9ngZAyvC30L2DpOXJcZB616kQ0hIXb/INuN
Q9o/aos31r7jzAivCAFjViLD1ONTBuWFolPdF4mtY05+tfMzI4oOCqyLKQIh
tZBb0uXH6AggBcvjUoEv2rTZrb/Sne5nFiANFDJozGYS/c+c54zZtreFe8c4
CIp1WMml4sYlUteX/T9EIju0VqLz87MDIgopOhKk81xcS6Sfz/uZycKAudeV
n1sSWChNSV+j/sU4d7RjrO0hGgtWjZYs09wkGjiTwq0bLksCUdpobVC0VUDG
Mq502bRsc38ABy9mT9+mbvHkGLyHvfN4+GALjWzvdOLHjPoZ/JSJh3FeROFy
bgCzHaEQsDj2GPxeCJj0yMdM01Ef8KheMWEnJ0D4PoRH44bfsx3GujZ8JEdB
4ZUVyN5ANY/FCfzmajDiAydBZwxCkPyyRPC1rifVU6bAoZU8nAzLVh8LIfq8
YPqMGXParplJNIP7UnAySBcnJsC5zqUvd0RGA3IQuMVCMEH36IZDoVjdkiur
KTo197JT9873I9BQtcRz2DcNK+QD6xc5Fwr7+MIUVlV71jpL0UgzAZ6I2QiV
m60IVGYmThc0Y0UwHB+IxZNin34KDiSR7sTT7PkebbbUNVA3s94YWNWZesTr
WO4Yxj1339D4AX1Rbhnulefjd9aGeVSGMCzrG+xNZFpiRE2bpnoFLJnEACUx
jFlpaIYUcLUqYwDUwZHCpOMq60P0XGjWIsU4JQe688GRdgOibRkLSZwPAS4s
QeqbeEuGffPoglQqqBfgZEkkFj8VaW0o9SRrvu/roXiMA8R2vnA3tfRakSR6
bXAhbaS8Ah8qgzc8hhp8PubK10rceEHs47EEtzCsSu7BLSPnBin5w9fFmJqz
guLbWC1mozY7tBdNURpFMDIO7My4dghythZg6aqYJHaoVUxkGCppwxNM7xtX
f9EF570H4YfcrAF6T6pQzdGrac9f5hIjMzm7kJiWpJooaWfiCzKcKp+uEnEV
rR7P/iBR8w85gPbSlcUBdK4QJOlsgOU+PH+mrifSdl6QAFv7csDRVy82FVmu
2R/vhphz/CHL3t/hs8o+OOVDTitthuK9EMKxJNeri5sezfjG3mec6Bk74eba
pSP4o6IvCtVXOXuAx4GbyoW/Xh+/EBRjbODDRrd/srVdxwoTE9Z47OiC7wQN
uecDEhT22hfyP4DqDK97HV/nh9t/oQh90q9WQHLaBCbU1+LziPWQl5X3QWKs
NTDltXgjTKoXpDTHsFlcXOYpZSz+npBuIFYO/o1YSNK1Ci9NmmisxRKUuhgA
NgqWR6OeZsP+LfaTbpykK3NwG6ME1YDRs8dSh9jvyCaWDlQGnJTkCTH3tVTP
Z+8Y3LzSFiGaf0GZCpc1bIuHnIRYED/fpRDQADHcS0dLSZcZstJvaT0zSVmd
383gV3RJGxp2HHI8R8vuD+W0JFjEkjFqgw9sGQF8DY5BlFQIFAbuB/Sjv2eT
tFAFFxxOkq5F0Sg7mEXq+gjJuNHb6GLzPvbMzSs4dsVITfO1R/6dECBk+Aks
w2wIzll/HOajYshXZA1S6FEsS5FYmSH0rYkpg+I7EgFHQHwQB0+j7zESHuRX
cIcskNdax7I9MykddqGORplMSFdkDAuXTmoVdopPuCJIKxzvYi2ljszKm2mV
CaG2lCXJrAQxmJR5i+oc7gAK7g+LGgimeFnOAMNiVAEyDzt7+8CrSXRDeopr
IsGmp8nyQx4xCZmObxl8HBqVpVuJkXziLsOnjGL6MYFoKnx45XyWrXRhkAot
HmHuOQy36hm8OEkFTbIqRlsWnsa+tWkZOWF4w1f461y3dCs6rYrC7v+DS9F9
bKWl0WByyYYOatf5XIAE7YTJryDG5SvOrluEzWAmeuGnxfC/UABFOrx9ZEO3
kjcpx6PrksqOEpGGY2QzU9dMUl4v/XnwUIQJ8M61+RF8RMA+MaecVY7u3WJ6
BNh/2ictBtn6zUzerroJirSh6RTgaYKs9KpHamWhW/AH6YberO6meRm77qUm
lfhfiXnHjgdYrz3IVj4azkcse4py2aQ1tCGuMjAjdG4+lbEbXr9M1wHjhevY
sVS6KlWU8LdaQEiBnh5Tm3N3VUYseMZzJIb8UjhXo7PSUjsLDZ5z+D0R9h7G
QDZfhN5iO7Sa1W/cChk9Yw/wB9u7iRLmJxZ2xqtcoQZFaFP44XX7wQ8unR3e
+hoyK3hPxU/QBl+cDDMEFlTcOuPWs5+e3ZYSTe2udvVu8+vRYvgeTCWBgRnz
OsR5Yvh2gC9I8RWBpY8WhNoP0VfMgsrrFIxAh8gOvw28f8d7ODk0IMjVT+Kb
AdYiBRWOp0fi6t/zh775b/7v+cXwVfTJ69F7/j0/TdcXHyjki7gpyXd35DsJ
k/w73klmrygC9NTji4vv6T/3vvoz/vcu/ueP/M+v/bPh0RP2ueKpP/Jj9D/f
4gd/3h8VZhE98u1XYdA/8z+/5Ud/2o5f/+XdP+HrL/mZr/G/fxo/Gd7+p2/w
A37+KxqPdnz0pLz8mz+H5775Gs99iRVlkrgutUZVheJjdq4agrPh6ooYQb/l
XOPBLrJBY6bxySZJG9AlE+kakrRRDwnlAEajjAlL5cQkY6WDO4x/0ea//DJs
RM6OV86cZHmjY706e85UrJBJouS3GkM5e/WHPxPJbMyOEZSN/S4PGYppJ5dQ
thA1SlkfT8sLi1QBw+UiklEfgxehs0lGj3bDzoZrdYf6WXXrpAADrWqEY/MZ
kYyqStIKRqc12sYsqJ4hdD0AG4jE10xTu9CBxR0ihb6qXWYWi3ZkPaaMJwVS
aLloBEWboNIQHyQlJTMoM9xVlpSnq4FHpLEcHyejRCfb2oG0m3AVbmKmGWMb
D8kO0TIUdeb1SO+SFJB5Tysj624uEV/nJYCXG6H0D4hHjAL5YbrXqIIAIHWW
VjwspfeCRel4pNvs8pfnfH4D0CGTDOsLdjF6j59wEyV+xDR/xT2TRae+qQUk
wGkarM49L56ZmQd2yFy+vHvvbsENv/OHAoJheK3Uu0UUQzu08i9TSIYPu4jY
uuEME6FsX7EkJGZrdTOdtEQoOrAg8fhtvSNBgkIsfEMsAmVr54BQt2r6Sb/i
RFG9hdYYoQR2QCjua1JKvVxdNC25olSZn/Qz2l0aLVQPyGKBmtFN8VpfyR7M
JM1C69tKvJns5MaXpcARjxqTpbJcni6k9pvMUWxI7XSbBTgvu6cEmpLiRj33
+nCpcIY3iPE+xIAMSD4hj2Q/teZil3oIQ+fq1rdcx46o/extJuijSOXk3vFI
+tFya7Jb+9ljSY/UdFLsU8F8dJnq1g/1HpKQVfCYhvDAwArv1joPpSqGuaed
ypJTYaUdEcCW2+fuSZ10c3zYUCITvps8Rnh8ckKyKOmdsNdKjfS95w+Ob2dc
H8VsZOn0s+jTFhfYAYWqZorbSSkZBcVnUu7AastfxqDK7UCtfD64ZQnw1P4W
S/KPDXMPP0gLMKcA5ogN1PKfwxlyNGhcqZMzHIXljSohpVjrD2/72FntYwxd
6elmp6UicIWAO6eNhzeESw9OEDe0m7LfHGX4haLH4kVZKHRk/Nq08ozR2kky
hGwk79OCbHaod4Ls4zLdo2HUET/MGtPS8lyODCP+H5wKEdKLYJIa9f2Uwywm
rkKYRaeDoFdCJjzn5KqXMCopIl+krofS+bB8esrp4wlE5PBe3lqSksGw0DQV
eFCXJkDLfIutIlS9QSyOfiF5KeAlTJKo7ZP72j6+69++Xvmxu6nCoQ0edY6K
ZNovV5zyUqgG5U5jV/MgkJCjq5BQO1hOBiCr1sKYxDSmMu1EPdiLtuykpZLQ
DN0pwC65cEdoWaGIjXDfDixP6925fFvWojVplRcxMYPdnt9KXj7BzZt4fQvx
SIZwIK4XKgrUbsHwLxUryF041lIfcWjGah+4Hj6SMIS8thlXQdDsvwECGsmE
+2LRR5KklqkkX2eKHw/JSWmy29YaVNbtY3MFtL/QKkvI/WYliMlmeZBYaME6
10GpqeFxWml8Qa/J+kGl9Ygti85RbbS1TO9jio6beitC5BY6NZxJ9vKgY42v
aYBuDIWkN6PtjWqaje8cNbYQfO94iEe6bSs0EbbSEVmcv4ci7kGn8YG8FFYU
svQlfWrsV4SmUlUh8DP/rcMeCGqwlwAlNmJBYw6xJ0FdLrDLsQ52Q46c85OA
LkWoOqJLUZevjSEk7mABBlahFEi7ZgjpV5L3a5sCvYib/TWIGA4dUEbg1hAz
Ei8uEVShfn/TAHBR8Qtjscok9rE3hFe/xIcLRutVvcZGhy9Ofz9kwXLdH3FX
piVgFRkBIGwr6ewBhqshTuaH/ixuvbONiyETTbzBieCLQX7c7VztUn8ZbhHJ
I6wSPsk++nOa96GGJfudopP74U9kcYCspbJTDPTmR+sdmdg1QmIScjsi417h
grLNqYe2HBTMlYwAdDkweYxqJWUlgSSgbZVYCOesDOCYmlFwjSbvJBkweZ/F
wTQ89mAhMxFdnTl5NNX3Q5XbFPsoQcgIBdZUrHroaZVkrPGb5rJ0+3ZuvSqQ
+s5mJKz5KkArMKSzkAZh+aJcDM1SQUVJhTOfHseTlebj19Yn9XGNc47Icj7D
RNQkheCzKRvQgeLljaiQYRaUxNxDIpTWpfE71aJKXfQ8BFMhdfXodYjvk6AE
chm9bFGBw02JmDIncvuRDJDU1Bh1WQYt1Vppv/FPjOWcxulp6t/sH0uM4URj
V+v1CIZb0hjTYJAE+/dt5XrhtQ8tt5iCX7XhQUj71IZZnYvGMSdhB/QjG+vA
H4ewt/Q74iJpXKtn2EDzg/V6hs3DQtGe9NdqRo5jNpOhfy1YDaTRofAyGl7V
Kaew0laXa/4H8JrHJ9q3hmupcA2z0Lkid3hymqe95MTaRW8FRuKyT1TdDokh
FDzc2zUZMOI5vSm92wn/YoaTFsPw4iCtOZ1JzcAItYsYrtiRA00k+IpEs3xY
TTtL6nUEmxxn5qvveQ+uhPhYewCGLE1QQJJs2qx34DwJVbNDQjO98cvpV0/W
72ity9L3b+FM5iXC+wLBZroKDj6Gr4h9VYYSBTfcdiIdmDXcrzE0J5GxEcNE
O/PexcRnyM4tKZOR9EieHvDPepJfcyOnpKW9xMsFEuQV5ASnxdnj2gGCVQpF
LyzGyFnhL145jdH+JAXa1ekRJBlEe6UlsqwYtGvOwz1s+rqOsZpx2poaff5C
Z1BQ2lKaiST1ErkGG1hhMjuJDfLP9bzp1xBiHiipA9DAyOZ0zRjQU3atrZai
92CmPXMLGoSvZKfFYpyv3eaJCmgbnc6UVh1wT/uFXKQQqWNndLIoFGTPRzXD
YvInY0a9sVUvVfWJRh/+khFG9aFS/9r+CbWY64kWRfvVKbIaGJMBkx3M8qGX
XybOnR0kRCKjQCdgCohV4WN5uEHTlsYG/U01ZCcWEy/Av3k6amIIL7aaot4Z
Le1wOZGIjSCU9sTRMhZDeMqN2WXcBkKS5/JDND8ula8Ji3X4VRZGD+VJJXf/
lmGoZdvd5vkIwonWdZ8vB8Pzjc6CjU6z7fxdHFy6ZWSct4iZ+nY5vPOx7tLt
IGH5aIOvJyyOB+z8JLcMAteyJejbu9W+G5PQeiUfCI1uyI4ESZo4AlZm46FT
eGHZbngeDFlUvqakKPddsd3MVKPvE4V8FupmQep5LVqYL3BlQ3ugmVVQGDhZ
i6qpLPmkaOqCM7e0SkzHAENfUENo+WaNDqjbUq1cHIcgNIzkFqCKP9NOUh8/
yU0QKJxLugvKBfY8l3HkHg8/gCGmBysuS/gB1SvLE5EMWIDyd5KMJ7ecixCA
EWmuR7QQW8Vd+/CiGu3n0ghWt/1Eyz9n2YnqUBrW/EK6XOOOciVuvNPrBkkq
Y1rKMrFg0e6sUdgmB6Fis6R6oQBYKVq51dh8+I36l8Yl+wQXwvuf1DNm5Y7r
RCcNmNIMg0RJk20xrVyzFGHbRGr16TOp1/1AacJcAWKYA4I8qD7ia/T66qwo
DR2EFpeZfBu/3Cmw07CRXHLj4Xy0B3pbU4Dl3nihbSlRRi+/M9UdhL3NCuSd
QBDj8ONmJAdnHMv3CA2zNj4Wk54x3skHuaz7E/e5gx8ZhDXJO7Irh1GQOl0J
L4Z766sIp05TGoMTSX5FX5L6FoOy075Psu6B4yrW6FGpEiqwQSaji0P3f3hS
THUenbS3sfFMZPGsUPClKCNL7iV0JupFiAPLAj9CI9Lps2zZJTq4k2zdaUPj
d8JJ9xjQ/svToK3MZUiwbeAwpPh7H8aZpuoMTaksU/4fnKD6mHSPFkUptmhg
6zzYF5lqqpvUu8aGRGJW7nf/0xK/UscG/KaBEjIHg1/YAo1ADvA3tuiFcXBF
VU79iMlCmqBQL+HOilnmfjWCG144Zthump19iBBVyfe+n1FugLbeRQPUlH3k
4tCUipCjmEeqv4aKV2jm6icjHkRWMdiBQcbZhsOcfu4kZa+AnShj2xjfnWbg
/h1gScanXGhpH9VwmFcGpKXnQrEri8a93MCtkeVB8x7idrZGOgAkS78PL0Je
+MC7l7ujsMpCHVrnxydP79D/w0HJGQ5DIKt6ieXLQBwYeoT7ZI0JtNCojkxL
fOvhMDJQEtBHhahBqQuds+ZpPyvr/m0AKGinx/n2koiM3aQ+P91n9RTsDtTX
BJXTk0eanY8zZyZ2QJXU31/h5lWFdMuMRRaGLXAabaTJYOFrUwm/zRlTKrMZ
zsO7sPeHVfrgrsE44+fH5z/+9CjpbsdZ+v6ch4I4CV7nmrMWc0z9nh7XsQYq
0zmDIKMqmd6aO/m6pI/R608LTMMPFrtLDJx8g9KcEucJs/Yv56U9e/kkaNjW
bnxJL1aI4VXYGD7OOaIxC9+TlOGXOoFoUOi1u9Aa9KchTPdYcoxT39PLAUM5
5LgasBxtQuhuamI0cD0qC0m7ErZbFHLcY05mAd7BQWNc1oVjxDoCqewRHPGl
5B6MJU6WFJ4ZdCuQkNSGuykJxGQ8u40U6kvOKqQ/axS+86259QUI3zdv9wP3
2nAkr9xqpRGKjQlgeBaI0dIK5bQTxZELEnmLWOEKcKFHLUG4eRKaFM2CA7XJ
CWJAkjmqqq/NZisxk3RDyzYLfs6NeeMatet9hb8EaKS6UwIPV+M+O2B67FUL
H8zM76tWUn908ThTJzeCd94KE8lJ1LXSPkNp5UarXdb4wpCpcWV3g3lEC3+i
trlWou9QYnMEUfHjZL7EgK+oos3fvJurjSzCSnEz31K200zKAFlAbFqmk6k7
p2i52msbkgo1sIAuLayP+gJHe9HVtbuRSplJq7u0K4G4Q+inJpT3HpiVWcx9
8XToa+kObsHo1kjBuixtroR2B3tWdEokCo6M3YSlDtKMiGR/ZwY08SHerM20
tlHbGxj5wtEehVjpyYDoH6jI/V7BFkwKFyFl4yyIiQ865ov1TdFtibu99pBH
s7hOVKSktMhE/nCteLSDFyYGX7Jbz746v50GKehGBXrLQ9liNPbukpKJXPVE
MOJZcD6xWN/LPonedCajii+Lx5pg79mnQMRHl61lGt+vwZwEGvgbTSZIcRxa
/Fx8xVnEqPj0OC0uC7cCZ9+ZDQqMh2aGjDDSg0e+Cq0LscH1IfEQ13j28KdY
EMf3UKepAp9ISlcWUKkCZfwYbnIira9Yx73h2HitOidCBoBUMv9KSiNxqTxJ
OW8CHufCaQvvtt/yFAPsK9MqLEji3bT7bbhq35dQkqaS6izomSNxglJKkgU1
keXQmx5dieka3sSIBlA40jrZ97IUEuGz0Na+pLs3bZdKDe1zjx5FnQbAwJWS
WbZeSVRIp4w0EJNJsreesfCUeJyLReRSyblncu5I2T/3XJcjcuHgT89YmgAS
qKX3uPeKAjlY0tarmJGJnYITm2G7A30TlyiZMsM4Fgupd5ndu7sJzjvdqda9
cX2jKYEIFgbrg72thnsfH+1cj6G+iGWR2Q+NjxfswlQudoQ62ys7yQ74CgQZ
pKIlnaOk34xMLIZIZXKhUlnYDYBPwzhQ61uPXKWMQMq0t33ZpSooEeXbLrkK
h9iLr0vAHPIZX+ABeCdlnSSTA2anyM+TiiEpcNK+Jd3Pw/XG/Y6TNldsooww
7BpvUqpXd486Lcs6RhuvW4lBZnnsWY2Iivr158GbCf8qSoKQRoEyEYWUiUBV
NVafAiXDoxU0AS4RHNB4TL+FD92m+Zj8zhe/EmHLPxJhy/LfGGOLLq0DgYcs
/z3hNvZ0anw7yRuTPGWGAiC1Q9XN4KVeHHBFnjhSeEIKL7e828sP3/emBbcs
fF160fZNgoNBqPMSZyGFRYK6ygrLAZ+HKhp5VDSyfC+moL3ao5FY5Gf7PgRF
7wQ3Rch/CGU3o+6cpz2C1eMID5j61nClvJ8N1yjiFuQE43OtHSh/QxE+ABFl
YxAR18ipFbTr6sgK2s5uWy76qhnkSndkLGW+p51vajnNXy5Tp7cvvIO7I+CD
mUds4XeDWp3RYJCiemyoSZlf3zmq2SlwSqrjB5SBb42a1HNM+zf4vk3T/awl
dhcobxl7nVDZKm2wm3zrF4Qig21Se066yvvABEZBrnsdY7sRIB/KhKXzgcbT
ctwr9L5eiEYz8GWnZsOKW170o96Gpb3hDM6o/rBbaggIBgI2GpSjyoWMqTNa
22zwdilB1UE4ovke6hCEpCBazknUuCTk7ffGZChzQ1p5s/Mu1UlauBO/ENQ4
J4HhNb5fxiw1ZNLifQFIqTWm325ld+U3RooHFpxjClWIpsZ/ZL6EF8eIjgeF
rGIZLBZKI8YeBwwwmUns65fGhWOPj+G2hxYamQ/Jssle+tbkik/her46XckL
9F0lDk4hhKoHUYqFnfF99rvkJVuume1wd0CzKLkaqeUkHO/tdk0s8qURrjTP
WBg+57qZbQwH0HkXRBCgLQFZnX7gBnlClwuiGLda6E8cFFpvjJRxT+ujhnCR
5gZXiFU7FIBKzOMsKTvgC2SNtIyiSC5Dy+qzPKmuZZNU2pdCjkOYnTc8uEwF
p4urtVgKCZUombkZT5QpG0LAK7SKWQ8cY2WRD7MVb6IvQMEJfjK7AlMFGqQ0
oUyEDCGZhb42dX6rdmVrb2c6ufwRt31fqt4p0SnimkXUSFT/DNMMAap6kexn
2iZBzTb2y3jlh9VX0YeEPlEc52c/K5lqOLKZHSaFhKpUULkArWhQz02hq7HB
TtJ/eRLyJ7Rk8WSA/fSNRZNaiJlU6katcrtxw1rHg86C+YuAdE5SLbMECYc3
xBKVnHU4EUr3q2v6mu91YhzvcwNu4+FBeElVViaxgHueeCjjJLfdfJqlcm1Q
hcQDR6Qo2n414H7bcioiznURM7NHCaWxCkWqciYpODpcmnUoqWbMGIiyuIbK
MGYhjhv6ATp5hjufairwv0XYX9DFxHQONbER1oCLdU8J4xyXIGq/h8x26A+r
qleIKiTprCGVKrYiRZ22uvsuP+WabsGLFRoXS5VUaCJOfynFariCcVLn87s8
f1lrk5UCHZmqiRTRYx5Ek21vMBpX2D8i7lKqY/1okh35GLxvSemnOSOuCFzF
kcBS1U/jh5JuQaWcCu8c7c21RVUHqSHATyeh0EE9aNeEer1IDZpkMjOyPYth
d47EzcL9e3FhjiYeYIbza7kFErf01t4lcqeTJg6PtIyoJwNt+DvQATeKGpMc
BBa4IW9S+qGoF8Zqr1bdpphvEJ8f1vxuNHtAC30lthhtgA4KZSEWrfY1YxU+
MDNq6V1xDrFWlfRTDYidQlj7gapbifIjuBBTp5oXn5Lk8ORpH+odq0p8u4EC
0S7b4anYZ+c28DCtuFm1DmQYQaWxl9HIO8LJwdEUsJr+YZKU43zBcbVQjhIW
Av3OBvlaoXOeSxJHRs2SFc8qOcCqWHHr7bbjPmZVTGfOL9Y9cactumarBtYw
iQTDS+v+Qgxlw6hH1LpShZq4yHwvuyC/tUwrvfq+x0LdWpRIwh6sVZ+ePD8L
ZarB5kU9NZl62NJ5AHt1m9tMjeqNJpuiHNyfjjemPCOOurEv3TIq7+/l5ChL
ZsI5sEEkSjapcImVPqDFiLSJnLC3JBdBep2dhtQFuTrtlXfRS2oW+8QYIOGW
S22wDN6AhBG16IRokwTAzJew95HX+/ol/1r7SGnMxBMSs/DUGZ8xPnPwtVZZ
SXrS5k/MxmpXcjY9U1PD23dSJ01qJy3L0KNe9XxFuaLYBGJNy1gqDp7DNJ6C
k9M6cU1sFNDaiB9Rh9/SlJw1WktJGEAufB605tXxDvA43yWYlJzhp0lSfGxk
zyVLOQSLAPp1q04iaCcjHaPVUpcytTCCpIxqIyLiDI73bXSSy8Zx3S/8Min5
qXXks+xk4DKL/hg5mEKLXu8Cg5T7C0EkifZZ0h817UoW8F9i+yUzqrgGPHNp
zlbJlJEKcSpBBP5Mq0xvLoid2GVpFFjHvyNSQatOLu2hfuMN7xzAPUb2NAwI
5/aak6ZR9q/f8Lq1RTUXTyfd6A3jKRxzD3VdzMSN4SEHB0RdisiPMortYMli
Cy4uAY1K0yuufhaLW3L6K9ZDNEAinuEe0ikyTTtoArnhC1+EA/NCPyHfojG8
T/NS09J1voB/fhotsTSfBvIrSwOaJD/4vnnkvlT0X1uzDYEJsMCSTI4bc609
QLhy8y3hY7cHkO42JMtKK1oVHD585deEopzs/PCRBXA/pk9/vB4zyf4J12l4
W/sKwg8oElqaOoMBeVVqrzCDB+16eJl47yUdlPtSo0Q655xyM861XuyBo9iX
9jOD2ug6JNkEPo5y2D9gmlUfYeyyXiWyARacZCQceoE/cNMIRjok4Qpo4PTQ
bemTKmGySLJD03er0pUPW4jqlsmPROocSd86cCjDqiUmknn+qt9CmeEW9ikA
jtO4PWvd5b4/+dZXJMtUkVWlm8H8zMIr15Dhy5WwOcOONDBAsO5INP4OAMIT
YhuLOzuLyMsdvLPmhc6I3b2zd1auWtwB3KqkFd/2EMbgmI3hEY6CYJqhGJLe
Ag86S9vondGsn3MbPe6i197WZ37yzX4SgDg+1yxg/evbb/9tkJQd7jNN4Tkq
kIqTnpXGodHEOgXU0XELbagCwTRl97pwtlBbkY+Sdsd0sXKrz6mix5+5IfZE
3hW9yt4IBumQlW8lmBFIMYZu0ugWBn4gsIllr+XMtcQOrmSAbRjUsUkQ7Yz+
855rGuPRMJOVUeGJy8mkVCMUyv3NanE4c6I9Dp7R8FXfarLfPP71nozZmLEv
temRsr9Xj/EGHTU4+A9YCR1hYplBF+Zk9twXZNc+0psNPKvJ4+HU/sgFoxlf
SLfmtSS2NvAmzU3D2P7aRxdZjai9UoNrfYvF5yJEeGcWQg/qsI8voafBbXVO
CkLKzXtMQeIORGnC/Zwk7WjN5YrlssYh1AOm+UCNNqUlY8g1go/VMlB8wWPx
LQzdZjHD0RdAIpnE9g+CiiwWdO543rtnKxPQxcQYvD9AubsEYJ6IkE72lDOs
H8TGpW0CxffobBEhMXWfKwKMOvMGzJm3JOp//v0/pJfqqPGuZDyJvot7zY8c
KUooPHUEo4x7CaZuC/GMp81VBeIp/om0TZbkX6v3mgN6RjLguDzyoejWP//+
Pw5+wQm4pg4Ollg9D97ofiUhs6/i2w6OLjpc4kIX7O58XV6z29BIiVxS49SP
iOl4UztxnNwZ9d9LfU9fo0EEGJp4MH36hCoC0bdTtmm5Btq4UQlUZhY+gsk+
ce4QQzSIho++xYCRrLWB8IYsHZV5GOeDIS362CsYydQgnbiXiHgXg84Q8oLC
A0WBVzCgOPTMCUlOMW2ZIWgo9IFgRig+KD2JOFaX5ykpTbM/YWJDP4Sn6tB5
OhY99YadlAMAT9Yqx975A3HpHacJDxO8PKlNtJ12ESEId2K7tSz781SkZn5Q
at4Wu0j2cKK+rCRFf5p9G6kRz+l2cI4JNFw6X7WVFBC5MjGrPhHAXJVPCW8y
rFLTbllHYdUzu3d3msga4bsa6ebIH7dbYuzCj+48iTEmxyaZo5w7mrocxSvK
LFWcj/Sye9NhqmwqcH3ENfY59EUBhWB8jwi8anA/U4dJrIlVJ4SsJSykmECs
kcKBfJrVl1yuV4T0AvxyhqDiCkqma5Kmxx5u3bR3fH2iVvU5HliFctDfoRMl
99MXLUmyrrnX07X1vHBYiiVxwBy+1sTgafbELF/Ymz1PM58XPfSje5S2A9Rz
g2zQwk88ViZYdy6ozd49sTOQGeBhUUwq0tFPWuUKCDg2k+LeJQr6vLZDj2aw
h9IDiH7dpEFcUh/SaMWN2J9d7BiMQcsSH7pgqlLrDbzNx+bVDBVXg7/Jslgu
YFsJYBPAS7F7ktsu4EfNJbz3dVrXJmirUFw/pKOqXBwy+nvf+E1cNr1AzGWq
aY/vpERAIwanGnw883aIIklrWwG75X9D7/oTWoaYK7smg2DgVsCMRcrTnVrV
qobQy1Eaodbacd47I+D/08GPU/CF61Czxg6FO9wpyLOaD6wCvXJjnIH461hT
6exWmXyIZGu/DE2cyX27w+jTyO79WVoiSGeygqYRc/DYc3gA6ZYEjnhUO+cQ
FXtgQ7GoUL+CEdukgK12YlbQO4VRpwl8QCAA1nmNHDUuEwetQxrYIGRd76Re
GdMqv5Rd67E9Pfz+YM63Q9OiZMpw+km/NEUst71ZWL24HH0hnXrhGs0gDAga
cbjIq1NcmhD+C+7WAIBLH0q5DnTM44Wvh55eDtbNEVtvAu9elIu4FXFMrbEg
VoKJ0F7RJW+sKMbstGIvhO/U5ZV5lboCdJx3GauU/Pqgakutod93GBK5zDlM
K+5w2kdkinPyT3RafOnLyEYneUDKeu4iUkqCHPMY+U7kki9iE83VIcjU29sP
uZmUuGy8K0ZzgxNxVi5zxXryFeaBBczq8wRktBdBg2fiYaDuo3rFyY/QUZME
jnTenumxHv5Z+AfZoQJZLZw2qtSmK6h0E4FfaQm5YH7K8YuHKuk8mCXJ8b5A
qnbfUFIbu2IUy6g/gpzP9IoxG+eqLr7s41t1nCETJBSn43F9OOz07Oz5BOHg
F5Lomr86fn3bt1URY6fiYlECmglOAOV1ARKanb6iV8jQvFEh1XQ+goAdK8Ra
vwaQ9Fp42RaSb3glxsXo02bg3jb0VT24Bl2Rv8ZOhTaPGjqLioPvMRp8j6Qb
mlCVSeGIQHIGvGdSI4odfYKBRNg79SxHh428AxUwk2QkoMck4d/kDx9y8Q/T
dWi1QnMmOxNdV7BtgzzUZzDmOf0r5mmyPsG8xi1H9R3Wjtue1Qu7GNIex89M
gGIEHCztF27Y8Rplc4+rhZnNzErS234gSz4/bq6uRKE7Q1cZ0swfmIoYyKwx
qD8hBb7ylxXJN3rBA2KucIcAeIHPH74xG1vRx9D+Vd154Gb5A7LD5k6eeQVF
8kHjbmSo47o29GcPuIQqWWd20bj8xLSa+fkzilKe0FXhv87JRiPOdbK27brU
9z619ZsSv1k7fYZ0K/qLzT9Sc8TCzH9w6zo/KRF6kndX9q3h+mYnxK7k5c96
2u7nU6660hAJyxTOzSY/acwN3VY2t4hBkV360FwjPQF/Pylrlz9UUza/cKTo
QgI8ml9ZaSdCKhjJt0erlf/7B0Tjyk3+mEydWubzxDXNjj4omzUSDORFJZm7
jxFQ73RDnxOxP/bz+IFOgFnkY2IPVzqZH2jYJ6TC7eTPV+YN/eTJ2pS6nYt1
Se+yklb5mm4IcnqIWTY6JG1b/r1tZq7TxSHgmH+PUJVf35XNv//H/ywr2r7i
h3/8z2YFnKnusqETPd3ZemVkwJMeHvX8B8s1hVp/YrlUguI/H13TALv86dqR
2JMHSJpdl/nTsrq6KnVedK2flnJM/21ta9o3ui+yJ+XVGp6CZ2U/cyvXrkkZ
uNbZ0OnT59DHyBaTsXcomi4Vg5Wi4YBFWXOzoNlv0cVQ9sq2tJjnpnZvKtJU
RfV55fBmpCz688APITX8PaK79dz4tNGnjQTOns/JsN8pxb5CSi5ZRs/tTrf9
cWPqf/y/DtRHN0QP5wkZc/R3016J9XhCAnOXP3fNSud9XOEvfy7n70rYZO+g
vL+gpehe09wqXGWiJZKHrZJjmf+1zF+uK39ucl3cdp1L/Tm9DnNiYMSiFr3V
o99sdvRBX8mMnuGUzkjVXMnVw50gkWLtQud3behivrKkCRq/e3QF5OGnjmb1
EK1u13a71Y2QEc7RKN68k2N4QHtDYi8PZe+UaPiXDk2y9ZZ2tOiKBFG96red
0ctQw/NFB7wsbekPe0sySEjvJWr14pOyMfp+elGJ4st6Xd6Q4DsnIcEsUHe0
oeO+ys97JFbbVj9+CO3twqw7z+iu6K9dpTbqgzd0T+r8lAbKLywyMf0Fd8Rh
6SP4P/UGnyHARSoBaZOeeYEPv2vznw2dvW4KEuA5Zeq1qdq1XoT52hALfm2r
dwkJvV6TvJRZkZBN/nyKGGf+mkM86mR4hmS2Vf6XXrfP1FeG/v6rkTN7QsdT
o/+sVF6Ei2jV9CXdSHoK8uz0+AGRtfRKV1l+oQlxxwAhkJ4KmIZ6bo/hjYTq
CPXogYP1tBn+WLugJQ5xKF8ZAxquffyBDTb1OFjuxKhXMRVpDy2yGdcsblbq
jMsf0C6emM12hrQ3uRlDRr3HIA/xM7qjb0z+9B//i24aCe3FISY1Zgzjq3SC
fii4S6SztR+6CqPzGxwPbf7xHPiwyi5W3gqCLuELs4v7kNFnUpWxvhqqDal2
IWEHPQxf7pgON2LAtMg8a0wlqSFqAiFs5ZHMYUCumVyq+95DAPBUkj0hdt3D
xiwlPszNENA7gxaBqcJ08Oll1nFCo2+A6/smcChDHgLBYCilhUcNXdjjntZy
Yyo5n/MSKKJnRPh6F0lPoct/QWvagBowm9d+a05IwDTtqKLZnD8UhZdDKW6p
r9ujmn0RN+A86srh/Tzx+6kVc/a2+cD7RqT+IQ3t13UpUXVcoursaSAj/UK0
AafKgEt1gT3N4aAuQGy7ebcjQfvUtfaqOH/nui69Vb1Nb9UhiT6Wwfvi9aAQ
PCjyxtdyxEBHHJPu3BP6oJ+RrtB4YMGAW/kGSEMW5nuM5UlOOHt7FCfurQd2
xwOivsjWwtjWXbdt79+5s+LXTokq7tChwkSbgX3eUZukUK9z4Ym1EGM/y/4/
9ibTwJUOAQA=
<reference anchor="Briscoe2021" target="https://www.iab.org/wp-content/IAB
-uploads/2021/09/single-delay-metric-1.pdf">
<front>
<title>A Single Common Metric to Characterize Varying Packet Delay</ti
tle>
<author initials="B." surname="Briscoe">
<organization/>
</author>
<author initials="G." surname="White">
<organization/>
</author>
<author initials="V." surname="Goel">
<organization/>
</author>
<author initials="K." surname="De Schepper">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<reference anchor="McIntyre2021" target="https://www.iab.org/wp-content/IA
B-uploads/2021/09/Internet-Score-2.pdf">
<front>
<title>An end-user approach to an Internet Score</title>
<author initials="C." surname="Paasch">
<organization/>
</author>
<author initials="K." surname="McIntyre">
<organization/>
</author>
<author initials="O." surname="Shapira">
<organization/>
</author>
<author initials="R." surname="Meyer">
<organization/>
</author>
<author initials="S." surname="Cheshire">
<organization/>
</author>
<date year="2021" month="September"/>
</front>
</reference>
<reference anchor="Speedtest" target="https://www.speedtest.net">
<front>
<title>Speedtest</title>
<author>
<organization>Ookla</organization>
</author>
</front>
</reference>
<reference anchor="NetworkQuality" target="https://support.apple.com/en-gb
/HT212313">
<front>
<title>Network Quality</title>
<author>
<organization>Apple</organization>
</author>
</front>
</reference>
<reference anchor="SamKnows" target="https://www.samknows.com/">
<front>
<title>SamKnows</title>
<author>
<organization/>
</author>
</front>
</reference>
</references>
<section anchor="program-committee" numbered="true" toc="default">
<name>Program Committee</name>
<t>The program committee consisted of:</t>
<ul empty="true" spacing="compact">
<li><t><contact fullname="Jari Arkko"/></t></li>
<li><t><contact fullname="Olivier Bonaventure"/></t></li>
<li><t><contact fullname="Vint Cerf"/></t></li>
<li><t><contact fullname="Stuart Cheshire"/></t></li>
<li><t><contact fullname="Sam Crowford"/></t></li>
<li><t><contact fullname="Nick Feamster"/></t></li>
<li><t><contact fullname="Jim Gettys"/></t></li>
<li><t><contact fullname="Toke Hoiland-Jorgensen"/></t></li>
<li><t><contact fullname="Geoff Huston"/></t></li>
<li><t><contact fullname="Cullen Jennings"/></t></li>
<li><t><contact fullname="Katarzyna Kosek-Szott"/></t></li>
<li><t><contact fullname="Mirja Kühlewind"/></t></li>
<li><t><contact fullname="Jason Livingood"/></t></li>
<li><t><contact fullname="Matt Mathis"/></t></li>
<li><t><contact fullname="Randall Meyer"/></t></li>
<li><t><contact fullname="Kathleen Nichols"/></t></li>
<li><t><contact fullname="Christoph Paasch"/></t></li>
<li><t><contact fullname="Tommy Pauly"/></t></li>
<li><t><contact fullname="Greg White"/></t></li>
<li><t><contact fullname="Keith Winstein"/></t></li>
</ul>
</section>
<section anchor="workshop-chairs" numbered="true" toc="default">
<name>Workshop Chairs</name>
<t>The workshop chairs consisted of:</t>
<ul empty="true" spacing="compact">
<li><t><contact fullname="Wes Hardaker"/></t></li>
<li><t><contact fullname="Evgeny Khorov"/></t></li>
<li><t><contact fullname="Omer Shapira"/></t></li>
</ul>
</section>
<section anchor="participants-list" numbered="true" toc="default">
<name>Workshop Participants</name>
<t>The following is a list of participants who attended the workshop over
a remote connection:</t>
<ul empty="true" spacing="compact">
<li><t><contact fullname="Ahmed Aldabbagh"/></t></li>
<li><t><contact fullname="Jari Arkko"/></t></li>
<li><t><contact fullname="Praveen Balasubramanian"/></t></li>
<li><t><contact fullname="Olivier Bonaventure"/></t></li>
<li><t><contact fullname="Djamel Bousaber"/></t></li>
<li><t><contact fullname="Bob Briscoe"/></t></li>
<li><t><contact fullname="Rich Brown"/></t></li>
<li><t><contact fullname="Anna Brunstrom"/></t></li>
<li><t><contact fullname="Pedro Casas"/></t></li>
<li><t><contact fullname="Vint Cerf"/></t></li>
<li><t><contact fullname="Stuart Cheshire"/></t></li>
<li><t><contact fullname="Kenjiro Cho"/></t></li>
<li><t><contact fullname="Steve Christianson"/></t></li>
<li><t><contact fullname="John Cioffi"/></t></li>
<li><t><contact fullname="Alexander Clemm"/></t></li>
<li><t><contact fullname="Luis M. Contreras"/></t></li>
<li><t><contact fullname="Sam Crawford"/></t></li>
<li><t><contact fullname="Neil Davies"/></t></li>
<li><t><contact fullname="Gino Dion"/></t></li>
<li><t><contact fullname="Toerless Eckert"/></t></li>
<li><t><contact fullname="Lars Eggert"/></t></li>
<li><t><contact fullname="Joachim Fabini"/></t></li>
<li><t><contact fullname="Gorry Fairhurst"/></t></li>
<li><t><contact fullname="Nick Feamster"/></t></li>
<li><t><contact fullname="Mat Ford"/></t></li>
<li><t><contact fullname="Jonathan Foulkes"/></t></li>
<li><t><contact fullname="Jim Gettys"/></t></li>
<li><t><contact fullname="Rajat Ghai"/></t></li>
<li><t><contact fullname="Vidhi Goel"/></t></li>
<li><t><contact fullname="Wes Hardaker"/></t></li>
<li><t><contact fullname="Joris Herbots"/></t></li>
<li><t><contact fullname="Geoff Huston"/></t></li>
<li><t><contact fullname="Toke Høiland-Jørgensen"/></t></li>
<li><t><contact fullname="Jana Iyengar"/></t></li>
<li><t><contact fullname="Cullen Jennings"/></t></li>
<li><t><contact fullname="Ken Kerpez"/></t></li>
<li><t><contact fullname="Evgeny Khorov"/></t></li>
<li><t><contact fullname="Kalevi Kilkki"/></t></li>
<li><t><contact fullname="Joon Kim"/></t></li>
<li><t><contact fullname="Zhenbin Li"/></t></li>
<li><t><contact fullname="Mikhail Liubogoshchev"/></t></li>
<li><t><contact fullname="Jason Livingood"/></t></li>
<li><t><contact fullname="Kyle MacMillan"/></t></li>
<li><t><contact fullname="Sharat Madanapalli"/></t></li>
<li><t><contact fullname="Vesna Manojlovic"/></t></li>
<li><t><contact fullname="Robin Marx"/></t></li>
<li><t><contact fullname="Matt Mathis"/></t></li>
<li><t><contact fullname="Jared Mauch"/></t></li>
<li><t><contact fullname="Kristen McIntyre"/></t></li>
<li><t><contact fullname="Randall Meyer"/></t></li>
<li><t><contact fullname="François Michel"/></t></li>
<li><t><contact fullname="Greg Mirsky"/></t></li>
<li><t><contact fullname="Cindy Morgan"/></t></li>
<li><t><contact fullname="Al Morton"/></t></li>
<li><t><contact fullname="Szilveszter Nadas"/></t></li>
<li><t><contact fullname="Kathleen Nichols"/></t></li>
<li><t><contact fullname="Lai Yi Ohlsen"/></t></li>
<li><t><contact fullname="Christoph Paasch"/></t></li>
<li><t><contact fullname="Lucas Pardue"/></t></li>
<li><t><contact fullname="Tommy Pauly"/></t></li>
<li><t><contact fullname="Levi Perigo"/></t></li>
<li><t><contact fullname="David Reed"/></t></li>
<li><t><contact fullname="Alvaro Retana"/></t></li>
<li><t><contact fullname="Roberto"/></t></li>
<li><t><contact fullname="Koen De Schepper"/></t></li>
<li><t><contact fullname="David Schinazi"/></t></li>
<li><t><contact fullname="Brandon Schlinker"/></t></li>
<li><t><contact fullname="Eve Schooler"/></t></li>
<li><t><contact fullname="Satadal Sengupta"/></t></li>
<li><t><contact fullname="Jinous Shafiei"/></t></li>
<li><t><contact fullname="Shapelez"/></t></li>
<li><t><contact fullname="Omer Shapira"/></t></li>
<li><t><contact fullname="Dan Siemon"/></t></li>
<li><t><contact fullname="Vijay Sivaraman"/></t></li>
<li><t><contact fullname="Karthik Sundaresan"/></t></li>
<li><t><contact fullname="Dave Taht"/></t></li>
<li><t><contact fullname="Rick Taylor"/></t></li>
<li><t><contact fullname="Bjørn Ivar Teigen"/></t></li>
<li><t><contact fullname="Nicolas Tessares"/></t></li>
<li><t><contact fullname="Peter Thompson"/></t></li>
<li><t><contact fullname="Balazs Varga"/></t></li>
<li><t><contact fullname="Bren Tully Walsh"/></t></li>
<li><t><contact fullname="Michael Welzl"/></t></li>
<li><t><contact fullname="Greg White"/></t></li>
<li><t><contact fullname="Russ White"/></t></li>
<li><t><contact fullname="Keith Winstein"/></t></li>
<li><t><contact fullname="Lisong Xu"/></t></li>
<li><t><contact fullname="Jiankang Yao"/></t></li>
<li><t><contact fullname="Gavin Young"/></t></li>
<li><t><contact fullname="Mingrui Zhang"/></t></li>
</ul>
</section>
<section anchor="iab-members-at-the-time-of-approval" numbered="false" toc="
default">
<name>IAB Members at the Time of Approval</name>
<t>Internet Architecture Board members at the time this document was
approved for publication were:</t>
<ul empty="true" spacing="compact">
<li><t><contact fullname="Jari Arkko"/></t></li>
<li><t><contact fullname="Deborah Brungard"/></t></li>
<li><t><contact fullname="Lars Eggert"/></t></li>
<li><t><contact fullname="Wes Hardaker"/></t></li>
<li><t><contact fullname="Cullen Jennings"/></t></li>
<li><t><contact fullname="Mallory Knodel"/></t></li>
<li><t><contact fullname="Mirja Kühlewind"/></t></li>
<li><t><contact fullname="Zhenbin Li"/></t></li>
<li><t><contact fullname="Tommy Pauly"/></t></li>
<li><t><contact fullname="David Schinazi"/></t></li>
<li><t><contact fullname="Russ White"/></t></li>
<li><t><contact fullname="Qin Wu"/></t></li>
<li><t><contact fullname="Jiankang Yao"/></t></li>
</ul>
</section>
<section anchor="acknowledgments" numbered="false" toc="default">
<name>Acknowledgments</name>
<t>The authors would like to thank the workshop participants, the members
of the IAB, and the program committee for creating and participating
in many interesting discussions.</t>
</section>
<section anchor="draft-contributors" numbered="false" toc="default">
<name>Contributors</name>
<t>Thank you to the people that contributed edits to this document:</t>
<ul empty="true" spacing="compact">
<li><t><contact fullname="Erik Auerswald"/></t></li>
<li><t><contact fullname="Simon Leinen"/></t></li>
<li><t><contact fullname="Brian Trammell"/></t></li>
</ul>
</section>
</back>
</rfc> </rfc>
 End of changes. 213 change blocks. 
2192 lines changed or deleted 1653 lines changed or added

This html diff was produced by rfcdiff 1.48.