rfc8963.original   rfc8963.txt 
Network Working Group C. Huitema Independent Submission C. Huitema
Internet-Draft Private Octopus Inc. Request for Comments: 8963 Private Octopus Inc.
Intended status: Informational October 25, 2020 Category: Informational January 2021
Expires: April 28, 2021 ISSN: 2070-1721
Evaluation of a Sample of RFC Produced in 2018 Evaluation of a Sample of RFCs Produced in 2018
draft-huitema-rfc-eval-project-07
Abstract Abstract
This document presents the author's effort to understand the delays This document presents the author's effort to understand the delays
involved in publishing an idea in the IETF or through the Independent involved in publishing an idea in the IETF or through the Independent
Stream, from the first individual draft to the publication of the Stream, from the first individual draft to the publication of the
RFC. We analyze a set of randomly chosen RFC approved in 2018, RFC. We analyze a set of randomly chosen RFCs approved in 2018,
looking for history and delays. We also use two randomly chosen sets looking for history and delays. We also use two randomly chosen sets
of RFC published in 2008 and 1998 for comparing delays seen in 2018 of RFCs published in 2008 and 1998 for comparing delays seen in 2018
to those observed 10 or 20 years ago. The average RFC in the 2018 to those observed 10 or 20 years ago. The average RFC in the 2018
sample was produced in 3 years and 4 months, of which 2 years and 10 sample was produced in 3 years and 4 months, of which 2 years and 10
months were spent in the Working Group, 3 to 4 months for IETF months were spent in the working group, 3 to 4 months for IETF
consensus and IESG review, and 3 to 4 months in RFC production. The consensus and IESG review, and 3 to 4 months in RFC production. The
main variation in RFC production delays comes from the AUTH-48 phase. main variation in RFC production delays comes from the AUTH48 phase.
We also measure the number of citations of the chosen RFC using We also measure the number of citations of the chosen RFC using
Semantic Scholar, and compare citation counts with what we know about Semantic Scholar, and compare citation counts with what we know about
deployment. We show that citation counts indicate academic interest, deployment. We show that citation counts indicate academic interest,
but correlate only loosely with deployment or usage of the but correlate only loosely with deployment or usage of the
specifications. Counting web references could complement that. specifications. Counting web references could complement that.
The RFCs selected for this survey were chosen at random and represent
a small sample of all RFCs produced, and only approximately 10% of
the RFCs produced in each of 1998, 2008, and 2018. It is possible
that different samples would produce different results. Furthermore,
the conclusions drawn from the observations made in this document
represent the author's opinions and do not have consensus of the
IETF.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This is a contribution to the RFC Series, independently of any other
and may be updated, replaced, or obsoleted by other documents at any RFC stream. The RFC Editor has chosen to publish this document at
time. It is inappropriate to use Internet-Drafts as reference its discretion and makes no statement about its value for
material or to cite them other than as "work in progress." implementation or deployment. Documents approved for publication by
the RFC Editor are not candidates for any level of Internet Standard;
see Section 2 of RFC 7841.
This Internet-Draft will expire on April 28, 2021. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8963.
Copyright Notice Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document.
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction
2. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Methodology
2.1. Defining the Important Milestones . . . . . . . . . . . . 5 2.1. Defining the Important Milestones
2.2. Selecting a Random Sample of RFCs . . . . . . . . . . . . 6 2.2. Selecting a Random Sample of RFCs
3. Analysis of 20 Selected RFCs . . . . . . . . . . . . . . . . 6 2.3. Conventions Used in This Document
3.1. 8411 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Analysis of 20 Selected RFCs
3.2. 8456 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.1. RFC 8411
3.3. 8446 . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2. RFC 8456
3.4. 8355 . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3. RFC 8446
3.5. 8441 . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.4. RFC 8355
3.6. 8324 . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.5. RFC 8441
3.7. 8377 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.6. RFC 8324
3.8. 8498 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.7. RFC 8377
3.9. 8479 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.8. RFC 8498
3.10. 8453 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.9. RFC 8479
3.11. 8429 . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.10. RFC 8453
3.12. 8312 . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.11. RFC 8429
3.13. 8492 . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.12. RFC 8312
3.14. 8378 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.13. RFC 8492
3.15. 8361 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.14. RFC 8378
3.16. 8472 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.15. RFC 8361
3.17. 8471 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.16. RFC 8472
3.18. 8466 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.17. RFC 8471
3.19. 8362 . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.18. RFC 8466
3.20. 8468 . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.19. RFC 8362
4. Analysis of Process and Delays . . . . . . . . . . . . . . . 19 3.20. RFC 8468
4.1. First Draft to RFC Delays . . . . . . . . . . . . . . . . 20 4. Analysis of Process and Delays
4.2. Working Group Processing Time . . . . . . . . . . . . . . 25 4.1. Delays from First Draft to RFC
4.3. Preparation and Publication Delays . . . . . . . . . . . 28 4.2. Working Group Processing Time
4.4. Copy Editing . . . . . . . . . . . . . . . . . . . . . . 31 4.3. Preparation and Publication Delays
4.5. Independent Stream . . . . . . . . . . . . . . . . . . . 34 4.4. Copy Editing
5. Citation Counts . . . . . . . . . . . . . . . . . . . . . . . 34 4.5. Independent Stream
5.1. Citation Numbers . . . . . . . . . . . . . . . . . . . . 35 5. Citation Counts
5.2. Comparison to 1998 and 2008 . . . . . . . . . . . . . . . 37 5.1. Citation Numbers
5.3. Citations Versus Deployments . . . . . . . . . . . . . . 40 5.2. Comparison to 1998 and 2008
5.4. Citations Versus Web References . . . . . . . . . . . . . 42 5.3. Citations versus Deployments
6. Observations and Next Steps . . . . . . . . . . . . . . . . . 44 5.4. Citations versus Web References
7. Security Considerations . . . . . . . . . . . . . . . . . . . 46 6. Observations and Next Steps
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 46 7. Security Considerations
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 8. IANA Considerations
10. Informative References . . . . . . . . . . . . . . . . . . . 46 9. Informative References
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 50 Acknowledgements
Author's Address
1. Introduction 1. Introduction
As stated on the organization's web site, "The IETF is a large open As stated on the organization's web site, "The IETF is a large open
international community of network designers, operators, vendors, and international community of network designers, operators, vendors, and
researchers concerned with the evolution of the Internet architecture researchers concerned with the evolution of the Internet architecture
and the smooth operation of the Internet." The specifications and the smooth operation of the Internet." The specifications
produced by the IETF are published in the RFC series, along with produced by the IETF are published in the RFC series, along with
independent submissions, research papers and IAB documents. In this documents from the IAB, IRTF, and Independent streams (as per RFC
memo, the author attempts to understand the delays involved in 8729). In this memo, the author attempts to understand the delays
publishing an idea in the IETF or through the Independent Stream, involved in publishing an idea in the IETF or through the Independent
from the first individual draft to the publication of the RFC. This Stream, from the first individual draft to the publication of the
is an individual effort, and the author's conclusions presented here RFC. This is an individual effort, and the author's conclusions
are personal. There was no attempt to seek IETF consensus. presented here are personal. There was no attempt to seek IETF
consensus.
The IETF keeps records of documents and process actions in the IETF The IETF keeps records of documents and process actions in the IETF
datatracker [TRKR]. The IETF datatracker provides information about Datatracker [TRKR]. The IETF Datatracker provides information about
RFCs and drafts, from which we can infer statistics about the RFCs and drafts, from which we can infer statistics about the
production system. We can measure how long it takes to drive a production system. We can measure how long it takes to drive a
proposition from initial draft to final publication, and how these proposition from initial draft to final publication, and how these
delays can be split between Working Group discussions, IETF reviews, delays can be split between working group discussions, IETF reviews,
IESG assessment, RFC Editor delays and final reviews by the authors - IESG assessment, RFC Editor delays and final reviews by the authors
or, for Independent Stream RFCs, draft production, reviews by the -- or, for Independent Stream RFCs, draft production, reviews by the
Independent Stream Editor, conflict reviews, RFC Editor delays and Independent Submissions Editor, conflict reviews, RFC Editor delays
final reviews. Tracker data is available for all RFCs, not just IETF and final reviews. Tracker data is available for all RFCs, not just
stream RFCs. IETF Stream RFCs.
Just measuring production delays may be misleading. If the IETF or Just measuring production delays may be misleading. If the IETF or
the editors of the other series simply rubber-stamped draft proposals the other streams simply rubber-stamped draft proposals and published
and published them, the delays would be short but the quality and them, the delays would be short but the quality and impact might
impact might suffer. We hope that most of the RFC that are published suffer. We hope that most of the RFCs that are published are useful,
are useful, but we need a way to measure that usefulness. We try to but we need a way to measure that usefulness. We try to do that by
do that by measuring the number of references of the published RFCs measuring the number of references of the published RFCs in Semantic
in Semantic Scholar [SSCH], and also by asking the authors of each Scholar [SSCH], and also by asking the authors of each RFC in the
RFC in the sample whether the protocols and technologies defined in sample whether the protocols and technologies defined in the RFCs
the RFCs were implemented and used on the Internet. The citations were implemented and used on the Internet. The citations measured by
measured by the Semantic Scholar include citations in other RFCs and the Semantic Scholar include citations in other RFCs and in Internet-
in Internet drafts. We also measure the number of references on the Drafts. We also measure the number of references on the web, which
web, which provides some results but would be hard to automate. provides some results but would be hard to automate.
In order to limit the resource required for this study, we selected In order to limit the resources required for this study, we selected
at random 20 RFCs published in 2018, as explained in Section 2.2. at random 20 RFCs published in 2018, as explained in Section 2.2.
The statistical sampling picked both IETF stream and Independent The statistical sampling picked both IETF Stream and Independent
Stream documents. For comparison purposes, we also selected at Stream documents. For comparison purposes, we also selected at
random 20 RFC published in 1998 and 20 published in 2008. Limiting random 20 RFCs published in 1998 and 20 published in 2008. Limiting
the sample to 20 out of 209 RFCs published in 2018 allows for in the sample to 20 out of 209 RFCs published in 2018 allows for in-
depth analysis of each RFC, but readers should be reminded that the depth analysis of each RFC, but readers should be reminded that the
this is a small sample. The sample is too small to apply general this is a small sample. The sample is too small to apply general
statistical techniques and quantify specific ratios, and discussions statistical techniques and quantify specific ratios, and discussions
of correlation techniques would be inappropriate. Instead, the of correlation techniques would be inappropriate. Instead, the
purpose is to identify trends, spot issues and document future work. purpose is to identify trends, spot issues, and document future work.
The information gathered for every RFC in the sample is presented in The information gathered for every RFC in the sample is presented in
Section 3. In Section 4 we analyze the production process and the Section 3. In Section 4, we analyze the production process and the
sources of delays, comparing the 2018 sample to the selected samples sources of delays, comparing the 2018 sample to the selected samples
for 1998 and 2018. In Section 5.1 we present citation counts for the for 1998 and 2018. In Section 5.1, we present citation counts for
RFCs in the samples, and analyze whether citation counts could be the RFCs in the samples, and analyze whether citation counts could be
used to evaluate the quality of RFCs. used to evaluate the quality of RFCs.
The measurement of delays could be automated by processing dates and The measurement of delays could be automated by processing dates and
events recorded in the datatracker. The measurement of published events recorded in the Datatracker. The measurement of published
RFCs could be complemented by statistics on abandoned drafts, which RFCs could be complemented by statistics on abandoned drafts, which
would measure the efficiency of the IETF triaging process. More would measure the efficiency of the IETF triaging process. More
instrumentation would help understanding how large delays happen instrumentation would help understanding how large delays happen
during Working Group processes. These potential next steps are during working group processes. These potential next steps are
developed in Section 6. developed in Section 6.
2. Methodology 2. Methodology
The study reported here started with a simple idea: take a sample of The study reported here started with a simple idea: take a sample of
RFCs, and perform an in-depth analysis of the path from the first RFCs, and perform an in-depth analysis of the path from the first
presentation of the idea to its publication, while also trying to presentation of the idea to its publication, while also trying to
access the success of the resulting specification. This requires access the success of the resulting specification. This requires
defining the key milestones that we want to track, and drawing a defining the key milestones that we want to track, and drawing a
random sample using an unbiased process. random sample using an unbiased process.
2.1. Defining the Important Milestones 2.1. Defining the Important Milestones
The IETF datatracker records a list of events for each document The IETF Datatracker records a list of events for each document
processed by IETF Working Groups. This has a high granularity, and processed by IETF working groups. This has a high granularity, and
also a high variability. Most documents start life as an individual also a high variability. Most documents start life as an individual
draft, are adopted by a Working Group, undergo a Working Group last draft, are adopted by a working group, undergo a Working Group Last
call, are submitted to the IESG, undergo an IETF last call and an Call, are submitted to the IESG, undergo an IETF Last Call and an
IESG review, get eventually approved by the IESG, and are processed IESG review, get eventually approved by the IESG, and are processed
for publication by the RFC Editor, but there are exceptions. Some for publication by the RFC Editor, but there are exceptions. Some
documents are first submitted to one Working Group and then moved to documents are first submitted to one working group and then moved to
another. Some documents are published through the Independent another. Some documents are published through the Independent
Stream, and are submitted to the Independent Stream Editor instead of Stream, and are submitted to the Independent Submissions Editor
the IESG. instead of the IESG.
In order to simplify tabulation, we break the delay from between the In order to simplify tabulation, we break the period from the
submission of the first draft and the publication of the RFC in three submission of the first draft to the publication of the RFC into
big components: three big components:
o The Working Group processing time, from the first draft to the * The working group processing time, from the first draft to the
start of the IETF last call; start of the IETF last call;
o The IETF processing time, which lasts from the beginning of the * The IETF processing time, which lasts from the beginning of the
IETF last call to the approval by the IESG, including the reviews IETF last call to the approval by the IESG, including the reviews
by various directorates; by various directorates;
o The RFC production, from approval by the IESG to publication, * The RFC production, from approval by the IESG to publication,
including the AUTH-48 reviews. including the AUTH48 reviews.
For submissions to the Independent Stream, we don't have a Working For submissions to the Independent Stream, we don't have a working
Group. We consider instead the progression of the individual draft group. We consider instead the progression of the individual draft
until the adoption by the ISE as the equivalent of the "Working until the adoption by the Independent Submissions Editor (ISE) as the
Group" period, and the delay from adoption by the ISE until equivalent of the "Working Group" period, and the delay from adoption
submission to the RFC Editor as the equivalent of the IETF delay. by the ISE until submission to the RFC Editor as the equivalent of
the IETF processing time.
We measure the staring point of the process using the date of We measure the starting point of the process using the date of
submission of the first draft listed on that RFC page in the IETF submission of the first draft listed on that RFC page in the IETF
datatracker. In most case, this first draft is an individual draft Datatracker. In most cases, this first draft is an individual draft
that then resubmitted as a Working Group draft, or maybe resubmitted that then resubmitted as a working group draft, or maybe resubmitted
with a new name as the draft was searching for a home in an IETF with a new name as the draft was searching for a home in an IETF
Working Group, or before deciding for submission on the Independent working group, or before deciding for submission on the Independent
Stream. Stream.
The IETF datatracker entries for RFCs and drafts do not list Working The IETF Datatracker entries for RFCs and drafts do not _always_ list
Group events like Working Group Last Call. The only intermediate working group events like Working Group Last Call. The only
event that we list between the first draft and the submission to the intermediate event that we list between the first draft and the
IESG is the Working Group Adoption. For that, we use the date of submission to the IESG is the working group adoption, for which we
submission of the version 00 of the draft eventually published as use the date of submission of version 00 of the draft eventually
RFC. We use the same definition for drafts submitted to the published as RFC. We also use that date (of submission of version
Independent Stream. 00) for drafts submitted to the Independent Stream.
2.2. Selecting a Random Sample of RFCs 2.2. Selecting a Random Sample of RFCs
Basic production mechanisms could be evaluated by processing data Basic production mechanisms could be evaluated by processing data
from the IETF datatracker, but subjective data requires manual from the IETF Datatracker, but subjective data requires manual
assessment of results, which can be time consuming. Since our assessment of results, which can be time-consuming. Since our
resources are limited, we will only perform this analysis for a small resources are limited, we will only perform this analysis for a small
sample of RFCs, selected at random from the list of RFCs approved in sample of RFCs, selected at random from the list of RFCs approved in
2018. Specifically, we will pick 20 RFC numbers at random between: 2018. Specifically, we will pick 20 RFC numbers at random between:
o RFC 8307, published in January 2018, and * RFC 8307, published in January 2018, and
o RFC 8511, published December 2018. * RFC 8511, published December 2018.
The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC
8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC 8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC
8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471, 8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471,
RFC 8466, RFC 8362, and RFC 8468. RFC 8466, RFC 8362, and RFC 8468.
When evaluating delays and impact, we will compare the year 2018 to When evaluating delays and impact, we will compare the year 2018 to
2008 and 1998, 10 and 20 years ago. To drive this comparison, we 2008 and 1998, 10 and 20 years ago. To drive this comparison, we
pick 20 RFCs at random among those published in 2008, and another 20 pick 20 RFCs at random among those published in 2008, and another 20
among those published in 1998. among those published in 1998.
The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC
5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC 5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC
5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC
5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301. 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301.
The list of the 20 randomly selected RFCs from 2008 is: RFC 2431, RFC The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC
2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC 2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC
2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC
2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323. 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323.
2.3. Conventions Used in This Document
The following abbreviations are used in the tables:
BCP Best Current Practice
Exp Experimental
Info Informational
PS Proposed Standard
DS Draft Standard [This maturity level was retired by RFC 6410.]
In addition, Status is as defined in RFC 2026, and Stream is as
defined in RFC 8729.
3. Analysis of 20 Selected RFCs 3. Analysis of 20 Selected RFCs
We review each of the RFCs listed in Section 2.2 for the year 2018, We review each of the RFCs listed in Section 2.2 for the year 2018,
trying both to answer the known questions and to gather insight for trying both to answer the known questions and to gather insight for
further analyzes. In many cases, the analysis of the data is further analyses. In many cases, the analysis of the data is
complemented by direct feedback from the RFC authors. complemented by direct feedback from the RFC authors.
3.1. 8411 3.1. RFC 8411
IANA Registration for the Cryptographic Algorithm Object Identifier "IANA Registration for the Cryptographic Algorithm Object Identifier
Range [RFC8411]: Range" [RFC8411]:
Informational, 5 pages Status (Length): Informational (5 pages)
4 drafts (personal), first 2017-05-08. Overview: 4 individual drafts
Last call announced 2017-10-09 First draft: 2017-05-08
IESG evaluation starts 2017-12-28 Last Call start: 2017-10-09
Approved 2018-02-26, draft 03 IESG eval. start: 2017-12-28
AUTH-48 2018-04-20 IESG approved: 2018-02-26 (draft 03)
AUTH-48 complete 2018-07-17 AUTH48 start: 2018-04-20
Published 2018-08-06 AUTH48 complete: 2018-07-17
IANA action: create table Published: 2018-08-06
IANA action: create table
This RFC was published from the individual draft, which was not This RFC was published from the individual draft, which was not
resubmitted as a Working Group draft. resubmitted as a working group draft.
The draft underwent minor copy edit before publication. The draft underwent minor copy editing before publication.
Some but not all of the long delay in AUTH-48 is due to clustering Some but not all of the long delay in AUTH48 is due to clustering
with [RFC8410]. MISSREF was cleared on 2018-05-09 and the document with [RFC8410]. MISSREF state concluded on 2018-05-09 and the
re-entered AUTH-48 at once. AUTH-48 lasted over two months after document re-entered AUTH48 at once. AUTH48 lasted over two months
that. after that. (For state definitions, see <https://www.rfc-
editor.org/about/queue/#state_def>.)
The time after AUTH-48 and before publication (3 weeks) partly The time after AUTH48 and before publication (3 weeks) partly
overlaps with travel for IETF-102 and is partly due to coordinating overlaps with travel for IETF 102 and is partly due to coordinating
the cluster. the cluster.
3.2. 8456 3.2. RFC 8456
Benchmarking Methodology for Software-Defined Networking (SDN) "Benchmarking Methodology for Software-Defined Networking (SDN)
Controller Performance [RFC8456]: Controller Performance" [RFC8456]:
Informational, 64 pages Status (Length): Informational (64 pages)
2 personal drafts, 9 WG drafts, first 2015-03-23 Overview: 2 individual drafts; 9 WG drafts
WG adoption on 2015-10-18 First draft: 2015-03-23
Last call announced 2018-01-19 WG adoption: 2015-10-18
IESG evaluation starts 2018-02-27 Last Call start: 2018-01-19
IESG approved 2018-05-25 IESG eval. start: 2018-02-27
AUTH-48 2018-08-31 IESG approved: 2018-05-25
AUTH-48 complete 2018-10-16 AUTH48 start: 2018-08-31
Published 2018-10-30 AUTH48 complete: 2018-10-16
Published: 2018-10-30
The draft underwent very extensive copy editing, covering use of The draft underwent extensive copy editing, covering use of articles,
articles, turn of phrases, choice of vocabulary. The changes are syntax, and word choice. The changes are enough to cause pagination
enough to cause pagination differences. The "diff" tool marks pretty differences. The "diff" tool marks pretty much every page as
much every page as changed. Some diagrams see change in protocol changed. Some diagrams see change in protocol elements like message
elements like message names. names.
According to the author, the experience of producing this draft According to the author, the experience of producing this document
mirrors a typical one in the Benchmarking Methodologies Working Group mirrors a typical one in the Benchmarking Methodologies Working Group
(BMWG). There were multiple authors in multiple time zones, which (BMWG). There were multiple authors in multiple time zones, which
slowed down the AUTH-48 process somewhat, although the AUTH-48 delay slowed down the AUTH48 process somewhat, although the AUTH48 delay of
of 46 days is only a bit longer than the average draft. 46 days is only a bit longer than the average draft.
The RFC was part of cluster with [RFC8455]. The RFC was part of cluster with [RFC8455].
BMWG publishes informational RFCs centered around benchmarking, and BMWG publishes Informational RFCs centered around benchmarking, and
the methodologies in RFC 8456 have been implemented in benchmarking the methodologies in RFC 8456 have been implemented in benchmarking
products. products.
3.3. 8446 3.3. RFC 8446
The Transport Layer Security (TLS) Protocol Version 1.3 [RFC8446], as "The Transport Layer Security (TLS) Protocol Version 1.3" [RFC8446],
the title indicates, defines the new version of the TLS protocol. as the title indicates, defines the new version of the TLS protocol.
From the IETF datatracker, we extract the following: From the IETF Datatracker, we extract the following:
Proposed standard Status (Length): Proposed Standard (160 pages)
160 pages Overview: 29 WG drafts
29 WG drafts first 2014-04-17. First draft: 2014-04-17
Last call announced 2018-02-15 Last Call start: 2018-02-15
IESG evaluation starts 2018-03-02 IESG eval. start: 2018-03-02
Approved 2018-03-21, draft 28 IESG approved: 2018-03-21 (draft 28)
AUTH-48 2018-06-14 AUTH48 start: 2018-06-14
AUTH-48 complete 2018-08-10 AUTH48 complete: 2018-08-10
Published 2018-08-10 Published: 2018-08-10
This draft started as a WG effort. This draft started as a WG effort.
The RFC was a major effort in the IETF. Working Group members The RFC was a major effort in the IETF. Working group participants
developed and tested several implementations. Researchers analyzed developed and tested several implementations. Researchers analyzed
the specifications and performed formal verifications. Deployment the specifications and performed formal verifications. Deployment
tests outlined issues that caused extra work when the specification tests outlined issues that caused extra work when the specification
was almost ready. These complexity largely explains the time spent was almost ready. This complexity largely explains the time spent in
in the Working Group. the working group.
Comparing the final draft to the published version, we find Comparing the final draft to the published version, we find
relatively light copy editing. It includes explaining acronyms on relatively light copy editing. It includes explaining acronyms on
first use, clarifying some definitions standardizing punctiation and first use, clarifying some definitions standardizing punctuation and
capitalization, and spelling out some numbers in text. This capitalization, and spelling out some numbers in text. This
generally fall in the category of "style", although some of the generally fall in the category of "style", although some of the
clarifications go into message definitions. However, that simple clarifications go into message definitions. However, that simple
analysis does not explain why the AUTH-48 phase took almost two analysis does not explain why the AUTH48 phase took almost two
months. months.
This document's AUTH-48 process was part of the "Github experiment", This document's AUTH48 process was part of the "GitHub experiment",
which tried to use github pull requests to track the AUTH-48 changes which tried to use GitHub pull requests to track the AUTH48 changes
and review comments. The RPC staff had to learn using Github for and review comments. The RFC Production Center (RPC) staff had to
that process, and this required more work than the usual RFC. Author learn using GitHub for that process, and this required more work than
and AD thoroughly reviewed each proposed edit, accepting some and the usual RFC. The author and AD thoroughly reviewed each proposed
rejecting some. The concern there was that any change in a complex edit, accepting some and rejecting some. The concern there was that
specification might affect a protocol that was extensively reviewed any change in a complex specification might affect a protocol that
in the Working Group, but of course these reviews added time to the was extensively reviewed in the working group, but of course these
AUTH-48 delays. reviews added time to the AUTH48 delays.
There are 21 implementations listed in the Wiki of the TLS 1.3 There are 21 implementations listed in the Wiki of the TLS 1.3
project [TLS13IMP]. It has been deployed on major browsers, and is project [TLS13IMP]. It has been deployed on major browsers, and is
already used in a large fraction of TLS connections. already used in a large fraction of TLS connections.
3.4. 8355 3.4. RFC 8355
Resiliency Use Cases in Source Packet Routing in Networking (SPRING) "Resiliency Use Cases in Source Packet Routing in Networking (SPRING)
Networks [RFC8355] is an informational RFC. It originated from a use Networks" [RFC8355] is an Informational RFC. It originated from an
case informational draft that was mostly used for the BOF creating informational use-case draft; it was mostly used for the BOF creating
the WG, and then to drive initial work/evolutions from the WG. the WG, and then to drive initial work and evolutions from the WG.
Informational, 13 pages. Status (Length): Informational (13 pages)
2 personal drafts (personal), first 2014-01-31. 13 WG drafts. Overview: 2 individual drafts; 13 WG drafts
WG adoption on 2014-05-13 First draft: 2014-01-31
Last call announced 2017-04-20 WG adoption: 2014-05-13
IESG evaluation starts 2017-05-04, draft 09 Last Call start: 2017-04-20
Approved 2017-12-19, draft 12 IESG eval. start: 2017-05-04 (draft 09)
AUTH-48 2018-03-12 IESG approved: 2017-12-19 (draft 12)
AUTH-48 complete 2018-03-27 AUTH48 start: 2018-03-12
Published 2018-03-28 AUTH48 complete: 2018-03-27
Published: 2018-03-28
Minor set of copy edits, mostly for style. Minor set of copy edits, mostly for style.
No implementation of the RFC itself, but the technology behind it No implementation of the RFC itself, but the technology behind it
such as Segment Routing (architecture RFC 8402, TI-LFA draft-ietf- (such as Segment Routing Architecture [RFC8402] and TI-LFA [TI-LFA])
rtgwg-segment-routing-ti-lfa) is widely implemented and deployment is is widely implemented and deployment is ongoing.
ongoing.
According to participants in the discussion, the process of adoption According to participants in the discussion, the process of adoption
of the source packet routing standards was very contentious. The of the source packet routing standards was very contentious. The
establishment of consensus at both the Working Group level and the establishment of consensus at both the working group level and the
IETF level was difficult and time consuming. IETF level was difficult and time-consuming.
3.5. 8441 3.5. RFC 8441
Bootstrapping WebSockets with HTTP/2 [RFC8441] "Bootstrapping WebSockets with HTTP/2" [RFC8441]
Proposed standard, 8 pages. Updates RFC 6455.
3 personal drafts (personal), first 2017-10-15. 8 WG drafts. Status (Length): Proposed Standard (8 pages)
WG adoption on 2017-12-19 Overview: 3 individual drafts; 8 WG drafts; Updates RFC
Last call announced 2018-05-07, draft 05 6455
IESG evaluation starts 2018-05-29, draft 06 First draft: 2017-10-15
Approved 2018-06-07, draft 07 WG adoption: 2017-12-19
AUTH-48 2018-08-13 Last Call start: 2018-05-07 (draft 05)
AUTH-48 complete 2018-09-15 IESG eval. start: 2018-05-29 (draft 06)
Published 2018-09-21 IESG approved: 2018-06-18 (draft 07)
IANA Action: table entries AUTH48 start: 2018-08-13
AUTH48 complete: 2018-09-15
Published: 2018-09-18
IANA action: table entries
This RFC defines the support of WebSockets in HTTP/2, which is This RFC defines the support of WebSockets in HTTP/2, which is
different from the mechanism defined for HTTP/1.1 in [RFC6455]. The different from the mechanism defined for HTTP/1.1 in [RFC6455]. The
process was relatively straightforward, involving the usual type of process was relatively straightforward, involving the usual type of
discussions, some on details and some on important points. discussions, some on details and some on important points.
Comparing final draft and published RFC shows a minor set of copy Comparing the final draft and published RFC shows a minor set of copy
edit, mostly for style. However, the author recalls a painful edits, mostly for style. However, the author recalls a painful
process. The RFC includes many charts and graphs that were very process. The RFC includes many charts and graphs that were very
difficult to format correctly in the author's production process that difficult to format correctly in the author's production process that
involve conversions from markdown to XML, and then from XML to text. involved conversions from markdown to XML, and then from XML to text.
The author had to get substantial help from the RFC editor. The author had to get substantial help from the RFC Editor.
There are several implementations, including Firefox and Chrome, There are several implementations, including Firefox and Chrome,
making RFC 8441 a very successful specification. making RFC 8441 a very successful specification.
3.6. 8324 3.6. RFC 8324
DNS Privacy, Authorization, Special Uses, Encoding, Characters, "DNS Privacy, Authorization, Special Uses, Encoding, Characters,
Matching, and Root Structure: Time for Another Look? [RFC8324]. Matching, and Root Structure: Time for Another Look?" [RFC8324].
This is an opinion piece on DNS development, published on the This is an opinion piece on DNS development, published on the
Independent Stream. Independent Stream.
Informational, 29 pages. Independent Stream. Status (Length): Informational (29 pages)
5 personal drafts (personal), first 2017-06-02. Overview: 5 individual drafts; Independent Stream
ISE review started 2017-07-10, draft 03 First draft: 2017-06-02
IETF conflict review and IESG review started 2017-10-29 ISE review start: 2017-07-10 (draft 03)
Approved 2017-12-18, draft 04 IETF conflict review start: 2017-10-29
AUTH-48 2018-01-29, draft 05 Approved: 2017-12-18 (draft 04)
AUTH-48 complete 2018-02-26 AUTH48 start: 2018-01-29 (draft 05)
Published 2018-02-27 AUTH48 complete: 2018-02-26
Published: 2018-02-27
This RFC took only 9 months from first draft to publication, which is This RFC took only 9 months from first draft to publication, which is
the shortest in the 2018 sample set. In part, this is because the the shortest in the 2018 sample set. In part, this is because the
text was privately circulated and reviewed by ISE designated experts text was privately circulated and reviewed by the ISE's selected
before the first draft was published. The nature of the document is experts before the first draft was published. The nature of the
another reason for the short delay. It is an opinion piece, and does document is another reason for the short delay. It is an opinion
not require the same type of consensus building and reviews than a piece and does not require the same type of consensus building and
protocol specification. reviews as a protocol specification.
Comparing the final draft and the published version shows only minor Comparing the final draft and the published version shows only minor
copy edit, mostly for style. According to the author, because this copy edits, mostly for style. According to the author, this is
is because he knows how to write in RFC Style with the result that because he knows how to write in RFC style with the result that his
his documents often need a minimum of editing. He also makes sure documents often need a minimum of editing. He also makes sure that
that the document on which the Production Center starts working the document on which the RFC Production Center starts working
already has changes discussed and approved during Last Call and IESG already has changes discussed and approved during Last Call and IESG
review incorporated rather than expecting the Production Center to review incorporated, rather than expecting the Production Center to
operate off of notes about changed to be made. operate off of notes about changes to be made.
3.7. 8377 3.7. RFC 8377
Transparent Interconnection of Lots of Links (TRILL): Multi-Topology "Transparent Interconnection of Lots of Links (TRILL): Multi-
[RFC8377] Topology" [RFC8377]
Proposed standard, 20 pages. Updates RFC 6325, 7177. Status (Length): Proposed Standard (20 pages)
3 personal drafts (personal), first 2013-09-03. 7 WG drafts. Overview: 3 individual drafts; 7 WG drafts; Updates RFCs
WG adoption on 2015-09-01 6325 and 7177
Last call announced 2018-02-19, draft 05 First draft: 2013-09-03
IESG evaluation starts 2018-03-02, draft 06 WG adoption: 2015-09-01
Approved 2018-03-12, draft 05 Last Call start: 2018-02-19 (draft 05)
AUTH-48 2018-04-20, draft 06 IESG eval. start: 2018-03-06 (draft 05)
AUTH-48 complete 2018-07-31 IESG approved: 2018-03-12 (draft 06)
Published 2018-07-31 AUTH48 start: 2018-04-20 (draft 06)
IANA Table, table entries AUTH48 complete: 2018-07-31
Published: 2018-07-31
IANA action: table entries
Minor set of copy edits, mostly for style, also clarity. Minor set of copy edits, mostly for style, also clarity.
3.8. 8498 3.8. RFC 8498
A P-Served-User Header Field Parameter for an Originating Call "A P-Served-User Header Field Parameter for an Originating Call
Diversion (CDIV) Session Case in the Session Initiation Protocol Diversion (CDIV) Session Case in the Session Initiation Protocol
(SIP) [RFC8498]. (SIP)" [RFC8498].
Informational, 15 pages. Status (Length): Informational (15 pages)
5 personal drafts (personal), first 2016-03-21. 9 WG drafts. Overview: 5 individual drafts; 9 WG drafts
WG adoption on 2017-05-15 First draft: 2016-03-21
Last call announced 2018-10-12, draft 05 WG adoption: 2017-05-15
IESG evaluation starts 2018-11-28, draft 07 Last Call start: 2018-10-12 (draft 05)
Approved 2018-12-10, draft 08 IESG eval. start: 2018-11-28 (draft 07)
AUTH-48 2019-01-28 IESG approved: 2018-12-11 (draft 08)
AUTH-48 complete 2019-02-13 AUTH48 start: 2019-01-28
Published 2019-02-15 AUTH48 complete: 2019-02-13
IANA Action, table rows added. Published: 2019-02-14
IANA action: table rows added.
Copy edit for style, but also clarification of ambiguous sentences. Copy edits for style, but also clarification of ambiguous sentences.
3.9. 8479 3.9. RFC 8479
Storing Validation Parameters in PKCS#8 [RFC8479] "Storing Validation Parameters in PKCS#8" [RFC8479]
Informational, 8 pages. Independent Stream. Status (Length): Informational (8 pages)
5 personal drafts (personal), first 2017-08-08. Overview: 5 individual drafts; Independent Stream
ISE review started 2018-12-10, draft 00 First draft: 2017-08-08
IETF conflict review and IESG review started 2018-03-29 ISE review start: 2018-12-10 (draft 00)
Approved 2018-08-20, draft 03 IETF conflict review start: 2018-03-29
AUTH-48 2018-09-20, draft 04 Approved: 2018-08-20 (draft 03)
AUTH-48 complete 2018-09-25 AUTH48 start: 2018-09-20 (draft 04)
Published 2018-09-26 AUTH48 complete: 2018-09-25
Published: 2018-09-26
The goal of the draft was to document what the gnutls implementation The goal of the draft was to document what the gnutls implementation
was using for storing provably generated RSA keys. This is a short was using for storing provably generated RSA keys. This is a short
RFC that was published relatively quickly, although discussion RFC that was published relatively quickly, although discussion
between the author, the Independent Series Editor and the IESG lasted between the author, the Independent Submissions Editor, and the IESG
several months. In the initial conflict review, The IESG asked the lasted several months. In the initial conflict review, the IESG
ISE to not publish this document before IETF Working Groups had an asked the ISE to not publish this document before IETF working groups
opportunity to pick up the work. The author met that requirement by had an opportunity to pick up the work. The author met that
a presentation to the SECDISPATCH WG in IETF 102. Since no WG was requirement by a presentation to the SECDISPATCH WG during IETF 102.
interested in pickup the work, the document progressed on the Since no WG was interested in picking up the work, the document
Independent Stream. progressed on the Independent Stream.
Very minor set of copy edit, moving some references from normative to Very minor set of copy edits, moving some references from normative
informative. to informative.
The author is not aware of other implementations than gnutls relying The author is not aware of other implementations than gnutls relying
on this RFC. on this RFC.
3.10. 8453 3.10. RFC 8453
Framework for Abstraction and Control of TE Networks (ACTN) [RFC8453] "Framework for Abstraction and Control of TE Networks (ACTN)"
[RFC8453]
Informational, 42 pages. Status (Length): Informational (42 pages)
3 personal drafts, first 2015-06-15. 16 WG drafts. Overview: 3 individual drafts; 16 WG drafts
WG adoption on 2016-07-15 First draft: 2015-06-15
Out of WG 2018-01-26, draft 11 WG adoption: 2016-07-15
Expert review requested, 2018-02-13 Out of WG: 2018-01-26 (draft 11)
Last call announced 2018-04-16, draft 13 Expert review requested: 2018-02-13
IESG evaluation starts 2018-05-16, draft 14 Last Call start: 2018-04-16 (draft 13)
Approved 2018-06-01, draft 15 IESG eval. start: 2018-05-16 (draft 14)
AUTH-48 2018-08-13 IESG approved: 2018-06-01 (draft 15)
AUTH-48 complete 2018-08-20 AUTH48 start: 2018-08-13
Published 2018-08-20 AUTH48 complete: 2018-08-20
IANA Action, table rows added. Published: 2018-08-23
IANA action: table rows added.
Minor copy editing. Minor copy editing.
3.11. 8429 3.11. RFC 8429
Deprecate Triple-DES (3DES) and RC4 in Kerberos [RFC8429] "Deprecate Triple-DES (3DES) and RC4 in Kerberos" [RFC8429]
BCP, 10 pages. Status (Length): BCP (10 pages)
6 WG drafts, first 2017-05-01. Overview: 6 WG drafts
Last call announced 2017-07-16, draft 03 First draft: 2017-05-01
IESG evaluation starts 2017-08-18, draft 04 Last Call start: 2017-07-16 (draft 03)
Approved 2018-05-25, draft 05 IESG eval. start: 2017-08-18 (draft 04)
AUTH-48 2018-07-24 IESG approved: 2018-05-25 (draft 05)
AUTH-48 complete 2018-10-31 AUTH48 start: 2018-07-24
Published 2018-10-31 AUTH48 complete: 2018-10-31
IANA Action, table rows added. Published: 2018-10-31
IANA action: table rows added.
This draft started as a Working Group effort. This draft started as a working group effort.
This RFC recommends to deprecate two encryption algorithms that are This RFC recommends deprecating two encryption algorithms that are
now considered obsolete and possibly broken. The document was sent now considered obsolete and possibly broken. The document was sent
back to the WG after the first last call, edited, and then there was back to the WG after the first Last Call, edited, and then there was
a second last call. The delay from first draft to Working Group last a second Last Call. The delay from first draft to Working Group Last
call was relatively short, but the number may be misleading. The Call was relatively short, but the number may be misleading. The
initial draft was a replacement of a similar draft in the KITTEN initial draft was a replacement of a similar draft in the KITTEN
Working Group, which stagnated for some time before the CURDLE Working Group, which stagnated for some time before the CURDLE
Working Group took up the work. The deprecation of RC4 was somewhat Working Group took up the work. The deprecation of RC4 was somewhat
contentious, but the WG had already debated this prior to the contentious, but the WG had already debated this prior to the
production of this draft, and the draft was not delayed by this production of this draft, and the draft was not delayed by this
debate. debate.
Most of the 280 days between IETF LC and IESG approval was because Most of the 280 days between IETF LC and IESG approval were because
the IESG had to talk about whether this document should obsolete or the IESG had to talk about whether this document should obsolete RFC
move to historic RFC 4757, and no one was really actively pushing 4757 or move it to Historic status, and no one was really actively
that discussion for a while. pushing that discussion for a while.
The 99 days in AUTH-48 are mostly because one of the authors was a The 99 days in AUTH48 are mostly because one of the authors was a
sitting AD, and those duties ended up taking precedence over sitting AD, and those duties ended up taking precedence over
reviewing this document. reviewing this document.
Minor copy editing, for style. Minor copy editing, for style.
The implementation of the draft would be the actual removal of The implementation of the draft would be the actual removal of
support for 3DES and RC4 in major implementations. This is support for 3DES and RC4 in major implementations. This is
happening, but very slowly. happening, but very slowly.
3.12. 8312 3.12. RFC 8312
CUBIC for Fast Long-Distance Networks [RFC8312] "CUBIC for Fast Long-Distance Networks" [RFC8312]
Informational, 18 pages.
2 personal drafts, first 2014-09-01. 8 WG drafts Status (Length): Informational (18 pages)
WG adoption on 2015-06-08 Overview: 2 individual drafts; 8 WG drafts
Last call announced 2017-09-18, draft 06 First draft: 2014-09-01
IESG evaluation starts 2017-11-14 WG adoption: 2015-06-08
Approved 2017-10-04, draft 07 Last Call start: 2017-09-18 (draft 06)
AUTH-48 2018-01-08 IESG eval. start: 2017-10-04
AUTH-48 complete 2018-02-07 IESG approved: 2017-11-14 (draft 07)
Published 2018-02-07 AUTH48 start: 2018-01-08
IANA Action, table rows added. AUTH48 complete: 2018-02-07
Published: 2018-02-07
IANA action: table rows added.
Minor copy editing, for style. Minor copy editing, for style.
The TCP congestion control algorithm Cubic was defined first in 2005, The TCP congestion control algorithm Cubic was first defined in 2005,
was implemented in Linux soon after, and was implemented in major was implemented in Linux soon after, and was implemented in major
OSes after that. After some debates from 2015 to 2015, the TCPM OSes after that. After some debates from 2015 to 2015, the TCPM
Working Group adopted the draft, with a goal of documenting Cubic in Working Group adopted the draft, with a goal of documenting Cubic in
the RFc series. According to the authors, this was not a high the RFC Series. According to the authors, this was not a high-
priority effort, as Cubic was already implemented in multiple OSes priority effort, as Cubic was already implemented in multiple OSes
and documented in research papers. At some point, only one of the and documented in research papers. At some point, only one of the
authors was actively working on the draft. Ths may explain why authors was actively working on the draft. This may explain why
another two years was spent progressing the draft after adoption by another two years was spent progressing the draft after adoption by
the WG. the WG.
The RFC publication may or may not have triggered further The RFC publication may or may not have triggered further
implementations. On the other hand, several OSes picked up bug fixes implementations. On the other hand, several OSes picked up bug fixes
from the draft and the RFC. from the draft and the RFC.
3.13. 8492 3.13. RFC 8492
Secure Password Ciphersuites for Transport Layer Security (TLS) "Secure Password Ciphersuites for Transport Layer Security (TLS)"
[RFC8492] [RFC8492]
Informational, 40 pages. (Independent Stream) Status (Length): Informational (40 pages)
10 personal drafts, first 2012-09-07. 8 WG drafts Overview: 10 individual drafts; 8 WG drafts; Independent
Targeted to ISE stream 2016-08-05 Stream
ISE review started 2017-05-10, draft 01 First draft: 2012-09-07
IETF conflict review and IESG review started 2017-09-04 Targeted to ISE: 2016-08-05
Approved 2017-10-29, draft 04 ISE review start: 2017-05-10 (draft 01)
AUTH-48 2018-10-19, draft 05 IETF conflict review start: 2017-09-04
AUTH-48 complete 2019-02-19 Approved: 2017-10-29 (draft 02)
Published 2019-02-21 AUTH48 start: 2018-10-19 (draft 05)
IANA Action, table rows added. AUTH48 complete: 2019-02-19
Published: 2019-02-21
IANA action: table rows added.
This RFC has a complex history. The first individual draft was This RFC has a complex history. The first individual draft was
submitted to the TLS Working Group on September 7, 2012. It submitted to the TLS Working Group on September 7, 2012. It
progressed there, and was adopted by the WG after 3 revisions. There progressed there, and was adopted by the WG after 3 revisions. There
were then 8 revisions in the TLS WG, until the WG decided to not were then 8 revisions in the TLS WG, until the WG decided to not
progress it. The draft was parked in 2013 by the WG chairs after progress it. The draft was parked in 2013 by the WG chairs after
failing to get consensus in WG last call. The AD finally pulled the failing to get consensus in WG Last Call. The AD finally pulled the
plug in 2016, and the draft was then resubmitted to the ISE. plug in 2016, and the draft was then resubmitted to the ISE.
At that point, the author was busy and was treating this RFC with a At that point, the author was busy and was treating this RFC with a
low priority because, in his words, it would not be a "real RFC". low priority because, in his words, it would not be a "real RFC".
There were problems with the draft that only came up late. In There were problems with the draft that only came up late. In
particular, it had to wait for a change in registry policy that only particular, it had to wait for a change in registry policy that only
came about with the publication of TLS 1.3, which caused the draft to came about with the publication of TLS 1.3, which caused the draft to
only be published after RFC 8446, and also required adding references be published after RFC 8446, and also required adding references to
to TLS 1.3. The author also got a very late comment while in AUTH-48 TLS 1.3. The author also got a very late comment while in AUTH48
that caused some rewrite. Finally, there was some IANA issue with that caused some rewriting. Finally, there was some IANA issue with
the extension registry where a similar extension was added by someone the extension registry where a similar extension was added by someone
else. The draft was changed to just use it. else. The draft was changed to just use it.
Changes in AUTH-48 include added reference to TLS 1.3, copy-editing Changes in AUTH48 include adding a reference to TLS 1.3, copy editing
for style, some added requirements, added paragraphs, and changes in for style, some added requirements, added paragraphs, and changes in
algorithms specification. algorithms specification.
3.14. 8378 3.14. RFC 8378
Signal-Free Locator/ID Separation Protocol (LISP) Multicast [RFC8378] "Signal-Free Locator/ID Separation Protocol (LISP) Multicast"
is an experimental RFC, defining how to implement Multicast in the [RFC8378] is an Experimental RFC, defining how to implement Multicast
LISP architecture. in the LISP architecture.
Experimental, 21 pages. Status (Length): Experimental (21 pages)
5 personal drafts, first 2014-02-28. 10 WG drafts Overview: 5 individual drafts; 10 WG drafts
WG adoption on 2015-12-21 First draft: 2014-02-28
Last call announced 2018-02-13, draft 07 WG adoption: 2015-12-21
IESG evaluation starts 2018-02-28, draft 08 Last Call start: 2018-02-13 (draft 07)
Approved 2018-03-12, draft 09 IESG eval. start: 2018-02-28 (draft 08)
AUTH-48 2018-04-23 IESG approved: 2018-03-12 (draft 09)
AUTH-48 complete 2018-05-02 AUTH48 start: 2018-04-23
Published 2018-05-02 AUTH48 complete: 2018-05-02
Published: 2018-05-02
Preparing the RFC took more than 4 years. According to the authors, Preparing the RFC took more than 4 years. According to the authors,
they were not aggressive pushing it and just let the Working Group they were not aggressively pushing it and just let the working group
process decide to pace it. They also did implementations during that process decide to pace it. They also did implementations during that
time. time.
Minor copy editing, for style. Minor copy editing, for style.
The RFC was implemented by lispers.net and cisco, and was used in The RFC was implemented by lispers.net and Cisco, and it was used in
doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in
PyeungChang. The plan is to work on a proposedstandard once the PyeungChang. The plan is to work on a Proposed Standard once the
experiment concludes. experiment concludes.
3.15. 8361 3.15. RFC 8361
Transparent Interconnection of Lots of Links (TRILL): Centralized "Transparent Interconnection of Lots of Links (TRILL): Centralized
Replication for Active-Active Broadcast, Unknown Unicast, and Replication for Active-Active Broadcast, Unknown Unicast, and
Multicast (BUM) Traffic [RFC8361] Multicast (BUM) Traffic" [RFC8361]
Proposed Standard, 17 pages. Status (Length): Proposed Standard (17 pages)
3 personal drafts, first 2013-11-12. 14 WG drafts Overview: 3 individual drafts; 14 WG drafts
WG adoption on 2014-12-16 First draft: 2013-11-12
Last call announced 2017-11-28, draft 10 WG adoption: 2014-12-16
IESG evaluation starts 2017-12-18, draft 11 Last Call start: 2017-11-28 (draft 10)
Approved 2018-01-29, draft 13 IESG eval. start: 2017-12-18 (draft 11)
AUTH-48 2018-03-09 IESG approved: 2018-01-29 (draft 13)
AUTH-48 complete 2018-04-09 AUTH48 start: 2018-03-09
Published 2018-04-12 AUTH48 complete: 2018-04-09
Published: 2018-04-12
According to the authors, the long delays in producing this RFC was According to the authors, the long delays in producing this RFC were
due to a slow uptake of the technology in the industry. due to a slow uptake of the technology in the industry.
Minor copy editing, for style. Minor copy editing, for style.
There was at least 1 partial implementation. There was at least one partial implementation.
3.16. 8472 3.16. RFC 8472
Transport Layer Security (TLS) Extension for Token Binding Protocol "Transport Layer Security (TLS) Extension for Token Binding Protocol
Negotiation [RFC8472] Negotiation" [RFC8472]
Proposed Standard, 8 pages. Status (Length): Proposed Standard (8 pages)
1 personal drafts, 2015-05-29. 15 WG drafts Overview: 1 individual draft; 15 WG drafts
WG adoption on 2015-09-11 First draft: 2015-05-29
Last call announced 2017-11-13, draft 10 WG adoption: 2015-09-11
IESG evaluation starts 2018-03-19 Last Call start: 2017-11-13 (draft 10)
Approved 2018-07-20, draft 14 IESG eval. start: 2018-03-19
AUTH-48 2018-09-17 IESG approved: 2018-07-20 (draft 14)
AUTH-48 complete 2018-09-25 AUTH48 start: 2018-09-17
Published 2018-10-08 AUTH48 complete: 2018-09-25
Published: 2018-10-08
This is a pretty simple document, but it took over 3 years from This is a pretty simple document, but it took over 3 years from
individual draft to RFC. According to the authors,the biggest individual draft to RFC. According to the authors,the biggest
setbacks occurred at the start: it took a while to find a home for setbacks occurred at the start: it took a while to find a home for
this draft. It was presented in the TLS WG (because it's a TLS this draft. It was presented in the TLS WG (because it's a TLS
extension) and UTA WG (because it has to do with applications using extension) and UTA WG (because it has to do with applications using
TLS). Then the ADs determined that a new WG was needed, so the TLS). Then the ADs determined that a new WG was needed, so the
authors had to work through the WG creation process, including authors had to work through the WG creation process, including
running a BOF. running a BOF.
Minor copy editing, for style, with the addition of a reference to Minor copy editing, for style, with the addition of a reference to
TLS 1.3. TLS 1.3.
Perhaps partially due to the delays, some of the implementers lost Perhaps partially due to the delays, some of the implementers lost
interest in supporting this RFC. interest in supporting this RFC.
3.17. 8471 3.17. RFC 8471
The Token Binding Protocol Version 1.0 [RFC8471] "The Token Binding Protocol Version 1.0" [RFC8471]
Proposed Standard, 18 pages. Status (Length): Proposed Standard (18 pages)
1 personal drafts, 2014-10-13. 19 WG drafts Overview: 1 individual draft; 19 WG drafts
WG adoption on 2015-03-15 First draft: 2014-10-13
Last call announced 2017-11-13, draft 16 WG adoption: 2015-03-15
IESG evaluation starts 2018-03-19 Last Call start: 2017-11-13 (draft 16)
Approved 2018-07-20, draft 19 IESG eval. start: 2018-03-19
AUTH-48 2018-09-17 IESG approved: 2018-07-20 (draft 19)
AUTH-48 complete 2018-09-25 AUTH48 start: 2018-09-17
Published 2018-10-08 AUTH48 complete: 2018-09-25
Published: 2018-10-08
Presentation of a Token Binding Protocol for TLS. We can notice a This document presents a Token Binding Protocol for TLS. We can
delay of 5 months before adoption of the draft by the WG. That notice a period of 5 months before adoption of the draft by the WG.
explains in part the overall delay of almost 4 years from first draft That explains in part the overall time of almost 4 years from first
to publication. draft to publication.
Minor copy editing, for style. Minor copy editing, for style.
The web references indicates adoption in multiple development The web references indicate adoption in multiple development
projects. projects.
3.18. 8466 3.18. RFC 8466
A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service "A YANG Data Model for Layer 2 Virtual Private Network (L2VPN)
Delivery [RFC8466] Service Delivery" [RFC8466]
Proposed Standard, 158 pages. Status (Length): Proposed Standard (158 pages)
5 personal drafts, first 2016-09-01. 11 WG drafts Overview: 5 individual drafts; 11 WG drafts
WG adoption on 2017-02-26 First draft: 2016-09-01
Last call announced 2018-02-21, draft 07 WG adoption: 2017-02-26
IESG evaluation starts 2018-03-14, draft 08 Last Call start: 2018-02-21 (draft 07)
Approved 2018-06-25, draft 10 IESG eval. start: 2018-03-14 (draft 08)
AUTH-48 2018-09-17 IESG approved: 2018-06-25 (draft 10)
AUTH-48 complete 2018-10-09 AUTH48 start: 2018-09-17
Published 2018-10-12 AUTH48 complete: 2018-10-09
Published: 2018-10-12
Copy editing for style and clarity, with also corrections to the yang Copy editing for style and clarity, with also corrections to the YANG
model. model.
3.19. 8362 3.19. RFC 8362
OSPFv3 Link State Advertisement (LSA) Extensibility [RFC8362] is a "OSPFv3 Link State Advertisement (LSA) Extensibility" [RFC8362] is a
major extension to the OSPF protocol. It makes OSPFv3 fully major extension to the OSPF protocol. It makes OSPFv3 fully
extensible. extensible.
Proposed Standard, 33 pages. Status (Length): Proposed Standard (33 pages)
4 personal drafts, first 2013-02-17. 24 WG drafts Overview: 4 individual drafts; 24 WG drafts
WG adoption on 2013-10-15 First draft: 2013-02-17
Last call announced 2017-12-19, draft 19 WG adoption: 2013-10-15
IESG evaluation starts 2018-01-18, draft 20 Last Call start: 2017-12-19 (draft 19)
Approved 2018-01-29, draft 23 IESG eval. start: 2018-01-18 (draft 20)
AUTH-48 2018-03-19 IESG approved: 2018-01-29 (draft 23)
AUTH-48 complete 2018-03-30 AUTH48 start: 2018-03-19
Published 2018-04-03 AUTH48 complete: 2018-03-30
Published: 2018-04-03
The specification was first submitted as a personal draft in the IPv6 The specification was first submitted as an individual draft in the
WG, then moved to the OSPF WG. The long delay of producing this RFC IPv6 WG, then moved to the OSPF WG. The long delay of producing this
is due to the complexity of the problem, and the need to wait for RFC is due to the complexity of the problem, and the need to wait for
implementations. It is a very important change to OSPF that makes implementations. It is a very important change to OSPF that makes
OSPFv3 fully extensible. Since it was a non-backward compatible OSPFv3 fully extensible. Since it was a non-backward compatible
change, the developers started out with some very complex migration change, the developers started out with some very complex migration
scenarios but ended up with either legacy or extended OSPFv3 LSAs scenarios but ended up with either legacy or extended OSPFv3 LSAs
within an OSPFv3 routing domain. The initial attempts to have a within an OSPFv3 routing domain. The initial attempts to have a
hybrid mode of operation with both legacy and extended LSAs also hybrid mode of operation with both legacy and extended LSAs also
delayed implementation due to the complexity. delayed implementation due to the complexity.
Copy editing for style and clarity. Copy editing for style and clarity.
This specification either was or will be implemented by all the This specification either was or will be implemented by all the
router vendors. router vendors.
3.20. 8468 3.20. RFC 8468
IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP
Metrics (IPPM) Framework [RFC8468]. Performance Metrics (IPPM) Framework" [RFC8468].
Informational, 15 pages. Status (Length): Informational (15 pages)
3 personal drafts, first 2015-08-06. 7 WG drafts Overview: 3 individual drafts; 7 WG drafts
WG adoption on 2016-07-04 First draft: 2015-08-06
Last call announced 2018-04-11, draft 04 WG adoption: 2016-07-04
IESG evaluation starts 2018-05-24, draft 05 Last Call start: 2018-04-11 (draft 04)
Approved 2018-07-10, draft 06 IESG eval. start: 2018-05-24 (draft 05)
AUTH-48 2018-09-13 IESG approved: 2018-07-10 (draft 06)
AUTH-48 complete 2018-11-05 AUTH48 start: 2018-09-13
Published 2018-11-14 AUTH48 complete: 2018-11-05
RFC8468 was somehow special in that there was not a technical reason/ Published: 2018-11-14
interest that triggered it, but rather a formal requirement. While
writing RFC7312 the IP Performance Metrics Working Group (IPPM) RFC 8468 was somehow special in that there was not a technical reason
realized that RFC 2330, the IP Performance Metrics Framework or interest that triggered it, but rather a formal requirement.
While writing RFC 7312, the IP Performance Metrics (IPPM) Working
Group realized that RFC 2330, the IP Performance Metrics Framework
supported IPv4 only and explicitly excluded support for IPv6. supported IPv4 only and explicitly excluded support for IPv6.
Nevertheless, people used the metrics that were defined on top of RFC Nevertheless, people used the metrics that were defined on top of RFC
2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG
agreed that the work was needed, the interest of IPPM attendees in agreed that the work was needed, the interest of IPPM attendees in
progressing (and reading/reviewing) the IPv6 draft was limited. progressing (and reading/reviewing) the IPv6 draft was limited.
Resolving the IPv6 technical part was straight-forward, but Resolving the IPv6 technical part was straightforward, but
subsequently some people asked for a broader scope (topics like subsequently some people asked for a broader scope (topics like
header compression, 6lo, etc.) and it took some time to figure out header compression, 6LoWPAN, etc.), and it took some time to figure
and later on convince people that these topics are out of scope. The out and later on convince people that these topics are out of scope.
group also had to resolve contentious topics, for example how to The group also had to resolve contentious topics, for example, how to
measure the processing of IPv6 extension headers, which is sometimes measure the processing of IPv6 extension headers, which is sometimes
non-standard. nonstandard.
The AUTH-48 delay for this draft was longer than average. According The time in AUTH48 state for this document was longer than average.
to the authors, the main reasons include: According to the authors, the main reasons include:
o Work-load and travel caused by busy-work-periods of all co-authors * Workload and travel caused by busy work periods of all coauthors
o Time zone difference between co-authors and editor (at least US, * Time zone difference between coauthors and editor (at least US,
Europe, India, not considering travel) Europe, and India, not considering travel)
o Editor proposing and committing some unacceptable modifications * RFC Production Center proposed and committed some unacceptable
that needed to be reverted modifications that needed to be reverted
o Lengthy discussions on a new document title (required high effort * Lengthy discussions on a new document title (required high effort
and took a long time, in particular reaching consensus between co- and took a long time, in particular reaching consensus between
authors and editor was time-consuming and involved the AD) coauthors and editor was time-consuming and involved the AD)
o Editor correctly identifying some nits (obsoleted personal * RFC Production Center correctly identified some nits (obsoleted
websites of co-authors) and co-authors attempting to fix them. personal websites of coauthors) and coauthors attempting to fix
them.
The differences between the final draft and the publish RFC show copy The differences between the final draft and the published RFC show
editing for style and clarity, but do not account for the back and copy editing for style and clarity, but do not account for the back
forth between authors and editors mentioned by the authors. and forth between authors and editors mentioned by the authors.
4. Analysis of Process and Delays 4. Analysis of Process and Delays
We examine the 20 RFCs in the sample, measuring various We examine the 20 RFCs in the sample, measuring various
characteristics such as delay and citation counts, in an attempt to characteristics such as delay and citation counts, in an attempt to
identify patterns in the IETF processes. identify patterns in the IETF processes.
4.1. First Draft to RFC Delays 4.1. Delays from First Draft to RFC
We look at the distribution of delays between the submission of the We look at the distribution of delays between the submission of the
first draft and the publication of the RFC, using the three big first draft and the publication of the RFC, using the three
milestones defined in Section 2.1: processing time in the Working milestones defined in Section 2.1: processing time in the working
Group, IETF processing time, and publication delay. The following group, IETF processing time, and RFC production time. The following
table shows the delays for the 20 RFCs in the sample: table shows the number of days in each phase for the 20 RFCs in the
sample:
+------+------------------+-------+---------+------+------+------+ +======+============+=======+=========+======+======+======+
| RFC | Status | Pages | Overall | WG | IETF | Edit | | RFC | Status | Pages | Overall | WG | IETF | Edit |
+------+------------------+-------+---------+------+------+------+ +======+============+=======+=========+======+======+======+
| 8411 | Info | 5 | 455 | 154 | 140 | 161 | | 8411 | Info | 5 | 455 | 154 | 140 | 161 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8456 | Info | 64 | 1317 | 1033 | 126 | 158 | | 8456 | Info | 64 | 1317 | 1033 | 126 | 158 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8446 | PS | 160 | 1576 | 1400 | 34 | 142 | | 8446 | PS | 160 | 1576 | 1400 | 34 | 142 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8355 | Info | 13 | 1517 | 1175 | 243 | 99 | | 8355 | Info | 13 | 1517 | 1175 | 243 | 99 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8441 | PS | 8 | 341 | 204 | 31 | 106 | | 8441 | PS | 8 | 327 | 204 | 31 | 92 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8324 | ISE | 29 | 270 | 38 | 161 | 71 | | 8324 | Info (ISE) | 29 | 270 | 38 | 161 | 71 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8377 | PS | 8 | 1792 | 1630 | 21 | 141 | | 8377 | PS | 8 | 1792 | 1630 | 21 | 141 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8498 | Info | 15 | 1061 | 935 | 59 | 67 | | 8498 | Info | 15 | 1059 | 935 | 59 | 65 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8479 | ISE | 8 | 414 | 233 | 144 | 37 | | 8479 | Info (ISE) | 8 | 414 | 233 | 144 | 37 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8453 | Info | 42 | 1162 | 1036 | 46 | 80 | | 8453 | Info | 42 | 1165 | 1036 | 46 | 83 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8429 | BCP | 10 | 548 | 76 | 313 | 159 | | 8429 | BCP | 10 | 548 | 76 | 313 | 159 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8312 | Info | 18 | 1255 | 1113 | 16 | 126 | | 8312 | Info | 18 | 1214 | 1113 | 16 | 85 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8492 | ISE | 40 | 2358 | 1706 | 172 | 480 | | 8492 | Info (ISE) | 40 | 2358 | 1706 | 172 | 480 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 | | 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8361 | PS | 17 | 1612 | 1477 | 62 | 73 | | 8361 | PS | 17 | 1612 | 1477 | 62 | 73 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8472 | PS | 8 | 1228 | 899 | 249 | 80 | | 8472 | PS | 8 | 1228 | 899 | 249 | 80 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8471 | PS | 18 | 1228 | 899 | 249 | 80 | | 8471 | PS | 18 | 1228 | 899 | 249 | 80 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8466 | PS | 158 | 771 | 538 | 124 | 109 | | 8466 | PS | 158 | 771 | 538 | 124 | 109 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8362 | PS | 33 | 1871 | 1766 | 41 | 64 | | 8362 | PS | 33 | 1871 | 1766 | 41 | 64 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| 8468 | Info | 15 | 1196 | 979 | 90 | 127 | | 8468 | Info | 15 | 1196 | 979 | 90 | 127 |
| | | | | | | | +------+------------+-------+---------+------+------+------+
| | average | 35 | 1186 | 948 | 117 | 121 | | average | 35 | 1172 | 948 | 117 | 118 |
| | | | | | | | +-------------------+-------+---------+------+------+------+
| | average(not ISE) | 36 | 1217 | 999 | 110 | 107 | | average (not ISE) | 36 | 1200 | 999 | 110 | 104 |
+------+------------------+-------+---------+------+------+------+ +-------------------+-------+---------+------+------+------+
Table 1
The average delay from first draft to publication is about 3 years The average delay from first draft to publication is about 3 years
and 3 months, but this varies widely. Excluding the Independent and 3 months, but this varies widely. Excluding the RFCs from the
Stream submissions, the average delay from start to finish is 3 years Independent Stream, the average delay from start to finish is 3 years
and 4 months, of which on average 2 years and 9 months are spent and 4 months, of which on average 2 years and 9 months are spent
getting consensus in the Working Group, and 3 to 4 months each for getting consensus in the working group, and 3 to 4 months each for
IETF consensus and for RFC production. IETF consensus and for RFC production.
The longest delay is found for [RFC8492], 6.5 years from start to The longest delay is found for [RFC8492], 6.5 years from start to
finish. This is however a very special case, a draft that was finish. This is however a very special case -- a draft that was
prepared for the TLS Working Group and failed to reach consensus. prepared for the TLS Working Group and failed to reach consensus.
After that, it was resubmitted to the ISE, and incurred atypical After that, it was resubmitted to the ISE, and incurred atypical
production delays. production delays.
On average, we see that 80% of the delay is incurred in WG On average, we see that 80% of the delay is incurred in WG
processing, 10% in IETF review, and 10% for edition and publication. processing, 10% in IETF review, and 10% for edition and publication.
For IETF stream RFCs, it appears that the delays for informational For IETF Stream RFCs, it appears that the delays for Informational
documents are slightly shorter than those for protocol documents are slightly shorter than those for protocol
specifications, maybe six months shorter on average. However, there specifications, maybe six months shorter on average. However, there
are lots of differences between individual documents. The delays are lots of differences between individual documents. The delays
range from less than a year to more than 5 years for protocol range from less than a year to more than 5 years for protocol
specifications, and from a year and 3 months to a bit more than 4 specifications, and from a year and 3 months to a bit more than 4
years for informational documents. years for Informational documents.
We can compare the delays in the 2018 samples to those observed 10 We can compare the delays in the 2018 samples to those observed 10
years ago and 20 years before: years ago and 20 years before:
+------------+--------+-------+-------+ +============+============+=======+=======+
| RFC (2008) | Status | Pages | Delay | | RFC (2008) | Status | Pages | Delay |
+------------+--------+-------+-------+ +============+============+=======+=======+
| 5326 | Exp | 54 | 1584 | | 5326 | Exp | 54 | 1584 |
| | | | | +------------+------------+-------+-------+
| 5348 | PS | 58 | 823 | | 5348 | PS | 58 | 823 |
| | | | | +------------+------------+-------+-------+
| 5281 | Info | 51 | 1308 | | 5281 | Info | 51 | 1308 |
| | | | | +------------+------------+-------+-------+
| 5354 | Exp | 23 | 2315 | | 5354 | Exp | 23 | 2315 |
| | | | | +------------+------------+-------+-------+
| 5227 | PS | 21 | 2434 | | 5227 | PS | 21 | 2434 |
| | | | | +------------+------------+-------+-------+
| 5329 | PS | 12 | 1980 | | 5329 | PS | 12 | 1980 |
| | | | | +------------+------------+-------+-------+
| 5277 | PS | 35 | 912 | | 5277 | PS | 35 | 912 |
| | | | | +------------+------------+-------+-------+
| 5236 | ISE | 26 | 1947 | | 5236 | Info (ISE) | 26 | 1947 |
| | | | | +------------+------------+-------+-------+
| 5358 | BCP | 7 | 884 | | 5358 | BCP | 7 | 884 |
| | | | | +------------+------------+-------+-------+
| 5271 | Info | 22 | 1066 | | 5271 | Info | 22 | 1066 |
| | | | | +------------+------------+-------+-------+
| 5195 | PS | 10 | 974 | | 5195 | PS | 10 | 974 |
| | | | | +------------+------------+-------+-------+
| 5283 | PS | 12 | 1096 | | 5283 | PS | 12 | 1096 |
| | | | | +------------+------------+-------+-------+
| 5186 | Info | 6 | 2253 | | 5186 | Info | 6 | 2253 |
| | | | | +------------+------------+-------+-------+
| 5142 | PS | 13 | 1005 | | 5142 | PS | 13 | 1005 |
| | | | | +------------+------------+-------+-------+
| 5373 | PS | 24 | 1249 | | 5373 | PS | 24 | 1249 |
| | | | | +------------+------------+-------+-------+
| 5404 | PS | 27 | 214 | | 5404 | PS | 27 | 214 |
| | | | | +------------+------------+-------+-------+
| 5172 | PS | 7 | 305 | | 5172 | PS | 7 | 305 |
| | | | | +------------+------------+-------+-------+
| 5349 | Info | 10 | 1096 | | 5349 | Info | 10 | 1096 |
| | | | | +------------+------------+-------+-------+
| 5301 | PS | 6 | 396 | | 5301 | PS | 6 | 396 |
| | | | | +------------+------------+-------+-------+
| 5174 | Info | 8 | 427 | | 5174 | Info | 8 | 427 |
+------------+--------+-------+-------+ +------------+------------+-------+-------+
+------------+--------+-------+---------+ Table 2
| RFC (1998) | Status | Pages | Delay |
+------------+--------+-------+---------+ +============+============+=======+=========+
| 2289 | PS | 25 | 396 | | RFC (1998) | Status | Pages | Delay |
| | | | | +============+============+=======+=========+
| 2267 | Info | 10 | unknown | | 2289 | PS | 25 | 396 |
| | | | | +------------+------------+-------+---------+
| 2317 | BCP | 10 | 485 | | 2267 | Info | 10 | unknown |
| | | | | +------------+------------+-------+---------+
| 2404 | PS | 7 | 488 | | 2317 | BCP | 10 | 485 |
| | | | | +------------+------------+-------+---------+
| 2374 | PS | 12 | 289 | | 2404 | PS | 7 | 488 |
| | | | | +------------+------------+-------+---------+
| 2449 | PS | 19 | 273 | | 2374 | PS | 12 | 289 |
| | | | | +------------+------------+-------+---------+
| 2283 | PS | 9 | 153 | | 2449 | PS | 19 | 273 |
| | | | | +------------+------------+-------+---------+
| 2394 | Info | 6 | 365 | | 2283 | PS | 9 | 153 |
| | | | | +------------+------------+-------+---------+
| 2348 | DS | 5 | 699 | | 2394 | Info | 6 | 365 |
| | | | | +------------+------------+-------+---------+
| 2382 | Info | 30 | 396 | | 2348 | DS | 5 | 699 |
| | | | | +------------+------------+-------+---------+
| 2297 | ISE | 109 | 28 | | 2382 | Info | 30 | 396 |
| | | | | +------------+------------+-------+---------+
| 2381 | PS | 43 | 699 | | 2297 | Info (ISE) | 109 | 28 |
| | | | | +------------+------------+-------+---------+
| 2312 | Info | 20 | 365 | | 2381 | PS | 43 | 699 |
| | | | | +------------+------------+-------+---------+
| 2387 | PS | 10 | 122 | | 2312 | Info | 20 | 365 |
| | | | | +------------+------------+-------+---------+
| 2398 | Info | 15 | 396 | | 2387 | PS | 10 | 122 |
| | | | | +------------+------------+-------+---------+
| 2391 | PS | 10 | 122 | | 2398 | Info | 15 | 396 |
| | | | | +------------+------------+-------+---------+
| 2431 | PS | 10 | 457 | | 2391 | PS | 10 | 122 |
| | | | | +------------+------------+-------+---------+
| 2282 | Info | 14 | 215 | | 2431 | PS | 10 | 457 |
| | | | | +------------+------------+-------+---------+
| 2323 | ISE | 5 | unknown | | 2282 | Info | 14 | 215 |
| | | | | +------------+------------+-------+---------+
| 2448 | ISE | 7 | 92 | | 2323 | Info (ISE) | 5 | unknown |
+------------+--------+-------+---------+ +------------+------------+-------+---------+
| 2448 | Info (ISE) | 7 | 92 |
+------------+------------+-------+---------+
Table 3
We can compare the median delay, and the delays observed by the We can compare the median delay, and the delays observed by the
fastest and slowest quartiles in the three years: fastest and slowest quartiles in the three years:
+------+-------------+--------+-------------+ +======+=============+========+=============+
| Year | Fastest 25% | Median | Slowest 25% | | Year | Fastest 25% | Median | Slowest 25% |
+------+-------------+--------+-------------+ +======+=============+========+=============+
| 2018 | 604 | 1179 | 1522 | | 2018 | 604 | 1179 | 1522 |
| | | | | +------+-------------+--------+-------------+
| 2008 | 869 | 1081 | 1675 | | 2008 | 869 | 1081 | 1675 |
| | | | | +------+-------------+--------+-------------+
| 1998 | 169 | 365 | 442 | | 1998 | 169 | 365 | 442 |
+------+-------------+--------+-------------+ +------+-------------+--------+-------------+
The IETF takes three to four times more times to produce an RFC in Table 4
2018 than it did in 1998, but about the same time as it did in 2008.
We can get a rough estimate of how this translates in term of "level
of attention" per RFC by comparing the number of participants in the
IETF meetings of 2018, 2008 and 1998 [IETFCOUNT] to the number of RFC
published these years [RFCYEAR].
+------+------+---------+---------+------+----------+---------------+ The IETF takes three to four times more to produce an RFC in 2018
| Year | Nb | Spring | Summer | Fall | Average | Attendees/RFC | than it did in 1998, but about the same time as it did in 2008. We
| | RFC | P. | P. | | P. | | can get a rough estimate of how this translates in terms of "level of
+------+------+---------+---------+------+----------+---------------+ attention" per RFC by comparing the number of participants in the
| 2018 | 208 | 1235 | 1078 | 879 | 1064 | 5.1 | IETF meetings of 2018, 2008, and 1998 [IETFCOUNT] to the number of
| | | | | | | | RFCs published these years [RFCYEAR].
| 2008 | 290 | 1128 | 1181 | 962 | 1090 | 3.8 |
| | | | | | | | +======+=========+========+========+======+=========+============+
| 1998 | 234 | 1775 | 2106 | 1705 | 1862 | 9.0 | | Year | Number | Spring | Summer | Fall | Average | Attendees/ |
+------+------+---------+---------+------+----------+---------------+ | | of RFCs | P. | P. | P. | P. | RFC |
+======+=========+========+========+======+=========+============+
| 2018 | 208 | 1235 | 1078 | 879 | 1064 | 5.1 |
+------+---------+--------+--------+------+---------+------------+
| 2008 | 290 | 1128 | 1181 | 962 | 1090 | 3.8 |
+------+---------+--------+--------+------+---------+------------+
| 1998 | 234 | 1775 | 2106 | 1705 | 1862 | 8.0 |
+------+---------+--------+--------+------+---------+------------+
Table 5
The last column in the table provides the ratio of average number of The last column in the table provides the ratio of average number of
participants by number of RFC produced. If the IETF was a participants to the number of RFCs published. If the IETF were a
centralized organization, if all participants and documents were centralized organization, and if all participants and documents were
equivalent, this ratio would be the number of participants dedicated equivalent, this ratio would be the number of participants dedicated
to produce an RFC on a given year. This is of course a completely to produce an RFC on a given year. This is of course a completely
abstract figure because none of the hypotheses above is true, but it abstract figure because none of the hypotheses above are true, but it
still gives a vague indication of the "level of attention" applied to still gives a vague indication of the "level of attention" applied to
documents. We see that this ratio has increased from 2008 to 2018, documents. We see that this ratio has increased from 2008 to 2018,
as the number of participants was about the same for these two years as the number of participants was about the same for these two years
but the number of published RFCs decreased. However, that ratio was but the number of published RFCs decreased. However, this ratio was
much higher in 1998. The IETF had many more participants, and there much higher in 1998. The IETF had many more participants, and there
were probably many more eyes available to review any given draft. If were probably many more eyes available to review any given draft. If
we applied the ratios of 1998, the IETF would be producing 119 we applied the ratios of 1998, the IETF would be producing 119
documents in 2018 instead of 208. documents in 2018 instead of 208.
4.2. Working Group Processing Time 4.2. Working Group Processing Time
The largest part of the delays is spent in the Working Groups, before The largest part of the delays is spent in the working groups, before
the draft is submitted to the IESG for IETF review. As mentioned in the draft is submitted to the IESG for IETF review. As mentioned in
Section 2.1, the only intermediate milestone that we can extract from Section 2.1, the only intermediate milestone that we can extract from
the IETF datatracker is the date at which the document was adopted by the IETF Datatracker is the date at which the document was adopted by
the Working Group, or targeted for independent submission. The the working group, or targeted for independent submission. The
breakdown of the delays for the documents in our sample is: breakdown of the delays for the documents in our sample is:
+------+---------+------+----------------+----------------+ +======+============+======+================+================+
| RFC | Status | WG | Until adoption | After adoption | | RFC | Status | WG | Until adoption | After adoption |
+------+---------+------+----------------+----------------+ +======+============+======+================+================+
| 8411 | Info | 154 | 0 | 154 | | 8411 | Info | 154 | 0 | 154 |
| | | | | | +------+------------+------+----------------+----------------+
| 8456 | Info | 1033 | 209 | 824 | | 8456 | Info | 1033 | 209 | 824 |
| | | | | | +------+------------+------+----------------+----------------+
| 8446 | PS | 1400 | 0 | 1400 | | 8446 | PS | 1400 | 0 | 1400 |
| | | | | | +------+------------+------+----------------+----------------+
| 8355 | Info | 1175 | 102 | 1073 | | 8355 | Info | 1175 | 102 | 1073 |
| | | | | | +------+------------+------+----------------+----------------+
| 8441 | PS | 204 | 65 | 139 | | 8441 | PS | 204 | 65 | 139 |
| | | | | | +------+------------+------+----------------+----------------+
| 8324 | ISE | 38 | 0 | 38 | | 8324 | Info (ISE) | 38 | 0 | 38 |
| | | | | | +------+------------+------+----------------+----------------+
| 8377 | PS | 1630 | 728 | 902 | | 8377 | PS | 1630 | 728 | 902 |
| | | | | | +------+------------+------+----------------+----------------+
| 8498 | Info | 935 | 420 | 515 | | 8498 | Info | 935 | 420 | 515 |
| | | | | | +------+------------+------+----------------+----------------+
| 8479 | ISE | 233 | 0 | 233 | | 8479 | Info (ISE) | 233 | 0 | 233 |
| | | | | | +------+------------+------+----------------+----------------+
| 8453 | Info | 1036 | 396 | 640 | | 8453 | Info | 1036 | 396 | 640 |
| | | | | | +------+------------+------+----------------+----------------+
| 8429 | BCP | 76 | 0 | 76 | | 8429 | BCP | 76 | 0 | 76 |
| | | | | | +------+------------+------+----------------+----------------+
| 8312 | Info | 1113 | 280 | 833 | | 8312 | Info | 1113 | 280 | 833 |
| | | | | | +------+------------+------+----------------+----------------+
| 8492 | ISE | 1706 | 1428 | 278 | | 8492 | Info (ISE) | 1706 | 1428 | 278 |
| | | | | | +------+------------+------+----------------+----------------+
| 8378 | Exp | 1446 | 661 | 785 | | 8378 | Exp | 1446 | 661 | 785 |
| | | | | | +------+------------+------+----------------+----------------+
| 8361 | PS | 1477 | 399 | 1078 | | 8361 | PS | 1477 | 399 | 1078 |
| | | | | | +------+------------+------+----------------+----------------+
| 8472 | PS | 899 | 105 | 794 | | 8472 | PS | 899 | 105 | 794 |
| | | | | | +------+------------+------+----------------+----------------+
| 8471 | PS | 1127 | 153 | 794 | | 8471 | PS | 1127 | 153 | 794 |
| | | | | | +------+------------+------+----------------+----------------+
| 8466 | PS | 538 | 178 | 360 | | 8466 | PS | 538 | 178 | 360 |
| | | | | | +------+------------+------+----------------+----------------+
| 8362 | PS | 1766 | 240 | 1526 | | 8362 | PS | 1766 | 240 | 1526 |
| | | | | | +------+------------+------+----------------+----------------+
| 8468 | Info | 979 | 333 | 646 | | 8468 | Info | 979 | 333 | 646 |
| | | | | | +------+------------+------+----------------+----------------+
| | Average | 948 | 285 | 663 | | Average | 948 | 285 | 663 |
+------+---------+------+----------------+----------------+ +-------------------+------+----------------+----------------+
The time before Working Group adoption average to a bit more than 9 Table 6
months, compared to 1 years and almost 10 months for processing time
The time before working group adoption averages to a bit more than 9
months, compared to 1 year and almost 10 months for processing time
after adoption. We see that RFC 8492 stands out, with long delays after adoption. We see that RFC 8492 stands out, with long delays
spent attempting publication through a Working Group before spent attempting publication through a working group before
submission to the Independent Stream Editor. If we removed RFC 8492 submission to the Independent Submissions Editor. If we remove RFC
from the list, the average time until adoption drops to just over 7 8492 from the list, the average time until adoption drops to just
months, and becomes just 25% of the total processing time in the WG. over 7 months, and becomes just 25% of the total processing time in
the WG.
There are a few documents that started immediately as Working Group There are a few documents that started immediately as working group
efforts, or were immediately targeted for publication in the efforts, or were immediately targeted for publication in the
Independent Stream. Those documents tend to see short processing Independent Stream. Those documents tend to see short processing
times, with the exception of RFC 8446 on which the TLS Working Group times, with the exception of RFC 8446 on which the TLS Working Group
spent a long time working. spent a long time working.
4.3. Preparation and Publication Delays 4.3. Preparation and Publication Delays
The preparation and publication delays include three components: The preparation and publication delays include three components:
o the delay from submission to the RFC Editor to beginning of AUTH- * the delay from submission to the RFC Editor to beginning of
48, during which the document is prepared; AUTH48, during which the document is prepared (referred to as "RFC
edit" below);
o the AUTH-48 delay, during which authors review and eventually * the AUTH48 delay, during which authors review and eventually
approve the changes proposed by the editors; approve the changes proposed by the editors (referred to as
"AUTH48" below);
o the publication delay, from final agreement by authors and editors * the publication delay, from final agreement by authors and editors
to actual publication. to actual publication (referred to as "RFC Pub" below).
The breakdown of the publication delays for each RFC is shown in the The breakdown of the publication delays for each RFC is shown in the
following table. following table.
+-------+---------+-------+--------+---------+--------+-------------+ +=======+========+=======+==========+========+=========+=========+
| RFC | Status | Pages | RFC | AUTH-48 | RFC | Edit(total) | | RFC | Status | Pages | RFC edit | AUTH48 | RFC Pub | Edit |
| | | | edit | | Pub | | | | | | | | | (total) |
+-------+---------+-------+--------+---------+--------+-------------+ +=======+========+=======+==========+========+=========+=========+
| 8411 | Info | 5 | 53 | 88 | 20 | 161 | | 8411 | Info | 5 | 53 | 88 | 20 | 161 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8456 | Info | 64 | 98 | 46 | 14 | 158 | | 8456 | Info | 64 | 98 | 46 | 14 | 158 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8446 | PS | 160 | 85 | 57 | 0 | 142 | | 8446 | PS | 160 | 85 | 57 | 0 | 142 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8355 | Info | 13 | 83 | 15 | 1 | 99 | | 8355 | Info | 13 | 83 | 15 | 1 | 99 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8441 | PS | 8 | 67 | 33 | 6 | 106 | | 8441 | PS | 8 | 56 | 33 | 3 | 92 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8324 | ISE | 29 | 42 | 28 | 1 | 71 | | 8324 | Info | 29 | 42 | 28 | 1 | 71 |
| | | | | | | | | | (ISE) | | | | | |
| 8377 | PS | 8 | 39 | 102 | 0 | 141 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8377 | PS | 8 | 39 | 102 | 0 | 141 |
| 8498 | Info | 15 | 49 | 16 | 2 | 67 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8498 | Info | 15 | 48 | 16 | 1 | 65 |
| 8479 | ISE | 8 | 31 | 5 | 1 | 37 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8479 | Info | 8 | 31 | 5 | 1 | 37 |
| 8453 | Info | 42 | 73 | 7 | 0 | 80 | | | (ISE) | | | | | |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8429 | BCP | 10 | 60 | 99 | 0 | 159 | | 8453 | Info | 42 | 73 | 7 | 3 | 83 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8312 | Info | 18 | 96 | 30 | 0 | 126 | | 8429 | BCP | 10 | 60 | 99 | 0 | 159 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8492 | ISE | 40 | 355 | 123 | 2 | 480 | | 8312 | Info | 18 | 55 | 28 | 2 | 85 |
| | | | | | | | +-------+--------+-------+----------+--------+---------+---------+
| 8378 | Exp | 21 | 42 | 9 | 0 | 51 | | 8492 | Info | 40 | 355 | 123 | 2 | 480 |
| | | | | | | | | | (ISE) | | | | | |
| 8361 | PS | 17 | 39 | 31 | 3 | 73 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8378 | Exp | 21 | 42 | 9 | 0 | 51 |
| 8472 | PS | 8 | 59 | 8 | 13 | 80 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8361 | PS | 17 | 39 | 31 | 3 | 73 |
| 8471 | PS | 18 | 59 | 8 | 13 | 80 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8472 | PS | 8 | 59 | 8 | 13 | 80 |
| 8466 | PS | 158 | 84 | 22 | 3 | 109 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8471 | PS | 18 | 59 | 8 | 13 | 80 |
| 8362 | PS | 33 | 49 | 11 | 4 | 64 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8466 | PS | 158 | 84 | 22 | 3 | 109 |
| 8468 | Info | 15 | 65 | 53 | 9 | 127 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8362 | PS | 33 | 49 | 11 | 4 | 64 |
| | Average | | 76 | 40 | 5 | 121 | +-------+--------+-------+----------+--------+---------+---------+
| | | | | | | | | 8468 | Info | 15 | 65 | 53 | 9 | 127 |
| -8492 | Average | | 62 | 35 | 5 | 102 | +-------+--------+-------+----------+--------+---------+---------+
+-------+---------+-------+--------+---------+--------+-------------+ | Average | | 74 | 39 | 5 | 118 |
+----------------+-------+----------+--------+---------+---------+
| Average | | 59 | 35 | 5 | 99 |
| (without 8492) | | | | | |
+----------------+-------+----------+--------+---------+---------+
Table 7
On average, the total delay appears to be about four months, but the On average, the total delay appears to be about four months, but the
average is skewed by the extreme values encountered for [RFC8492]. average is skewed by the extreme values encountered for [RFC8492].
If we exclude that RFC from the computations, the average delay drops If we exclude that RFC from the computations, the average delay drops
to a just a bit more than 3 months: about 2 months for the to a just a bit more than 3 months: about 2 months for the
preparation, a bit more than one month for the AUTH-48 phase, and 5 preparation, a bit more than one month for the AUTH48 phase, and 5
days for the publishing. days for the publishing.
Of course, these delays vary from RFC to RFC. To try explain the Of course, these delays vary from RFC to RFC. To try explain the
causes of the delay, we compute the correlation factor between the causes of the delay, we compute the correlation factor between the
observed delays and several plausible explanation factors: observed delays and several plausible explanation factors:
o The number of pages in the document, * the number of pages in the document,
o The amount of copy edit, as discussed in Section 4.4, * the amount of copy editing, as discussed in Section 4.4,
o Whether or not an IANA action was required, * whether or not IANA actions were required,
o The number of authors, * the number of authors,
o The number of drafts revisions, * the number of draft revisions,
o The Working Group delay. * the working group delay.
We find the following values: We find the following values:
+-------------+----------+---------+-------------+ +===================+==========+========+=============+
| Correlation | RFC edit | AUTH-48 | Edit(total) | | Correlation | RFC edit | AUTH48 | Edit(total) |
+-------------+----------+---------+-------------+ +===================+==========+========+=============+
| Nb pages | 0.50 | -0.04 | 0.21 | | Number of pages | 0.50 | -0.04 | 0.21 |
| | | | | +-------------------+----------+--------+-------------+
| Copy-Edit | 0.42 | 0.24 | 0.45 | | Copy-Edit | 0.42 | 0.24 | 0.45 |
| | | | | +-------------------+----------+--------+-------------+
| IANA | -0.14 | -0.21 | 0.12 | | IANA | -0.14 | -0.21 | 0.12 |
| | | | | +-------------------+----------+--------+-------------+
| Nb Authors | 0.39 | -0.07 | 0.18 | | Number of Authors | 0.39 | -0.07 | 0.18 |
| | | | | +-------------------+----------+--------+-------------+
| Nb drafts | 0.18 | -0.33 | -0.19 | | Number of drafts | 0.18 | -0.33 | -0.19 |
| | | | | +-------------------+----------+--------+-------------+
| WG delay | 0.03 | -0.16 | -0.15 | | WG delay | 0.03 | -0.16 | -0.15 |
+-------------+----------+---------+-------------+ +-------------------+----------+--------+-------------+
Table 8
We see some plausible explanations for the production delay. It will We see some plausible explanations for the production delay. It will
be somewhat longer for longer documents, or for documents that be somewhat longer for longer documents or for documents that require
require a lot of copy editing (see Section 4.4). Somewhat a lot of copy editing (see Section 4.4). Somewhat surprisingly, it
surprisingly, it also tend to increase with the number of authors. also tends to increase with the number of authors. It does not
It does not appear significantly correlated with the presence or appear significantly correlated with the presence or absence of IANA
absence of IANA action. action.
The analysis of RFC 8324 in Section 3.6 explains its short editing The analysis of RFC 8324 in Section 3.6 explains its short editing
delays by the experience of the author. This makes sense: if a delays by the experience of the author. This makes sense: if a
document needs less editing, the editing delays would be shorter. document needs less editing, the editing delays would be shorter.
This is partially confirmed by the relation between the amount of This is partially confirmed by the relation between the amount of
copy editing and the publication delay. copy editing and the publication delay.
We see fewer plausible explanations for the AUTH48 delays. These We see fewer plausible explanations for the AUTH48 delays. These
delays vary much more than the preparation delay, with a standard delays vary much more than the preparation delay, with a standard
deviation of 20 days for AUTH-48 versus 10 days for the preparation deviation of 20 days for AUTH48 versus 10 days for the preparation
delay. In theory, AUTH-48 is just a final verification: the authors delay. In theory, AUTH48 is just a final verification: the authors
receive the document prepared by the RFC production center, and just receive the document prepared by the RFC production center, and just
have to give their approval, or maybe request a last minute have to give their approval, or maybe request a last minute
correction. The name indicates that this is expected to last just correction. The name indicates that this is expected to last just
two days, but in average it lasts more than a month. two days, but in average it lasts more than a month.
We often hypothesize that the number of authors influences the We often hypothesize that the number of authors influences the AUTH48
AUTH-48 delay, or that authors who have spent a long time working on delay, or that authors who have spent a long time working on the
the document in the Working Group somehow get demotivated and spend document in the working group somehow get demotivated and spend even
even longer to answer questions during AUTH-48. This may happen longer to answer questions during AUTH48. This may happen sometimes,
sometimes, but our statistics don't show that - if anything, the but our statistics don't show that - if anything, the numerical
numerical results point in the opposite direction. results point in the opposite direction.
After asking the authors of the RFCs in the sample why the AUTH-48 After asking the authors of the RFCs in the sample why the AUTH48
phase took a long time, we got three explanations: phase took a long time, we got three explanations:
1- Some RFCs have multiple authors in multiple time zones. This 1. Some RFCs have multiple authors in multiple time zones. This
slows down the coordination required for approving changes. slows down the coordination required for approving changes.
2- Some authors found some of the proposed changes unnecessary or 2. Some authors found some of the proposed changes unnecessary or
undesirable, and asked that they be reversed. This required long undesirable, and asked that they be reversed. This required long
exchanges between authors and editors. exchanges between authors and editors.
3- Some authors were not giving high priority to AUTH-48 responses. 3. Some authors were not giving high priority to AUTH48 responses.
As mentioned above, we were not able to verify these hypotheses by As mentioned above, we were not able to verify these hypotheses by
looking at the data. The author's experience with this document looking at the data. The author's experience with this document
suggests another potential delay for the Independent Stream RFC: suggests another potential delay for the Independent Stream RFC:
processing delay by the Independent Stream Editor, discussed in processing delay by the Independent Submissions Editor, discussed in
Section 4.5. Section 4.5.
4.4. Copy Editing 4.4. Copy Editing
We can assess the amount of copy editing applied to each published We can assess the amount of copy editing applied to each published
RFC by comparing the text of the draft approved for publication and RFC by comparing the text of the draft approved for publication and
the text of the RFC. We do expect differences in the "boilerplate" the text of the RFC. We do expect differences in the "boilerplate"
and in the IANA section, but we will also see differences due to copy and in the IANA section, but we will also see differences due to copy
editing. Assessing the amount of copy editing is subjective, and we editing. Assessing the amount of copy editing is subjective, and we
do it using a scale of 1 to 4: do it using a scale of 1 to 4:
1- Minor editing 1: Minor editing
2- Editing for style, such as capitalization, hyphens, that versus 2: Editing for style, such as capitalization, hyphens, "that" versus
which, and expending all acronyms at least one. "which", and expanding all acronyms at least once.
3- Editing for clarity in addition to style, such as rewriting 3: Editing for clarity in addition to style, such as rewriting
ambiguous sentences and clarifying use of internal references. For ambiguous sentences and clarifying use of internal references.
Yang models, that may include model corrections suggested by the For YANG models, that may include model corrections suggested by
verifier. the verifier.
4- Extensive editing. 4: Extensive editing.
The following table shows that about half of the RFCs required The following table shows that about half of the RFCs required
editing for style, and the other half at least some editing for editing for style, and the other half at least some editing for
clarity. clarity.
+------+--------+-----------+ +======+============+===========+
| RFC | Status | Copy Edit | | RFC | Status | Copy Edit |
+------+--------+-----------+ +======+============+===========+
| 8411 | Info | 2 | | 8411 | Info | 2 |
| | | | +------+------------+-----------+
| 8456 | Info | 4 | | 8456 | Info | 4 |
| | | | +------+------------+-----------+
| 8446 | PS | 3 | | 8446 | PS | 3 |
| | | | +------+------------+-----------+
| 8355 | Info | 2 | | 8355 | Info | 2 |
| | | | +------+------------+-----------+
| 8441 | PS | 2 | | 8441 | PS | 2 |
| | | | +------+------------+-----------+
| 8324 | ISE | 2 | | 8324 | Info (ISE) | 2 |
| | | | +------+------------+-----------+
| 8377 | PS | 3 | | 8377 | PS | 3 |
| | | | +------+------------+-----------+
| 8498 | Info | 3 | | 8498 | Info | 3 |
| | | | +------+------------+-----------+
| 8479 | ISE | 1 | | 8479 | Info (ISE) | 1 |
| | | | +------+------------+-----------+
| 8453 | Info | 2 | | 8453 | Info | 2 |
| | | | +------+------------+-----------+
| 8429 | BCP | 2 | | 8429 | BCP | 2 |
| | | | +------+------------+-----------+
| 8312 | Info | 2 | | 8312 | Info | 2 |
| | | | +------+------------+-----------+
| 8492 | ISE | 3 | | 8492 | Info (ISE) | 3 |
| | | | +------+------------+-----------+
| 8378 | Exp | 2 | | 8378 | Exp | 2 |
| | | | +------+------------+-----------+
| 8361 | PS | 2 | | 8361 | PS | 2 |
| | | | +------+------------+-----------+
| 8472 | PS | 2 | | 8472 | PS | 2 |
| | | | +------+------------+-----------+
| 8471 | PS | 2 | | 8471 | PS | 2 |
| | | | +------+------------+-----------+
| 8466 | PS | 3 | | 8466 | PS | 3 |
| | | | +------+------------+-----------+
| 8362 | PS | 3 | | 8362 | PS | 3 |
| | | | +------+------------+-----------+
| 8468 | Info | 3 | | 8468 | Info | 3 |
+------+--------+-----------+ +------+------------+-----------+
Table 9
This method of assessment does not take into account the number of This method of assessment does not take into account the number of
changes proposed by the editors and eventually rejected by the changes proposed by the editors and eventually rejected by the
authors, since these changes are not present in either the final authors, since these changes are not present in either the final
draft or the published RFC. It might be possible to get an draft or the published RFC. It might be possible to get an
evaluation of these "phantom changes" from the RFC Production Center. evaluation of these "phantom changes" from the RFC Production Center.
4.5. Independent Stream 4.5. Independent Stream
Out of 20 randomly selected RFCs, 3 were published through the Out of 20 randomly selected RFCs, 3 were published through the
Independent Stream. One is an independent opinion, another a Independent Stream. One is an independent opinion, another a
description of a non-IETF protocol format, and the third was description of a non-IETF protocol format, and the third was
[RFC8492], which is a special case. Apart from this special case, [RFC8492], which is a special case. Apart from this special case,
the publication delays were significantly shorter for the Independent the publication delays were significantly shorter for the Independent
Stream than for the IETF stream. Stream than for the IETF Stream.
The authors of these 3 RFCs are regular IETF contributors. This The authors of these 3 RFCs are regular IETF contributors. This
observation motivated a secondary analysis of all the RFCs published observation motivated a secondary analysis of all the RFCs published
in the Independent Stream in 2018. There are 14 such RFCs: 8507, in the Independent Stream in 2018. There are 14 such RFCs: 8507,
8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351,
8328 and 8324. (RFC 8367 and 8369 were published on 1 April 2018.) 8328, and 8324. (RFCs 8367 and 8369 were published on 1 April 2018.)
The majority of the documents were published by regular IETF The majority of the documents were published by regular IETF
participants, but two of them were not. One describes "The BagIt participants, but two of them were not. One describes "The BagIt
File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS
Testbed" [RFC8483]. They document a data format and a system Testbed" [RFC8483]. They document a data format and a system
developed outside the IETF, and illustrate the outreach function of developed outside the IETF and illustrate the outreach function of
the Independent Stream. In both cases, the authors include one the Independent Stream. In both cases, the authors include one
experienced IETF participant, who presumably helped outsiders experienced IETF participant, who presumably helped outsiders
navigate the publication process. navigate the publication process.
Th present document experienced some publication delays due to the Th present document experienced some publication delays due to the
Independent Stream Editor. The ISE is a bottleneck and is a Independent Submissions Editor. The ISE is a bottleneck and is a
volunteer resource. Although the ISE as a lone person operating as a volunteer resource. Although the ISE as a lone person operating as a
volunteer is still roughly adequate resource for the job, the volunteer is still roughly adequate resource for the job, the
delivery will necessarily be best effort with delays caused by spikes delivery will necessarily be best effort with delays caused by spikes
in ISE load, work commitments, and other life events. These delays in ISE load, work commitments, and other life events. These delays
may not be fundamentally critical to RFC delivery, but they are may not be fundamentally critical to RFC delivery, but they are
capable of introducing a significant percentage delay into what might capable of introducing a significant percentage delay into what might
otherwise be a smooth process. otherwise be a smooth process.
5. Citation Counts 5. Citation Counts
In this exploration, we want to assess whether citation counts In this exploration, we want to examine whether citation counts
provide a meaningful assessment of the popularity of RFCs. We obtain provide a meaningful assessment of the popularity of RFCs. We obtain
the citation counts through the Semantic Scholar API, using queries the citation counts through the Semantic Scholar API, using queries
of the form: of the form: <https://api.semanticscholar.org/v1/paper/10.17487/
rfc8446?include_unknown_references=true>
http://api.semanticscholar.org/
v1/paper/10.17487/rfc8446?include_unknown_references=true
In these queries, the RFC is uniquely identified by its DOI In these queries, the RFC is uniquely identified by its DOI
reference, which is composed of the RFC Series prefix 10.17487 and reference, which is composed of the RFC Series prefix 10.17487 and
the RFC identifier. The queries return a series of properties, the RFC identifier. The queries return a series of properties,
including a list of citations for the RFC. Based on that list of including a list of citations for the RFC. Based on that list of
citations, we compute three numbers: citations, we compute three numbers:
o The total number of citations * The total number of citations
o The number of citations in the year of publication and the year * The number of citations in the year of publication and the year
after that after that
o For the RFC published in 1998 or 2008 that we use for comparison, * For the RFC published in 1998 or 2008 that we use for comparison,
the number of citations in the years 2018 and 2019. the number of citations in the years 2018 and 2019.
All the numbers were retrieved on October 6, 2019. All the numbers were retrieved on October 6, 2019.
5.1. Citation Numbers 5.1. Citation Numbers
As measured on October 6, 2019, the citation counts for the RFC in As measured on October 6, 2019, the citation counts for the RFC in
our sample set were: our sample set were:
+-----------+--------+-------+-----------+ +============+============+=======+===========+
| RFC(2018) | Status | Total | 2018-2019 | | RFC (2018) | Status | Total | 2018-2019 |
+-----------+--------+-------+-----------+ +============+============+=======+===========+
| 8411 | Info | 1 | 0 | | 8411 | Info | 1 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8456 | Info | 1 | 1 | | 8456 | Info | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8446 | PS | 418 | 204 | | 8446 | PS | 418 | 204 |
| | | | | +------------+------------+-------+-----------+
| 8355 | Info | 3 | 3 | | 8355 | Info | 3 | 3 |
| | | | | +------------+------------+-------+-----------+
| 8441 | PS | 1 | 1 | | 8441 | PS | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8324 | ISE | 0 | 0 | | 8324 | Info (ISE) | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8377 | PS | 0 | 0 | | 8377 | PS | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8498 | Info | 0 | 0 | | 8498 | Info | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8479 | ISE | 0 | 0 | | 8479 | Info (ISE) | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8453 | Info | 3 | 3 | | 8453 | Info | 3 | 3 |
| | | | | +------------+------------+-------+-----------+
| 8429 | BCP | 0 | 0 | | 8429 | BCP | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8312 | Info | 25 | 16 | | 8312 | Info | 25 | 16 |
| | | | | +------------+------------+-------+-----------+
| 8492 | ISE | 4 | 4 | | 8492 | Info (ISE) | 4 | 4 |
| | | | | +------------+------------+-------+-----------+
| 8378 | Exp | 1 | 1 | | 8378 | Exp | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8361 | PS | 0 | 0 | | 8361 | PS | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8472 | PS | 1 | 1 | | 8472 | PS | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8471 | PS | 1 | 1 | | 8471 | PS | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8466 | PS | 0 | 0 | | 8466 | PS | 0 | 0 |
| | | | | +------------+------------+-------+-----------+
| 8362 | PS | 1 | 1 | | 8362 | PS | 1 | 1 |
| | | | | +------------+------------+-------+-----------+
| 8468 | Info | 1 | 1 | | 8468 | Info | 1 | 1 |
+-----------+--------+-------+-----------+ +------------+------------+-------+-----------+
Table 10
The results indicate that [RFC8446] is by far the most cited of the The results indicate that [RFC8446] is by far the most cited of the
20 RFC in our sample. This is not surprising, since TLS is a key 20 RFC in our sample. This is not surprising, since TLS is a key
Internet Protocol. The TLS 1.3 protocol was also the subject of Internet Protocol. The TLS 1.3 protocol was also the subject of
extensive studies by researchers, and thus was mentioned in a number extensive studies by researchers, and thus was mentioned in a number
of published papers. Surprisingly, the Semantic Scholar mentions a of published papers. Surprisingly, the Semantic Scholar mentions a
number of citations that predate the publication date. These are number of citations that predate the publication date. These are
probably citations of the various draft versions of the protocol. probably citations of the various draft versions of the protocol.
The next most cited RFC in the sample is [RFC8312] which describes The next most cited RFC in the sample is [RFC8312] which describes
the Cubic congestion control algorithm for TCP. That protocol was the Cubic congestion control algorithm for TCP. That protocol was
also the target of a large number of academic publications.The other also the target of a large number of academic publications. The
RFC in the sample only have a small number of citations. other RFCs in the sample only have a small number of citations.
There is probably a small bias when measuring citations at a fixed There is probably a small bias when measuring citations at a fixed
date. An RFC published in January 2018 would have more time to date. An RFC published in January 2018 would have more time to
accrue citations than one published in December. That may be true to accrue citations than one published in December. That may be true to
some extent, as the second most cited RFC in the set was published in some extent, as the second most cited RFC in the set was published in
January. However, the effect has to be limited. The most cited RFC January. However, the effect has to be limited. The most cited RFC
was published in August, and the second most cited was published in was published in August, and the second most cited was published in
2019. (That RFC got an RFC number in 2018, but publication was 2019. (That RFC got an RFC number in 2018, but publication was
slowed by long AUTH-48 delays.) slowed by long AUTH48 delays.)
5.2. Comparison to 1998 and 2008 5.2. Comparison to 1998 and 2008
In order to get a baseline, we can look at the number of references In order to get a baseline, we can look at the number of references
for the RFCs published in 2008 and 1998. However, we need totake for the RFCs published in 2008 and 1998. However, we need to take
time into account. Documents published a long time ago are expected time into account. Documents published a long time ago are expected
to have accrued more references. We try to address this by looking to have accrued more references. We try to address this by looking
at three counts for each document: the overall number of references at three counts for each document: the overall number of references
over the document's lifetime, the number of references obtained in over the document's lifetime, the number of references obtained in
the year following publication, and the number of references observed the year following publication, and the number of references observed
since 2018: since 2018:
+-----------+--------+-------+-----------+-----------+ +===========+============+=======+===========+===========+
| RFC(2008) | Status | Total | 2008-2009 | 2018-2019 | | RFC(2008) | Status | Total | 2008-2009 | 2018-2019 |
+-----------+--------+-------+-----------+-----------+ +===========+============+=======+===========+===========+
| 5326 | Exp | 138 | 14 | 15 | | 5326 | Exp | 138 | 14 | 15 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5348 | PS | 14 | 3 | 0 | | 5348 | PS | 14 | 3 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5281 | Info | 69 | 15 | 7 | | 5281 | Info | 69 | 15 | 7 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5354 | Exp | 17 | 13 | 0 | | 5354 | Exp | 17 | 13 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5227 | PS | 19 | 1 | 2 | | 5227 | PS | 19 | 1 | 2 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5329 | PS | 24 | 6 | 1 | | 5329 | PS | 24 | 6 | 1 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5277 | PS | 32 | 3 | 2 | | 5277 | PS | 32 | 3 | 2 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5236 | ISE | 25 | 5 | 4 | | 5236 | Info (ISE) | 25 | 5 | 4 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5358 | BCP | 21 | 2 | 0 | | 5358 | BCP | 21 | 2 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5271 | Info | 7 | 2 | 0 | | 5271 | Info | 7 | 2 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5195 | PS | 7 | 4 | 2 | | 5195 | PS | 7 | 4 | 2 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5283 | PS | 8 | 1 | 0 | | 5283 | PS | 8 | 1 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5186 | Info | 14 | 4 | 2 | | 5186 | Info | 14 | 4 | 2 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5142 | PS | 8 | 4 | 0 | | 5142 | PS | 8 | 4 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5373 | PS | 5 | 2 | 0 | | 5373 | PS | 5 | 2 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5404 | PS | 1 | 1 | 0 | | 5404 | PS | 1 | 1 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5172 | PS | 2 | 0 | 0 | | 5172 | PS | 2 | 0 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5349 | Info | 8 | 0 | 2 | | 5349 | Info | 8 | 0 | 2 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5301 | PS | 5 | 1 | 0 | | 5301 | PS | 5 | 1 | 0 |
| | | | | | +-----------+------------+-------+-----------+-----------+
| 5174 | Info | 0 | 0 | 0 | | 5174 | Info | 0 | 0 | 0 |
+-----------+--------+-------+-----------+-----------+ +-----------+------------+-------+-----------+-----------+
+-----------+--------+-------+-----------+-----------+
| RFC(1998) | Status | Total | 1998-1999 | 2018-2019 | Table 11
+-----------+--------+-------+-----------+-----------+
| 2289 | PS | 2 | 0 | 1 | +============+============+=======+===========+===========+
| | | | | | | RFC (1998) | Status | Total | 1998-1999 | 2018-2019 |
| 2267 | Info | 982 | 5 | 61 | +============+============+=======+===========+===========+
| | | | | | | 2289 | PS | 2 | 0 | 1 |
| 2317 | BCP | 9 | 1 | 2 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2267 | Info | 982 | 5 | 61 |
| 2404 | PS | 137 | 6 | 1 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2317 | BCP | 9 | 1 | 2 |
| 2374 | PS | 42 | 4 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2404 | PS | 137 | 6 | 1 |
| 2449 | PS | 7 | 2 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2374 | PS | 42 | 4 | 0 |
| 2283 | PS | 17 | 3 | 2 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2449 | PS | 7 | 2 | 0 |
| 2394 | Info | 13 | 2 | 1 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2283 | PS | 17 | 3 | 2 |
| 2348 | DS | 5 | 0 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2394 | Info | 13 | 2 | 1 |
| 2382 | Info | 17 | 12 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2348 | DS | 5 | 0 | 0 |
| 2297 | ISE | 36 | 11 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2382 | Info | 17 | 12 | 0 |
| 2381 | PS | 39 | 12 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2297 | Info (ISE) | 36 | 11 | 0 |
| 2312 | Info | 14 | 3 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2381 | PS | 39 | 12 | 0 |
| 2387 | PS | 4 | 1 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2312 | Info | 14 | 3 | 0 |
| 2398 | Info | 17 | 0 | 1 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2387 | PS | 4 | 1 | 0 |
| 2391 | PS | 31 | 3 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2398 | Info | 17 | 0 | 1 |
| 2431 | PS | 3 | 0 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2391 | PS | 31 | 3 | 0 |
| 2282 | Info | 8 | 0 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2431 | PS | 3 | 0 | 0 |
| 2323 | ISE | 1 | 0 | 0 | +------------+------------+-------+-----------+-----------+
| | | | | | | 2282 | Info | 8 | 0 | 0 |
| 2448 | ISE | 0 | 0 | 0 | +------------+------------+-------+-----------+-----------+
+-----------+--------+-------+-----------+-----------+ | 2323 | Info (ISE) | 1 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 2448 | Info (ISE) | 0 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
Table 12
We can compare the median number of citations and the numbers of We can compare the median number of citations and the numbers of
citations for the least and most popular quartiles in the three citations for the least and most popular quartiles in the three
years: years:
+----------------------------+-----------+--------+------------+ +============================+===========+========+============+
| References | Lower 25% | Median | Higher 25% | | References | Lower 25% | Median | Higher 25% |
+----------------------------+-----------+--------+------------+ +============================+===========+========+============+
| RFC (2018) | 0 | 1 | 3 | | RFC (2018) | 0 | 1 | 3 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (2008) | 6.5 | 11 | 21.75 | | RFC (2008) | 6.5 | 11 | 21.75 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (2008), until 2009 | 1 | 2.5 | 4.5 | | RFC (2008), until 2009 | 1 | 2.5 | 4.5 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (2008), 2018 and after | 0 | 0 | 2 | | RFC (2008), 2018 and after | 0 | 0 | 2 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (1998) | 4.75 | 13.5 | 32.25 | | RFC (1998) | 4.75 | 13.5 | 32.25 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (1998), until 1999 | 0 | 2 | 4.25 | | RFC (1998), until 1999 | 0 | 2 | 4.25 |
| | | | | +----------------------------+-----------+--------+------------+
| RFC (1998), 2018 and after | 0 | 0 | 1 | | RFC (1998), 2018 and after | 0 | 0 | 1 |
+----------------------------+-----------+--------+------------+ +----------------------------+-----------+--------+------------+
Table 13
The total numbers show new documents with fewer citations than the The total numbers show new documents with fewer citations than the
older ones. This can be explained to some degree by the passage of older ones. This can be explained to some degree by the passage of
time. If we restrict the analysis to the number of citations accrued time. If we restrict the analysis to the number of citations accrued
in the year of publishing and the year after that, we still see about in the year of publishing and the year after that, we still see about
the same distribution for the three samples. the same distribution for the three samples.
We also see that the number of references to RFC fades over time. We also see that the number of references to RFCs fades over time.
Only the most popular of the RFC produced in 1998 are still cited in Only the most popular of the RFC produced in 1998 are still cited in
2019. 2019.
5.3. Citations Versus Deployments 5.3. Citations versus Deployments
The following table shows side by side the number of citations as The following table shows side by side the number of citations as
measured in Section 5.1 and the estimation of deployment as indicated measured in Section 5.1 and the estimation of deployment as indicated
in Section 3. in Section 3.
+-----------+--------+-----------+------------+ +============+============+===========+============+
| RFC(2018) | Status | Citations | Deployment | | RFC (2018) | Status | Citations | Deployment |
+-----------+--------+-----------+------------+ +============+============+===========+============+
| 8411 | Info | 1 | medium | | 8411 | Info | 1 | medium |
| | | | | +------------+------------+-----------+------------+
| 8456 | Info | 1 | medium | | 8456 | Info | 1 | medium |
| | | | | +------------+------------+-----------+------------+
| 8446 | PS | 418 | high | | 8446 | PS | 418 | high |
| | | | | +------------+------------+-----------+------------+
| 8355 | Info | 3 | medium | | 8355 | Info | 3 | medium |
| | | | | +------------+------------+-----------+------------+
| 8441 | PS | 1 | high | | 8441 | PS | 1 | high |
| | | | | +------------+------------+-----------+------------+
| 8324 | ISE | 0 | N/A | | 8324 | Info (ISE) | 0 | N/A |
| | | | | +------------+------------+-----------+------------+
| 8377 | PS | 0 | unknown | | 8377 | PS | 0 | unknown |
| | | | | +------------+------------+-----------+------------+
| 8498 | Info | 0 | unknown | | 8498 | Info | 0 | unknown |
| | | | | +------------+------------+-----------+------------+
| 8479 | ISE | 0 | one | | 8479 | Info (ISE) | 0 | one |
| | | | | +------------+------------+-----------+------------+
| 8453 | Info | 3 | unknown | | 8453 | Info | 3 | unknown |
| | | | | +------------+------------+-----------+------------+
| 8429 | BCP | 0 | some | | 8429 | BCP | 0 | some |
| | | | | +------------+------------+-----------+------------+
| 8312 | Info | 25 | high | | 8312 | Info | 25 | high |
| | | | | +------------+------------+-----------+------------+
| 8492 | ISE | 4 | one | | 8492 | Info (ISE) | 4 | one |
| | | | | +------------+------------+-----------+------------+
| 8378 | Exp | 1 | some | | 8378 | Exp | 1 | some |
| | | | | +------------+------------+-----------+------------+
| 8361 | PS | 0 | one | | 8361 | PS | 0 | one |
| | | | | +------------+------------+-----------+------------+
| 8472 | PS | 1 | medium | | 8472 | PS | 1 | medium |
| | | | | +------------+------------+-----------+------------+
| 8471 | PS | 1 | medium | | 8471 | PS | 1 | medium |
| | | | | +------------+------------+-----------+------------+
| 8466 | PS | 0 | unknown | | 8466 | PS | 0 | unknown |
| | | | | +------------+------------+-----------+------------+
| 8362 | PS | 1 | medium | | 8362 | PS | 1 | medium |
| | | | | +------------+------------+-----------+------------+
| 8468 | Info | 1 | some | | 8468 | Info | 1 | some |
+-----------+--------+-----------+------------+ +------------+------------+-----------+------------+
Table 14
From looking at these results, it is fairly obvious that citation From looking at these results, it is fairly obvious that citation
counts cannot be used as proxies for the "value" of an RFC. In our counts cannot be used as proxies for the "value" of an RFC. In our
sample, the two RFCs that have high citation counts were both widely sample, the two RFCs that have high citation counts were both widely
deployed, and can certainly be described as successful, but we also deployed, and can certainly be described as successful, but we also
see many RFCs that saw significant deployment without garnering a see many RFCs that saw significant deployment without garnering a
high level of citations. high level of citations.
Citation counts are driven by academic interest, but are only loosely Citation counts are driven by academic interest, but are only loosely
correlated with actual deployment. We saw that [RFC8446] was widely correlated with actual deployment. We saw that [RFC8446] was widely
skipping to change at page 42, line 24 skipping to change at line 1807
extensions to an experimental delay tolerant transport protocol. extensions to an experimental delay tolerant transport protocol.
This protocol does not carry a significant proportion of Internet This protocol does not carry a significant proportion of Internet
traffic, but has been the object of a fair number of academic traffic, but has been the object of a fair number of academic
studies. studies.
The citation process tends to privilege the first expression of a The citation process tends to privilege the first expression of a
concept. We see that with the most cited RFC in the 1998 set is concept. We see that with the most cited RFC in the 1998 set is
[RFC2267], an informational RFC defining Network Ingress Filtering [RFC2267], an informational RFC defining Network Ingress Filtering
that was obsoleted in May 2000 by [RFC2827]. It is still cited that was obsoleted in May 2000 by [RFC2827]. It is still cited
frequently in 2018 and 2019, regardless of its formal status in the frequently in 2018 and 2019, regardless of its formal status in the
RFC series. We see the same effect at work with [RFC8441], which RFC Series. We see the same effect at work with [RFC8441], which
garners very few citations although it obsoletes [RFC6455] that has a garners very few citations although it updates [RFC6455] that has a
large number of citations. The same goes for [RFC8468], which is large number of citations. The same goes for [RFC8468], which is
sparsely cited while the [RFC2330] is widely cited. Just counting sparsely cited while the [RFC2330] is widely cited. Just counting
citations will not indicate whether developers still use an old citations will not indicate whether developers still use an old
specification or have adopted the revised RFC. specification or have adopted the revised RFC.
5.4. Citations Versus Web References 5.4. Citations versus Web References
Web references might be another indicator of the popularity of an Web references might be another indicator of the popularity of an
RFC. In order to evaluate these references, we list here the number RFC. In order to evaluate these references, we list here the number
of results returned by searches on Google and Bing, looking for the of results returned by searches on Google and Bing, looking for the
search term "RFCnnnn" (e.g., RFC8411), and copying the number of search term "RFCnnnn" (e.g., "RFC8411"), and copying the number of
results returned by the search engines. The table below presents the results returned by the search engines. The table below presents the
results of these searches, performed on April 4, 2020. results of these searches, performed on April 4, 2020.
+-----------+--------+-----------+--------+-------+ +===========+============+===========+========+=======+
| RFC(2018) | Status | Citations | Google | Bing | | RFC(2018) | Status | Citations | Google | Bing |
+-----------+--------+-----------+--------+-------+ +===========+============+===========+========+=======+
| 8411 | Info | 1 | 301 | 94 | | 8411 | Info | 1 | 301 | 94 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8456 | Info | 1 | 266 | 8456 | | 8456 | Info | 1 | 266 | 8456 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8446 | PS | 418 | 25900 | 47800 | | 8446 | PS | 418 | 25900 | 47800 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8355 | Info | 3 | 521 | 114 | | 8355 | Info | 3 | 521 | 114 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8441 | PS | 1 | 2430 | 59500 | | 8441 | PS | 1 | 2430 | 59500 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8324 | ISE | 0 | 393 | 138 | | 8324 | Info (ISE) | 0 | 393 | 138 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8377 | PS | 0 | 264 | 10900 | | 8377 | PS | 0 | 264 | 10900 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8498 | Info | 0 | 335 | 10100 | | 8498 | Info | 0 | 335 | 10100 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8479 | ISE | 0 | 564 | 11000 | | 8479 | Info (ISE) | 0 | 564 | 11000 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8453 | Info | 3 | 817 | 11400 | | 8453 | Info | 3 | 817 | 11400 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8429 | BCP | 0 | 391 | 41600 | | 8429 | BCP | 0 | 391 | 41600 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8312 | Info | 25 | 1620 | 2820 | | 8312 | Info | 25 | 1620 | 2820 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8492 | ISE | 4 | 323 | 9400 | | 8492 | Info (ISE) | 4 | 323 | 9400 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8378 | Exp | 1 | 418 | 11600 | | 8378 | Exp | 1 | 418 | 11600 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8361 | PS | 0 | 499 | 92 | | 8361 | PS | 0 | 499 | 92 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8472 | PS | 1 | 496 | 169 | | 8472 | PS | 1 | 496 | 169 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8471 | PS | 1 | 1510 | 11600 | | 8471 | PS | 1 | 1510 | 11600 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8466 | PS | 0 | 766 | 173 | | 8466 | PS | 0 | 766 | 173 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8362 | PS | 1 | 67 | 147 | | 8362 | PS | 1 | 67 | 147 |
| | | | | | +-----------+------------+-----------+--------+-------+
| 8468 | Info | 1 | 453 | 127 | | 8468 | Info | 1 | 453 | 127 |
+-----------+--------+-----------+--------+-------+ +-----------+------------+-----------+--------+-------+
The results counts from Bing are sometimes surprising. Why would RFC Table 15
The result counts from Bing are sometimes surprising. Why would RFC
8441 gather 59,500 web references? Looking at the results in detail, 8441 gather 59,500 web references? Looking at the results in detail,
we find a mix of data. Some of them are logs of development projects we find a mix of data. Some of them are logs of development projects
implementing Web Sockets, which is exactly what we are looking for, implementing Web Sockets, which is exactly what we are looking for,
but others appear spurious. For example, a shop selling rugby but others appear spurious. For example, a shop selling rugby
jerseys is listed because its phone number ends with "8441". Other jerseys is listed because its phone number ends with "8441". Other
pages were listed because street numbers or product numbers matched pages were listed because street numbers or product numbers matched
the RFC number. The same type of collision may explain the large the RFC number. The same type of collision may explain the large
reference counts on Bing for RFC 8377, 8498, 8479, 8453, 8429, 8378, reference counts on Bing for RFCs 8377, 8498, 8479, 8453, 8429, 8378,
and 8471. The result counts on Bing do not appear to provide a good and 8471. The result counts on Bing do not appear to provide a good
metric. metric.
On Google, all RFC garner at least a 250 references, largely because On Google, all RFCs garner at least a 250 references, largely because
the whole RFC catalog is replicated on a large number of web servers. the whole RFC catalog is replicated on a large number of web servers.
Deviations from that base line are largely correlated with the number Deviations from that baseline are largely correlated with the number
of citations in the Semantic Scholar, with a couple of exception: RFC of citations in the Semantic Scholar, with a couple of exception: RFC
8441, and 8471 garner more references than the low citation counts 8441 and RFC 8471 garner more references than the low citation counts
would predict. Looking at the results, we find many references in would predict. Looking at the results, we find many references in
development databases explaining how these protocols are implemented development databases explaining how these protocols are implemented
in various code bases and open source projects. This means that in various code bases and open source projects. This means that
counting Google results would give some indication about an RFC's counting Google results would give some indication about an RFC's
popularity, complementing the citation counts. popularity, complementing the citation counts.
There are some practical problems in using the counts of Google There are some practical problems in using the counts of Google
results. Google searches are personalized, the results depend on the results. Google searches are personalized, the results depend on the
source of the queries, and the counts may vary as well. The search source of the queries, and the counts may vary as well. The search
result depend on the search algorithm, and there is no guarantee that results depend on the search algorithm, and there is no guarantee
counts will not change when the algorithm changes. On the other that counts will not change when the algorithm changes. On the other
hand, the results do indicate that some of the RFC in our sample are hand, the results do indicate that some of the RFCs in our sample are
beeing used by developers or in deployments. being used by developers or in deployments.
6. Observations and Next Steps 6. Observations and Next Steps
The author's goal was to get a personal understanding of the "chain The author's goal was to get a personal understanding of the "chain
of production" of the RFCs, and in particular to look at the various of production" of the RFCs, and in particular to look at the various
causes of delays in the process. As shown in Section 4, the average causes of delays in the process. As shown in Section 4, the average
RFC was produced in 3 years and 4 months, which is similar to what RFC was produced in 3 years and 4 months, which is similar to what
was found in the 2008 sample, but more than three times larger than was found in the 2008 sample, but more than three times larger than
the delays for the 1998 sample. the delays for the 1998 sample.
The Working Group process appears to be the main source of delays. The working group process appears to be the main source of delays.
Efforts to diminish delays should probably focus there, instead of on Efforts to diminish delays should probably focus there, instead of on
the IETF and IESG reviews of the RFC production. For the RFC the IETF and IESG reviews or the RFC production. For the RFC
production phase, most of the variability originates in the AUTH-48 production phase, most of the variability originates in the AUTH48
process, which is influenced by a variety of factors such as number process, which is influenced by a variety of factors such as number
of authors or level of engagement of these authors. of authors or level of engagement of these authors.
Most of the delay is spent in the Working Group, but the IETF Most of the delay is spent in the working group, but the IETF
datatracker does not hold much information about what happens inside Datatracker does not hold much information about what happens inside
the Working Groups. For example, events like Working Group Last the working groups. For example, events like Working Group Last
Calls were not recorded in the history of the selected drafts Calls were not recorded in the history of the selected drafts
available in the datatracker. Such information would have been available in the Datatracker. Such information would have been
interesting. Of course, requiring that information would create an interesting. Of course, requiring that information would create an
administrative burden, so there is clearly a trade-off between administrative burden, so there is clearly a trade-off between
requiring more work from working group chairs and providing better requiring more work from working group chairs and providing better
data for process analysis. (It appears that this information can be data for process analysis. (It appears that this information can be
available in the datatracker for more recent drafts, if the WG chairs available in the Datatracker for more recent drafts, if the WG chairs
use the datatracker properly.) use the Datatracker properly.)
The Independent Stream operates as expected. The majority of the The Independent Stream operates as expected. The majority of the
authors of the Independent Stream RFCs appear to be in IETF insiders, authors of the Independent Stream RFCs appear to be in IETF insiders,
but there is significant amount of engagement by outside parties. but there is significant amount of engagement by outside parties.
The analysis of citations in Section 5.1 shows that citation numbers The analysis of citations in Section 5.1 shows that citation numbers
are a very poor indication of the "value" of an RFC. Citation are a very poor indication of the "value" of an RFC. Citation
numbers measure the engagement of academic researchers with specific numbers measure the engagement of academic researchers with specific
topics, but have little correlation with the level of adoption and topics, but have little correlation with the level of adoption and
deployment of a specific RFC. The result counts of Google searches deployment of a specific RFC. The result counts of Google searches
do capture references outside academia, such as logs of development do capture references outside academia, such as logs of development
projects. This might be informative, but it is not clear that the projects. This might be informative, but it is not clear that the
counts would not change over time due to algorithm changes or counts would not change over time due to algorithm changes or
personaliztion. personalization.
This document analyses a small sample of RFCs "in depth". This This document analyses a small sample of RFCs "in depth". This
allowed gathering of detailed feedback on the process and the allowed gathering of detailed feedback on the process and the
deployments. On the other hand, much of the data on delays is deployments. On the other hand, much of the data on delays is
available from the IETF datatracker. It may be worth considering available from the IETF Datatracker. It may be worth considering
adding an automated reporting of delay metrics in the IETF adding an automated reporting of delay metrics in the IETF
datatracker. Datatracker.
This document only considers the RFCs that were published in a given This document only considers the RFCs that were published in a given
year. This approach can be criticized as introducing a form of year. This approach can be criticized as introducing a form of
"survivor bias". There are many drafts proposed to the IETF, and "survivor bias". There are many drafts proposed to the IETF, and
only a fraction of them end up being published as RFCs. On one hand only a fraction of them end up being published as RFCs. On one hand,
this is expected, because part of the process is to triage between this is expected, because part of the process is to triage between
ideas that can gather consensus and those that don't. On the other ideas that can gather consensus and those that don't. On the other
hand, we don't know whether that triage is too drastic and hand, we don't know whether that triage is too drastic and has
discouraged progress on good ideas. discouraged progress on good ideas.
One way to evaluate the triage process would be to look at One way to evaluate the triage process would be to look at
publication attempts that were abandoned, for example drafts that publication attempts that were abandoned -- for example, drafts that
expired without progressing or being replaced. The sampling expired without progressing or being replaced. The sampling
methodology could also be used for that purpose. Pick maybe 20 methodology could also be used for that purpose. Pick maybe 20
drafts at random, among those abandoned in a target year, and drafts at random, among those abandoned in a target year, and
investigate why they were abandoned. Was it because better solutions investigate why they were abandoned. Was it because better solutions
emerged in the Working Group? Or maybe because the authors emerged in the working group? Or maybe because the authors
discovered a flaw in their proposal? Or was it because some discovered a flaw in their proposal? Or was it because some
factional struggle blocked a good idea? Was the idea pursued in a factional struggle blocked a good idea? Was the idea pursued in a
different venue? Hopefully, someone will try this kind of different venue? Hopefully, someone will try this kind of
investigation. investigation.
7. Security Considerations 7. Security Considerations
This draft does not specify any protocol. This document does not specify any protocol.
We might want to analyze whether security issues were discovered We might want to analyze whether security issues were discovered
after publication of specific standards. after publication of specific standards.
8. IANA Considerations 8. IANA Considerations
This draft does not require any IANA action. This document has no IANA actions.
Peliminary analysis does not indicate that IANA is causing any Preliminary analysis does not indicate that IANA is causing any
particular delay in the RFC publication process. particular delay in the RFC publication process.
9. Acknowledgements 9. Informative References
Many thanks to the authors of the selected RFCs who were willing to
provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah
Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini,
Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak
Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos
Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei
Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao
Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful
advice, to Stephen Farrell and Colin Perkins for their guidance on
the use of citations, and to Dave Crocker for a comprehensive review.
10. Informative References
[IETFCOUNT] [IETFCOUNT]
IETF, "Past IETF Meetings", 2020, IETF, "Past IETF Meetings",
<https://www.ietf.org/how/meetings/past/>. <https://www.ietf.org/how/meetings/past/>.
[RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering: [RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering:
Defeating Denial of Service Attacks which employ IP Source Defeating Denial of Service Attacks which employ IP Source
Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January
1998, <https://www.rfc-editor.org/info/rfc2267>. 1998, <https://www.rfc-editor.org/info/rfc2267>.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
DOI 10.17487/RFC2330, May 1998, DOI 10.17487/RFC2330, May 1998,
skipping to change at page 48, line 5 skipping to change at line 2052
[RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent [RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent
Interconnection of Lots of Links (TRILL): Multi-Topology", Interconnection of Lots of Links (TRILL): Multi-Topology",
RFC 8377, DOI 10.17487/RFC8377, July 2018, RFC 8377, DOI 10.17487/RFC8377, July 2018,
<https://www.rfc-editor.org/info/rfc8377>. <https://www.rfc-editor.org/info/rfc8377>.
[RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID [RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID
Separation Protocol (LISP) Multicast", RFC 8378, Separation Protocol (LISP) Multicast", RFC 8378,
DOI 10.17487/RFC8378, May 2018, DOI 10.17487/RFC8378, May 2018,
<https://www.rfc-editor.org/info/rfc8378>. <https://www.rfc-editor.org/info/rfc8378>.
[RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L.,
Decraene, B., Litkowski, S., and R. Shakir, "Segment
Routing Architecture", RFC 8402, DOI 10.17487/RFC8402,
July 2018, <https://www.rfc-editor.org/info/rfc8402>.
[RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for [RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for
Ed25519, Ed448, X25519, and X448 for Use in the Internet Ed25519, Ed448, X25519, and X448 for Use in the Internet
X.509 Public Key Infrastructure", RFC 8410, X.509 Public Key Infrastructure", RFC 8410,
DOI 10.17487/RFC8410, August 2018, DOI 10.17487/RFC8410, August 2018,
<https://www.rfc-editor.org/info/rfc8410>. <https://www.rfc-editor.org/info/rfc8410>.
[RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the [RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the
Cryptographic Algorithm Object Identifier Range", Cryptographic Algorithm Object Identifier Range",
RFC 8411, DOI 10.17487/RFC8411, August 2018, RFC 8411, DOI 10.17487/RFC8411, August 2018,
<https://www.rfc-editor.org/info/rfc8411>. <https://www.rfc-editor.org/info/rfc8411>.
skipping to change at page 49, line 45 skipping to change at line 2142
Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493, Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493,
DOI 10.17487/RFC8493, October 2018, DOI 10.17487/RFC8493, October 2018,
<https://www.rfc-editor.org/info/rfc8493>. <https://www.rfc-editor.org/info/rfc8493>.
[RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an [RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an
Originating Call Diversion (CDIV) Session Case in the Originating Call Diversion (CDIV) Session Case in the
Session Initiation Protocol (SIP)", RFC 8498, Session Initiation Protocol (SIP)", RFC 8498,
DOI 10.17487/RFC8498, February 2019, DOI 10.17487/RFC8498, February 2019,
<https://www.rfc-editor.org/info/rfc8498>. <https://www.rfc-editor.org/info/rfc8498>.
[RFCYEAR] RFC Editor, "Number of RFC Published per YEAR", 2020, [RFCYEAR] RFC Editor, "Number of RFC Published per YEAR",
<https://www.rfc-editor.org/rfcs-per-year/>. <https://www.rfc-editor.org/rfcs-per-year/>.
[SSCH] Allen Institute for AI, "Semantic Scholar", 2020, [SSCH] Allen Institute for AI, "Semantic Scholar | AI-Powered
<https://www.semanticscholar.org/>. Research Tool", <https://www.semanticscholar.org/>.
[TLS13IMP] [TI-LFA] Litkowski, S., Bashandy, A., Filsfils, C., Decraene, B.,
TLS WG, "TLS 1.3 Implementations", 2020, and D. Voyer, "Topology Independent Fast Reroute using
<https://github.com/tlswg/tlswg-wiki/blob/master/ Segment Routing", Work in Progress, Internet-Draft, draft-
IMPLEMENTATIONS.md>. ietf-rtgwg-segment-routing-ti-lfa-05, 15 November 2020,
<https://tools.ietf.org/html/draft-ietf-rtgwg-segment-
routing-ti-lfa-05>.
[TRKR] IETF, "IETF Data Tracker", 2020, [TLS13IMP] TLS WG, "TLS 1.3 Implementations", commit dcb7890, 14
<https://datatracker.ietf.org/>. October 2019, <https://github.com/tlswg/tlswg-
wiki/blob/master/IMPLEMENTATIONS.md>.
[TRKR] IETF, "IETF Datatracker", <https://datatracker.ietf.org/>.
Acknowledgements
Many thanks to the authors of the selected RFCs who were willing to
provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah
Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini,
Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak
Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos
Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei
Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao
Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful
advice, to Stephen Farrell and Colin Perkins for their guidance on
the use of citations, and to Dave Crocker for a comprehensive review.
Thanks also to Alice Russo and the RFC Editor team for their work
improving this document and checking the accuracy of the data.
Author's Address Author's Address
Christian Huitema Christian Huitema
Private Octopus Inc. Private Octopus Inc.
427 Golfcourse Rd 427 Golfcourse Rd
Friday Harbor WA 98250 Friday Harbor, WA 98250
U.S.A United States of America
Email: huitema@huitema.net Email: huitema@huitema.net
 End of changes. 266 change blocks. 
1176 lines changed or deleted 1256 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/