rfc9307.original   rfc9307.txt 
Network Working Group N. ten Oever Internet Architecture Board (IAB) N. ten Oever
Internet-Draft University of Amsterdam Request for Comments: 9307 University of Amsterdam
Intended status: Informational C. Cath Category: Informational C. Cath
Expires: 1 December 2022 ISSN: 2070-1721 University of Cambridge
M. Kühlewind M. Kühlewind
Ericsson Ericsson
C. S. Perkins C. S. Perkins
University of Glasgow University of Glasgow
30 May 2022 August 2022
Report from the IAB Workshop on Analyzing IETF Data (AID), 2021 Report from the IAB Workshop on Analyzing IETF Data (AID) 2021
draft-iab-aid-workshop-01
Abstract Abstract
The 'Show me the numbers: Workshop on Analyzing IETF Data (AID)' was The "Show me the numbers: Workshop on Analyzing IETF Data (AID)"
convened by the Internet Architecture Board (IAB) from November 29 to workshop was convened by the Internet Architecture Board (IAB) from
December 2 and hosted by the IN-SIGHT.it project at the University of November 29 to December 2, 2021 and hosted by the IN-SIGHT.it project
Amsterdam, however, converted to an online only event. The workshop at the University of Amsterdam; however, it was converted to an
was conducted based on two discussion parts and a hackathon activity online-only event. The workshop was organized into two discussion
in between. This report summarizes the workshop's discussion and parts with a hackathon activity in between. This report summarizes
identifies topics that warrant future work and consideration. the workshop's discussion and identifies topics that warrant future
work and consideration.
Note that this document is a report on the proceedings of the Note that this document is a report on the proceedings of the
workshop. The views and positions documented in this report are workshop. The views and positions documented in this report are
those of the workshop participants and do not necessarily reflect IAB those of the workshop participants and do not necessarily reflect IAB
views and positions. views and positions.
Discussion Venues
This note is to be removed before publishing as an RFC.
Source for this draft and an issue tracker can be found at
https://github.com/intarchboard/workshop-aid.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Architecture Board (IAB)
and may be updated, replaced, or obsoleted by other documents at any and represents information that the IAB has deemed valuable to
time. It is inappropriate to use Internet-Drafts as reference provide for permanent record. It represents the consensus of the
material or to cite them other than as "work in progress." Internet Architecture Board (IAB). Documents approved for
publication by the IAB are not candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on 1 December 2022. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9307.
Copyright Notice Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. Code Components carefully, as they describe your rights and restrictions with respect
extracted from this document must include Revised BSD License text as to this document.
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction
2. Workshop Scope and Discussion . . . . . . . . . . . . . . . . 3 2. Workshop Scope and Discussion
2.1. Tools, data, and methods . . . . . . . . . . . . . . . . 3 2.1. Tools, Data, and Methods
2.2. Observations on affiliation and industry control . . . . 4 2.2. Observations on Affiliation and Industry Control
2.3. Community and diversity . . . . . . . . . . . . . . . . . 5 2.3. Community and Diversity
2.4. Publications, process, and decision-making . . . . . . . 6 2.4. Publications, Process, and Decision Making
2.5. Environmental Sustainability . . . . . . . . . . . . . . 7 2.5. Environmental Sustainability
3. Hackathon Report . . . . . . . . . . . . . . . . . . . . . . 7 3. Hackathon Report
4. Position Papers . . . . . . . . . . . . . . . . . . . . . . . 8 4. Position Papers
4.1. Tools, data, and methods . . . . . . . . . . . . . . . . 8 4.1. Tools, Data, and Methods
4.2. Observations on affiliation and industry control . . . . 8 4.2. Observations on Affiliation and Industry Control
4.3. Community and diversity . . . . . . . . . . . . . . . . . 9 4.3. Community and Diversity
4.4. Publications, process, and decision-making . . . . . . . 9 4.4. Publications, Process, and Decision Making
4.5. Environmental Sustainability . . . . . . . . . . . . . . 10 4.5. Environmental Sustainability
5. Workshop participants . . . . . . . . . . . . . . . . . . . . 10 5. Informative References
6. Program Committee . . . . . . . . . . . . . . . . . . . . . . 10 Appendix A. Data Taxonomy
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 10 Appendix B. Program Committee
8. Annexes . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Appendix C. Workshop Participants
8.1. Annex 1 - Data Taxonomy . . . . . . . . . . . . . . . . . 11 IAB Members at the Time of Approval
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13 Acknowledgments
Authors' Addresses
1. Introduction 1. Introduction
The IETF, as an international Standards Developing Organization The IETF, as an international Standards Developing Organization
(SDO), hosts a diverse set of data including on the organization's (SDO), hosts a diverse set of data about the IETF's history and
history, development, and current standardization activities, development, current standardisation activities, Internet protocols,
including of Internet protocols and its institutions. A large and the institutions that comprise the IETF. A large portion of this
portion of this data is publicly available, yet it is underutilized data is publicly available, yet it is underutilized as a tool to
as a tool to inform the work in the IETF proper or the broader inform the work in the IETF or the broader research community that is
research community focused on topics like Internet governance and focused on topics like Internet governance and trends in information
trends in ICT standard-setting. and communication technologies (ICT) standard setting.
The aim of the IAB Workshop on Analyzing IETF Data (AID) 2021 was to The aim of the "IAB Workshop on Analyzing IETF Data (AID) 2021"
study how IETF data is currently used, understand what insights can workshop was to study how IETF data is currently used, to understand
be drawn from that data, and to explore open questions around how what insights can be drawn from that data, and to explore open
that data may be further used in future. questions around how that data may be further used in the future.
These questions can inform a research agenda drawing from IETF data, These questions can inform a research agenda drawing from IETF data
that fosters further collaborative work among interested parties, that fosters further collaborative work among interested parties,
ranging from academia and civil society to industry and IETF ranging from academia and civil society to industry and IETF
leadership. leadership.
2. Workshop Scope and Discussion 2. Workshop Scope and Discussion
The workshop was organized with two all-group discussion slots at the The workshop was organized with two all-group discussion slots at the
beginning and the end of the workshop. In between the workshop beginning and the end of the workshop. In between, the workshop
participants organized hacakthon activities, based on topics participants organized hackathon activities based on topics
identifed during the initial discussion and submitted position identified during the initial discussion and in submitted position
papers. The follow topic areas have been identified and discussed. papers. The following topic areas were identified and discussed.
2.1. Tools, data, and methods 2.1. Tools, Data, and Methods
The IETF holds a wide range of data sources. The main ones used are The IETF holds a wide range of data sources. The main ones used are
the mailinglist archives (https://mailarchive.ietf.org/arch/), RFCs the mailinglist archives [Mail-Arch], RFCs [IETF-RFCs], and the
(https://www.ietf.org/standards/rfcs/), and the datatracker datatracker [Datatracker]. The latter provides information on
(https://datatracker.ietf.org/). The latter provides information on participants, authors, meeting proceedings, minutes, and more
participants, authors, meeting proceedings, minutes and more [Data-Overview]. Furthermore, there are statistics for the IETF
(https://notes.ietf.org/iab-aid-datatracker-database-overview#). websites [IETF-Statistics], the working group Github repositories,
Furthermore there are statistics for the IETF websites and the IETF survey data [Survey-Data]. There was discussion about
(https://www.ietf.org/policies/web-analytics/), working group Github the utility of download statistics for the RFCs themselves from
repositories, IETF survey data (https://www.ietf.org/blog/ietf- different repos.
community-survey-2021/) and there was discussion about the utility of
download statistics for the RFCs itself from different repos.
There are a wide range of tools to analyze this data, produced by There is a wide range of tools to analyze this data produced by IETF
IETF participants or researchers interestested in the work of the participants or researchers interested in the work of the IETF. Two
IETF. Two projects that presented their work at the workshop were projects that presented their work at the workshop were BigBang
BigBang (https://bigbang-py.readthedocs.io/en/latest/) and [BigBang] and Sodestream's IETFdata [ietfdata] library. The RFC
Sodestream's IETFdata (https://github.com/glasgow-ipl/ietfdata) Prolog Database was described in a submitted paper (see Section 4
library; the RFC Prolog Database was described in a submitted paper below). These projects could provide additional insight into
(see Section Section 4 below). These projects could be used to add existing IETF statistics [ArkkoStats] and datatracker statistics
additional insights to the existing IETF statistics [DatatrackerStats], e.g., gender-related information. Privacy issues
(https://www.arkko.com/tools/docstats.html) page and the datatracker and the implications of making such data publicly available were
statistics (https://datatracker.ietf.org/stats/), e.g., related to discussed as well.
gender questions, however, privacy issues andd implication of making
such data publicly available were discussed as well.
The datatracker itself is a community tool that welcomes The datatracker itself is a community tool that welcomes
contributions, e.g. for additions to the existing interfaces or the contributions; for example, for additions to the existing interfaces
statistics page directly (see https://notes.ietf.org/iab-aid- or the statistics page directly, see the Datatracker Database
datatracker-database-overview (https://notes.ietf.org/iab-aid- Overview [Data-Overview]. At the time of the workshop, instructions
datatracker-database-overview)). Instructions how to set up a local about how to set up a local development environment could be found at
development environment can be found, at the time of the workshop, at IAB AID Workshop Data Resources [DataResources]. Questions or
https://notes.ietf.org/iab-aid-data-resources discussion about the datatracker and possible enhancements can be
(https://notes.ietf.org/iab-aid-data-resources). Questions or any sent to tools-discuss@ietf.org.
discussion can be issued to tools-discuss@ietf.org.
2.2. Observations on affiliation and industry control 2.2. Observations on Affiliation and Industry Control
A large portion of the submitted position papers indicated interest A large portion of the submitted position papers indicated interest
in researching questions about industry control in the in researching questions about industry control in the
standardization process (vs. individual contributions in personal standardization process (as opposed to individual contributions in a
capacity), where industry control covers both, technical contribution personal capacity), where industry control covers both a) technical
and the ability to successfully standardize these contribution as contributions and the ability to successfully standardize these
well as competition on leadership roles. To assess these question it contributions and b) competition on leadership roles. To assess
has ben discussed to investigate participant's affiliations including these questions, investigating participant affiliations, including
"indirect" affiliation e.g. by funding and changes in affiliation as "indirect" affiliations (e.g., by tracking funding and changes in
well as the nessecarity to model company characteristics or affiliation) was discussed. The need to model company
stakeholder groups. characteristics or stakeholder groups was also discussed.
Discussions about the analysis of IETF data shows that affiliation Discussion about the analysis of IETF data shows that affiliation
dynamics are hard to capture, due to the specifics of how the data is dynamics are hard to capture due to the specifics of how the data is
entered but also because of larger social dynamics. On the side of entered and because of larger social dynamics. On the side of IETF
IETF data capture, affiliation is an open text field, which causes data capture, affiliation is an open text field that causes people to
people to write their affiliation down in different ways write their affiliation down in different ways (e.g., capitalization,
(capitilization, space, word seperation, etc). A common data format space, word separation, etc). A common data format could contribute
could contribute to analyses that compare SDO performance and to analyses that compare SDO performance and behavior of actors
behavior of actors inside and across standards bodies. To help this inside and across standards bodies. To help with this, a draft data
a draft data model has been developed during hackathon portion of the model was developed during the hackathon portion of the workshop; the
workshop which can found as Annex 1 - Data Taxonomy. data model can be found in Appendix A.
Furthermore, there is the issue of mergers and acquisitions and Furthermore, there is the issue of mergers, acquisitions, and
subsidiary companies. There is no authorotative exogenous source of subsidiary companies. There is no authoritative exogenous source of
variation for affiliation changes, so hand-collected and curated data variation for affiliation changes, so hand-collected and curated data
is used to analyze changes in affiliation over time. While this is used to analyze changes in affiliation over time. While this
approach is imperfect, conclusions can be drawn from the data. For approach is imperfect, conclusions can be drawn from the data. For
example, in the case of mergers or acquisition where a small example, in the case of mergers or acquisition where a small
organizations joins a large organization, this results in a organization joins a large organization, this results in a
statistically significant increase in liklihood of an individual statistically significant increase in likelihood of an individual
being put in a working group chair position (see BaronKanevskaia being put in a working group chair position (see the document by
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Baron.pdf)). Baron and Kanevskaia [LEADERSHIP-POSITIONS]).
2.3. Community and diversity 2.3. Community and Diversity
High interest from the workshop participants was also on using The workshop participants were highly interested in using existing
existing data to better understand who the current IETF community is, data to better understand who the current IETF community is. They
especially in terms of diversity, and how to potentially increase were also interested in the community's diversity and how to
diversity and thereby inclusivity, e.g. understanding if are there potentially increase it and thereby increase inclusivity, e.g.,
certain groups or lists that "drive people away" and why. understanding if there are certain factors that "drive people away"
Inclusivity and transparency about the standardization process are and why. Inclusivity and transparency about the standardization
generally important to keep the Internet and its development process process are generally important to keep the Internet and its
viable. As commented during the workshop discussion, when measuring development process viable. As commented during the workshop
and evaluating different angles of diversity it is also important to discussion, when measuring and evaluating different angles of
understand the actual goals that are intended when increasing diversity, it is also important to understand the actual goals that
diversity, e.g. in order to increase competence (mainly technical are intended when increasing diversity, e.g., in order to increase
diversity from different companies and stakeholder groups) or competence (mainly technical diversity from different companies and
relevance (also regional diversity and international footprint). stakeholder groups) or relevance (also regional diversity and
international footprint).
The discussion on community and diversity spanned from methods that The discussion on community and diversity spanned from methods that
draw from novel text mining, time series clustering, graph mining and draw from novel text mining, time series clustering, graph mining,
psycholinguistic approaches to understand the consensus mechanism to and psycholinguistic approaches to understand the consensus mechanism
more speculative approaches about what it would take to build a to more speculative approaches about what it would take to build a
feminist Internet. The discussion also covered the data needed to feminist Internet. The discussion also covered the data needed to
measure who is in the community and how diverse it is. measure who is in the community and how diverse it is.
The discussion highlighted that part of the challenge is defining The discussion highlighted that part of the challenge is defining
what diversity means, how to measure it, or as one participant what diversity means and how to measure it, or as one participant
highlighted to define "who the average IETF is". The question was highlighted, defining "who the average IETFer is". There was a
also raised what to do about missing data or non-participating or question about what to do about missing data or non-participating or
underrepresented communities, like women, individuals from the underrepresented communities, like women, individuals from the
African continent and network operators. In terms of how IETF data African continent, and network operators. In terms of how IETF data
is structured, various researchers mentioned that it is hard track is structured, various researchers mentioned that it is hard to track
conversations as mail threads, split, merge and change. The ICANN- conversations because mail threads split, merge, and change. The
at-large model came up as an example of how to involve relevant ICANN-at-large model came up as an example of how to involve relevant
stakeholders in the IETF that are currently not present. Vice versa, stakeholders in the IETF that are currently not present. Conversely,
it is also interesting for outside communities (especially policy it is also interesting for outside communities (especially policy
makers) to get a sense of who the IETF community is and keep them makers) to get a sense of who the IETF community is and keep them
updated. updated.
The human element of the community and diversity was stressed, in The human element of the community and diversity was highlighted. In
order to understand the IETF community's diversity it is important to order to understand the IETF community's diversity, it is important
talk to people (beyond text analysis) and in order to ensure to talk to people (beyond text analysis). In order to ensure
inclusivity individual participants must make an effort to, as one inclusivity, individual participants must make an effort to, as one
participant recounted, tell them their participation is valuable. participant recounted, tell them their participation is valuable.
2.4. Publications, process, and decision-making 2.4. Publications, Process, and Decision Making
A number of submissions focussed on the RFC publication process, on A number of submissions focused on the RFC publication process, on
the development of standards and other RFCs in the IETF, and on how the development of standards and other RFCs in the IETF, and on how
the IETF makes decisions. This included work on both technical the IETF makes decisions. This included work on technical decisions
decisions about the content of the standards, but also procedural and about the content of the standards, on procedural and process
process decisions, and questions around how we can understand, model, decisions, and on questions around how we can understand, model, and
and perhaps improve the standards process. Some of the work perhaps improve the standards process. Some of the work considered
considered what makes a successful RFC, how are RFCs used and what makes an RFC successful, how RFCs are used and referenced, and
referenced, and what we can learn about importance of a topic by what we can learn about the importance of a topic by studying the
studying the RFCs, drafts, and email discussion. RFCs, Internet-Drafts, and email discussions.
There were three sets of questions to consider in this area. The There were three sets of questions to consider in this area. The
first related to success and failure of standards, and considered first question related to the success and failure of standards and
what makes a successful/good RFC? What makes the process of RFC considered:
making successful? And how are RFCs used and referenced once
published? Discussion considered how to better understand the path * What makes a successful and good RFC?
from an internet draft to an RFC, to see if there are specific
factors that lead to successful development of a draft into an RFC. * What makes the process of making RFCs successful?
* How are RFCs used and referenced once published?
Discussion considered how to better understand the path from an
Internet-Draft to an RFC, to see if there are specific factors that
lead to successful development of an Internet-Draft into an RFC.
Participants explored the extent to which this depends on the Participants explored the extent to which this depends on the
seniority and experience of the authors, on the topic and IETF area, seniority and experience of the authors, on the topic and IETF area,
extent and scope of mailing list discussion, and other factors, to on the extent and scope of mailing list discussion, and other
understand whether success of a draft can be predicted, and whether factors, to understand whether success of an Internet-Draft can be
interventions can be developed to increase the likelihood of success predicted and whether interventions can be developed to increase the
for work. likelihood of success for work.
The second question focused on decision making.
* How does the IETF make design decisions?
* What are the bottlenecks in effective decision making?
* When is a decision made? And what is the decision?
The second question was around decision making. How does the IETF
make design decisions? What are the bottlenecks in effective
decision making? When is a decision made? And what is the decision?
Difficulties here lie in capturing decisions and the results of Difficulties here lie in capturing decisions and the results of
consensus calls early in the process, and understanding the factors consensus calls early in the process, and understanding the factors
that lead to effective decision making. that lead to effective decision making.
Finally, there were questions around what can be learn about Finally, there were questions regarding what can be learned about
protocols by studying IETF publications, processes, and decision protocols by studying IETF publications, processes, and decision
making? For example, are there insights to be gained around how making. For example:
security concerns are discussed and considered in the development of
standards? Is it possible to verify correctness of protocols and/or
detect ambiguities? What can be learnt by extracting insights from
implementations and activities on implementation efforts?
Answers to these questions come from analysis of IETF emails, RFCs * Are there insights to be gained around how security concerns are
and Internet-Drafts, meeting minutes, recordings, Github data, and discussed and considered in the development of standards?
external data such as surveys, etc.
* Is it possible to verify correctness of protocols and detect
ambiguities?
* What can be learned by extracting insights from implementations
and activities on implementation efforts?
Answers to these questions will come from analysis of IETF emails,
RFCs and Internet-Drafts, meeting minutes, recordings, Github data,
and external data such as surveys, etc.
2.5. Environmental Sustainability 2.5. Environmental Sustainability
The final discussion session considered environmental sustainability. The final discussion session considered environmental sustainability.
It discussed what is the IETF's role with respect to climate change Topics included what the IETF's role with respect to climate change,
both in terms on what is the environmental impact of the way the IETF both in terms of what is the environmental impact of the way the IETF
develops standards, and in terms of what is the environmental impact develops standards and in terms of what is the environmental impact
of the standards the IETF develops? of the standards the IETF develops.
Discussion started by considering how sustainable are IETF meetings, Discussion started by considering how sustainable IETF meetings are,
focussing on how much CO2 emissions are IETF meetings responsible for focusing on the amount of carbon dioxide (CO2) emissions IETF
and how can we make the IETF more sustainable. Analysis looked at meetings are responsible for and how can we make the IETF more
the home locations of participants, meeting locations, and carbon sustainable. Analysis looked at the home locations of participants,
footprint of air travel and remote attendance, to estimate the carbon meeting locations, and carbon footprint of air travel and remote
costs of an IETF meeting. Initial results suggest that the costs of attendance to estimate the CO2 costs of an IETF meeting. While the
analysis is ongoing, initial results suggest that the costs of
holding multiple in-person IETF meetings per year are likely holding multiple in-person IETF meetings per year are likely
unsustainable in terms of carbon emission, although the analysis is unsustainable in terms of CO2 emission.
ongoing.
Discussion also considered to what extent are climate impacts The extent to which climate impacts are considered during the
considered in the development and standardization of Internet development and standardization of Internet protocols was discussed.
protocols? It reviewed the text of RFCs and active working group RFCs and Internet-Drafts of active working groups were reviewed for
drafts, looking for relevant keywords to highlight the extent to relevant keywords to highlight the extent to which climate change,
which climate change, energy efficiency, and related topics are energy efficiency, and related topics were considered in the design
considered in the design of Internet protocols, to show the limited of Internet protocols. This review revealed the limited extent to
extent to which these topics have been considered. Ongoing work is which these topics have been considered. There is ongoing work to
considering meeting minutes and mail archives, to get a fuller get a fuller picture by reviewing meeting minutes and mail archives
picture, but initial results show only limited consideration of these as well, but initial results show only limited consideration of these
important issues. important issues.
3. Hackathon Report 3. Hackathon Report
The middle two days of the workshop were organized as a hackathon. The middle two days of the workshop were organized as a hackathon.
The aims of the hackathon were to 1) acquaint people with the The aims of the hackathon were to 1) acquaint people with the
different data sources and analysis methods, 2) seek to answer some different data sources and analysis methods, 2) seek to answer some
of the questions that came up during presentations on the first day of the questions that came up during presentations on the first day
of the workshop, 3) foster collaboration among researchers to grow a of the workshop, and 3) foster collaboration among researchers to
community of IETF data researchers. grow a community of IETF data researchers.
At the end of Day 1, the plenary presentation day, people were At the end of Day 1, the plenary presentation day, people were
invited to divide themselves in groups who selected their own invited to divide themselves into groups and select their own
respective facilitators. All groups had their own work space and respective facilitators. All groups had their own work space and
could use their own communication methods and channels, or use IETF's could use their own communication methods and channels. Furthermore,
gather.town. Furthermore, daily check-ins were organized during the daily check-ins were organized during the two hackathon days. On the
two hackathon days. At the final day the hackathon groups presented final day, the hackathon groups presented their work in a plenary
their work in a plenary session. session.
The objectives of the hackathon, according to the co-chairs, have According to the co-chairs, the objectives of the hackathon have been
been met, and the output significantly exceeded expectations. It met, and the output significantly exceeded expectations. It allowed
allowed for more interaction then academic conferences and produced more interaction than academic conferences and produced some actual
some actual research results by people who had not collaborated research results by people who had not collaborated before the
before the workshop. workshop.
Future workshops that choose to integrate a hackathon could consider Future workshops that choose to integrate a hackathon could consider
to ask participants to submit groups, issues, and questions asking participants to submit issues and questions beforehand
beforehand (potentially as part of the positions paper or the sign-up (potentially as part of the position papers or the sign-up process)
process) to facilitate the formation of groups. to facilitate the formation of groups.
4. Position Papers 4. Position Papers
4.1. Tools, data, and methods 4.1. Tools, Data, and Methods
Sebastian Benthall Using Complex Systems Analysis to Identify Sebastian Benthall, "Using Complex Systems Analysis to Identify
Organizational Interventions (https://www.iab.org/wp-content/IAB- Organizational Interventions" [COMPLEX-SYSTEMS]
uploads/2021/11/Benthall.pdf)
Stephen McQuistin, Colin Perkins The ietfdata Library Stephen McQuistin and Colin Perkins, "The ietfdata Library"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/McQuistin.pdf) [ietfdata-Library]
Marc Petit-Huguenin The RFC Prolog Database (https://www.iab.org/wp- Marc Petit-Huguenin, "The RFC Prolog Database" [PROLOG-DATABASE]
content/IAB-uploads/2021/11/Petit-Huguenin.txt)
Jari Arkko Observations about IETF process measurements Jari Arkko, "Observations about IETF process measurements"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Arkko.pdf) [MEASURING-IETF-PROCESSES]
4.2. Observations on affiliation and industry control 4.2. Observations on Affiliation and Industry Control
Justus Baron, Olia Kanevskaia Competition for Leadership Positions in Justus Baron and Olia Kanevskaia, "Competition for Leadership
Standards Development Organizations (https://www.iab.org/wp-content/ Positions in Standards Development Organizations"
IAB-uploads/2021/11/Baron.pdf) [LEADERSHIP-POSITIONS]
Nick Doty Analyzing IETF Data: Changing affiliations Nick Doty, "Analyzing IETF Data: Changing affiliations"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Doty.pdf) [ANALYZING-AFFILIATIONS]
Don Le Position Paper (https://www.iab.org/wp-content/IAB- Don Le, "Analysing IETF Data Position Paper" [ANALYSING-IETF]
uploads/2021/11/Le.pdf)
Elizaveta Yachmeneva Research Proposal (https://www.iab.org/wp- Elizaveta Yachmeneva, "Research Proposal" [RESEARCH-PROPOSAL]
content/IAB-uploads/2021/11/Yachmeneva.pdf)
4.3. Community and diversity 4.3. Community and Diversity
Priyanka Sinha, Michael Ackermann, Pabitra Mitra, Arvind Singh, Amit Priyanka Sinha, Michael Ackermann, Pabitra Mitra, Arvind Singh, and
Kumar Agrawal Characterizing the IETF through its consensus Amit Kumar Agrawal, "Characterizing the IETF through its consensus
mechanisms (https://www.iab.org/wp-content/IAB-uploads/2021/11/ mechanisms" [CONSENSUS-MECHANISMS]
Sinha.pdf)
Mallory Knodel Would feminists have built a better internet? Mallory Knodel, "Would feminists have built a better internet?"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Knodel.pdf) [FEMINIST-INTERNET]
Wes Hardaker, Genevieve Bartlett Identifying temporal trends in IETF Wes Hardaker and Genevieve Bartlett, "Identifying temporal trends in
participation (https://www.iab.org/wp-content/IAB-uploads/2021/11/ IETF participation" [TEMPORAL-TRENDS]
Hardaker.pdf)
Lars Eggert Who is the Average IETF Participant? Lars Eggert, "Who is the Average IETF Participant?"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Eggert.pdf) [AVERAGE-PARTICIPANT]
Emanuele Tarantino, Justus Baron, Bernhard Ganglmair, Nicola Persico, Emanuele Tarantino, Justus Baron, Bernhard Ganglmair, Nicola Persico,
Timothy Simcoe Representation is Not Sufficient for Selecting Gender and Timothy Simcoe, "Representation is Not Sufficient for Selecting
Diversity (https://www.iab.org/wp-content/IAB-uploads/2021/11/ Gender Diversity" [GENDER-DIVERSITY]
Tarantino.pdf)
4.4. Publications, process, and decision-making 4.4. Publications, Process, and Decision Making
Michael Welzl, Carsten Griwodz, Safiqul Islam Understanding Internet Michael Welzl, Carsten Griwodz, and Safiqul Islam, "Understanding
Protocol Design Decisions (https://www.iab.org/wp-content/IAB- Internet Protocol Design Decisions" [DESIGN-DECISIONS]
uploads/2021/11/Welzl.pdf)
Ignacio Castro et al Characterising the IETF through the lens of RFC Ignacio Castro et al., "Characterising the IETF through the lens of
deployment (https://dl.acm.org/doi/abs/10.1145/3487552.3487821) RFC deployment" [RFC-DEPLOYMENT]
Carsten Griwodz, Safiqul Islam, Michael Welzl The Impact of Carsten Griwodz, Safiqul Islam, and Michael Welzl, "The Impact of
Continuity (https://www.iab.org/wp-content/IAB-uploads/2021/11/ Continuity" [CONTINUITY]
Griwodz.pdf)
Paul Hoffman RFCs Change (https://www.iab.org/wp-content/IAB- Paul Hoffman, "RFCs Change" [RFCs-CHANGE]
uploads/2021/11/Hoffman.pdf)
Xue Li, Sara Magliacane, Paul Groth The Challenges of Cross-Document Xue Li, Sara Magliacane, and Paul Groth, "The Challenges of
Coreference Resolution in Email (https://www.iab.org/wp-content/IAB- Cross-Document Coreference Resolution in Email"
uploads/2021/11/Groth.pdf) [CROSS-DOC-COREFERENCE]
Amelia Andersdotter Project in time series analysis: e-mailing lists Amelia Andersdotter, "Project in time series analysis: e-mailing
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Andersdotter.pdf) lists" [E-MAILING-LISTS]
Mark McFadden Position Paper (https://www.iab.org/wp-content/IAB- Mark McFadden, "A Position Paper by Mark McFadden" [POSITION-PAPER]
uploads/2021/11/McFadden.pdf)
4.5. Environmental Sustainability 4.5. Environmental Sustainability
Christoph Becker Towards Environmental Sustainability with the IETF Christoph Becker, "Towards Environmental Sustainability with the
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Becker.pdf) IETF" [ENVIRONMENTAL]
Daniel Migault CO2eq: Estimating Meetings' Air Flight CO2 Equivalent Daniel Migault, "CO2eq: Estimating Meetings' Air Flight CO2
Emissions: An Illustrative Example with IETF meetings Equivalent Emissions: An Illustrative Example with IETF meetings"
(https://www.iab.org/wp-content/IAB-uploads/2021/11/Migault.pdf) [CO2eq]
5. Workshop participants 5. Informative References
Bernhard Ganglmair, Carsten Griwodz, Christoph Becker, Colin Perkins, [ANALYSING-IETF]
Corinne Cath, Daniel Migault, Don Le, Effy Xue Li, Elizaveta Article 19, "Analysing IETF Position Paper",
Yachmeneva, Francois Ortolan, Greg Wood, Ignacio Castro, Jari Arkko, <https://www.iab.org/wp-content/IAB-uploads/2021/11/
Justus Baron, Karen O'Donoghue, Lars Eggert, Mallory Knodel, Marc Le.pdf>.
Petit-Huguenin, Mark McFadden, Michael Welzl, Mirja Kuehlewind, Nick
Doty, Niels ten Oever, Priyanka Sinha, Safiqul Islam, Sebastian
Benthall, Stephen McQuistin, Wes Hardaker, and Zhenbin Li.
6. Program Committee [ANALYZING-AFFILIATIONS]
Doty, N., "Analyzing IETF Data: Changing affiliations",
September 2021, <https://www.iab.org/wp-content/IAB-
uploads/2021/11/Doty.pdf>.
The workshop Program Committee members were Niels ten Oever (chair, [ArkkoStats]
University of Amsterdam), Colin Perkins (chair, IRTF, University of "Document Statistics",
Glasgow), Corinne Cath (chair, Oxford Internet Institute), Mirja <https://www.arkko.com/tools/docstats.html>.
Kuehlewind (IAB, Ericsson), Zhenbin Li (IAB, Huawei), and Wes
Hardaker (IAB, USC/ISI).
7. Acknowledgments [AVERAGE-PARTICIPANT]
Eggert, L., "Who is the Average IETF Participant?",
November 2021, <https://www.iab.org/wp-content/IAB-
uploads/2021/11/Eggert.pdf>.
The Program Committee wishes to extend its thanks to Cindy Morgan for [BigBang] BigBang, "Welcome to BigBang’s documentation!",
logistics support and to Kate Pundyk for notetaking. <https://bigbang-py.readthedocs.io/en/latest/>.
Efforts put in this workshop by Niels ten Oever was made possible [CO2eq] Migault, D., "CO2eq: Estimating Meetings' Air Flight CO2
through funding from the Dutch Research Council (NWO) through grant Equivalent Emissions: An Illustrative Example with IETF
MVI.19.032 as part of the programme 'Maatschappelijk Verantwoord meeting", <https://www.iab.org/wp-content/IAB-
Innoveren (MVI)'. uploads/2021/11/Migault.pdf>.
We would like to thank the Ford Foundation for their support that [COMPLEX-SYSTEMS]
made participation of Corinne Cath, Kate Pundyk, and Mallory Knodel Benthall, S., "Using Complex Systems Analysis to Identify
possible (grant number, 136179, 2020). Organizational Interventions", 2021, <https://www.iab.org/
wp-content/IAB-uploads/2021/11/Benthall.pdf>.
Efforts in the organization of this workshop by Niels ten Oever were [CONSENSUS-MECHANISMS]
supported in part by the UK Engineering and Physical Sciences Sinha, P., Ackermann, M., Mitra, P., Singh, A., and A.
Research Council under grant EP/S036075/1. Kumar Agrawal, "Characterizing the IETF through its
consensus mechanisms", <https://www.iab.org/wp-content/
IAB-uploads/2021/11/Sinha.pdf>.
8. Annexes [CONTINUITY]
Griwodz, C., Islam, S., and M. Welzl, "The Impact of
Continuity", <https://www.iab.org/wp-content/IAB-
uploads/2021/11/Griwodz.pdf>.
8.1. Annex 1 - Data Taxonomy [CROSS-DOC-COREFERENCE]
Li, X., Magliacane, S., and P. Groth, "The Challenges of
Cross-Document Coreference Resolution in Email",
<https://www.iab.org/wp-content/IAB-uploads/2021/11/
Groth.pdf>.
[Data-Overview]
"Datatracker Database Overview", for the IAB AID Workshop,
<https://notes.ietf.org/iab-aid-datatracker-database-
overview#>.
[DataResources]
"IAB AID Workshop Data Resources",
<https://notes.ietf.org/iab-aid-data-resources#>.
[Datatracker]
IETF, "Datatracker", <https://datatracker.ietf.org/>.
[DatatrackerStats]
IETF, "Statistics", <https://datatracker.ietf.org/stats/>.
[DESIGN-DECISIONS]
Welzl, M., Griwodz, C., and S. Islam, "Understanding
Internet Protocol Design Decisions", <https://www.iab.org/
wp-content/IAB-uploads/2021/11/Welzl.pdf>.
[E-MAILING-LISTS]
Andersdotter, A., "Project in time series analysis:
e-mailing lists", May 2018, <https://www.iab.org/wp-
content/IAB-uploads/2021/11/Andersdotter.pdf>.
[ENVIRONMENTAL]
Becker, C., "Towards Environmental Sustainability with the
IETF", <https://www.iab.org/wp-content/IAB-
uploads/2021/11/Becker.pdf>.
[FEMINIST-INTERNET]
Knodel, M., "Would feminists have built a better
internet?", September 2021, <https://www.iab.org/wp-
content/IAB-uploads/2021/11/Knodel.pdf>.
[GENDER-DIVERSITY]
Baron, J., Ganglmair, B., Persico, N., Simcoe, T., and E.
Tarantino, "Representation is Not Sufficient for Selecting
Gender Diversity", August 2021, <https://www.iab.org/wp-
content/IAB-uploads/2021/11/Tarantino.pdf>.
[IETF-RFCs]
IETF, "RFCs", <https://www.ietf.org/standards/rfcs/>.
[IETF-Statistics]
IETF, "Web analytics",
<https://www.ietf.org/policies/web-analytics/>.
[ietfdata] "IETF Data", Internet Protocols Laboratory, commit
c53bf15, August 2022,
<https://github.com/glasgow-ipl/ietfdata>.
[ietfdata-Library]
McQuistin, S. and C. Perkins, "The ietfdata Library",
<https://www.iab.org/wp-content/IAB-uploads/2021/11/
McQuistin.pdf>.
[LEADERSHIP-POSITIONS]
Baron, J. and O. Kanevskaia, "Competition for Leadership
Positions in Standards Development Organizations", October
2021, <https://www.iab.org/wp-content/IAB-uploads/2021/11/
Baron.pdf>.
[Mail-Arch]
IETF, "Mail Archive",
<https://mailarchive.ietf.org/arch/>.
[MEASURING-IETF-PROCESSES]
Arkko, J., "Observations about IETF process measurements",
<https://www.iab.org/wp-content/IAB-uploads/2021/11/
Arkko.pdf>.
[POSITION-PAPER]
McFadden, M., "A Position Paper", <https://www.iab.org/wp-
content/IAB-uploads/2021/11/McFadden.pdf>.
[PROLOG-DATABASE]
Huguenin, P., "The RFC Prolog Database", September 2021,
<https://www.iab.org/wp-content/IAB-uploads/2021/11/Petit-
Huguenin.txt>.
[RESEARCH-PROPOSAL]
Yachmeneva, E., "Research Proposal", <https://www.iab.org/
wp-content/IAB-uploads/2021/11/Yachmeneva.pdf>.
[RFC-DEPLOYMENT]
Castro, I., Healey, P., Iqbal, W., Karan, M., Khare, P.,
McQuistin, S., Perkins, C., Purver, M., Qadir, J., and G.
Tyson, "Characterising the IETF through the lens of RFC
deployment", November 2021,
<https://dl.acm.org/doi/abs/10.1145/3487552.3487821>.
[RFCs-CHANGE]
Hoffman, P., "RFCs Change", September 2021,
<https://www.iab.org/wp-content/IAB-uploads/2021/11/
Hoffman.pdf>.
[Survey-Data]
IETF, "IETF Community Survey 2021", 11 August 2021,
<https://www.ietf.org/blog/ietf-community-survey-2021/>.
[TEMPORAL-TRENDS]
Hardaker, W. and G. Bartlett, "Identifying temporal trends
in IETF participation", September 2021,
<https://www.iab.org/wp-content/IAB-uploads/2021/11/
Hardaker.pdf>.
Appendix A. Data Taxonomy
A Draft Data Taxonomy for SDO Data: A Draft Data Taxonomy for SDO Data:
Organization: Organization:
Organization Subsidiary Organization Subsidiary
Time Time
Email domain Email domain
Website domain Website domain
Size Size
Revenue, annual Revenue, annual
skipping to change at page 11, line 45 skipping to change at line 614
Regulatory Body Regulatory Body
Research and Development Institution Research and Development Institution
Software Provider Software Provider
Testing and Certification Testing and Certification
Telecommunications Provider Telecommunications Provider
Satellite Operator Satellite Operator
Org - Stakeholder Group : 1 - 1 Org - Stakeholder Group : 1 - 1
Academia Academia
Civil Society Civil Society
Private Sector -- including industry consortia and associations; state-owned and government-funded businesses Private Sector -- including industry consortia and associations;
state-owned and government-funded businesses
Government Government
Technical Community (IETF, ICANN, ETSI, 3GPP, oneM2M, etc) Technical Community (IETF, ICANN, ETSI, 3GPP, oneM2M, etc)
Intergovernmental organization Intergovernmental organization
SDO: SDO:
Membership Types (SDO) Membership Types (SDO)
Members (Organizations for some, individuals for others…) Members (Organizations for some, individuals for others...)
Membership organization Membership organization
Regional SDO Regional SDO
ARIB ARIB
ATIS ATIS
CCSA CCSA
ETSI ETSI
TSDSI TSDSI
TTA TTA
TTC TTC
Consortia Consortia
skipping to change at page 13, line 4 skipping to change at line 671
Time Time
Email Domain (personal domain) Email Domain (personal domain)
(Contribution data is in other tables) (Contribution data is in other tables)
Document Document
Status of Document Status of Document
Internet Draft Internet Draft
Work Item Work Item
Standard Standard
Author - Author -
Name Name
Affiliation - Organization Affiliation - Organization
Person/Participant Person/Participant
(Affiliation from Authors only?) (Affiliation from Authors only?)
Data Source - Provenance for any data imported from an external data set Data Source - Provenance for any data imported from an external data set
Meeting Meeting
Time Time
Place Place
Agenda Agenda
Registrations Registrations
Name Name
Email Email
Affiliation Affiliation
Appendix B. Program Committee
The workshop Program Committee members were Niels ten Oever (Chair,
University of Amsterdam), Colin Perkins (Chair, IRTF, University of
Glasgow), Corinne Cath (Chair, Oxford Internet Institute), Mirja
Kuehlewind (IAB, Ericsson), Zhenbin Li (IAB, Huawei), and Wes
Hardaker (IAB, USC/ISI).
Appendix C. Workshop Participants
The Workshop Participants were Bernhard Ganglmair, Carsten Griwodz,
Christoph Becker, Colin Perkins, Corinne Cath, Daniel Migault, Don
Le, Effy Xue Li, Elizaveta Yachmeneva, Francois Ortolan, Greg Wood,
Ignacio Castro, Jari Arkko, Justus Baron, Karen O'Donoghue, Lars
Eggert, Mallory Knodel, Marc Petit-Huguenin, Mark McFadden, Michael
Welzl, Mirja Kuehlewind, Nick Doty, Niels ten Oever, Priyanka Sinha,
Safiqul Islam, Sebastian Benthall, Stephen McQuistin, Wes Hardaker,
and Zhenbin Li.
IAB Members at the Time of Approval
Internet Architecture Board members at the time this document was
approved for publication were:
Jari Arkko
Deborah Brungard
Lars Eggert
Wes Hardaker
Cullen Jennings
Mallory Knodel
Mirja Kühlewind
Zhenbin Li
Tommy Pauly
David Schinazi
Russ White
Quin Wu
Jiankang Yao
Acknowledgments
The Program Committee wishes to extend its thanks to Cindy Morgan for
logistics support and to Kate Pundyk for note-taking.
We would like to thank the Ford Foundation for their support that
made participation of Corinne Cath, Kate Pundyk, and Mallory Knodel
possible (grant number, 136179, 2020).
Efforts put in this workshop by Niels ten Oever were made possible
through funding from the Dutch Research Council (NWO) through grant
MVI.19.032 as part of the program 'Maatschappelijk Verantwoord
Innoveren (MVI)'.
Efforts in the organization of this workshop by Colin Perkins were
supported in part by the UK Engineering and Physical Sciences
Research Council under grant EP/S036075/1.
Authors' Addresses Authors' Addresses
Niels ten Oever Niels ten Oever
University of Amsterdam University of Amsterdam
Email: mail@nielstenoever.net Email: mail@nielstenoever.net
Corinne Cath Corinne Cath
University of Cambridge
Email: corinnecath@gmail.com Email: corinnecath@gmail.com
Mirja Kühlewind Mirja Kühlewind
Ericsson Ericsson
Email: mirja.kuehlewind@ericsson.com Email: mirja.kuehlewind@ericsson.com
Colin Perkins Colin Perkins
University of Glasgow University of Glasgow
Email: csp@csperkins.org Email: csp@csperkins.org
 End of changes. 89 change blocks. 
318 lines changed or deleted 487 lines changed or added

This html diff was produced by rfcdiff 1.48.