rfc9004xml2.original.xml   rfc9004.xml 
<?xml version="1.0" encoding="US-ASCII"?> <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?> <!-- updated by Chris 01/08/21 -->
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?> <!DOCTYPE rfc SYSTEM "rfc2629-xhtml.ent">
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?> <rfc xmlns:xi="http://www.w3.org/2001/XInclude" docName="draft-ietf-bmwg-b2b-fra
<?rfc sortrefs="yes"?> me-04"
<?rfc comments="yes"?> number="9004" ipr="trust200902" updates="2544" obsoletes="" submissionType="IETF
<?rfc inline="yes"?> "
<?rfc compact="yes"?> category="info" consensus="true" xml:lang="en" tocInclude="true" tocDepth="3"
<?rfc subcompact="no"?> symRefs="true" sortRefs="true" version="3">
<rfc category="info" docName="draft-ietf-bmwg-b2b-frame-04" ipr="trust200902"
updates="2544"> <!-- xml2rfc v2v3 conversion 3.5.0 -->
<front> <front>
<title abbrev="B2B Frame Update">Updates for the Back-to-back Frame <title abbrev="B2B Frame Update">Updates for the Back-to-Back Frame
Benchmark in RFC 2544</title> Benchmark in RFC 2544</title>
<seriesInfo name="RFC" value="9004"/>
<author fullname="Al Morton" initials="A." surname="Morton"> <author fullname="Al Morton" initials="A." surname="Morton">
<organization>AT&amp;T Labs</organization> <organization>AT&amp;T Labs</organization>
<address> <address>
<postal> <postal>
<street>200 Laurel Avenue South</street> <street>200 Laurel Avenue South</street>
<city>Middletown</city>
<city>Middletown,</city>
<region>NJ</region> <region>NJ</region>
<code>07748</code> <code>07748</code>
<country>United States of America</country>
<country>USA</country>
</postal> </postal>
<phone>+1 732 420 1571</phone> <phone>+1 732 420 1571</phone>
<facsimile>+1 732 368 1192</facsimile>
<email>acmorton@att.com</email> <email>acmorton@att.com</email>
<uri/> <uri/>
</address> </address>
</author> </author>
<date month="May" year="2021"/>
<date day="18" month="December" year="2020"/> <keyword>Buffer size</keyword>
<keyword>Buffer delay</keyword>
<keyword>Correction Factor</keyword>
<abstract> <abstract>
<t>Fundamental Benchmarking Methodologies for Network Interconnect <t>Fundamental benchmarking methodologies for network interconnect
Devices of interest to the IETF are defined in RFC 2544. This memo devices of interest to the IETF are defined in RFC 2544. This memo
updates the procedures of the test to measure the Back-to-back frames updates the procedures of the test to measure the Back-to-Back Frames
Benchmark of RFC 2544, based on further experience.</t> benchmark of RFC 2544, based on further experience.</t>
<t>This memo updates Section 26.4 of RFC 2544.</t> <t>This memo updates Section 26.4 of RFC 2544.</t>
</abstract> </abstract>
<note title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14<xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when,
they appear in all capitals, as shown here.</t>
<t/>
</note>
</front> </front>
<middle> <middle>
<section title="Introduction"> <section numbered="true" toc="default">
<t>The IETF's fundamental Benchmarking Methodologies are defined in <name>Introduction</name>
<xref target="RFC2544"> </xref>, supported by the terms and definitions <t>The IETF's fundamental benchmarking methodologies are defined in
in <xref target="RFC1242"/>, and <xref target="RFC2544"/> actually <xref target="RFC2544" format="default"> </xref>, supported by the terms a
obsoletes an earlier specification, <xref target="RFC1944"/>. Over time, nd definitions
the benchmarking community has updated <xref target="RFC2544"/> several in <xref target="RFC1242" format="default"/>. <xref target="RFC2544" form
times, including the Device Reset Benchmark <xref target="RFC6201"/>, at="default"/> actually
and the important Applicability Statement <xref target="RFC6815"/> obsoletes an earlier specification, <xref target="RFC1944" format="default
"/>. Over time,
the benchmarking community has updated <xref target="RFC2544" format="defa
ult"/> several
times, including the Device Reset benchmark <xref target="RFC6201" format=
"default"/>
and the important Applicability Statement <xref target="RFC6815" format="d
efault"/>
concerning use outside the Isolated Test Environment (ITE) required for concerning use outside the Isolated Test Environment (ITE) required for
accurate benchmarking. Other specifications implicitly update <xref accurate benchmarking. Other specifications implicitly update <xref target
target="RFC2544"/>, such as the IPv6 Benchmarking Methodologies in <xref ="RFC2544" format="default"/>, such as the IPv6 benchmarking methodologies in <x
target="RFC5180"/>.</t> ref target="RFC5180" format="default"/>.</t>
<t>Recent testing experience with the Back-to-Back Frame test and
<t>Recent testing experience with the Back-to-back Frame test and benchmark in <xref target="RFC2544" sectionFormat="of" section="26.4"/> in
Benchmark in Section 26.4 of <xref target="RFC2544"/> indicates that an dicates that an
update is warranted <xref target="OPNFV-2017"/> <xref update is warranted <xref target="OPNFV-2017" format="default"/> <xref tar
target="VSPERF-b2b"/>. In particular, analysis of the results indicates get="VSPERF-b2b" format="default"/>. In particular, analysis of the results indi
that buffer size matters when compensating for interruptions of software cates
packet processing, and this finding increases the importance of the that buffer size matters when compensating for interruptions of software-p
Back-to-back frame characterization described here. This memo describes acket processing, and this finding increases the importance of the
additional rationale and provides the updated method.</t> Back-to-Back Frame characterization described here. This memo provides
additional rationale and the updated method.</t>
<t><xref target="RFC2544"/> (which obsoletes <xref target="RFC1944"/>) <t><xref target="RFC2544" format="default"/>
provides its own Requirements Language consistent with <xref provides its own requirements language consistent with <xref target="RFC21
target="RFC2119"/>, since <xref target="RFC1944"/> pre-dates <xref 19" format="default"/>, since <xref target="RFC1944" format="default"/> (which i
target="RFC2119"/> and all three memos share common authorship. t obsoletes) predates <xref target="RFC2119" format="default"/>. All three memo
Today,<xref target="RFC8174"/> clarifies the usage of Requirements s share common authorship.
Language, so the requirements presented in this memo are expressed in Today, <xref target="RFC8174" format="default"/> clarifies the usage of re
<xref target="RFC8174"/> terms, and intended for those quirements
language, so the requirements language presented in this memo are expresse
d in accordance with
<xref target="RFC8174" format="default"/>. They are intended for those
performing/reporting laboratory tests to improve clarity and performing/reporting laboratory tests to improve clarity and
repeatability, and for those designing devices that facilitate these repeatability, and for those designing devices that facilitate these
tests.</t> tests.</t>
</section> </section>
<section>
<name>Requirements Language</name>
<t>
The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQU
IRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL
NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>
RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
"<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to
be interpreted as
described in BCP&nbsp;14 <xref target="RFC2119"/> <xref target="RFC8174"/>
when, and only when, they appear in all capitals, as shown here.
</t>
</section>
<section numbered="true" toc="default">
<name>Scope and Goals</name>
<section title="Scope and Goals">
<t>The scope of this memo is to define an updated method to <t>The scope of this memo is to define an updated method to
unambiguously perform tests, measure the benchmark(s), and report the unambiguously perform tests, measure the benchmark(s), and report the
results for Back-to-back Frames (presently described in Section 26.4 of results for Back-to-Back Frames (as described in
<xref target="RFC2544"/>).</t> <xref target="RFC2544" sectionFormat="of" section="26.4"/>).</t>
<t>The goal is to provide more efficient test procedures where possible
<t>The goal is to provide more efficient test procedures where possible, and expand reporting with additional interpretation of the results.
and to expand reporting with additional interpretation of the results.
The tests described in this memo address the cases in which the maximum The tests described in this memo address the cases in which the maximum
frame rate of a single ingress port cannot be transferred loss-free to frame rate of a single ingress port cannot be transferred to
an egress port (for some frame sizes of interest).</t> an egress port without loss (for some frame sizes of interest).</t>
<t>Benchmarks as described in <xref target="RFC2544" format="default"/> re
<t><xref target="RFC2544"/> Benchmarks rely on test conditions with ly on test conditions with
constant frame sizes, with the goal of understanding what network device constant frame sizes, with the goal of understanding what network-device
capability has been tested. Tests with the smallest size stress the capability has been tested. Tests with the smallest size stress the
header processing capacity, and tests with the largest size stress the header-processing capacity, and tests with the largest size stress the
overall bit processing capacity. Tests with sizes in-between may overall bit-processing capacity. Tests with sizes in between may
determine the transition between these two capacities. However, determine the transition between these two capacities.
conditions simultaneously sending a mixture of Internet frame sizes However,
(IMIX), such as those described in <xref target="RFC6985"/>, MUST NOT be conditions simultaneously sending a mixture of Internet (IMIX) frame sizes
used in Back-to-back Frame testing.</t> , such as those described in <xref target="RFC6985" format="default"/>, <bcp14>M
UST NOT</bcp14> be
<t>Section 3 of <xref target="RFC8239"/> describes buffer size testing used in Back-to-Back Frame testing.</t>
for physical networking devices in a data center. The <xref <t><xref target="RFC8239" sectionFormat="of" section="3"/> describes buffe
target="RFC8239"/> methods measure buffer latency directly with traffic r-size testing
for physical networking devices in a data center. Those methods measure bu
ffer latency directly with traffic
on multiple ingress ports that overload an egress port on the Device on multiple ingress ports that overload an egress port on the Device
Under Test (DUT) and are not subject to the revised calculations Under Test (DUT) and are not subject to the revised calculations
presented in this memo. Likewise, the methods of <xref presented in this memo. Likewise, the methods of <xref target="RFC8239" fo
target="RFC8239"/> SHOULD be used for test cases where the egress port rmat="default"/> <bcp14>SHOULD</bcp14> be used for test cases where the egress-p
ort
buffer is the known point of overload.</t> buffer is the known point of overload.</t>
</section> </section>
<section numbered="true" toc="default" anchor="motivation">
<section title="Motivation"> <name>Motivation</name>
<t>Section 3.1 of <xref target="RFC1242"/> describes the rationale for <t><xref target="RFC1242" sectionFormat="of" section="3.1"/> describes the
the Back-to-back Frames Benchmark. To summarize, there are several rationale for
the Back-to-Back Frames benchmark. To summarize, there are several
reasons that devices on a network produce bursts of frames at the reasons that devices on a network produce bursts of frames at the
minimum allowed spacing; and it is, therefore, worthwhile to understand minimum allowed spacing; and it is, therefore, worthwhile to understand
the Device Under Test (DUT) limit on the length of such bursts in the DUT limit on the length of such bursts in
practice. Also, <xref target="RFC1242"/> states: <figure> practice. The same document also states:</t>
<artwork><![CDATA[ "Tests of this parameter are intended to dete
rmine the extent
of data buffering in the device."]]></artwork>
</figure></t>
<t>After this test was defined, there have been occasional discussions <blockquote>
Tests of this parameter are intended to determine the extent
of data buffering in the device.</blockquote>
<t>Since this test was defined, there have been occasional discussions
of the stability and repeatability of the results, both over time and of the stability and repeatability of the results, both over time and
across labs. Fortunately, the Open Platform for Network Function across labs. Fortunately, the Open Platform for Network Function
Virtualization (OPNFV) VSPERF project's Continuous Integration (CI) Virtualization (OPNFV) project on Virtual Switch Performance (VSPERF) Cont
<xref target="VSPERF-CI"/> testing routinely repeats Back-to-back Frame inuous Integration (CI)
<xref target="VSPERF-CI" format="default"/> testing routinely repeats Back
-to-Back Frame
tests to verify that test functionality has been maintained through tests to verify that test functionality has been maintained through
development of the test control programs. These tests were used as a development of the test-control programs. These tests were used as a
basis to evaluate stability and repeatability, even across lab set-ups basis to evaluate stability and repeatability, even across lab setups
when the test platform was migrated to new DUT hardware at the end of when the test platform was migrated to new DUT hardware at the end of
2016.</t> 2016.</t>
<t>When the VSPERF CI results were examined <xref target="VSPERF-b2b" form
<t>When the VSPERF CI results were examined <xref target="VSPERF-b2b"/>, at="default"/>,
several aspects of the results were considered notable:<list several aspects of the results were considered notable:</t>
style="numbers"> <ol spacing="normal" type="1"><li>Back-to-Back Frame benchmark was very co
<t>Back-to-back Frame Benchmark was very consistent for some fixed nsistent for some fixed
frame sizes, and somewhat variable for other frame sizes.</t> frame sizes, and somewhat variable for other frame sizes.</li>
<li>The number of Back-to-Back Frames with zero loss reported for
<t>The number of Back-to-back Frames with zero loss reported for
large frame sizes was unexpectedly long (translating to 30 seconds large frame sizes was unexpectedly long (translating to 30 seconds
of buffer time), and no explanation or measurement limit condition of buffer time), and no explanation or measurement limit condition
was indicated. It was important that the buffering time calculations was indicated. It was important that the buffering time calculations
were part of the referenced testing and analysis<xref were part of the referenced testing and analysis <xref target="VSPERF-
target="VSPERF-b2b"> </xref>, because the calculated buffer times of b2b" format="default"> </xref>, because the calculated buffer time of
30 seconds for some frame sizes were clearly wrong or highly 30 seconds for some frame sizes was clearly wrong or highly
suspect. On the other hand, a result expressed only as a large suspect. On the other hand, a result expressed only as a large
number of Back-to-back Frames does not permit such an easy number of Back-to-Back Frames does not permit such an easy
comparison with reality.</t> comparison with reality.</li>
<li>Calculation of the extent of buffer time in the DUT helped to
<t>Calculation of the extent of buffer time in the DUT helped to explain the results observed with all frame sizes. For example,
explain the results observed with all frame sizes (for example, tests with some frame sizes cannot exceed the frame-header-processing
tests with some frame sizes cannot exceed the frame header rate of the DUT, thus, no buffering occurs. Therefore,
processing rate of the DUT and thus no buffering occurs; therefore, the results depended on the test equipment and not the DUT.</li>
the results depended on the test equipment and not the DUT).</t> <li>It was found that a better estimate of the DUT buffer time could
<t>It was found that a better estimate of the DUT buffer time could
be calculated using measurements of both the longest burst in frames be calculated using measurements of both the longest burst in frames
without loss and results from the Throughput tests conducted without loss and results from the Throughput tests conducted
according to Section 26.1 of <xref target="RFC2544"/>. It is according to <xref target="RFC2544" sectionFormat="of" section="26.1"/
apparent that the DUT's frame processing rate empties the buffer >. It is
during a trial and tends to increase the "implied" buffer size apparent that the DUT's frame-processing rate empties the buffer
estimate (measured according to Section 26.4 of <xref during a trial and tends to increase the "implied" buffer-size
target="RFC2544"/> because many frames have departed the buffer when estimate (measured according to <xref target="RFC2544" sectionFormat="
of" section="26.4"/> because many frames have departed the buffer when
the burst of frames ends). A calculation using the Throughput the burst of frames ends). A calculation using the Throughput
measurement can reveal a "corrected" buffer size estimate.</t> measurement can reveal a "corrected" buffer-size estimate.</li>
</list></t> </ol>
<t>Further, if the Throughput tests of <xref target="RFC2544" sectionForma
<t>Further, if the Throughput tests of Section 26.1 of <xref t="of" section="26.1"/> are conducted as a prerequisite, the number of
target="RFC2544"/> are conducted as a prerequisite test, the number of frame sizes required for Back-to-Back Frame benchmarking can be reduced
frame sizes required for Back-to-back Frame Benchmarking can be reduced
to one or more of the small frame sizes, or the results for large frame to one or more of the small frame sizes, or the results for large frame
sizes can be noted as invalid in the results if tested anyway (these are sizes can be noted as invalid in the results if tested anyway. These are
the larger frame sizes for which the back-to-back frame rate cannot the larger frame sizes for which the Back-to-Back Frame rate cannot
exceed the frame header processing rate of the DUT and little or no exceed the frame-header-processing rate of the DUT and little or no
buffering occurs).</t> buffering occurs.</t>
<t>The material below provides the details of the calculation to <t>The material below provides the details of the calculation to
estimate the actual buffer storage available in the DUT, using results estimate the actual buffer storage available in the DUT, using results
from the Throughput tests for each frame size, and the maximum from the Throughput tests for each frame size and the Max
theoretical frame rate for the DUT links (which constrain the minimum Theoretical Frame Rate for the DUT links (which constrain the minimum
frame spacing).</t> frame spacing).</t>
<t>In reality, there are many buffers and packet-header-processing steps
<t>In reality, there are many buffers and packet header processing steps
in a typical DUT. The simplified model used in these calculations for in a typical DUT. The simplified model used in these calculations for
the DUT includes a packet header processing function with limited rate the DUT includes a packet-header-processing function with limited rate
of operation, as shown below:</t> of operation, as shown in <xref target="simplified-model"/>.</t>
<figure anchor="simplified-model">
<t><figure> <name>Simplified Model for DUT Testing</name>
<artwork><![CDATA[ |------------ DUT --------| <artwork name="" type="" align="left" alt="" ><![CDATA[
|------------ DUT --------|
Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver
]]></artwork> ]]></artwork>
</figure></t> </figure>
<t>So, in the Back-to-Back Frame testing:</t>
<t>So, in the Back-to-back Frame testing:<list style="numbers"> <ol spacing="normal" type="1"><li>The ingress burst arrives at Max Theoret
<t>The ingress burst arrives at Max Theoretical Frame Rate, and ical Frame Rate, and
initially the frames are buffered.</t> initially the frames are buffered.</li>
<li>The packet-header-processing function (HeaderProc) operates at
<t>The packet header processing function (HeaderProc) operates at the "Measured Throughput" (<xref target="RFC2544" sectionFormat="of" s
the &ldquo;Measured Throughput&rdquo; (Section 26.1 of <xref ection="26.1"/>), removing frames from the buffer (this is the
target="RFC2544"/>), removing frames from the buffer (this is the best approximation we have, another acceptable approximation is the re
best approximation we have).</t> ceived frame rate
during Back-to-back Frame testing, if Measured Throughput is
<t>Frames that have been processed are clearly not in the buffer, so not available). </li>
the Corrected DUT buffer time equation (Section 5.4) estimates and <li>Frames that have been processed are clearly not in the buffer, so
the Corrected DUT Buffer Time equation (<xref target="bench-calc" />)
estimates and
removes the frames that the DUT forwarded on egress during the removes the frames that the DUT forwarded on egress during the
burst. We define buffer time as the number of frames occupying the burst. We define buffer time as the number of frames occupying the
buffer divided by the Maximum Theoretical Frame Rate (on ingress) buffer divided by the Max Theoretical Frame Rate (on ingress)
for the frame size under test.</t> for the frame size under test.</li>
<li>A helpful concept is the buffer-filling rate, which is the
<t>A helpful concept is the buffer filling rate, which is the
difference between the Max Theoretical Frame Rate (ingress) and the difference between the Max Theoretical Frame Rate (ingress) and the
Measured Throughput (HeaderProc on egress). If the actual buffer Measured Throughput (HeaderProc on egress). If the actual buffer
size in frames was known, the time to fill the buffer during a size in frames is known, the time to fill the buffer during a
measurement can be calculated using the filling rate as a check on measurement can be calculated using the filling rate, as a check on
measurements. However, the buffer in the model represents many measurements. However, the buffer in the model represents many
buffers of different sizes in the DUT data path.</t> buffers of different sizes in the DUT data path.</li>
</list></t> </ol>
<t>Knowledge of approximate buffer storage size (in time or bytes) may <t>Knowledge of approximate buffer storage size (in time or bytes) may
be useful to estimate whether frame losses will occur if DUT forwarding be useful in estimating whether frame losses will occur if DUT forwarding
is temporarily suspended in a production deployment, due to an is temporarily suspended in a production deployment due to an
unexpected interruption of frame processing (an interruption of duration unexpected interruption of frame processing (an interruption of duration
greater than the estimated buffer would certainly cause lost frames). In greater than the estimated buffer would certainly cause lost frames). In
Section 5, the calculations for the correct buffer time use the <xref target="b2b"/>, the calculations for the correct buffer time use the
combination of offered load at Max Theoretical Frame Rate and header combination of offered load at Max Theoretical Frame Rate and header-proce
processing speed at 100% of Measured Throughput. Other combinations are ssing speed at 100% of Measured Throughput. Other combinations are
possible, such as changing the percent of measured Throughput to account possible, such as changing the percent of Measured Throughput to account
for other processes reducing the header processing rate.</t> for other processes reducing the header processing rate.</t>
<t>The presentation of OPNFV VSPERF evaluation and development of <t>The presentation of OPNFV VSPERF evaluation and development of
enhanced search algorithms <xref target="VSPERF-BSLV"/> was discussed at enhanced search algorithms <xref target="VSPERF-BSLV" format="default"/> w
IETF-102. The enhancements are intended to compensate for transient as given and discussed at
interrupts that may cause loss at near-Throughput levels of offered IETF 102. The enhancements are intended to compensate for transient
processor interrupts that may cause loss at near-Throughput levels of offe
red
load. Subsequent analysis of the results indicates that buffers within load. Subsequent analysis of the results indicates that buffers within
the DUT can compensate for some interrupts, and this finding increases the DUT can compensate for some interrupts, and this finding increases
the importance of the Back-to-back frame characterization described the importance of the Back-to-Back Frame characterization described
here.</t> here.</t>
</section> </section>
<section numbered="true" toc="default">
<section title="Prerequisites"> <name>Prerequisites</name>
<t>The Test Setup MUST be consistent with Figure 1 of <xref <t>The test setup <bcp14>MUST</bcp14> be consistent with Figure 1 of <xref
target="RFC2544"/>, or Figure 2 when the tester's sender and receiver target="RFC2544" format="default"/>, or Figure 2 of that document when the test
er's sender and receiver
are different devices. Other mandatory testing aspects described in are different devices. Other mandatory testing aspects described in
<xref target="RFC2544"/> MUST be included, unless explicitly modified in <xref target="RFC2544" format="default"/> <bcp14>MUST</bcp14> be included, unless explicitly modified in
the next section.</t> the next section.</t>
<t>The ingress and egress link speeds and link-layer protocols <bcp14>MUST
<t>The ingress and egress link speeds and link layer protocols MUST be </bcp14> be
specified and used to compute the maximum theoretical frame rate when specified and used to compute the Max Theoretical Frame Rate when
respecting the minimum inter-frame gap.</t> respecting the minimum interframe gap.</t>
<t>The test results for the Throughput benchmark conducted according to
<t>The test results for the Throughput Benchmark conducted according to <xref target="RFC2544" sectionFormat="of" section="26.1"/> for all frame s
Section 26.1 of <xref target="RFC2544"/> for all <xref izes <bcp14>RECOMMENDED</bcp14> by <xref target="RFC2544" format="default"/> <bc
target="RFC2544"/>-RECOMMENDED frame sizes MUST be available to reduce p14>MUST</bcp14> be available to reduce
the tested frame size list, or to note invalid results for individual the tested-frame-size list or to note invalid results for individual
frame sizes (because the burst length may be essentially infinite for frame sizes (because the burst length may be essentially infinite for
large frame sizes).</t> large frame sizes).</t>
<t>Note that:</t>
<t>Note that:<list style="symbols"> <ul spacing="normal">
<t>the Throughput and the Back-to-back Frame measurement <li>the Throughput and the Back-to-Back Frame measurement-configuration
configuration traffic characteristics (unidirectional or traffic characteristics (unidirectional or
bi-directional, and number of flows generated) MUST match.</t> bidirectional, and number of flows generated) <bcp14>MUST</bcp14> matc
h.</li>
<t>the Throughput measurement MUST be under zero-loss conditions, <li>the Throughput measurement <bcp14>MUST</bcp14> be taken under zero-l
according to Section 26.1 of <xref target="RFC2544"/>.</t> oss conditions,
</list>The Back-to-back Benchmark described in Section 3.1 of <xref according to <xref target="RFC2544" sectionFormat="of" section="26.1"/
target="RFC1242"/> MUST be measured directly by the tester, where buffer >.</li>
size is inferred from Back-to-back Frame bursts and associated packet </ul>
loss measurements. Therefore, sources of packet loss that are unrelated <t>The Back-to-Back Benchmark described in <xref target="RFC1242" sectionF
to consistent evaluation of buffer size SHOULD be identified and removed ormat="of" section="3.1"/> <bcp14>MUST</bcp14> be measured directly by the teste
or mitigated. Example sources include:<list style="symbols"> r, where buffer
<t>On-path active components that are external to the DUT</t> size is inferred from Back-to-Back Frame bursts and associated packet-loss
measurements. Therefore, sources of frame loss that are unrelated
<t>Operating system environment interrupting DUT operation</t> to consistent evaluation of buffer size <bcp14>SHOULD</bcp14> be identifie
d and removed
<t>Shared resource contention between the DUT and other off-path or mitigated. Example sources include:</t>
component(s) impacting DUT's behaviour, sometimes called the "noisy <ul spacing="normal">
neighbour" problem with virtualized network functions.</t> <li>On-path active components that are external to the DUT</li>
</list></t> <li>Operating-system environment interrupting DUT operation</li>
<li>Shared-resource contention between the DUT and other off-path
component(s) impacting DUT's behavior, sometimes called the "noisy
neighbor" problem with virtualized network functions.</li>
</ul>
<t>Mitigations applicable to some of the sources above are discussed in <t>Mitigations applicable to some of the sources above are discussed in
Section 5.2, with the other measurement requirements described below in <xref target="frame-size"/>, with the other measurement requirements descr
Section 5.</t> ibed below in
<xref target="b2b"/>.</t>
</section> </section>
<section numbered="true" toc="default" anchor="b2b">
<section title="Back-to-back Frames"> <name>Back-to-Back Frames</name>
<t>Objective: To characterize the ability of a DUT to process <t>Objective: To characterize the ability of a DUT to process
back-to-back frames as defined in <xref target="RFC1242"/>.</t> Back-to-Back Frames as defined in <xref target="RFC1242" format="default"/
>.</t>
<t>The Procedure follows.</t> <t>The procedure follows.</t>
<section numbered="true" toc="default">
<section title="Preparing the list of Frame sizes"> <name>Preparing the List of Frame Sizes</name>
<t>From the list of RECOMMENDED frame sizes (Section 9 of <xref <t>From the list of <bcp14>RECOMMENDED</bcp14> frame sizes (<xref target
target="RFC2544"/>), select the subset of frame sizes whose measured ="RFC2544" sectionFormat="of" section="9"/>), select the subset of frame sizes w
Throughput (during prerequisite testing) was less than the Maximum hose Measured
Theoretical Frame Rate of the DUT/test-set-up. These are the only Throughput (during prerequisite testing) was less than the Max
Theoretical Frame Rate of the DUT/test setup. These are the only
frame sizes where it is possible to produce a burst of frames that frame sizes where it is possible to produce a burst of frames that
cause the DUT buffers to fill and eventually overflow, producing one cause the DUT buffers to fill and eventually overflow, producing one
or more discarded frames.</t> or more discarded frames.</t>
</section> </section>
<section numbered="true" toc="default" anchor="frame-size">
<section title="Test for a Single Frame Size"> <name>Test for a Single Frame Size</name>
<t>Each trial in the test requires the tester to send a burst of <t>Each trial in the test requires the tester to send a burst of
frames (after idle time) with the minimum inter-frame gap, and to frames (after idle time) with the minimum interframe gap and to
count the corresponding frames forwarded by the DUT.</t> count the corresponding frames forwarded by the DUT.</t>
<t>The duration of the trial includes three <bcp14>REQUIRED</bcp14> comp
<t>The duration of the trial includes three REQUIRED components: <list onents: </t>
style="numbers"> <ol spacing="normal" type="1"><li>The time to send the burst of frames (
<t>The time to send the burst of frames (at the back-to-back at the back-to-back
rate), determined by the search algorithm.</t> rate), determined by the search algorithm.</li>
<li>The time to receive the transferred burst of frames (at the
<t>The time to receive the transferred burst of frames (at the <xref target="RFC2544" format="default"/> Throughput rate), possibly
<xref target="RFC2544"/> Throughput rate), possibly truncated by truncated by
buffer overflow, and certainly including the latency of the buffer overflow, and certainly including the latency of the
DUT.</t> DUT.</li>
<li>At least 2 seconds not overlapping the time to receive the
<t>At least 2 seconds not overlapping the time to receive the burst (Component 2, above), to ensure that DUT buffers have depleted
burst (2.), to ensure that DUT buffers have depleted. Longer times . Longer times
MUST be used when conditions warrant, such as when buffer times <bcp14>MUST</bcp14> be used when conditions warrant, such as when bu
ffer times
&gt;2 seconds are measured or when burst sending times are &gt;2 &gt;2 seconds are measured or when burst sending times are &gt;2
seconds, but care is needed since this time component directly seconds, but care is needed, since this time component directly
increases trial duration and many trials and tests comprise a increases trial duration, and many trials and tests comprise a
complete benchmarking study.</t> complete benchmarking study.</li>
</list>The upper search limit for the time to send each burst MUST </ol>
be configurable, to values as high as 30 seconds (buffer time results <t>The upper search limit for the time to send each burst <bcp14>MUST</b
cp14>
be configurable to values as high as 30 seconds (buffer time results
reported at or near the configured upper limit are likely invalid, and reported at or near the configured upper limit are likely invalid, and
the test MUST be repeated with a higher search limit).</t> the test <bcp14>MUST</bcp14> be repeated with a higher search limit).</t
>
<t>If all frames have been received, the tester increases the length <t>If all frames have been received, the tester increases the length
of the burst according to the search algorithm and performs another of the burst according to the search algorithm and performs another
trial.</t> trial.</t>
<t>If the received frame count is less than the number of frames in <t>If the received frame count is less than the number of frames in
the burst, then the limit of DUT processing and buffering may have the burst, then the limit of DUT processing and buffering may have
been exceeded, and the burst length is determined by the search been exceeded, and the burst length for the next trial is determined by
algorithm for the next trial (the burst length is typically reduced, the search
algorithm (the burst length is typically reduced,
but see below).</t> but see below).</t>
<t>Classic search algorithms have been adapted for use in <t>Classic search algorithms have been adapted for use in
benchmarking, where the search requires discovery of a pair of benchmarking, where the search requires discovery of a pair of
outcomes, one with no loss and another with loss, at load conditions outcomes, one with no loss and another with loss, at load conditions
within the acceptable tolerance or accuracy. Conditions encountered within the acceptable tolerance or accuracy. Conditions encountered
when benchmarking the Infrastructure for Network Function when benchmarking the infrastructure for network function
Virtualization require algorithm enhancement. Fortunately, the virtualization require algorithm enhancement. Fortunately, the
adaptation of Binary Search, and an enhanced Binary Search with Loss adaptation of Binary Search, and an enhanced Binary Search with Loss
Verification have been specified in clause 12.3 of <xref Verification, have been specified in Clause 12.3 of <xref target="TST009
target="TST009"/>. These algorithms can easily be used for " format="default"/>. These algorithms can easily be used for
Back-to-back Frame benchmarking by replacing the Offered Load level Back-to-Back Frame benchmarking by replacing the offered load level
with burst length in frames. <xref target="TST009"/> Annex B describes with burst length in frames. <xref target="TST009" format="default"/>, A
nnex B describes
the theory behind the enhanced Binary Search with Loss Verification the theory behind the enhanced Binary Search with Loss Verification
algorithm.</t> algorithm.</t>
<t>There are also promising works in progress that may prove useful in
<t>There is also promising work-in-progress that may prove useful in Back-to-Back Frame benchmarking. <xref target="I-D.vpolak-mkonstan-bmwg-
Back-to-back Frame benchmarking. <xref mlrsearch" format="default"/> and <xref target="I-D.vpolak-bmwg-plrsearch" forma
target="I-D.vpolak-mkonstan-bmwg-mlrsearch"/> and <xref t="default"/> are two such examples.</t>
target="I-D.vpolak-bmwg-plrsearch"/> are two such examples.</t> <t>Either the <xref target="TST009" format="default"/> Binary Search or
Binary Search
<t>Either the <xref target="TST009"/> Binary Search or Binary Search with Loss Verification algorithms <bcp14>MUST</bcp14> be used, and input
with Loss Verification algorithms MUST be used, and input parameters parameters
to the algorithm(s) MUST be reported.</t> to the algorithm(s) <bcp14>MUST</bcp14> be reported.</t>
<t>The tester usually imposes a (configurable) minimum step size for <t>The tester usually imposes a (configurable) minimum step size for
burst length, and the step size MUST be reported with the results (as burst length, and the step size <bcp14>MUST</bcp14> be reported with the results (as
this influences the accuracy and variation of test results).</t> this influences the accuracy and variation of test results).</t>
<t>The original <xref target="RFC2544" sectionFormat="of" section="26.4"
<t>The original Section 26.4 of <xref target="RFC2544"/> definition is /> definition is
stated below:<list style="empty"> stated below:</t>
<t>The Back-to-back Frame value is the longest burst of frames <blockquote>
that the DUT can successfully process and buffer without frame The back-to-back value is the number of frames in the longest burst tha
loss, as determined from the series of trials.</t> t the DUT will handle without the loss of any frames.
</list></t> </blockquote>
</section> </section>
<section numbered="true" toc="default" anchor="test-rep">
<section title="Test Repetition and Benchmark"> <name>Test Repetition and Benchmark</name>
<t>On this topic, Section 26.4 of <xref target="RFC2544"/> <t>On this topic, <xref target="RFC2544" sectionFormat="of" section="26.
requires:<list style="empty"> 4"/>
<t>The trial length MUST be at least 2 seconds and SHOULD be requires:</t>
<blockquote>
The trial length <bcp14>MUST</bcp14> be at least 2 seconds and <bcp14>
SHOULD</bcp14> be
repeated at least 50 times with the average of the recorded values repeated at least 50 times with the average of the recorded values
being reported.</t> being reported.</blockquote>
</list></t> <t>Therefore, the Back-to-Back Frame benchmark is the average of burst-l
ength values over repeated tests to determine the longest burst of
<t>Therefore, the Back-to-back Frame Benchmark is the average of burst
length values over repeated tests to determine the longest burst of
frames that the DUT can successfully process and buffer without frame frames that the DUT can successfully process and buffer without frame
loss. Each of the repeated tests completes an independent search loss. Each of the repeated tests completes an independent search
process.</t> process.</t>
<!--[rfced] *AD - FYI, we edited the following sentence for clarity. Please chec k that the meaning of the requirements language has not changed. AD approval is necessary for changes to requirements language.
<t>In this update, the test MUST be repeated N times (the number of Original:
repetitions is now a variable that must be reported),for each frame In this update, the test MUST be repeated N times (the number of
size in the subset list, and each Back-to-back Frame value made repetitions is now a variable that must be reported),for each frame
size in the subset list, and each Back-to-back Frame value made
available for further processing (below).
Changed to:
In this update, the test MUST be repeated N times (the number of
repetitions is now a variable that must be reported) for each frame
size in the subset list, and each back-to-back frame value MUST be
made available for further processing (below).
-->
<t>In this update, the test <bcp14>MUST</bcp14> be repeated N times (the
number of
repetitions is now a variable that must be reported) for each frame
size in the subset list, and each Back-to-Back Frame value <bcp14>MUST</
bcp14> be made
available for further processing (below).</t> available for further processing (below).</t>
</section> </section>
<section numbered="true" toc="default" anchor="bench-calc">
<section title="Benchmark Calculations"> <name>Benchmark Calculations</name>
<t>For each frame size, calculate the following summary statistics for <t>For each frame size, calculate the following summary statistics for
longest Back-to-back Frame values over the N tests:<list longest Back-to-Back Frame values over the N tests:</t>
style="symbols"> <ul spacing="normal">
<t>Average (Benchmark)</t> <li>Average (Benchmark)</li>
<li>Minimum</li>
<t>Minimum</t> <li>Maximum</li>
<li>Standard Deviation</li>
<t>Maximum</t> </ul>
<t>Standard Deviation</t>
</list></t>
<t>Further, calculate the Implied DUT Buffer Time and the Corrected <t>Further, calculate the Implied DUT Buffer Time and the Corrected
DUT Buffer Time in seconds, as follows:<figure> DUT Buffer Time in seconds, as follows:</t>
<artwork><![CDATA[Implied DUT Buffer Time = <sourcecode>
Implied DUT buffer time =
Average num of Back-to-back Frames / Max Theoretical Frame Rate Average num of Back-to-back Frames / Max Theoretical Frame Rate
]]></artwork> </sourcecode>
</figure>The formula above is simply expressing the burst of frames <t>The formula above is simply expressing the burst of frames
in units of time.</t> in units of time.</t>
<t>The next step is to apply a correction factor that accounts for the <t>The next step is to apply a correction factor that accounts for the
DUT's frame forwarding operation during the test (assuming the simple DUT's frame forwarding operation during the test (assuming the simple
model of the DUT composed of a buffer and a forwarding function, model of the DUT composed of a buffer and a forwarding function,
described in Section 3).</t> described in <xref target="motivation"/>).</t>
<artwork name="" type="" align="left" alt=""><![CDATA[Corrected DUT Buff
<t><figure> er Time =
<artwork><![CDATA[Corrected DUT Buffer Time =
/ \ / \
Implied DUT |Implied DUT Measured Throughput | Implied DUT |Implied DUT Measured Throughput |
= Buffer Time - |Buffer Time * -------------------------- | = Buffer Time - |Buffer Time * -------------------------- |
| Max Theoretical Frame Rate | | Max Theoretical Frame Rate |
\ /]]></artwork> \ /]]></artwork>
</figure></t> <t>where:</t>
<ol spacing="normal" type="1"><li>The "Measured Throughput" is the <xref
<t>where:<list style="numbers"> target="RFC2544" format="default"/> Throughput Benchmark for the frame size tes
<t>The &ldquo;Measured Throughput&rdquo; is the <xref ted,
target="RFC2544"/> Throughput Benchmark for the frame size tested,
as augmented by methods including the Binary Search with Loss as augmented by methods including the Binary Search with Loss
Verification algorithm in <xref target="TST009"/> where Verification algorithm in <xref target="TST009" format="default"/> w
applicable, and MUST be expressed in frames per second in this here
equation.</t> applicable and <bcp14>MUST</bcp14> be expressed in frames per second
in this
<t>The &ldquo;Max Theoretical Frame Rate&rdquo; is a calculated equation.</li>
value for the interface speed and link layer technology used, and <li>The "Max Theoretical Frame Rate" is a calculated
MUST be expressed in frames per second in this equation.</t> value for the interface speed and link-layer technology used, and it
</list></t> <bcp14>MUST</bcp14> be expressed in frames per second in this equati
on.</li>
</ol>
<t>The term on the far right in the formula for Corrected DUT Buffer <t>The term on the far right in the formula for Corrected DUT Buffer
Time accounts for all the frames in the Burst that were transmitted by Time accounts for all the frames in the burst that were transmitted by
the DUT *while the Burst of frames were sent in*. So, these frames are the DUT <strong>while the burst of frames was sent in</strong>.&nbsp; So
not in the buffer and the buffer size is more accurately estimated by , these frames are
excluding them.</t> not in the buffer, and the buffer size is more accurately estimated by
excluding them. If Measured Throughput is not available,
an acceptable approximation is the received frame rate (see Forwarding
Rate in <xref target="RFC2889" format="default"/> measured during Back-to-back
Frame testing).</t>
</section> </section>
</section> </section>
<section numbered="true" toc="default">
<section title="Reporting"> <name>Reporting</name>
<t>The back-to-back frame results SHOULD be reported in the format of a <t>The Back-to-Back Frame results <bcp14>SHOULD</bcp14> be reported in the
table with a row for each of the tested frame sizes. There SHOULD be format of a
columns for the frame size and for the resultant average frame count for table with a row for each of the tested frame sizes. There <bcp14>SHOULD</
bcp14> be
columns for the frame size and the resultant average frame count for
each type of data stream tested.</t> each type of data stream tested.</t>
<t>The number of tests averaged for the benchmark, N, <bcp14>MUST</bcp14>
<t>The number of tests Averaged for the Benchmark, N, MUST be be
reported.</t> reported.</t>
<t>The minimum, maximum, and standard deviation across all complete
<t>The Minimum, Maximum, and Standard Deviation across all complete tests <bcp14>SHOULD</bcp14> also be reported (they are referred to as "Min
tests SHOULD also be reported (they are referred to as "Min,Max,StdDev" ,Max,StdDev"
in the table below).</t> in <xref target="frame-results"/>).</t>
<t>The Corrected DUT Buffer Time <bcp14>SHOULD</bcp14> also be reported.</
<t>The Corrected DUT Buffer Time SHOULD also be reported.</t> t>
<t>If the tester operates using a limited maximum burst length in <t>If the tester operates using a limited maximum burst length in
frames, then this maximum length SHOULD be reported.</t> frames, then this maximum length <bcp14>SHOULD</bcp14> be reported.</t>
<table align="center" anchor="frame-results">
<texttable style="full" title="Back-to-Back Frame Results"> <name>Back-to-Back Frame Results</name>
<ttcol>Frame Size, octets</ttcol> <thead>
<tr>
<ttcol>Ave B2B Length, frames</ttcol> <th align="left">Frame Size, octets</th>
<th align="left">Ave B2B Length, frames</th>
<ttcol>Min,Max,StdDev</ttcol> <th align="left">Min,Max,StdDev</th>
<th align="left">Corrected Buff Time, Sec</th>
<ttcol>Corrected Buff Time, Sec</ttcol> </tr>
</thead>
<c>64</c> <tbody>
<tr>
<c>26000</c> <td align="left">64</td>
<td align="left">26000</td>
<c>25500,27000,20</c> <td align="left">25500,27000,20</td>
<td align="left">0.00004</td>
<c>0.00004</c> </tr>
</texttable> </tbody>
</table>
<t>Static and configuration parameters (reported with the table
above):</t>
<t>Number of test repetitions, N</t>
<t>Minimum Step Size (during searches), in frames.</t>
<t>Static and configuration parameters (reported with <xref target="frame-
results"/>):</t>
<ul>
<li>Number of test repetitions, N</li>
<li>Minimum Step Size (during searches), in frames.</li>
</ul>
<t/> <t/>
<t>If the tester has a specific (actual) frame rate of interest (less <t>If the tester has a specific (actual) frame rate of interest (less
than the Throughput rate), it is useful to estimate the buffer time at than the Throughput rate), it is useful to estimate the buffer time at
that actual frame rate:</t> that actual frame rate:</t>
<artwork name="" type="" align="left" alt=""><![CDATA[Actual Buffer Time =
<figure>
<artwork><![CDATA[Actual Buffer Time =
Max Theoretical Frame Rate Max Theoretical Frame Rate
= Corrected DUT Buffer Time * -------------------------- = Corrected DUT Buffer Time * --------------------------
Actual Frame Rate Actual Frame Rate
]]></artwork> ]]></artwork>
</figure>
<t>and report this value, properly labeled.</t> <t>and report this value, properly labeled.</t>
</section> </section>
<section numbered="true" toc="default">
<section title="Security Considerations"> <name>Security Considerations</name>
<t>Benchmarking activities as described in this memo are limited to <t>Benchmarking activities as described in this memo are limited to
technology characterization using controlled stimuli in a laboratory technology characterization using controlled stimuli in a laboratory
environment, with dedicated address space and the other constraints environment, with dedicated address space and the other constraints
of<xref target="RFC2544"/>.</t> of <xref target="RFC2544" format="default"/>.</t>
<t>The benchmarking network topology will be an independent test setup <t>The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test traffic and <bcp14>MUST NOT</bcp14> be connected to devices that may forward the t
into a production network, or misroute traffic to the test management est traffic
network. See <xref target="RFC6815"/>.</t> into a production network or misroute traffic to the test management
network. See <xref target="RFC6815" format="default"/>.</t>
<t>Further, benchmarking is performed on an "opaque-box" (a.k.a. <t>Further, benchmarking is performed on an "opaque-box" (a.k.a.
"black-box") basis, relying solely on measurements observable external "black-box") basis, relying solely on measurements observable external
to the DUT/SUT.</t> to the Device or System Under Test (SUT).</t>
<t>The DUT developers are commonly independent from the personnel and <t>The DUT developers are commonly independent from the personnel and
institutions conducting benchmarking studies. DUT developers might have institutions conducting benchmarking studies. DUT developers might have
incentives to alter the performance of the DUT if the test conditions incentives to alter the performance of the DUT if the test conditions
can be detected. Special capabilities SHOULD NOT exist in the DUT/SUT can be detected. Special capabilities <bcp14>SHOULD NOT</bcp14> exist in t he DUT/SUT
specifically for benchmarking purposes. Procedures described in this specifically for benchmarking purposes. Procedures described in this
document are not designed to detect such activity. Additional testing document are not designed to detect such activity. Additional testing
outside of the scope of this document would be needed and has been used outside of the scope of this document would be needed and has been used
successfully in the past to discover such malpractices.</t> successfully in the past to discover such malpractices.</t>
<t>Any implications for network security arising from the DUT/SUT <bcp14>S
<t>Any implications for network security arising from the DUT/SUT SHOULD HOULD</bcp14>
be identical in the lab and in production networks.</t> be identical in the lab and in production networks.</t>
</section> </section>
<section anchor="IANA" numbered="true" toc="default">
<section anchor="IANA" title="IANA Considerations"> <name>IANA Considerations</name>
<t>This memo makes no requests of IANA.</t> <t>This document has no IANA actions.</t>
</section>
<section title="Acknowledgments">
<t>Thanks to Trevor Cooper, Sridhar Rao, and Martin Klozik of the VSPERF
project for many contributions to the early testing <xref
target="VSPERF-b2b"/>. Yoshiaki Itou has also investigated the topic,
and made useful suggestions. Maciek Konstantyowicz and Vratko Polak also
provided many comments and suggestions based on extensive integration
testing and resulting search algorithm proposals - the most up-to-date
feedback possible. Tim Carlin also provided comments and support for the
draft. Warren Kumari's review improved readability in several key
passages. David Black, Martin Duke, and Scott Bradner's comments
improved the clarity and configuration advice on trial duration. Malisa
Vucinic suggested additional text on DUT design cautions in the Security
Considerations section.</t>
</section> </section>
</middle> </middle>
<back> <back>
<references title="Normative References">
<?rfc include='reference.RFC.2544'?>
<?rfc include='reference.RFC.1242'?>
<?rfc include="reference.RFC.2119"?>
<?rfc include='reference.RFC.6985'?>
<?rfc include='reference.RFC.8174'?>
<?rfc include='reference.RFC.8239'?>
<reference anchor="TST009"
target="https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/00
9/03.04.01_60/gs_NFV-TST009v030401p.pdf">
<front>
<title>ETSI GS NFV-TST 009 V3.4.1 (2020-12), "Network Functions
Virtualisation (NFV) Release 3; Testing; Specification of Networking
Benchmarks and Measurement Methods for NFVI"</title>
<author fullname="Rapporteur: Al Morton" initials="A."
surname="Morton">
<organization>ETSI Network Function Virtualization
ISG</organization>
</author>
<date month="December" year="2020"/> <displayreference target="I-D.vpolak-bmwg-plrsearch" to="BMWG-PLRSEARCH"/>
</front> <displayreference target="I-D.vpolak-mkonstan-bmwg-mlrsearch" to="BMWG-MLRSEARCH
</reference> "/>
<?rfc ?> <references>
<name>References</name>
<references>
<name>Normative References</name>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.2544.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.1242.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.2119.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.6985.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.8174.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.8239.xml"/>
<?rfc ?> <!--[rfced] FYI - when reviewing the documents listed below, we do not see Al Mo rton listed as an author.
<?rfc ?> [TST009] Morton, A., "ETSI GS NFV-TST 009 V3.4.1 (2020-12),
"Network Functions Virtualisation (NFV) Release 3;
Testing; Specification of Networking Benchmarks and
Measurement Methods for NFVI"", December 2020,
<https://www.etsi.org/deliver/etsi_gs/NFV-
TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf>.
<?rfc ?> [VSPERF-b2b]
Morton, A., "Back2Back Testing Time Series (from CI)",
June 2017, <https://wiki.opnfv.org/display/vsperf/
Traffic+Generator+Testing#TrafficGeneratorTesting-
AppendixB:Back2BackTestingTimeSeries(fromCI)>.
<?rfc ?> Linked wiki page states:
Traffic Generator Testing
Created by Trevor Cooper, last modified by Bob Fubel on Oct 02, 2018
-->
<?rfc ?> <reference anchor="TST009" target="https://www.etsi.org/deliver/etsi_gs/
NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf">
<front>
<title>Network Functions
Virtualisation (NFV) Release 3; Testing; Specification of Networking
Benchmarks and Measurement Methods for NFVI</title>
<author>
<organization>ETSI</organization>
</author>
<date month="December" year="2020"/>
</front>
<seriesInfo name="ETSI GS NFV-TST 009" value="v3.4.1"/>
<refcontent>Rapporteur: A. Morton</refcontent>
</reference>
<?rfc ?> </references>
</references> <references>
<name>Informative References</name>
<references title="Informative References"> <reference anchor="OPNFV-2017" target="https://wiki.anuket.io/download/a
<reference anchor="OPNFV-2017" ttachments/4404001/VSPERF-Dataplane-Perf-Cap-Bench.pdf?version=1&amp;modificatio
target="https://wiki.opnfv.org/download/attachments/10293193/VS nDate=1621191833500&amp;api=v2">
PERF-Dataplane-Perf-Cap-Bench.pptx?api=v2"> <front>
<front> <title>Dataplane Performance, Capacity, and Benchmarking in
<title>Dataplane Performance, Capacity, and Benchmarking in
OPNFV</title> OPNFV</title>
<author fullname="Trevor Cooper" initials="T." surname="Cooper">
<organization>Intel Corp.</organization>
</author>
<author fullname="Sridhar Rao" initials="S." surname="Rao">
<organization>Spirent Communications</organization>
</author>
<author fullname="Al Morton" initials="A." surname="Morton">
<organization>AT&amp;T Labs</organization>
</author>
<date day="15" month="June" year="2017"/>
</front>
</reference>
<author fullname="Trevor Cooper" initials="T." surname="Cooper"> <reference anchor="VSPERF-b2b" target="https://wiki.anuket.io/display/HO
<organization>Intel Corp.</organization> ME/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingT
</author> imeSeries(fromCI)">
<front>
<author fullname="Al Morton" initials="A." surname="Morton"> <title>Back2Back Testing Time Series (from CI)</title>
<organization>AT&amp;T Labs</organization> <author fullname="Al Morton" initials="A." surname="Morton">
</author> <organization>AT&amp;T Labs</organization>
</author>
<author fullname="Sridhar Rao" initials="S." surname="Rao"> <date month="June" year="2017"/>
<organization>Spirent Communications</organization> </front>
</author> </reference>
<date day="15" month="June" year="2017"/>
</front>
</reference>
<reference anchor="VSPERF-b2b" <reference anchor="VSPERF-CI" target="https://wiki.anuket.io/display/HOM
target="https://wiki.opnfv.org/display/vsperf/Traffic+Generator E/VSPERF+CI">
+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)"> <front>
<front> <title>OPNFV VSPERF CI</title>
<title>Back2Back Testing Time Series (from CI)</title> <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
<organization>Intel Corporation</organization>
</author>
<date month="June" year="2019"/>
</front>
</reference>
<author fullname="Al Morton" initials="A." surname="Morton"> <reference anchor="VSPERF-BSLV" target="https://datatracker.ietf.org/mee
<organization>AT&amp;T Labs</organization> ting/102/materials/slides-102-bmwg-evolution-of-repeatability-in-benchmarking-fr
</author> aser-plugfest-summary-for-ietf-bmwg-00">
<front>
<title>Evolution of Repeatability in Benchmarking: Fraser Plugfest
(Summary for IETF BMWG)</title>
<author fullname="Sridhar Rao" initials="S." surname="Rao">
<organization>Spirent Communications</organization>
</author>
<author fullname="Al Morton" initials="A." surname="Morton">
<organization>AT&amp;T Labs</organization>
</author>
<date month="July" year="2018"/>
</front>
</reference>
<date month="June" year="2017"/> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
</front> FC.1944.xml"/>
</reference> <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.6201.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.6815.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.R
FC.5180.xml"/>
<reference anchor="VSPERF-CI" <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2889.
target="https://wiki.opnfv.org/display/vsperf/VSPERF+CI"> xml"/>
<front>
<title>OPNFV VSPERF CI</title>
<author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> <!-- [I-D.vpolak-bmwg-plrsearch] IESG state Expired -->
<organization>Intel Corporation</organization>
</author>
<date month="June" year="2019"/> <reference anchor='I-D.vpolak-bmwg-plrsearch'>
</front> <front>
</reference> <title>Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)</title>
<reference anchor="VSPERF-BSLV" <author role="editor" initials='M' surname='Konstantynowicz' fullname='Maciek Ko
target="https://datatracker.ietf.org/meeting/102/materials/slid nstantynowicz'>
es-102-bmwg-evolution-of-repeatability-in-benchmarking-fraser-plugfest-summary-f <organization />
or-ietf-bmwg-00"> </author>
<front>
<title>Evolution of Repeatability in Benchmarking: Fraser Plugfest
(Summary for IETF BMWG)</title>
<author fullname="Al Morton" initials="A." surname="Morton"> <author role="editor" initials='V' surname='Polak' fullname='Vratko Polak'>
<organization>AT&amp;T Labs</organization> <organization />
</author> </author>
<author fullname="Sridhar Rao" initials="S." surname="Rao"> <date month='March' day='6' year='2020' />
<organization>Spirent Communications</organization>
</author>
<date month="July" year="2018"/> </front>
</front> <seriesInfo name='Internet-Draft' value='draft-vpolak-bmwg-plrsearch-03' />
</reference> <format type='TXT'
target='http://www.ietf.org/internet-drafts/draft-vpolak-bmwg-plrsearch-
03.txt' />
</reference>
<?rfc include='reference.RFC.1944'?> <!-- [I-D.vpolak-mkonstan-bmwg-mlrsearch] IESG state Expired -->
<?rfc include='reference.RFC.6201'?> <reference anchor='I-D.vpolak-mkonstan-bmwg-mlrsearch'>
<front>
<title>Multiple Loss Ratio Search for Packet Throughput (MLRsearch)</title>
<?rfc include='reference.RFC.6815'?> <author role="editor" initials='M' surname='Konstantynowicz' fullname='Maciek Ko
nstantynowicz'>
<organization />
</author>
<?rfc include='reference.RFC.5180'?> <author role="editor" initials='V' surname='Polak' fullname='Vratko Polak'>
<organization />
</author>
<?rfc ?> <date month='March' day='6' year='2020' />
<?rfc include='reference.I-D.vpolak-bmwg-plrsearch'?> </front>
<?rfc include='reference.I-D.vpolak-mkonstan-bmwg-mlrsearch'?> <seriesInfo name='Internet-Draft' value='draft-vpolak-mkonstan-bmwg-mlrsearch-03
' />
<format type='TXT'
target='http://www.ietf.org/internet-drafts/draft-vpolak-mkonstan-bmwg-m
lrsearch-03.txt' />
</reference>
<?rfc ?> </references>
</references> </references>
<section numbered="false" toc="default">
<name>Acknowledgments</name>
<t>Thanks to <contact fullname="Trevor Cooper"/>, <contact fullname="Sridh
ar Rao"/>, and <contact fullname="Martin Klozik"/> of the VSPERF
project for many contributions to the early testing <xref target="VSPERF-b
2b" format="default"/>. <contact fullname="Yoshiaki Itou"/> has also investigate
d the topic
and made useful suggestions. <contact fullname="Maciek Konstantyowicz"/> a
nd <contact fullname="Vratko Polak"/> also
provided many comments and suggestions based on extensive integration
testing and resulting search-algorithm proposals -- the most up-to-date
feedback possible. <contact fullname="Tim Carlin"/> also provided comments
and support for the
document. <contact fullname="Warren Kumari"/>'s review improved readabilit
y in several key
passages. <contact fullname="David Black"/>, <contact fullname="Martin Duk
e"/>, and <contact fullname="Scott Bradner"/>'s comments
improved the clarity and configuration advice on trial duration. <contact
fullname="Malisa Vucinic"/> suggested additional text on DUT design cautions in
the Security
Considerations section.</t>
</section>
</back> </back>
</rfc> </rfc>
 End of changes. 127 change blocks. 
523 lines changed or deleted 608 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/