RMCAT Working Group G. Van der Auwera Internet Draft M. Coban Intended status: Informational Qualcomm Technologies Inc. Expires: April 13, 2014 October 13, 2013 RMCAT Video Quality Evaluation and Double Bottleneck Test Scenario draft-vanderauwera-rmcat-video-quality-00.txt Abstract The first part of this document proposes video quality test scenarios and desired quality behaviors to evaluate RMCAT congestion control solutions. The purpose is to identify undesired video quality behaviors. The second part proposes a double bottleneck test scenario to provide additional insight into the rate allocation behavior of RMCAT solutions. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 13, 2014. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. Van der Auwera & Coban Expires April 13, 2014 [Page 1] Internet-Draft RMCAT Video Quality Evaluation October 2013 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction...................................................2 2. Video Quality Test Scenarios...................................3 2.1. Test Scenario A...........................................3 2.1.1. Configuration........................................3 2.1.2. Desired Video Quality Behavior.......................4 2.2. Test Scenario B...........................................4 2.2.1. Configuration........................................4 2.2.2. Desired Video Quality Behavior.......................5 2.3. Test Scenario C...........................................5 2.3.1. Configuration........................................5 2.3.2. Desired Video Quality Behavior.......................6 2.4. Video Coding and Communication Framework Discussion.......6 2.4.1. Test Sequences.......................................6 2.4.2. Configuration........................................6 2.5. Video Quality Assessment..................................7 3. Double Bottleneck Test Scenario................................7 3.1. Configuration.............................................7 4. Security Considerations........................................8 5. IANA Considerations............................................8 6. Conclusions....................................................8 7. References.....................................................9 7.1. Informative References....................................9 8. Acknowledgments................................................9 1. Introduction The impact of congestion control on media streams is important in real-world deployments. Therefore, the purpose of this evaluation is to verify or inspect the video quality of RMCAT congestion control solutions under different test scenarios. RMCAT based rate adaptations should result in expected or desired video quality when implemented in a state-of-the-art video coding and communication framework. In other words, the purpose is to verify that there are no unexpected or undesired video quality problems caused by congestion Van der Auwera & Coban Expires April 13, 2014 [Page 2] Internet-Draft RMCAT Video Quality Evaluation October 2013 control behaviors, while the main purpose is not to directly compare RMCAT solutions against each other. It is proposed to perform such video quality evaluation of candidate RMCAT solutions in addition to network traffic evaluations. An additional test scenario is proposed to evaluate the rate allocation behavior of RMCAT solutions when asymmetric flow conditions exist. In this test the video quality is not evaluated. 2. Video Quality Test Scenarios It is proposed to evaluate the visual quality of a subset of test scenarios described in [1]. The specifics are open for further discussion in the working group. Specific configuration parameters, such as bottleneck link rates, should be chosen so that video quality changes are visible. The video quality of the following congestion control behaviors is interesting to evaluate: o Startup o Varying bottleneck link bandwidth o Staggered flow starts o Background traffic (bursty) The following are video quality test scenarios based on [1]. 2.1. Test Scenario A This scenario consists of a single bottleneck link and a single media flow. The purpose is to evaluate video quality under congestion control startup and varying bottleneck bandwidth behaviors. 2.1.1. Configuration Topology: o Single bottleneck link o Single RMCAT sender and receiver Bottleneck link rate varies between 1Mbps and 500kbps as follows: o 0-20s: 1Mbps Van der Auwera & Coban Expires April 13, 2014 [Page 3] Internet-Draft RMCAT Video Quality Evaluation October 2013 o 20-80s: 500kbps o 80-100s: 1Mbps One-way propagation delay: 10ms Bottleneck queue type: drop-tail Bottleneck queue size: 32 packets Random loss rate over link: 0% Initial rate: 200kbps 2.1.2. Desired Video Quality Behavior Startup: Video quality improves and stabilizes within 4 seconds Bottleneck bandwidth drops from 1Mbps to 500kbps: Video quality decreases and stabilizes within 2 seconds Bottleneck bandwidth increases from 500kbps to 1Mbps: Video quality improves and stabilizes within 4 seconds 2.2. Test Scenario B Scenario B consists of a single bottleneck link and two media flows with different start times. The purpose is to evaluate video quality under the congestion control behavior when a second competing flow joins and leaves the bottleneck link. 2.2.1. Configuration Topology: o Single bottleneck link o Two RMCAT senders and receivers Second flow joins 20s after first flow and leaves after 40s. Bottleneck link rate is 1Mbps Van der Auwera & Coban Expires April 13, 2014 [Page 4] Internet-Draft RMCAT Video Quality Evaluation October 2013 One-way propagation delay: 10ms Bottleneck queue type: drop-tail Bottleneck queue size: 32 packets Random loss rate over link: 0% 2.2.2. Desired Video Quality Behavior First media flow: After second flow joins, video quality of first flow decreases and stabilizes within 4 seconds. After second flow leaves, video quality increases and stabilizes within 4 seconds. Second media flow: At startup, the video quality improves and stabilizes within 4 seconds. First and second flow: Video quality of both streams is similar. 2.3. Test Scenario C This scenario consists of a single bottleneck link with background traffic. The purpose is to evaluate video quality under congestion control behavior in the presence of bursty TCP flows. 2.3.1. Configuration Topology: o Single bottleneck link o Single RMCAT sender and receiver o Single TCP sender to receiver Bursty TCP flow starts at 0s and stops at 60s. Media flow joins link after 20s. Bottleneck link speed is 1Mbps Van der Auwera & Coban Expires April 13, 2014 [Page 5] Internet-Draft RMCAT Video Quality Evaluation October 2013 One-way propagation delay: 10ms Bottleneck queue type: drop-tail Bottleneck queue size: 32 packets Random loss rate over link: 0% Bursty TCP flow parameters: to be defined 2.3.2. Desired Video Quality Behavior After media flow joins link (>20s): Video quality improves and stabilizes within 4 seconds. After TCP flow ends (>60s): Video quality improves and stabilizes within 4 seconds. 2.4. Video Coding and Communication Framework Discussion It is proposed that RMCAT solutions are implemented in a state-of- the-art video coding and communication framework to provide proof-of- concept evidence. The purpose is to demonstrate that implementations of the congestion control solutions are feasible and that the video quality behavior under the above test scenarios is as desired. The following provides some details about test sequences, configuration and quality assessment. 2.4.1. Test Sequences Content types: video telephony and conferencing Length: 100 seconds VGA resolution or higher Frame rate is 15fps or higher File based feed for repeatability 2.4.2. Configuration Video codec: up to proponent (AVC/H.264, VP8, HEVC/H.265, VP9, etc.) Van der Auwera & Coban Expires April 13, 2014 [Page 6] Internet-Draft RMCAT Video Quality Evaluation October 2013 Error resiliency and concealment mechanisms: allowed Traffic shaping: allowed Proponents would be required to implement the congestion control method exactly as proposed and deliver the target bitrate directly to the video encoder's rate control without further processing. The reasoning is that if extra processing is required, then this should be part of the congestion control method under evaluation. The video encoder's rate control must achieve the target bit rate with reasonable accuracy and speed, which could be defined if necessary. The source code is to be provided for cross checking by non- proponents with the exception of proprietary modules that are not directly relevant for the evaluation. 2.5. Video Quality Assessment Since the purpose is to determine that the video quality behavior of the RMCAT solutions is as desired under the described test scenarios, it is proposed that the assessment is performed by a panel of experts that are non-proponents, for example, five experts. The experts report undesired video quality behaviors of the proposed RMCAT solutions. Alternatively, formal subjective quality testing (MOS) can be performed, if the MOS results are determined to be useful in the decision process. 3. Double Bottleneck Test Scenario This part of the document describes an additional test scenario for network traffic evaluation (not video quality). In this scenario, which is based on [2], one media flow encounters two bottleneck links that are each shared with a second but different flow. Figure 1 depicts the test setup. Each source Si sends a flow to its corresponding receiver Ri. The second bottleneck is more restricted than the first. The purpose is to evaluate the congestion control's rate distribution among the flows under these asymmetric conditions. 3.1. Configuration Topology: o Two sequential bottleneck links (Figure 1) Van der Auwera & Coban Expires April 13, 2014 [Page 7] Internet-Draft RMCAT Video Quality Evaluation October 2013 o Three RMCAT senders and receivers First bottleneck link speed is 1.5Mbps Second bottleneck link speed is 1Mbps One-way propagation delay per link: 10ms Bottleneck queue type: drop-tail Bottleneck queue size: 32 packets Random loss rate over link: 0% +--+ +--+ |S3| +--+ |S1|=== \ +--+ / ===|R1| +--+ \\ || // +--+ \\ || // +--+ Bottleneck +--+ +--+ Bottleneck +--+ |N0|==============|N1|=====|N2|==============|N3| +--+ Link 1 +--+ +--+ Link 2 +--+ // || \\ +--+ // || \\ +--+ |S2|=== / || \ ===|R3| +--+ +--+ +--+ |R2| +--+ Figure 1 Double Bottleneck Test Setup 4. Security Considerations Security issues have not been discussed in this memo. 5. IANA Considerations There are no IANA impacts in this memo. 6. Conclusions This document proposes video quality test scenarios and desired quality behaviors to evaluate RMCAT congestion control solutions. In Van der Auwera & Coban Expires April 13, 2014 [Page 8] Internet-Draft RMCAT Video Quality Evaluation October 2013 addition, a double bottleneck test scenario is proposed to provide additional insight into the rate allocation behavior of RMCAT solutions. These evaluation scenarios are provided to be further discussed in the RMCAT working group. 7. References 7.1. Informative References [1] "RMCAT Solution Evaluations", https://sites.google.com/site/ietfrmcatsolutionevaluations/ [2] S. Holmer, "On Fairness, Delay and Signalling of Different Approaches to Real-time Congestion Control" 8. Acknowledgments The authors are grateful to Vadim Seregin from Qualcomm for valuable discussions. This document was prepared using 2-Word-v2.0.template.dot. Van der Auwera & Coban Expires April 13, 2014 [Page 9] Internet-Draft RMCAT Video Quality Evaluation October 2013 Authors' Addresses Geert Van der Auwera Qualcomm Technologies Inc. 5775 Morehouse Drive San Diego, CA 92121 USA Email: geertv@qti.qualcomm.com Muhammed Coban Qualcomm Technologies Inc. 5775 Morehouse Drive San Diego, CA 92121 USA Email: mcoban@qti.qualcomm.com Van der Auwera & Coban Expires April 13, 2014 [Page 10]