ARL Subjective Quality Evaluation Services

Overview

ARL staff have more than 20 years of experience in conducting subjective listening tests for evaluating the performance of speech and audio coding or processing technologies. We can perform tests on mono, stereo or 5.1 channel content, and presentation can be via either headphones or loudspeakers. Headphone listening is done in acoustically controlled IAC sound booth, while loudspeaker listening is done in a acoustically isolated, double-wall, BS-1116 qualified listening room. We have created our own software for computer-controlled presention of the stimulus to the subject and recording of the listener response (see STEP product information page).

ARL can conduct the following types of tests:

• BS.1116: Appropriate for evaluating high quality systems with very small impairments
• MUSHRA: Appropriate for evaluaing systems with medium impairments
• Comparative: A pair-wise comparisons methodology that permit differentiation of systems with very similar quality
• A/B/X: A pair-wise comparison methodology whose outcome is: transparent or not transparent?

ARL can perform all aspects of the listening test, including:

• Design of listening test: Determining the systems under test and the appropriate anchors, the operating points and the number of listeners so that the test will produce the desired information.
• Selection of test items: Select items from a larger list of candidate items such that the selected items "stress" the systems under test.
• Preparation of test items: Coding audio and training items and anonymizing the item names.
• Recruiting listeners:
• Creation of STEP scripts: Preparing STEP training and testing session files. Training is essential to insure that listeners have similar sensitivity to impairments. The test must be designed so that presentation is balanced within each session and each session is of appropriate length.
• Analysis of subjective data: This can a simple pooling of data and computation of means and associated 95% confidence intervals, or ANOVA which takes into account factors and interactions.
• Generation of test report: The experimental design, systems under test, test setup and test results are documented in a formal test report. There can be an in-person presentation of the results if desired.

Recent Work

Date	Project
1998 - Current	Client: MPEG Audio Subgroup Project: In my capacity as Chair of the MPEG Audio Subgroup I have participated in numerous subjective listening test activities. These efforts entail: design of listening test preparation of test items creation of STEP scripts for the test collection of subjective scores analysis of pooled scores via Excel pivot table generation of test report
2008	Client: Music Industry Trade Group Project: Conducted listening test to assess impact of steganographic processing on audio signals. Project consisted of designing test, identifying test location, recruiting listeners, conducting test, analyzing data and writing final report.
2006	Client: Major US Telecommunications Equipment Manufacturor Project: Conducted a subjective test to assess the audio quality of a set of six wide band telephone handsets, including two client handsets. Subjective quality was assessed via live, unscripted conversations between two subjects in isolated test rooms for all combination of handset and speakerphone operation. Tasks included designing test, recruiting subjects, administering subjective test, analysis of subjective scores, generation of final report and presentation of report to client.
2006	Client: Southwestern US technology company Project: Conducted a subjective test to assess the impact of watermarking technology on motion picture sound tracks. All test items were audio-visual. Tasks included designing test, administering test, analysis of subjective scores and generation of final report. Tests were conducted at a major motion picture audio mixing theatre and subjects were either professional re-mix engineers or experts in theatre sound.
2003 - 2004	Client: 3GPP (Third Generation Partnership Project) Project: Prepare speech and music test items for a very large subjective listening test conducted under the supervision of 3GPP working group TSG-SA WG4. The test comprised 8 low bit rate listening tests and 3 high bit rate listening tests. The item preparation report is 3GPP document S4-040026.
2003 - 2004	Client: 3GPP (Third Generation Partnership Project) Project: Conduct data analysis and generate final report for a very large subjective listening test conducted under the supervision of 3GPP working group TSG-SA WG4. The test comprised 8 low bit rate listening tests at 8 test sites and 3 high bit rate listening tests at 6 test sites. The final test reports are 3GPP documents S4-040099 and S4-040028.
2004	Client: MPEG Audio Subgroup Project: Prepare audio test items for a large subjective listening test conducted under the supervision of ISO/IEC WG11 (MPEG) Audio Subgroup. The results of the test were used to determine the core technology for an MPEG work item that would become MPEG Surround. The test consisted of five separate tests, each evaluating different operating modes. Three were 5.1 channel presentations and two were stereo presentations. In each test there were 8 systems under test evaluated at 7 test sites comprising a total of more than 275 listeners. My specific tasks were preparation of test items, creation of STEP scripts for the test , collection of subjective scores from the various test sites, analysis of pooled scores via Excel pivot table and generation of a written test report.