							23 Oct 2014


LISTENING TEST ON THE CLASSIFICATION OF ERRORS FOR A LOW-QUALITY 
SPEECH SYNTHESIS VOICE
(Data release from Simple4all project)


PURPOSE: 
This data was collected as a first step into investigation on spoken 
user feedback. The purpose is to investigate listeners' agreement on 
the types of mistakes that a synthetic voice of poor quality would 
do.

CONTENTS:
The data includes:
- synthetic speech waveforms generated from a model set trained with 
  300 sentences of speech from "Roger"-database used in the Blizzard 
  Challenge 2009;
- a demonstration web page of the listening test procedure;
- listening test results. The listening test was run on 200 sentences 
  with 34 listeners. Each listener listened to and classified 100 
  sentences into categories:
     a) Ok: Quality is good; 
     b) Ok: It's not great but it will do; 
     c) Not ok: Mispronunciation of word(s); 
     d) Not ok: Incomprehensible segments; 
     e) Not ok: Bad rhythm or intonation; 
     f) Not ok: Bad audio quality (artifacts etc.).
- a preliminary analysis of the listening test results

SAMPLES:
A demonstration web page of the test arrangement is available (as of 
October 2014) online in 

http://users.ics.aalto.fi/rkarhila/training_data_quality_listening_test/sampletestpage.html. 

This can also be found in the directory "demo_wwwpage"

LICENSE:
The synthetic speech samples are made public under the licensing and 
copyright terms of the Creative Commons Attribution 3.0 Unported 
License (CC-BY 3.0). The listening test results are free to use, but 
for any published work a citation to the preliminary report is 
expected.

USER GUIDELINES
The listening test results are in the directory "results." The file 
"results.txt" lists the ratings. On each line, there is information 
on the listener (age, gender, native accent) and ratings for the 
sentences this listener has evaluated. A minus-sign "-" marks the 
sentences this listener has not evaluated. The file "quality_key.txt" 
contains mapping from numbers in the results file to the category 
names used in the listening test.

DOWNLOAD
The dataset is available according the the license described above 
and is available from Simple4all data release repositories.


AUTHORS
Listening test designed by Reima Karhila (Aalto University School of 
Electric Engineering) and run by Rob Clark (University of Edinburgh 
Centre for Speech Technology Research).

Any questions about the data should be directed to 
reima.karhila@aalto.fi.
