EvaSy Evaluation Package

registration: 2008/03/25 07:28:39, last modified: 2016/04/01 08:55:20
title
  • EvaSy Evaluation Package
title.abbreviation
title.alternative
  • Package d’évaluation EvaSy
creator
  • developer EVALDA
  • researcher Khalid Choukri, ELDA (choukri@elda.org)
  • researcher Nicolas Moreau, ELDA (moreau@elda.org)
  • researcher Christophe d’Alessandro, LIMSI - Unité Propre de Recherche (cda@limsi.fr)
subject
  • e-mail text dataset
  • proper name dataset
  • phonetically balanced dataset
  • speech dataset
subject.linguisticField
  • computational_linguistics
  • phonetics
  • phonology
subject.monoMultilingual
  • monolingual
subject.resourceSubject
  • corpus
description
  • Desktop/Microphone
    The EvaSy Evaluation Package was produced within the French national project EvaSy (Evaluation of speech synthesis systems), as part of the Technolangue programme funded by the French Ministry of Research and New Technologies (MRNT). The EvaSy project enabled to carry out a campaign for the evaluation of speech synthesis systems using French text data. This project is an extension of the only campaign that was ever carried out for French in this field within the AUPELF campaigns (Actions de recherche Concertées, 1996-1999).

    This package includes the material that was used for the EvaSy evaluation campaign. It includes resources, protocols, scoring tools, results of the campaign, etc., that were used or produced during the campaign. The aim of these evaluation packages is to enable external players to evaluate their own system and compare their results with those obtained during the campaign itself.

    The campaign is distributed over three actions:
    1) Evaluation of grapheme-to-phoneme conversion: it consists in evaluating the capacity of speech synthesis systems to phonetize text data.
    2) Evaluation of prosody: it consists in evaluating the capacity of speech synthesis systems to forecast text prosody (duration and fundamental frequency of phonemes) from the text itself.
    3) Global evaluation of the quality of speech synthesis systems:
    - ACR tests (Absolute Category Rating): they consist in evaluating the overall quality of speech synthesis voices, by asking a number of subjects to evaluate several general characteristics of the speech synthesis voice, such as its naturalness, its fluency, its intelligibility.
    - SUS tests (Semantically Unpredictable Sentences): they consist in evaluating the intelligibility of the speech synthesis voice, by using syntactically correct as well as semantically unpredictable sentences (which have no meaning).

    The EvaSy evaluation package contains the following data and tools:
    1) For the evaluation of the grapheme-to-phoneme conversion module:
    1) About 8,000 proper names (4,115 pairs firstname-surname) were extracted from Le Monde newspaper of 1992–2000 (over 200 million words), manually phonetised with variants and annotated with linguistic tags. The reference phonetisation was checked and corrected after the adjudication phase.
    2) A corpus of emails (about 115,000 words) anonymised, segmented by paragraph and phonetised in SAMPA. The reference phonetisation was not checked. The evaluation of thos data was not carried out within EvaSy.
    3) The SCLITE tool (developed by NIST) was used to compare the reference phonetisation with the one from the evaluated system, and to calculate the number of mistaken phonemes (inserted, forgotten or substituted phonemes).
    4) The Post-align tool was used to align the reference phonetisation with the one from the evaluated system on a word-by-word basis.

    2) For the evaluation of the prosodic module:
    - Text data: 7 phonetically-balanced sentences extracted from the BREF corpus (cf. ELRA-S0067), with a duration lasting from 4 to 11 seconds.
    - Speech data: 7 sentences read by one speaker.
    - The Mbroli tool, which converts *.pho prosodic files into *.wav speech files, together with the MBROLA fr1 diphone database.
    - The Mbrolign tool, which aligns the phonemes with the signal, extracts the prosodic parameters of the signal and copy them in the MBROLA diphone databas.

    3) For the global evaluation of the quality of speech synthesis systems:
    a) For ACR tests (Absolute Category Rating):
    - Text data: 40 abstracts with 5 sentences each of 20 second duration, extracted from the EUROM1f French corpus (cf. ELRA-S0014-01).
    - Données audio : lecture des 40 passages par un locuteur EUROM1.
    b) For SUS tests (Semantically Unpredictable Sentences):
    - Text data: 24 lists of 12 SUS sentences. Phonemes are also distributed by list.
    - Speech data: 24 lists read by a professional speaker.

    A description of the project is available at the following address:
    http://www.technolangue.net/article.php3?id_article=202 (in French language)
description.price *This metadata is only as a guide
description.language
  • French
description.inputDevice
  • microphone
description.inputEnvironment
description.speakingStyle
  • read_speech
description.speechMode
  • monologue
description.samplingRate
description.additionalData
publisher
contributor
  • sponsor French Ministry of Research and New Technologies (MRNT)
contributor.attribute.author.motherTongue
  • native
contributor.attribute.author.dialect
  • standard_dialect
contributor.attribute.author.level
  • professional
  • amateur
contributor.attribute.author.age
  • adult
contributor.attribute.author.gender
  • male
  • female
contributor.attribute.speaker.motherTongue
  • native
contributor.attribute.speaker.dialect
  • standard_dialect
contributor.attribute.speaker.level
  • professional
  • amateur
contributor.attribute.speaker.age
  • adult
contributor.attribute.speaker.gender
  • male
  • female
contributor.attribute.speaker.number
  • number_of_total_speakers 2
date.created
  • 2002-12-04 2006-03-00
date.issued
  • 2007-06-28
date.modified
  • 2007-06-28
type
  • Sound
  • Text
type.discourseType
  • report
  • narrative
type.linguisticType
  • primary_text
type.purpose
  • developing_technologies speech synthesis
type.style
  • speech reading
type.form
  • fixed numbers
  • fixed CVC
  • fixed semantically unpredictable sentences
  • fixed phonetically balanced sentences
  • fixed proper names
  • unfixed
type.sentence
  • isolated_words numbers, proper names
  • short
  • long
type.annotation
  • annotated manually phonetised with variants in SAMPA (proper names)
  • annotated linguistic origin tags (proper names)
  • annotated anonymised, segmented by paragraph and phonetised in SAMPA (e-mails)
type.annotationSample
type.sample
format.extent
  • Number of words (proper name data) : about 8,000 proper names (4,115 pairs firstname-surname)
  • Number of words (e-mail data) : about 115,000 words
  • Text data for prosodic module evaluation : 7 phonetically-balanced sentences/speaker, lasting from 4 to 11 seconds/sentence
  • Speech data for prosodic module evaluation : 7 sentences read by one speaker
  • Text data for global evaluation of the quality of speech synthesis systems' ACR tests : 40 abstracts with 5 sentences each of 20 second duration
  • Speech data for global evaluation of the quality of speech synthesis systems' ACR tests : 40 passages read by a speaker
  • Text data for global evaluation of the quality of speech synthesis systems' SUS tests : 24 lists of 12 SUS sentences
  • Speech data for global evaluation of the quality of speech synthesis systems' SUS tests : 24 lists read by a speaker
format.medium
format.encoding
format.markup
format.functionality
  • The SCLITE tool :
    Developed by NIST. For comparing the reference phonetisation with the one from the evaluated system, and for calculating the number of mistaken phonemes (inserted, forgotten or substituted phonemes). Included in the package.
  • The Post-align tool :
    Aligns the reference phonetisation with the one from the evaluated system on a word-by-word basis. Included in the package.
  • The Mbroli tool :
    Converts *.pho prosodic files into *.wav speech files. Included in the package, together with the MBROLA fr1 diphone database.
  • The Mbrolign tool :
    Aligns the phonemes with the signal, extracts the prosodic parameters of the signal and copys them in the MBROLA diphone databas. Included in the package.
identifier
identifier.doi
identifier.islrn
  • 340-228-754-954-9
source
  • (proper name dataset for the grapheme-to-phoneme conversion module evaluation) : Le Monde newspaper of 1992–2000
  • (phonetically balanced sentences for the prosodic module evaluation) : C-000887,BREF-120 - A large corpus of French read speech (ELRA-S0067)
  • (text data for the quality of speech synthesis systems evaluation) : C-000915,EUROM1f (ELRA-S0014-01)
language
  • French
language.area
  • Europe
relation
relation.utilization *This metadata is automatically extracted
coverage.temporal
  • 2002-12-04 - 2006-03 (project period)
  • 1992 - 2000 (Le Monde articles)
coverage.spacial
  • France, Europe
rights