ESTER Evaluation Package

registration: 2008/03/27 07:44:04, last modified: 2016/04/02 09:07:49
title
  • ESTER Evaluation Package
title.abbreviation
title.alternative
  • Package d’évaluation ESTER
creator
  • developer EVALDA
  • researcher Khalid Choukri, ELDA, France (choukri@elda.org)
  • researcher Jean-François Bonastre, LIA, France (jean-francois.bonastre@lia.univ-avignon.fr)
  • researcher Guillaume Gravier, AFCP - IUP - Université d'Avignon, France (ggravier@irisa.fr)
  • researcher Edouard Geoffrois, CTA/GIP, France (Edouard.Geoffrois@etca.fr)
subject
  • speech corpus
  • named entity annotated transcripts
  • newspaper article database
  • spontaneous speech transcripts
subject.linguisticField
  • phonetics
  • computational_linguistics
subject.monoMultilingual
  • monolingual
subject.resourceSubject
  • corpus
description
  • Broadcast Resources
    The ESTER Evaluation Package was produced within the French national project ESTER (Evaluation of Broadcast News enriched transcription systems), as part of the Technolangue programme funded by the French Ministry of Research and New Technologies (MRNT). The ESTER project enabled to carry out a campaign for the evaluation of Broadcast News enriched transcription systems using French data. This project is an extension of the only campaign that was ever carried out for French in this field within the AUPELF campaigns (Actions de recherche Concertées, 1996-1999).

    This package includes the material that was used for the ESTER evaluation campaign. It includes resources, protocols, scoring tools, results of the campaign, etc., that were used or produced during the campaign. The aim of these evaluation packages is to enable external players to evaluate their own system and compare their results with those obtained during the campaign itself.

    The campaign is distributed over three actions:
    1) Orthographic transcription: it consists in producing an orthographic transcription of radio-broadcast news, which quality is measured by word error rates. There are two distinct tasks, one with and one without calculation time constraint.
    2) Segmentation: the segmentation tasks consist of segmentation in sound events, speaker tracking and speaker segmentation. For the sound event segmentation, the task consists of tracking the parts which contain music (with or without speech) and the parts which contain speech (with or without music). The speaker tracking task consists in detecting the parts of the document that correspond to a given speaker. The speaker segmentation consists of segmenting the document in speakers and grouping the parts spoken by the same speaker.
    3) Information extraction: it consists of an exploratory task on named entity tracking. The objective was to set up and test an evaluation protocol instead of measure performances. The systems must detect eight classes of entities (person, place, data, organisation, geo-political entity, amount, building and unknown) from the automatic transcription or the manual transcription.

    The ESTER evaluation package contains the following data and tools:
    1) About 100 hours of orthographically transcribed news broadcast, including annotations of named entities.
    2) The textual resources distributed within the ESTER campain are mainly based on the archives from Le Monde newspaper 1987-2003 (ELRA-W0015) and the debates from the European Parliament (ELRA-W0023).
    3) The evaluation tools allow to evaluation each task defined above.
    4) Two guides and manuals were produced and are provided in the package distributed by ELDA :
    o Guide for the annotation of named entities
    o Specifications and evaluation protocol

    A description of the project is available at the following address:
    http://www.technolangue.net/article.php3?id_article=60 (in French language)

    An extra corpus of 1,700 hours of non-transcribed radio broadcast news recordings can also be provided upon request, on hard disk, as an adding to this package at a cost of 100 Euro (plus shipment fee).

    For research or commercial use, please refer to ELRA-S0241 ESTER Corpus.
description.price *This metadata is only as a guide
description.language
  • French
description.inputDevice
  • broadcast
description.inputEnvironment
  • sound_proof_room radio broadcast studios
description.speakingStyle
  • read_speech broadcast news
description.speechMode
description.samplingRate
description.additionalData
  • analysis_parameter orthographic transcripts
publisher
contributor
  • sponsor French Ministry of Research and New Technologies (MRNT)
  • developer ELDA (http://www.elda.org/) (annotation guides)
contributor.attribute.author.motherTongue
  • native
contributor.attribute.author.dialect
  • standard_dialect
contributor.attribute.author.level
  • professional le Monde journalists
  • amateur European Parliament
contributor.attribute.author.age
  • adult
contributor.attribute.author.gender
  • male
  • female
contributor.attribute.speaker.motherTongue
  • native
contributor.attribute.speaker.dialect
  • standard_dialect
contributor.attribute.speaker.level
  • professional radio newscasters
contributor.attribute.speaker.age
  • adult
contributor.attribute.speaker.gender
  • male
  • female
contributor.attribute.speaker.number
date.created
  • 2002-12-04 2006-04-04
date.issued
  • 2007-06-28
date.modified
  • 2012-03-29
type
  • Sound
  • Text
type.discourseType
  • report
  • interactive_discourse
type.linguisticType
  • primary_text
type.purpose
  • developing_technologies automatic transcription system
  • developing_technologies named entity recognition system
type.style
  • speech broadcast news
  • speech debate
type.form
  • unfixed
type.sentence
  • short
  • long
type.annotation
  • annotated named entities (person, place, data, organisation, geo-political entity, amount, building and unknown) based on the guides produced by ELDA
type.annotationSample
type.sample
  • [Automatic Transcription]:

    20041006_0700_0800_CLASSIQUE 1 12.520 0.150 le
    20041006_0700_0800_CLASSIQUE 1 12.670 0.620 gouvernement
    20041006_0700_0800_CLASSIQUE 1 13.290 0.040 et
    20041006_0700_0800_CLASSIQUE 1 13.330 0.050 l'
    20041006_0700_0800_CLASSIQUE 1 13.380 0.640 opposition
    20041006_0700_0800_CLASSIQUE 1 14.090 0.300 enterre
    20041006_0700_0800_CLASSIQUE 1 14.390 0.130 la
    20041006_0700_0800_CLASSIQUE 1 14.520 0.500 polémique
    20041006_0700_0800_CLASSIQUE 1 15.020 0.160 sur
    20041006_0700_0800_CLASSIQUE 1 15.180 0.110 la
    20041006_0700_0800_CLASSIQUE 1 15.290 0.360 mission
    20041006_0700_0800_CLASSIQUE 1 15.650 0.170 de
    20041006_0700_0800_CLASSIQUE 1 15.820 0.370 Didier
    20041006_0700_0800_CLASSIQUE 1 16.190 0.330 Julia
    20041006_0700_0800_CLASSIQUE 1 16.520 0.130 pour
    20041006_0700_0800_CLASSIQUE 1 16.650 0.350 libérer
    20041006_0700_0800_CLASSIQUE 1 17.000 0.190 les
    20041006_0700_0800_CLASSIQUE 1 17.190 0.320 otages
    20041006_0700_0800_CLASSIQUE 1 17.510 0.100 en
    20041006_0700_0800_CLASSIQUE 1 17.610 0.450 Irak


    The rest is omitted.

    http://catalog.elra.info/product_info.php?action=download&filename=SMP_E0021_TRS_AUTO.pdf&osCsid=b7274ef695a0cc8e159925ca076577f5
  • [Manual Transcription]:


    le gouvernement et l'opposition enterrent la polémique sur la mission de Didier Julia pour libérer les otages en Irak


    après une réunion à Matignon hier ils ont décidé de rejouer la carte de la cohésion sociale


    du moins dans l'attente de la libération de Christian Chesnot et Georges Malbrunot .


    le premier ministre n'en a pas moins condamné l'initiative parallèle du député


    UMP

    celui-ci s'est expliqué hier à huis clos devant la commission des affaires
    étrangères de l'Assemblée nationale .


    le groupe UMP


    évoque de possibles sanctions à l'encontre de Didier Julia


    The rest is omitted.

    http://catalog.elra.info/product_info.php?action=download&filename=SMP_E0021_TRS_MANUEL.pdf&osCsid=b7274ef695a0cc8e159925ca076577f5
format.extent
  • Duration of speech : 60 hours
format.medium
format.encoding
  • ISO-8859-1
format.markup
  • XML
format.functionality
  • Evaluation tools for evaluating tasks for ESTER (orthographic transcription, segmentation and named-entity extraction) are included.
identifier
identifier.doi
identifier.islrn
  • 110-079-844-983-7
source
  • radio broadcast news
  • C-001465,MLCC Multilingual and Parallel Corpora (MLCC) (Official Journal of the European Commission, Annex: Debates of the European Parliament 1992-1994, ELRA-W0023)
  • Text corpus of "Le Monde" (1987 - 2003, ELRA-W0015)
language
  • French
language.area
  • Europe
relation
relation.utilization *This metadata is automatically extracted
coverage.temporal
  • 1992 - 1994 (textual resource "European Parliament")
  • 1987 - 2003 (textual resource "Le Monde")
coverage.spacial
  • Europe
rights