CESTA Evaluation Package

registration: 2008/03/24 07:49:56, last modified: 2016/01/17 10:20:25
title
  • CESTA Evaluation Package
title.abbreviation
title.alternative
  • Package d’évaluation CESTA
creator
  • developer EVALDA
  • researcher Olivier Hamon, ELDA, France (hamon@elda.org)
  • researcher Khalid Choukri, ELDA, France (choukri@elda.org)
  • researcher Widad Mustafa El Hadi, CERSATES, Université de Lille (mustafa@univ-lille3.fr)
subject
  • bilingual parallel corpus
  • adequacy/fluency annotated corpus
subject.linguisticField
  • computational_linguistics
  • translating_and_interpreting
subject.monoMultilingual
  • multilingual
subject.resourceSubject
  • corpus
description
  • Written Corpora
    The CESTA Evaluation Package was produced within the French national project CESTA (Evaluation of MT systems), as part of the Technolangue programme funded by the French Ministry of Research and New Technologies (MRNT). The CESTA project enabled to carry out a campaign for the evaluation of machine translation systems with English and Arabic texts translated into French.

    This package includes the material that was used for the CESTA evaluation campaign. It includes resources, protocols, scoring tools, results of the campaign, etc., that were used or produced during the campaign. The aim of these evaluation packages is to enable external players to evaluate their own system.

    The campaign is distributed over two actions:
    1) Evaluation on a restrictive vocabulary: an evaluation protocol was introduced and was dedicated to two translation directions: English into French and Arabic into French.
    2) Evaluation on a specialised domain (evaluation after terminology enrichment): it consists in observing the impact of the systems adaptation to the specialised domain.

    The CESTA evaluation package contains the following data and tools:
    1) Test run data:
    - English-French parallel corpus: 21,590 English words and 23,554 French words extracted from the Official Journal of the European Communities, 1993, Written Questions section of the European Parliament, from the MLCC corpus (catalogue ref. ELRA-W0023).
    - Arabic-French parallel corpus: 15,603 Arabic words and 18,257 French words extracted from Le Monde Diplomatique 2002 (catalogue ref. ELRA-W0036).

    2) First campaign data:
    - English-French parallel corpus: test corpus of 20,658 English words and 22,774 French words extracted from the Official Journal of the European Communities, 1993, Written Questions section of the European Parliament, from the MLCC corpus (catalogue ref. ELRA-W0023). Four translations in French are available.
    - Arabic-French parallel corpus: test corpus of 23,763 Arabic words and 28,664 French words extracted from Le Monde Diplomatique 2002 and 2003 (catalogue réf. ELRA-W0036). Four translations in French are available.

    3) Second campaign data:
    - English-French parallel corpus: adaptation corpus of 19,383 English words and 22,741 French words, extracted from the Santé Canada website. Translation in French is available.
    - Arabic-French parallel corpus: adaptation corpus of 19,560 Arabic words and 22,533 French words extracted from the UNICEF, WHO and FHI websites. Translation in French is available.
    - English-French parallel corpus: test corpus of 18,880 English words and 23,411 French words, extracted from the Santé Canada website. Four translations in French are available.
    - Arabic-French parallel corpus: test corpus of 17,305 Arabic words and 20,885 French words extracted from the UNICEF, WHO and FHI websites. Four translations in French are available.

    4) Anonymised submissions of systems and human judgments with adequacy and fluency annotations.
    5) French corpus of 13,000 words with adequacy and fluency tags.
    6) Evaluation infrastructure for human judgments and for automatic evaluation.
    7) Project documentation and publications.

    A description of the project is available at the following address:
    http://www.technolangue.net/article.php3?id_article=199 (in French language)
description.price *This metadata is only as a guide
description.language
  • French
description.inputDevice
description.inputEnvironment
description.speakingStyle
description.speechMode
description.samplingRate
description.additionalData
publisher
  • ELRA (European Language Resources Association)
contributor
  • sponsor French Ministry of Research and New Technologies (MRNT)
  • research_participant IDIS/CESARTES, University of Lille 3, France (http://www.univ-lille3.fr/fr/recherche/equipes-recherche/geriico/)
  • research_participant Artificial Intelligence Laboratory (LIA), Swiss Federal Institute of Technology, Lausanne, Switzerland (liawww.epfl.ch)
  • research_participant Centre for Translation Studies, University of Leeds, UK (http://www.leeds.ac.uk/)
  • research_participant Temis S.A.
  • research_participant Systran S.A.
  • research_participant Softissimo S.A.
  • research_participant CIMOS S.A.
  • research_participant IMAG, University of Grenoble, France
  • research_participant Department of Linguistics and Translation, University of Montreal, Canada (http://www.ling.umontreal.ca/)
  • research_participant RALI, University of Montreal, Canada
  • research_participant ISSCO, University of Geneva (http://www.issco.unige.ch/)
  • research_participant RWTH University of Aachen, Germany (http://www.rwth-aachen.de/go/id/bdz/)
  • research_participant Technical University of Catalonia (UPC), Barcelona, Spain (http://www.upc.edu/indexEng.htm)
  • research_participant SDL International
  • research_participant Comprendium S.L.
contributor.attribute.author.motherTongue
contributor.attribute.author.dialect
contributor.attribute.author.level
contributor.attribute.author.age
contributor.attribute.author.gender
contributor.attribute.speaker.motherTongue
contributor.attribute.speaker.dialect
contributor.attribute.speaker.level
contributor.attribute.speaker.age
contributor.attribute.speaker.gender
contributor.attribute.speaker.number
date.created
  • 2002-12-04 2006-03-00
date.issued
  • 2007-06-28
date.modified
  • 2007-06-28
type
  • Text
type.discourseType
  • report
type.linguisticType
  • primary_text
type.purpose
  • developing_technologies machine translation
type.style
  • written questions
  • written newspaper
  • written website
type.form
  • fixed questions
  • unfixed
type.sentence
  • short
  • long
type.annotation
  • annotated adequacy and fluency tags
type.annotationSample
type.sample
format.extent
  • Number of words - Test run data - English-French parallel corpus : 21,590 English words and 23,554 French words
  • Number of words - Test run data - Arabic-French parallel corpus : 15,603 Arabic words and 18,257 French words
  • Number of words - First campaign data - English-French parallel corpus : 20,658 English words and 22,774 French words
  • Number of words - First campaign data - Arabic-French parallel corpus : 23,763 Arabic words and 28,664 French words
  • Number of words - Second campaign data (adaptation corpus) - English-French parallel corpus : 19,383 English words and 22,741 French words
  • Number of words - Second campaign data (adaptation corpus) - Arabic-French parallel corpus : 19,560 Arabic words and 22,533 French words
  • Number of words - Second campaign data (test corpus) - English-French parallel corpus : 18,880 English words and 23,411 French words
  • Number of words - Second campaign data (test corpus) - Arabic-French parallel corpus : 17,305 Arabic words and 20,885 French words
  • Number of words in French corpus with adequacy and fluency tags : 13,000 words
format.medium
format.encoding
format.markup
format.functionality
identifier
identifier.doi
identifier.islrn
source
  • (For English-French test run/first campaign data) : the Official Journal of the European Communities, 1993, Written Questions section of the European Parliament, from MLCC Multilingual and Parallel Corpora (MLCC) (SHACHI-ID: C-001465, catalogue ref. ELRA-W0023)
  • (For Arabic-French test run/first campaign data) : Le Monde Diplomatique 2002 and 2003 (catalogue ref. ELRA-W0036)
  • (For English-French second campaign data) : data extracted from the Santé Canada website
  • (For Arabic-French second campaign data) : data extracted from the UNICEF, WHO and FHI websites
language
  • French Standard French spoken in France and Canadian French
  • English
  • Arabic
language.area
  • Europe
  • Asia Middle East
  • America Canada, North America
relation
relation.utilization *This metadata is automatically extracted
coverage.temporal
  • 2002 - 2006 (project period)
  • 1993 (English-French, test run and first campaign)
  • 2002 - 2003 (Arabic-French, test run and first campaign)
coverage.spacial
  • Europe
  • France, Europe
  • Canada, North America
rights