Annotated English Gigaword

registration: 2013/01/15 04:46:57, last modified: 2016/02/14 13:32:33
title
  • Annotated English Gigaword
title.abbreviation
title.alternative
  • Annotated Gigaword
creator
  • researcher Human Language Technology Center of Excellence, Johns Hopkins University
  • researcher Courtney Napoles
  • researcher Matthew R. Gormley
  • researcher Benjamin Van Durme
subject
  • syntactically annotated corpus
subject.linguisticField
  • computational_linguistics
  • discourse_analysis
  • syntax
subject.monoMultilingual
  • monolingual
subject.resourceSubject
  • corpus
description
  • *Introduction*

    Annotated English Gigaword was developed by Johns Hopkins Universitys Human Language Technology Center of Excellence. It adds automatically-generated syntactic and discourse structure annotation to English Gigaword Fifth Edition (LDC2011T07) and also contains an API and tools for reading the datasets XML files. The goal of the annotation is to provide a standardized corpus for knowledge extraction and distributional semantics which enables broader involvement in large-scale knowledge-acquisition efforts by researchers.

    *Data*

    Annotated English Gigaword contains the nearly ten million documents (over four billion words) of the original English Gigaword Fifth Edition from seven news sources:

    * Agence France-Presse, English Service (afp_eng)
    * Associated Press Worldstream, English Service (apw_eng)
    * Central News Agency of Taiwan, English Service (cna_eng)
    * Los Angeles Times/Washington Post Newswire Service (ltw_eng)
    * Washington Post/Bloomberg Newswire Service (wpb_eng)
    * New York Times Newswire Service (nyt_eng)
    * Xinhua News Agency, English Service (xin_eng)

    The following layers of annotation were added:

    * Tokenized and segmented sentences
    * Treebank-style constituent parse trees
    * Syntactic dependency trees
    * Named entities
    * In-document coreference chains

    The annotation was performed in a three-step process: (1) the data was preprocessed and sentences selected for annotation (sentences with more than 100 tokens were excluded) (2) syntactic parses were derived and (3) the parsed output was post-processed to derive syntactic dependencies, named entities and coreference chains. Over 183 million sentences were parsed.

    The data is stored in a form similar to the gigaword SGML format with XML annotations containing the additional markup. The included API provides object representations for the contents of the XML files.

    *Samples*

    Please the link for a sample.

    *Additional Licensing Information*

    Any 2011 member organization that licensed English Gigaword Fifth Edition (LDC2011T07) may request a no-cost copy of Annotated English Gigaword. Any non-member organization that licensed English Gigaword Fifth Edition may request a copy of Annotated English Gigaword for a $250 media fee. Please contact ldc@ldc.upenn.edu for licensing or with any additional questions.

    *Updates*

    None at this time.
description.price *This metadata is only as a guide
description.language
  • English
description.inputDevice
description.inputEnvironment
description.speakingStyle
description.speechMode
description.samplingRate
description.additionalData
publisher
contributor
contributor.attribute.author.motherTongue
  • native Agence France-Presse, English Service
  • native Associated Press Worldstream, English Service
  • native Central News Agency of Taiwan, English Service
  • native Los Angeles Times/Washington Post Newswire Service
  • native Washington Post/Bloomberg Newswire Service
  • native New York Times Newswire Service
  • native Xinhua News Agency, English Service
contributor.attribute.author.dialect
  • standard_dialect
contributor.attribute.author.level
  • professional newspaper writers
contributor.attribute.author.age
  • adult
contributor.attribute.author.gender
  • male
  • female
contributor.attribute.speaker.motherTongue
contributor.attribute.speaker.dialect
contributor.attribute.speaker.level
contributor.attribute.speaker.age
contributor.attribute.speaker.gender
contributor.attribute.speaker.number
date.created
date.issued
  • 2012-11-15
date.modified
type
  • Text
type.discourseType
  • report
type.linguisticType
  • primary_text
type.purpose
  • analysis
  • developing_technologies
type.style
  • written newswire
type.form
  • unfixed
type.sentence
  • short
  • long
type.annotation
  • annotated tokenized and segmented sentences, treebank-style constituent parse trees, syntactic dependency trees, named entities, and in-document coreference chains
type.annotationSample
type.sample
format.extent
  • Data size: nearly ten million documents (over four billion words)
  • Corpus size: 164392369 KB
format.medium
  • Hard Drive
format.encoding
format.markup
  • SGML
  • XML
format.functionality
  • A Java API for providing streaming access to the documents in the .xml.gz files
  • Command line tools for displaying human readable versions of the XML annotations
identifier
identifier.doi
identifier.islrn
  • 335-916-789-872-0
source
  • English Gigaword Fifth Edition
language
  • English
language.area
  • Europe
  • America
relation
  • references English Gigaword Fifth Edition
relation.utilization *This metadata is automatically extracted
coverage.temporal
  • 1994-2010/12
coverage.spacial
rights
  • Portions Copyrigh: © 1994-2010 Agence France Presse
  • Portions Copyrigh: © 1994-2010 The Associated Press
  • Portions Copyrigh: © 1997-2010 Central News Agency (Taiwan)
  • Portions Copyrigh: © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc.
  • Portions Copyrigh: © 1994-2010 New York Times
  • Portions Copyrigh: © 2010 The Washington Post News Service with Bloomberg News
  • Portions Copyrigh: © 1995-2010 Xinhua News Agency
  • Portions Copyrigh: © 2012 Matthew R. Gormley
  • Portions Copyrigh: © 2003, 2005, 2007, 2009, 2011, 2012 Trustees of the University of Pennsylvania
  • Licensing Instructions for Subscription & Standard Members, and Non-Members: http://www.ldc.upenn.edu/language-resources/data/obtaining