The GDELT Project

Announcing The Global Entity Graph’s (GEG) G1 Baseline Dataset: 8 Billion Entities From 2019

Kalev Leetaru

Today we are excited to announce the GDELT Global Entity Graph’s (GEG) second release: the G1 Baseline Dataset, a dataset comprised of more than 8 billion entity annotations computed from 83 million English-language worldwide online news articles monitored by GDELT since January 1, 2019 and updated every 15 minutes through present. Each article is annotated through a grammar-based extractor applied to a classical HMM POS tagger, with both common and proper maximal entities, contextualized counts and date references extracted. The end result is a reasonably accurate classical annotation dataset that covers the majority of English-language coverage monitored by GDELT each day and offers a baseline dataset to compare against the rest of the current and future GEG datasets.

The GEG G1 Baseline Dataset comprises the entirely of all worldwide English-language online news coverage monitored by GDELT since January 1, 2019. The only exclusion at present is an initial language model filter that is applied to each article to determine whether it will yield reasonable-quality Part of Speech tagging results from the Hidden Markov model tagger used here. This means that a small percentage of English language coverage is not represented in this dataset and the GDELT Global Knowledge Graph (GKG) still represents the canonical exhaustive annotation catalog.

Each article is tagged and processed through a specially crafted extraction grammar to identify the four classes of entities it currently recognizes. At this time common and proper entities are extracted as-is, though in future a postprocessing language model may be applied to refine the results and remove common error cases.

Common and proper entity names are extracted in maximal form, rather than subdivided into discrete component entities. Thus, “Kenyan Director of Health Operations Jane Doe” would be treated as a single entity rather than as three three separate entities “Kenyan”, “Director of Health Operations” and “Jane Doe” in order to preserve the semantic connection among the three. This is a notable difference between this dataset and other entity extractors and must be accounted for in analyses. In this case we believe preserving these linkages is more important, since entity subdivision can be performed in a postprocessing step in which the semantic relationships among the subentities can be properly preserved.

For all four entities, a “salience” score is calculated that represents its overall “importance” to the document. At present this score is based on the word position within the document of all mentions of each entity under the journalistic inverse pyramid rule that words mentioned more often and earlier in an article are more important than those mentioned sparingly at the end. Thus, an entity mentioned once in the lead of an article might be considered more salient than one mentioned three times in the last sentence. In future this may be updated to incorporate additional signals. All scores are normalized to 0.0 (little salience) to 1.0 (high salience).

For numeric entities, the text immediately following the number is analyzed to determine whether it provides information that would help understand what the number represents and if so, up to four words of this is represented in the “context” field. Thus, an article mentioning “There were 200 dead and 30 injured” will yield “200” with context of “dead” and “30” with context of “injured” helping to lend understandability to numeric references, rather than just extracting “200” and “30” without any context as to what those counts represent.

The goal of the GEG G1 Baseline Dataset is to provide a maximal-coverage classical baseline that offers reasonable entity extraction quality across as much of GDELT’s monitored coverage as possible and with sufficiently high quality and coverage that it can form the basis of research as-is and act as a baseline against which newer approaches, especially neural systems, can be compared.

The dataset is structured as a set of UTF8 newline-delimited JSON files at 15-minute resolution from January 1, 2019 through present in the format "http://data.gdeltproject.org/gdeltv3/geg_g1/YYYYMMDDHHMMSS.geg-g1.json.gz". Due to the dataset's experimental nature, there may be some brief gaps.

Within each file, each row represents a given article, with the date/time the article was seen (precise to the second GDELT saw the article), the URL of the article and a JSON array containing the list of entities identified by the API. The JSON structure of each row is as follows:

Note that the URLs found within a given 15-minute file historically have aligned with those found in the GKG for the same period, but in future this will be increasingly decoupled as GDELT 3.0 launches, meaning that analyses looking across both GKG and GEG annotations will need to use a rolling window to match them.

You can download all of the GEG-G1 files from:

You can check the LASTUPDATE.TXT file every 5 minutes to see when the most recent file is ready:

The dataset is also available in Google's BigQuery:

We are incredibly excited to see what you're able to do with this extraordinary new dataset!