For those of you wondering what Turkey's attempted coup looked like through GDELT's eyes, the timeline below shows the sudden massive vertical leap visible through the GDELT Stability API timeline for Turkey.
The Stability API timeline shows the first upward movement around 20:00GMT with things really taking off by 20:45GMT. The instability level as assessed by GDELT hits peak at around 00:15GMT and moves sharply down towards greater stability by 1:00GMT, reflecting the arrival earlier that hour of President Erdogan's plane and rapidly growing reports that the coup plotters were losing control of the country.
When using GDELT to assess any fast moving realtime situation, it is important to remember that GDELT currently uses a deep multi-stage processing pipeline operating on a 15 minute heartbeat, with content proceeding from stage to stage of the pipeline at each heartbeat. According to Reuter's timeline of events at 19:29GMT the first reports emerged of the military closing two key bridges, with social media beginning to show scattered images of the closed bridges and military movements around this time as well. Many of these earliest media reports emerged around this time and slightly after 19:30. Thus, GDELT's ingest pipeline primarily captured this content during its 19:45-20:00 cycle. Foreign language material was then forwarded to the GDELT Translingual infrastructure for machine translation where it was translated from 20:00 to 20:15. After translation it was then forwarded for processing from 20:15 to 20:30. From 20:30 to 20:45 it was then processed by the GDELT API infrastructure, which includes the Stability API timeline, which resulted in the final data point reflecting the massive military mobilization being reflected in the Stability timeline at 20:45GMT.
Given the absolutely massive amount of computation required to go from source articles to final point on the timeline below, even this delay is incredible. Yet, here at GDELT we've been busily working away at the next generation of the GDELT architecture which will roll out over the coming year and move GDELT away from its current 15 minute heartbeat and towards a continuous flow architecture in which content is ingested, translated and processed individually and effectively in true realtime, with tens of seconds to a few minutes minutes separating publication from final data point in the GDELT output streams. Obviously building architectures at these scales and the sheer magnitude of data points GDELT computes from all this data (three quarters of a trillion emotional scores from 2015 alone) requires some incredible architectural workflows and we are slowly laying all of this in and will be rolling it out bit by bit over the coming year.
We are also exploring a shortcircuit infrastructure for the stability assessment APIs that will perform low-level processing of each incoming document at time of ingest to cut out even those tens of seconds to provide truly physical realtime early alerting. Stay tuned for further developments there as well.
Finally, stay tuned for an immensely exciting new announcement coming later this summer regarding an incredible new API designed to pick up the earliest glimmers of "something unusual is happening" worldwide, which picked up some of the earliest reports of strange movements of troops and military hardware leading up to the actual start of the coup. Such alerting should provide some of the earliest possible warnings that something is about to happen in the immediate future.
Stay tuned!
