Using AI to watch a decade of television evening news broadcasts poses fascinating and novel challenges when it comes to reconciling the frame-level precision of AI with the coarse airtime annotations historically used by human content analysts. When machines watch television, they catalog what they see frame-by-frame, issuing timestamps recorded in nanoseconds and visual offsets in pixels. Humans, on the other hand, typically deal in seconds of airtime and rough approximations of scene location.
This disconnect creates unique challenges in the fast-paced world of television news in which a single second of airtime can feature five entirely different scenes cut together in rapid succession. The machine precisely records the starting and ending points of each scene at the nanosecond resolution, while no human annotator could begin to be so precise at scale.
The core Visual Global Entity Graph 2.0 (VGEG 2.0) dataset is designed to closely align with traditional human content analysis, aggregating the frame-level AI output into airtime second-level summaries. To support more advanced content analysis research that takes advantage of AI's frame-level precision and to enable research that explores more deeply the differences between frame-level and second-level representations and how they change our understanding of the news, today we are releasing for each broadcast the complete raw visual output of Google's Cloud Video AI for that video. This does not contain the speech recognition output, but does include the complete set of fields resulting from "LABEL_DETECTION", "TEXT_DETECTION" AND "SHOT_CHANGE_DETECTION" annotations. See the Cloud Video API's documentation for more detail on how to interpret these fields. The only change to the JSON compared with the raw output directly from the API is that Cloud Video API outputs by default "prettified" JSON that is easy for humans to visually skim whereas to minimize download times, the JSON files here are collapsed to remove all extraneous space.
We're tremendously excited to see what you're able to accomplish with this incredible new dataset!
DOWNLOAD
The complete set of daily files can be downloaded below (this file updates daily):