In collaboration with the TV News Archive, we have applied Cloud Vision to OCR more than 19 billion seconds of global television news, transcribing more than 294 billion words of onscreen text. While Cloud Vision records the precise pixel bounding boxes of every character, word and text block, our work to date has focused on searching the entirety of the onscreen text of each second of airtime. What if we want to search only the chyron text? While chyron text by definition appears in the "lower third" of the screen, its actual size, shape, extensions, color, text size/color/font and other characteristics varies immensely across the world and thus per-channel bounding boxes would be needed to isolate the chyron text across each channel. Moreover, the sheer density of onscreen text as a whole and the close proximity of scrolling "tickers" and other insets makes determining which text makes up the chyron and which is part of the myriad other onscreen textual elements immensely difficult and far beyond the ability of fixed bounding boxes or simple heuristics to navigate. Even manually-defined bounding boxes can be confounded by the increasing use of multiple chryon sizes and positions intermixed with logos and other visual elements. Even more complicated is the use by some channels of visual "tabs" and extensions that house the name and affiliation of the speaker during interviews and panel conversations.
Could Gemini's visual understanding allow us to fully autonomously identify the chyron text region and its bounding box in a given broadcast, regardless of the visual complexity of the scene and do so across such a globally diverse array of channels and languages? Here we took a screen shot of 14 different programs from channels across the world and used the following prompt to ask Gemini 3.1 Pro to identify whether it contains a chyron and if so, to compute the extended bounding box of that chyron to contain both its text and speaker name and affiliation information and to transcribe and translate that chyron text for the example image to verify its accuracy.
Remarkably, Gemini perfectly identifies and isolates the chyron text in every single example, including correctly flagging the images that lack a chyron region. In a production workflow, one could imagine maintaining a general chyron bounding box for each channel as a mask that is run against the Cloud Vision OCR and if the OCR'd text appears to extend beyond the chyron bounding box, the system would automatically trigger Gemini to recompute the bounding box for that program for all of the areas where text appears to extend beyond the bounds. Since Gemini only has to examine a single screen shot for each of the problematic snippets, token cost would be minimal. The results below suggest this approach has strong potential. As usual, only the public Gemini API was used and no data was used to train or tune any model.
Note that for these quick experiments we did not provide Gemini with the precise dimensions of each image in pixels (that information is lost in the tokenization process), so the pixel dimensions for the bounding boxes below assume 1280×720 in most cases, which is not correct for several of the images and thus the bounding boxes may be slightly off in these specific experiments – in production the actual precise dimensions would be provided to Gemini as part of the prompt.
Examine this still image from a TV news channel carefully. Identify if there is a chyron text section in the image (include attribution information on the speaker as part of the chyron). Ignore all other text, including any scrolling feeds, tickers and all other onscreen text OTHER THAN the chyron. If there is no obvious chyron text section, output "NO CHYRON" and stop. If there is a chyron, identify the bounding box of the chyron section. Not just the bounding box around the current chyron text, but the bounding box around the entire chyron section where chyron text might appear. Make sure the bounding box does not encompass non-chyron sections. Transcribe the chyron text ONLY and give me the chyron text in the language it appears onscreen. If the chyron text is not in English, then give me the translation of it. Output should be: CHYRON TEXT: TRANSLATION: (if applicable) CHYRON BOUNDING BOX: Top:, Bottom: Left: Right: (in px)
CHYRON TEXT: OFFICIALS: 2 KILLED, MORE INJURED IN SEATTLE FESTIVAL SHOOTING CHYRON BOUNDING BOX: Top: 565, Bottom: 660, Left: 45, Right: 1090 (in px) (Note: Pixel coordinates are estimated based on a standard 1280x720 resolution, bounding the white container reserved for the headline text, excluding the CNN logo block).
CHYRON TEXT: ► BENJAMIN NETANYAHU | ISRAELI PRIME MINISTER NETANYAHU: IRAN HAS TO END THEIR NUCLEAR PROGRAM CHYRON BOUNDING BOX: Top: 576, Bottom: 686, Left: 188, Right: 1007 (in px)
CHYRON TEXT: TRUMP ADMIN. WITHDRAWS SUBPOENA FOR LIST OF JEWS CHYRON BOUNDING BOX: Top: 590, Bottom: 660, Left: 35, Right: 900 (in px)
NO CHYRON
CHYRON TEXT: مسافرون يشتكون من غياب النقاط الطبية وضعف الاتصالات على طريق دير الزور دمشق TRANSLATION: Travelers complain about the lack of medical points and poor communications on the Deir ez-Zor - Damascus road. CHYRON BOUNDING BOX: Top: 805, Bottom: 925 Left: 110 Right: 1810 (in px)
CHYRON TEXT: GEO-POLITICAL ANALYST MAHRUKH HAMEED POLITICAL ECONOMY EXPERT US AND IRAN RESUME INDIRECT TALKS CHYRON BOUNDING BOX: Top: 362, Bottom: 450, Left: 44, Right: 836 (in px)
CHYRON TEXT: OIL SLIPS MORE THAN 5% AFTER US PAUSES STRIKES ON IRAN CHYRON BOUNDING BOX: Top: 398, Bottom: 450, Left: 41, Right: 839 (in px)
CHYRON TEXT: عبد اللطيف الجواهري والي بنك المغرب تقرير المغربية TRANSLATION: Abdellatif Jouahri Governor of Bank Al-Maghrib Report Al Maghribia CHYRON BOUNDING BOX: Top: 350, Bottom: 425, Left: 105, Right: 553 (in px)
CHYRON TEXT: کابل د مخابراتو او معلوماتي ټکنالوژۍ نندارتون پرانیست غونډه TRANSLATION: Kabul Opening ceremony of the telecommunications and information technology exhibition CHYRON BOUNDING BOX: Top: 850, Bottom: 1030 Left: 90 Right: 1830 (in px)* (Note: Pixel coordinates are approximate and estimated based on a standard 1920x1080 HD resolution scale).
Note that at first glance it might appear that Gemini got this case wrong by claiming no Chyron, you can see in the clip that both top and bottom text boxes are scrolling headline tickers rather than chyron text bars.
NO CHYRON
CHYRON TEXT: ФСБ РФ: ЗАДЕРЖАННЫЙ ПЕРЕДАВАЛ РАЗВЕДКЕ НОВОЙ ЗЕЛАНДИИ ДАННЫЕ, КОТОРЫЕ МОГЛИ ИСПОЛЬЗОВАТЬСЯ ПРОТИВ РФ TRANSLATION: FSB RF: The detainee transferred data to New Zealand intelligence that could be used against the Russian Federation CHYRON BOUNDING BOX: Top: 650, Bottom: 685, Left: 0, Right: 1280 (in px, assuming standard 1280x720 resolution)
CHYRON TEXT: 全台亂象...毒駕扣車劇增 塞爆拖吊場 TRANSLATION: Chaos across Taiwan... Sharp increase in impounded vehicles from drug driving overcrowds tow yards CHYRON BOUNDING BOX: Top: 855, Bottom: 965, Left: 185, Right: 1920
CHYRON TEXT: HONDURAS | SUSPENDEN ORDEN DE CAPTURA DE JUAN ORLANDO HERNÁNDEZ SURGEN LAS CRÍTICAS RESPECTO A LA LLEGADA DE HERNÁNDEZ TRANSLATION: HONDURAS | ARREST WARRANT FOR JUAN ORLANDO HERNÁNDEZ SUSPENDED CRITICISM ARISES REGARDING THE ARRIVAL OF HERNÁNDEZ CHYRON BOUNDING BOX: Top: 855, Bottom: 985, Left: 420, Right: 1920 (in px, estimated for a 1920x1080 frame)
CHYRON TEXT: نور الله عاطف استاد پوهنتون تخار TRANSLATION: Noorullah Atif Professor, Takhar University CHYRON BOUNDING BOX: Top: 292, Bottom: 328, Left: 51, Right: 561 (in px)