As we continue to explore Gemini 1.5 Pro's video analysis capabilities, we've examined a number of different prompt approaches to summarizing television news coverage, ranging from second-by-second breakdowns to asking Gemini to segment the video into discrete stories. Given the success of our story segmentation on an American evening news broadcast, let's return to our 6-minute Russian television news experiments and test how Gemini breaks it into stories. As a reminder, we're using this 6 minute clip from the start of a Russia 1 broadcast and using the Gemini 1.5 Pro edition available in Google's AI Studio.
What did we learn? Five times we asked Gemini to segment the clip into discrete stories. We received what appeared at first glance to be five high-precision and accurate segmentations of the broadcasts, but upon further inspection, only one of the five runs offered an accurate and usable segmentation of the video. The others were missing multiple central stories, had the wrong start and stop times for others, moved some stories to wildly different places in the broadcast or hallucinated details. Critical story details changed from run to run, making the results even more difficult to use. Notably, in the four runs that correctly captured the extensive footage of Biden's State of the Union address, all four described him as "confused" such as "Following his speech, Biden appears confused and disoriented, asking where to go and needing assistance from staff."
In a programmatic workflow, it is unclear how a fully automated processing pipeline would handle all of these issues. It took five runs before we got a truly accurate segmentation of the broadcast, but without human review and comparison against the original video all five outputs look entirely reasonable and likely highly accurate. Imagine an automated workflow that ran a video five times through Gemini and ended up with five very different segmentations: how does it select which is the correct one? That kind of quality assessment is not yet clear and represents an existential challenge for LMM use in real-world applications. Nevertheless, the results below capture the very real potential of LMMs for video segmentation – when they do work, the results are extremely impressive.
The 6-minute clip takes up 94,680 tokens out of Gemini's 1 million available. Let's use our standard segmentation prompt that we found to work well on American television:
Break this video down into individual discrete stories. Output a table in which each row is one story. The first column is the start time of the story. The second column is the end time of the story. The third column is a descriptive title of the story. The fourth column is a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that story. The fifth column should be a detailed textual description of that story that describes it in sufficient detail to someone who can't see the video.
This yields the following breakdown that is reasonable at first glance, but misses several stories, including Nigeria. However, there is a critical problem with the summary below: the timecodes. The first "podcast" story is accurate, as is the clock "News Intro", but then things get more complicated. Instead of Putin appearing at 00:28, there is a newsreader that appears first to introduce Putin, who does not actually appear until ten seconds later at 0:38. At the same time, that is quite reasonable to group the introduction into the story itself, but simply offers a caveat. Unfortunately, things go wrong here, as Putin's speech lasts until 0:52. At 00:54, Gemini claims there is a Polish farmers protest. In reality, the Women's Day celebrations continue through 1:02, where coverage switches to Ukraine, Biden's SOTU at 1:12 and Trump's rally at 1:30, with the farmers' protest not beginning until 1:34. Nigeria begins at 1:42, fish harvesting at 1:49, the deer farm at 1:58 and panda cub at 2:04. Thus, some stories appear near their actual start dates below, while others are entirely missing or subsumed into others or even moved to an entirely different time.
Thus, while at first glance the story list below appears to be a high-precision and accurate segmentation of the broadcast, it is missing several stories, has the wrong start and stop times for others and even moves some to wildly different places in the broadcast. This makes the story list below more difficult to use, since it is incomplete, hallucinates some details, has incorrect start/stop times for some stories and reorders some stories.
Breakdown of Video into Discrete Stories
| Start Time | End Time | Title | Keywords | Description |
| 00:00 | 00:09 | "Smotrim" Podcast Platform | podcasts, Russia, Smotrim | An advertisement for the "Smotrim" podcast platform, showing various podcasts available on the platform. |
| 00:12 | 00:28 | News Intro | Russia, news, clock | A clock animation with a world map in the background, transitioning to a news studio with a news anchor. |
| 00:28 | 00:42 | Putin Congratulates Women on International Women's Day | Russia, Putin, International Women's Day | Vladimir Putin delivers a speech congratulating women on International Women's Day, praising their strength, beauty, and contributions to society. |
| 00:42 | 00:54 | International Women's Day Celebrations Across Russia | Russia, International Women's Day, celebrations | Footage of various celebrations for International Women's Day across Russia, including astronauts on the International Space Station sending greetings, soldiers delivering flowers to female medical staff, and festive events in different cities. |
| 00:54 | 01:11 | Polish Farmers Protest | Poland, farmers, protest | Polish farmers stage a protest with tractors and trucks, blocking roads and demanding government action on agricultural issues. |
| 01:11 | 01:15 | News Break | Russia, news | A brief news break with a 3D animation of the word "Vesti". |
| 01:15 | 01:26 | Biden Delivers State of the Union Address | USA, Biden, State of the Union | Joe Biden delivers his State of the Union address to Congress, highlighting his administration's accomplishments and outlining his agenda for the future. |
| 01:26 | 01:33 | Biden Appears Confused After Speech | USA, Biden, confusion | Following his speech, Biden appears confused and disoriented, asking where to go and needing assistance from staff. |
| 01:33 | 01:34 | News Break | Russia, news | A brief news break with a 3D animation of the word "Vesti". |
| 01:34 | 01:43 | Trump Holds Rally in Iowa | USA, Trump, rally, Iowa | Donald Trump holds a rally in Iowa, promoting his 2024 presidential campaign and criticizing the Biden administration. |
| 01:43 | 02:00 | News Break & Economic Update | Russia, news, economy | A news break with a 3D animation of the word "Vesti", followed by a brief update on the Russian economy, showing footage of a factory and a cargo ship. |
| 02:00 | 02:07 | Reindeer Farm in Murmansk | Russia, Murmansk, reindeer | Footage of a reindeer farm in Murmansk, showing reindeer being fed and interacting with visitors. |
| 02:07 | 02:14 | Panda Cub Makes Public Debut in China | China, panda, zoo | A panda cub makes its public debut at a zoo in China, playing with its mother and exploring its enclosure. |
| 02:14 | 02:21 | "Live Sound" Show | Russia, music, competition | A singing competition called "Live Sound" takes place, with various performers competing and the audience voting for their favorites. |
| 02:21 | 05:00 | News Break & International Women's Day Celebrations | Russia, news, International Women's Day, celebrations | A news break with a 3D animation of the word "Vesti", followed by an extended segment on International Women's Day celebrations across Russia. This includes footage of Putin's speech, soldiers giving flowers to women on the street, and various festive events. |
Let's run with a slightly different prompt that asks for advertisements to be broken out as well:
Break this video down into individual discrete stories and advertisements. Output a table in which each row is one story or advertisement. The first column is the start time of the story. The second column is the end time of the story. The third column is a descriptive title of the story. The fourth column is a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that story. The fifth column should be a detailed textual description of that story that describes it in sufficient detail to someone who can't see the video.
This yields a highly similar result with the same overall issues:
Breakdown of Video into Discrete Stories and Advertisements
| Start Time | End Time | Title | Keywords | Description |
| 00:00 | 00:09 | "Smotrim" Podcast Platform (Advertisement) | podcasts, Russia, Smotrim | An advertisement for the "Smotrim" podcast platform, showing various podcasts available on the platform. |
| 00:12 | 00:28 | News Intro | Russia, news, clock | A clock animation with a world map in the background, transitioning to a news studio with a news anchor. |
| 00:28 | 00:42 | Putin Congratulates Women on International Women's Day | Russia, Putin, International Women's Day | Vladimir Putin delivers a speech congratulating women on International Women's Day, praising their strength, beauty, and contributions to society. |
| 00:42 | 00:54 | International Women's Day Celebrations Across Russia | Russia, International Women's Day, celebrations | Footage of various celebrations for International Women's Day across Russia, including astronauts on the International Space Station sending greetings, soldiers delivering flowers to female medical staff, and festive events in different cities. |
| 00:54 | 01:11 | Polish Farmers Protest | Poland, farmers, protest | Polish farmers stage a protest with tractors and trucks, blocking roads and demanding government action on agricultural issues. |
| 01:11 | 01:15 | News Break | Russia, news | A brief news break with a 3D animation of the word "Vesti". |
| 01:15 | 01:26 | Biden Delivers State of the Union Address | USA, Biden, State of the Union | Joe Biden delivers his State of the Union address to Congress, highlighting his administration's accomplishments and outlining his agenda for the future. |
| 01:26 | 01:33 | Biden Appears Confused After Speech | USA, Biden, confusion | Following his speech, Biden appears confused and disoriented, asking where to go and needing assistance from staff. |
| 01:33 | 01:34 | News Break | Russia, news | A brief news break with a 3D animation of the word "Vesti". |
| 01:34 | 01:43 | Trump Holds Rally in Iowa | USA, Trump, rally, Iowa | Donald Trump holds a rally in Iowa, promoting his 2024 presidential campaign and criticizing the Biden administration. |
| 01:43 | 02:00 | News Break & Economic Update | Russia, news, economy | A news break with a 3D animation of the word "Vesti", followed by a brief update on the Russian economy, showing footage of a factory and a cargo ship. |
| 02:00 | 02:07 | Reindeer Farm in Murmansk | Russia, Murmansk, reindeer | Footage of a reindeer farm in Murmansk, showing reindeer being fed and interacting with visitors. |
| 02:07 | 02:14 | Panda Cub Makes Public Debut in China | China, panda, zoo | A panda cub makes its public debut at a zoo in China, playing with its mother and exploring its enclosure. |
| 02:14 | 02:21 | "Live Sound" Show | Russia, music, competition | A singing competition called "Live Sound" takes place, with various performers competing and the audience voting for their favorites. |
| 02:21 | 05:00 | News Break & International Women's Day Celebrations | Russia, news, International Women's Day, celebrations | A news break with a 3D animation of the word "Vesti", followed by an extended segment on International Women's Day celebrations across Russia. This includes footage of Putin's speech, soldiers giving flowers to women on the street, and various festive events. |
And again with the same prompt. This time we get a more detailed breakdown that includes multiple additional stories. Interestingly, the panda cub is now born in the Moscow zoo rather than relating to China, the Polish farmer protests are about Ukraine now, Nigeria appears and a reindeer farm in Murmansk becomes a spotted deer population in Altai Krai. However, for the first time the timecodes are actually relatively accurate:
Breakdown of Video into Stories and Advertisements
| Start Time | End Time | Title | Keywords | Description |
| 00:00 | 00:06 | Advertisement for Smotrim podcast platform | Smotrim, podcasts, Russia, media | This advertisement showcases the Smotrim podcast platform, highlighting its variety of podcasts on various topics. It features snippets from different podcasts and emphasizes the platform's connection to Russia. |
| 00:07 | 00:12 | Advertisement for Smotrim podcast platform | Smotrim, podcasts, Russia, media | This is another advertisement for the Smotrim podcast platform, again emphasizing its diverse content and Russian connection. |
| 00:13 | 00:28 | News intro and countdown | Russia, news, Vesti, clock | This segment shows a clock ticking down to the start of the news program, Vesti, on the Rossiya channel. It features a globe and digital effects, signifying the program's focus on global news. |
| 00:29 | 00:37 | News intro and headlines | Russia, news, Vesti, headlines, March 8th | The news program, Vesti, begins with the anchor introducing the date and highlighting the main headlines of the day. |
| 00:38 | 00:51 | Putin congratulates women on International Women's Day | Russia, Putin, International Women's Day, congratulations, family, mothers, daughters, grandmothers, wives | President Putin delivers a speech congratulating women on International Women's Day. He emphasizes the importance of family and the role women play in it, expressing gratitude for their love, care, and support. |
| 00:52 | 00:54 | Astronauts on the ISS congratulate women | International Space Station, astronauts, International Women's Day, congratulations | Two Russian astronauts aboard the International Space Station send their congratulations to women on International Women's Day. |
| 00:55 | 01:00 | Soldiers congratulate female medical staff | Russia, soldiers, medical staff, International Women's Day, congratulations, flowers | Soldiers visit a medical facility and present flowers to the female staff in honor of International Women's Day. |
| 01:01 | 01:03 | News transition | Russia, news, Vesti, transition | This brief segment shows a visual transition between news stories. |
| 01:04 | 01:11 | Russian military operation in Ukraine | Russia, Ukraine, military operation, explosions, destruction | This segment shows footage of explosions and destruction in Ukraine, presumably as a result of the ongoing Russian military operation. |
| 01:12 | 01:23 | Biden delivers State of the Union address | USA, Biden, State of the Union, Congress, applause | President Biden delivers his State of the Union address to Congress, receiving applause from the audience. |
| 01:24 | 01:29 | Biden appears confused after speech | USA, Biden, confusion, direction | President Biden appears momentarily confused after his speech and asks for directions on where to go. |
| 01:30 | 01:33 | Trump speaks at a rally in Iowa | USA, Trump, rally, Iowa, 2024 election | Former President Trump speaks at a rally in Iowa, promoting his potential candidacy for the 2024 presidential election. |
| 01:34 | 01:41 | Polish farmers protest against Ukrainian grain imports | Poland, farmers, protest, Ukraine, grain imports | Polish farmers stage a protest with tractors and trucks, demanding a halt to Ukrainian grain imports, which they believe are harming their business. |
| 01:42 | 01:47 | Nigerian military operation against bandits | Nigeria, military operation, bandits, villagers | Footage shows the Nigerian military conducting an operation against bandits, with villagers cheering their arrival. |
| 01:48 | 02:00 | News transition | Russia, news, Vesti, transition | Another brief visual transition between news stories. |
| 02:01 | 02:04 | Spotted deer population grows in Altai Krai | Russia, Altai Krai, spotted deer, nature reserve | This segment showcases the growing population of spotted deer in a nature reserve in Altai Krai, Russia. |
| 02:05 | 02:07 | Panda cub born in Moscow Zoo | Moscow Zoo, panda, cub, birth | This segment shows a newborn panda cub at the Moscow Zoo, highlighting its playful interaction with its mother. |
| 02:08 | 02:13 | "Live Sound" show features folk music performance | Russia, Live Sound, folk music, performance | A group of performers in traditional costumes sing and dance on the "Live Sound" show, showcasing Russian folk music. |
| 02:14 | 02:20 | News transition and outro | Russia, news, Vesti, outro | The news program, Vesti, concludes with a visual transition and outro sequence. |
| 02:21 | 04:42 | Putin congratulates women on International Women's Day (extended) | Russia, Putin, International Women's Day, congratulations, family, mothers, daughters, grandmothers, wives, military service, special operation | This extended segment shows President Putin's full speech for International Women's Day. He emphasizes the importance of family, acknowledges the challenges women face, and expresses gratitude for their contributions, particularly those serving in the military and supporting the special operation. |
| 04:43 | 05:38 | Celebrations for International Women's Day across Russia | Russia, International Women's Day, celebrations, concerts, laser shows, sports competitions, flowers, soldiers | This segment showcases various celebrations for International Women's Day across Russia, including concerts, laser shows, sports competitions, and soldiers giving flowers to women. |
| 05:39 | 05:56 | "Give a Flower to a Woman" campaign in Yekaterinburg | Russia, Yekaterinburg, International Women's Day, flowers, soldiers, campaign | This segment focuses on the "Give a Flower to a Woman" campaign in Yekaterinburg, where soldiers distribute flowers to women on the streets and receive smiles and gratitude in return. |
Running again, we're back to where we were before:
Breakdown of Video into Stories and Advertisements
| Start Time | End Time | Title | Keywords | Description |
| 00:00 | 00:09 | Advertisement for Smotrim Podcast Platform | Smotrim, podcasts, Russia, media | This advertisement showcases the Smotrim podcast platform, highlighting its variety of podcasts on various topics. It features snippets from different podcasts and emphasizes the platform's accessibility. |
| 00:12 | 00:28 | News Intro | Russia, news, Vesti, time | This segment serves as the introduction to the news program, featuring a clock ticking and transitioning to the news studio with the anchor. |
| 00:28 | 00:40 | Putin Congratulates Women on International Women's Day | Russia, Putin, International Women's Day, congratulations | President Putin delivers a message congratulating women on International Women's Day. He praises their strength, beauty, and contributions to society. |
| 00:40 | 00:54 | Celebrations of International Women's Day Across Russia | Russia, International Women's Day, celebrations, flowers, astronauts, soldiers | This segment showcases how International Women's Day is celebrated across Russia. It features astronauts sending greetings from the International Space Station, soldiers giving flowers to women in hospitals, and various festive events taking place. |
| 00:54 | 01:11 | Polish Farmers Protest Against Ukrainian Grain Imports | Poland, farmers, protest, Ukraine, grain | Polish farmers stage a protest against the import of Ukrainian grain, which they believe is harming their livelihoods. They block roads with tractors and demand government intervention. |
| 01:11 | 01:33 | News Update | Russia, news, Vesti, economy, fish, deer, panda | This segment provides a brief update on various news stories, including the Russian economy, fish processing in Murmansk, deer breeding in the Altai region, and the birth of a panda cub in China. |
| 01:33 | 02:14 | "Live Sound" Show Features Folk Music Performance | Russia, Live Sound, music, folk, performance | The "Live Sound" show features a performance by a folk music group. The singers and musicians are dressed in traditional costumes and deliver an energetic performance. |
| 02:14 | 02:21 | News Transition | Russia, news, Vesti | This segment serves as a transition between news stories, featuring a visual effect of moving through a digital space. |
| 02:21 | 03:14 | Putin Addresses Women on International Women's Day | Russia, Putin, International Women's Day, address, family, mothers | President Putin delivers a longer address specifically dedicated to women on International Women's Day. He emphasizes the importance of family and motherhood, praising women's contributions to raising children and building a strong society. He also acknowledges the challenges women face and promises government support for families. |
| 03:14 | 04:24 | Putin Addresses Women Serving in the Military | Russia, Putin, International Women's Day, military, soldiers, families | President Putin specifically addresses women serving in the military, acknowledging their courage and sacrifice. He expresses gratitude to their families for their support and promises comprehensive assistance to those serving and their loved ones. |
| 04:24 | 05:36 | Celebrations of International Women's Day Across Russia | Russia, International Women's Day, celebrations, flowers, concerts, sports | This segment further showcases celebrations of International Women's Day across Russia, featuring concerts, laser shows, sporting events, and men giving flowers to women on the streets. |
| 05:36 | 05:57 | "Gift a Flower to a Woman" Campaign in Yekaterinburg | Russia, Yekaterinburg, International Women's Day, flowers, soldiers, campaign | This segment focuses on the "Gift a Flower to a Woman" campaign in Yekaterinburg, where soldiers and volunteers distribute flowers to women on the streets and in public spaces. The campaign aims to express gratitude and appreciation for women on their special day. |
Running one last time, we get perhaps our most accurate segmentation of the five, with all of the major stories present with relatively accurate timecodes for all:
Breakdown of Video into Stories and Advertisements
| Start Time | End Time | Title | Keywords | Description |
| 00:00 | 00:06 | Advertisement for Smotrim podcast platform | Smotrim, podcasts, Russia, entertainment, education | The advertisement shows clips from various podcasts available on the Smotrim platform, including news, sports, and entertainment. The advertisement ends with the Smotrim logo and website address. |
| 00:07 | 00:12 | Advertisement for Smotrim podcast platform | Smotrim, podcasts, Russia, entertainment, education | This advertisement shows a montage of podcast covers available on the Smotrim platform. The advertisement ends with the Smotrim logo and website address. |
| 00:13 | 00:28 | News intro and countdown | Russia, news, clock, countdown | A clock counts down from 15 seconds to 0, with the "Vesti" news logo appearing at the end. |
| 00:29 | 00:37 | News intro and headlines | Russia, news, headlines, March 8th | The news anchor introduces the program and highlights the main headlines for the day, including International Women's Day and the ongoing conflict in Ukraine. |
| 00:38 | 00:51 | Putin's International Women's Day address | Russia, Putin, International Women's Day, family, mothers, daughters, grandmothers, love, support | President Putin delivers a speech congratulating women on International Women's Day. He emphasizes the importance of family and the role of women in society. He also expresses his gratitude to women serving in the military and those supporting their families. |
| 00:52 | 00:54 | Astronauts on the International Space Station send greetings | International Space Station, astronauts, International Women's Day | Two Russian astronauts on the International Space Station send their greetings to women on International Women's Day. |
| 00:55 | 01:00 | Soldiers surprise women with flowers at a hospital | Russia, military, International Women's Day, flowers, hospital | Soldiers visit a hospital and surprise the female staff with flowers and congratulations for International Women's Day. |
| 01:01 | 01:03 | News transition | Russia, news, transition | A brief visual transition with the "Vesti" logo is shown. |
| 01:04 | 01:10 | Footage of the conflict in Ukraine | Ukraine, conflict, explosions, destruction | This segment shows footage of explosions and destruction in Ukraine, highlighting the ongoing conflict. |
| 01:11 | 01:12 | News transition | Russia, news, transition | Another brief visual transition with the "Vesti" logo is shown. |
| 01:13 | 01:23 | Biden's State of the Union address | USA, Biden, State of the Union, Congress, applause | President Biden delivers his State of the Union address to Congress, receiving applause from the audience. |
| 01:24 | 01:29 | Biden appears confused after his speech | USA, Biden, confusion, State of the Union | This segment shows President Biden appearing confused and disoriented after finishing his State of the Union address. He asks where he should go and is directed by an aide. |
| 01:30 | 01:33 | Trump speaks at a rally in Iowa | USA, Trump, rally, Iowa, 2024 election | Former President Trump speaks at a rally in Iowa, promoting his potential candidacy for the 2024 presidential election. |
| 01:34 | 01:36 | News transition | Russia, news, transition | A brief visual transition with the "Vesti" logo is shown. |
| 01:37 | 01:41 | Polish farmers protest against Ukrainian grain imports | Poland, farmers, protest, Ukraine, grain imports | Polish farmers stage a protest with tractors and trucks, demanding a halt to Ukrainian grain imports which they claim are harming their livelihoods. |
| 01:42 | 01:43 | News transition | Russia, news, transition | Another brief visual transition with the "Vesti" logo is shown. |
| 01:44 | 01:47 | Nigerian military confronts bandits in Zamfara state | Nigeria, military, bandits, Zamfara state | The Nigerian military conducts an operation against bandits in Zamfara state, showcasing their efforts to combat insecurity. |
| 01:48 | 01:54 | News transition and focus on Russian economy | Russia, news, economy, industry, fishing | A transition with the "Vesti" logo is shown, followed by footage of various industries in Russia, including metal production and fishing, highlighting the country's economic activity. |
| 01:55 | 02:02 | Report on deer breeding farm in the Murmansk region | Russia, Murmansk, deer, breeding farm | This segment showcases a deer breeding farm in the Murmansk region, focusing on the animals and the farm's operations. |
| 02:03 | 02:06 | Panda cub plays with its mother at a zoo in China | China, panda, zoo, cub, mother | This segment shows a playful panda cub interacting with its mother at a zoo in China. |
| 02:07 | 02:13 | "Live Sound" show features folk music performance | Russia, Live Sound, music, folk, performance | A clip from the "Live Sound" show features a performance by a folk music choir, showcasing traditional Russian music and costumes. |
| 02:14 | 02:20 | News transition and outro | Russia, news, outro | A final visual transition with the "Vesti" logo is shown, followed by the program's outro sequence. |
| 02:21 | 04:42 | Putin's International Women's Day address (continued) | Russia, Putin, International Women's Day, family, mothers, daughters, grandmothers, love, support, military, volunteers | This segment continues President Putin's International Women's Day address, focusing on his message to women serving in the military and those supporting their families. He also emphasizes the importance of family values and expresses his appreciation for women's contributions to society. |
| 04:43 | 05:39 | Celebrations for International Women's Day across Russia | Russia, International Women's Day, celebrations, concerts, sports, flowers, soldiers | This segment showcases various celebrations for International Women's Day across Russia, including concerts, laser shows, sporting events, and soldiers giving flowers to women on the streets. |
| 05:40 | 05:56 | Soldiers surprise women with flowers in Yekaterinburg | Russia, Yekaterinburg, International Women's Day, soldiers, flowers, surprise | This segment focuses on a specific event in Yekaterinburg where soldiers surprised women on the streets with flowers and congratulations for International Women's Day. The segment shows the positive reactions of the women and their appreciation for the gesture. |