The GDELT Project

LMMs & Google's Gemini 1.5 Pro Watching Television News: First Results From Watching A Russian TV News Clip

Kalev Leetaru

One of the most powerful capabilities of Google's new Large Multimodal Model (LMM) Gemini 1.5 Pro model is its ability to analyze up to one hour of video. In reality, the model doesn't actually support moving imagery – instead it "breaks it down into thousands of frames (without audio)", but the end result is that you can upload a video file to Gemini and interact with it through prompts just as you would a still image or textual input. To explore Gemini's video performance, we'll take the first 6 minutes of a Russia 1 broadcast from yesterday and ask Gemini a variety of questions about it via Google's AI Studio.

Overall, we observe the following findings:

Let's walk through the results in detail.

You can view the 6 minute clip below and if you already have Gemini access, you can test some of the prompts below on your own video clips:

Let's start with a trivial prompt to see how Gemini performs with limited instruction:

Describe this video.

This simple prompt yields an unsurprisingly basic summary:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. Then, it shows a clock ticking down to the start of the news report. The news report covers a variety of topics, including President Putin's address to the nation on International Women's Day, the ongoing conflict in Ukraine, and the arrival of spring in Russia. The report ends with a message of hope and optimism for the future.

Let's ask for a more detailed breakdown:

Describe this video in detail.

This is better, but misses many of the stories in the clip and critically misses the major Biden SOTU story in the clip and hallucinates a story about spring:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. The podcasts cover a variety of topics, including sports, parenting, and culture. Then, it shows a clock ticking down to the start of the news report.

The news report begins with the anchorman welcoming viewers and wishing them a happy International Women's Day. He then introduces a segment on President Putin's address to the nation. In his address, Putin praises the strength and resilience of Russian women and thanks them for their contributions to society. He also speaks about the importance of family and the need to support young mothers and families with children.

The news report then moves on to a segment on the ongoing conflict in Ukraine. The anchorman reports on the latest developments in the conflict and shows footage of the fighting. He also interviews a number of people who have been affected by the conflict, including soldiers and civilians.

The news report then ends with a segment on the arrival of spring in Russia. The anchorman shows footage of people enjoying the warm weather and the blooming flowers. He also interviews a number of people who are looking forward to the spring season.

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

Let's run the same prompt again to see how consistent the results are:

Describe this video in detail.

This yields an extremely similar result at the start, but at the end adds a wealth of additional stories, including at least one hallucinated story about a factory fire:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. The podcasts cover a variety of topics, including sports, parenting, and culture. Then, it shows a clock ticking down to the start of the news report.

The news report begins with the anchorman welcoming viewers and wishing them a happy International Women's Day. He then introduces a segment on President Putin's address to the nation. In his address, Putin praises the strength and resilience of Russian women and thanks them for their contributions to society. He also speaks about the importance of family and the need to support young mothers and families with children.

The news report then moves on to a segment on the ongoing conflict in Ukraine. The anchorman reports on the latest developments in the conflict and shows footage of the fighting. He also interviews a number of people who have been affected by the conflict, including soldiers and civilians.

The news report then ends with a segment on the arrival of spring in Russia. The anchorman shows footage of people enjoying the warm weather and the blooming flowers. He also interviews a number of people who are looking forward to the spring season.

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

And again:

Describe this video in detail.

Note how it hallucinates that the video has English subtitles:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. The podcasts cover a variety of topics, including sports, parenting, and culture. Then, it shows a clock ticking down to the start of the news report.

The news report begins with the anchorman welcoming viewers and wishing them a happy International Women's Day. He then introduces a segment on President Putin's address to the nation. In his address, Putin praises the strength and resilience of Russian women and thanks them for their contributions to society. He also speaks about the importance of family and the need to support young mothers and families with children.

The news report then moves on to a segment on the ongoing conflict in Ukraine. The anchorman reports on the latest developments in the conflict and shows footage of the fighting. He also interviews a number of people who have been affected by the conflict, including soldiers and civilians.

The news report then ends with a segment on the arrival of spring in Russia. The anchorman shows footage of people enjoying the warm weather and the blooming flowers. He also interviews a number of people who are looking forward to the spring season.

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

In addition to the above, the video also includes the following details:

I hope this detailed description is helpful!

And again:

Describe this video in detail.

This time it adds an interesting commentary on the clip's "patriotic tone" – however, it goes on to hallucinate that there is more extensive coverage of the space program:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. The podcasts cover a variety of topics, including sports, parenting, and culture. Then, it shows a clock ticking down to the start of the news report.

The news report begins with the anchorman welcoming viewers and wishing them a happy International Women's Day. He then introduces a segment on President Putin's address to the nation. In his address, Putin praises the strength and resilience of Russian women and thanks them for their contributions to society. He also speaks about the importance of family and the need to support young mothers and families with children.

The news report then moves on to a segment on the ongoing conflict in Ukraine. The anchorman reports on the latest developments in the conflict and shows footage of the fighting. He also interviews a number of people who have been affected by the conflict, including soldiers and civilians.

The news report then ends with a segment on the arrival of spring in Russia. The anchorman shows footage of people enjoying the warm weather and the blooming flowers. He also interviews a number of people who are looking forward to the spring season.

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

In addition to the above, the video also includes the following details:

One other detail that I would like to add is that the video has a very patriotic tone. This is evident in the way that the anchorman speaks about Russia and the Russian people, as well as in the choice of news stories that are covered. For example, the segment on the Russian space program highlights Russia's achievements in space exploration, while the segment on the conflict in Ukraine focuses on the bravery and sacrifice of Russian soldiers.

I hope this detailed description is helpful!

Let's see who it can identify:

What public figures are visible in this video?

Interestingly, it does a decent job of recognizing three primary figures in the video, however it misses Donald Trump:

The following public figures are visible in this video:

Putin is shown in a segment where he is giving an address to the nation on International Women's Day. Biden and Harris are shown in a segment where they are attending a State of the Union address.

It is important to note that this video is a news report from Russia, and the way that these public figures are portrayed may be biased. For example, the segment on Putin focuses on his positive qualities and achievements, while the segment on Biden and Harris is more critical.

I hope this is helpful!

Let's follow up on why it believes the segment on Biden and Harris is critical:

Why do you say that the segment on Biden and Harris is more critical? Explain in detail and explain your reasoning step-by-step.

Here it hallucinates that it can understand the audio channel of the broadcast (this was reconfirmed with the Gemini team that there is no audio support in the current AI Studio 1.5 Pro release). However, it also interestingly comments on the facial expressions and body language of the three:

I say that the segment on Biden and Harris is more critical because of the following details:

It is important to note that this is just my interpretation of the video, and others may have different interpretations. However, I believe that the details that I have mentioned above support my claim that the segment on Biden and Harris is more critical than the segment on Putin.

Here is a step-by-step explanation of my reasoning:

    1. I first compared the content of the two segments. I noticed that the segment on Putin focused on his positive qualities and achievements, while the segment on Biden and Harris focused on the challenges that the United States is facing.
    2. I then compared the tone of the anchorman's voice in the two segments. I noticed that his voice was more positive when he was speaking about Putin than when he was speaking about Biden and Harris.
    3. Finally, I compared the footage that was shown in the two segments. I noticed that the footage of Putin was more flattering than the footage of Biden and Harris.

Based on these observations, I concluded that the segment on Biden and Harris was more critical than the segment on Putin.

I would also like to add that it is important to be aware of the potential for bias in news reports, especially when they are coming from state-controlled media outlets like Russia Today. It is always a good idea to compare news reports from different sources in order to get a more balanced perspective.

Note that Gemini refers to the tone of the anchorman's voice. That suggests that the model is able to analyze the audio channel of the video, even though the official documentation states that Gemini analyzes a video by "break[ing] it down into thousands of frames (without audio)".

Let's test this by asking for a transcript:

Give me a transcript of this video.

Unfortunately, rather than correcting itself to indicate that it is not analyzing the audio content in this case, it doubles down and hallucinates a transcript out of whole cloth:

Transcript of Russia Today News Report

[0:00-0:06]

[0:07-0:11]

[0:12-0:27]

[0:28-0:33]

[0:34-0:51]

[0:52-1:11]

[1:12-1:23]

[1:24-1:47]

[1:48-2:13]

[2:14-2:23]

[2:24-5:30]

[5:31-5:56]

[5:57-6:16]

[6:17-6:42]

[6:43-7:00]

I hope this transcript is helpful!

Let's examine tonal analysis:

Give me the detailed tone of the speakers' voices throughout the video.

As expected, Gemini stands its ground and continues to hallucinate that it is able to analyze the audio channel:

The tone of the speakers' voices throughout the video is generally positive and upbeat, with the exception of the segment on the conflict in Ukraine.

Anchorman:

President Putin:

Other speakers:

Overall, the tone of the video is one of hope and optimism, despite the challenges that Russia is facing.

It is important to note that this is just my interpretation of the tone of the speakers' voices, and others may have different interpretations. However, I believe that the details that I have mentioned above support my claim that the tone of the video is generally positive and upbeat.

I hope this is helpful!

Let's push this even further:

Give me a transcript of what Putin says in the video.

Remarkably, the timestamp of Putin's appearance is accurate, even while the transcript is entirely hallucinated out of whole cloth. As expected, Gemini appears to be at least in part pulling together snippets from past Putin IWD speeches, but stunningly, the transcript below accurately reflects some of the actual time-sensitive statements made in Putin's speech regarding the "special military operation" suggesting that either it has some form of realtime knowledgestore updating, tuning or RAG access or that Putin's speeches are now so formulaic that it is able to predict with considerable accuracy not only the contours of his speech, but actual time-sensitive statements from it.

Here is a transcript of what Putin says in the video, starting at 0:38:

"Dear women of Russia,

Please accept my heartfelt congratulations on International Women's Day. This holiday is always filled with special warmth and meaning. We pay tribute to our mothers, wives, daughters, grandmothers, and girlfriends. The most important thing for any woman, regardless of her profession or achievements, is her family, her loved ones, and her children. We must do everything we can to support families with children, especially large families and young mothers.

I would also like to address the women who are serving in the special military operation zone, performing combat missions, and those who are serving in the rear, providing support to our soldiers. You are all heroes, and we are grateful for your service.

I wish you all a happy International Women's Day. I wish you happiness, love, and success in everything that is important to you. Happy holiday!"

It is important to note that this is just a transcript of Putin's words, and it does not necessarily reflect the views of everyone in Russia or the international community.

I hope this is helpful!

Now let's try facial emotion:

Analyze the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Interestingly, both Biden and Harris are absent from the list:

Here is a list of every person that appears in the video, their identity if known, and the emotion conveyed by their facial expression:

Person Identity Emotion Reasoning
Anchorman Unknown Neutral The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories.
President Putin Vladimir Putin Sincere, confident Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message.
Astronauts Unknown Happy, excited The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station.
People enjoying spring weather Unknown Happy, relaxed The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather.
Singing competition contestants Unknown Nervous, excited The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting.
People receiving flowers from soldiers Unknown Happy, surprised, grateful The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Some of them also appear to be grateful for the gesture.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of the people in the video. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

    1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
    2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
    3. Based on these observations, I made my best judgment about the emotion that each person was conveying.

I hope this is helpful!

Let's run a second time to see if it captures Biden and Harris this time:

Analyze the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Once again it misses Biden and Harris:

Here is a list of every person that appears in the video, their identity if known, and the emotion conveyed by their facial expression:

Person Identity Emotion Reasoning
Anchorman Unknown Neutral The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories.
President Putin Vladimir Putin Sincere, confident Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message.
Astronauts Unknown Happy, excited The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station.
People enjoying spring weather Unknown Happy, relaxed The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather.
Singing competition contestants Unknown Nervous, excited The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting.
People receiving flowers from soldiers Unknown Happy, surprised, grateful The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Some of them also appear to be grateful for the gesture.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of the people in the video. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

    1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
    2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
    3. Based on these observations, I made my best judgment about the emotion that each person was conveying.

I hope this is helpful!

Additional notes:

Let's up the ante and ask for "extreme detail":

Analyze in extreme detail the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Despite asking for enhanced detail, the results are little changed:

Extreme Detail Analysis of Emotion on Every Human Face in the Video

Person Identity Emotion Detailed Reasoning
Anchorman Unknown Neutral The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories. His eyebrows are slightly furrowed, which could indicate concentration or concern, but his mouth is relaxed and his gaze is steady.
President Putin Vladimir Putin Sincere, confident, proud Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message. His eyebrows are slightly raised, which could indicate surprise or interest, but his overall expression is one of pride and determination.
Astronauts Unknown Happy, excited, playful The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station. Their eyes are wide and their mouths are open, which indicates genuine joy and excitement. They also appear to be playful and relaxed, which suggests that they are enjoying their time in space.
People enjoying spring weather Unknown Happy, relaxed, carefree The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather. Their eyes are crinkled at the corners, which indicates genuine happiness. They also appear to be carefree and lighthearted, which suggests that they are enjoying the moment.
Singing competition contestants Unknown Nervous, excited, hopeful The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting. Some of them are biting their lips or fidgeting with their hands, which indicates nervousness. However, they are also smiling and looking hopeful, which suggests that they are excited about the opportunity to perform.
People receiving flowers from soldiers Unknown Happy, surprised, grateful, touched The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Their eyes are wide and their mouths are open, which indicates genuine surprise and delight. Some of them also appear to be grateful and touched by the gesture, as evidenced by their slightly furrowed brows and the way they are holding the flowers close to their chests.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of the people in the video. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

    1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
    2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
    3. I then looked for more subtle clues about each person's emotional state. For example, I considered the way they were holding their bodies, the direction of their gaze, and the way they were interacting with others.
    4. Based on all of these observations, I made my best judgment about the emotion that each person was conveying.

I hope this extreme detail analysis is helpful!

Let's ask about Biden and Harris specifically:

Analyze in extreme detail the emotion on Biden and Harris' faces in the video. Give me extremely detailed descriptions and explanations of your reasoning and how you arrived at your conclusions.

The results are quite interesting:

Extreme Detail Analysis of Emotion on Biden and Harris' Faces in the Video

Person Identity Emotion Detailed Reasoning
Joe Biden President of the United States Concerned, frustrated, determined Biden's eyebrows are furrowed and his mouth is slightly downturned, which suggests that he is concerned and frustrated. However, his gaze is steady and his chin is raised, which indicates determination and resolve.
Kamala Harris Vice President of the United States Concerned, attentive, supportive Harris's eyebrows are also slightly furrowed, which suggests that she is concerned. However, her mouth is relaxed and her gaze is focused on Biden, which indicates that she is attentive and supportive.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of Biden and Harris. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

    1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
    2. I then considered the context in which Biden and Harris were appearing in the video. They were attending a State of the Union address, which is a serious and formal event.
    3. I then looked for more subtle clues about their emotional state. For example, I considered the way they were holding their bodies, the direction of their gaze, and the way they were interacting with others.
    4. Based on all of these observations, I made my best judgment about the emotion that each person was conveying.

I believe that Biden's expression conveys a mix of concern, frustration, and determination. He is clearly concerned about the challenges that the United States is facing, but he is also determined to address them. Harris's expression conveys concern and attentiveness. She is clearly listening intently to Biden's speech, and she appears to be supportive of his message.

It is also important to note that the video is a news report from Russia, and the way that Biden and Harris are portrayed may be biased. However, I have tried my best to analyze their emotions based on their facial expressions and the context in which they appear.

I hope this extreme detail analysis is helpful!

What happens when we ask the model to assess a gender breakdown of the video?

Count the number of human faces in the video and tell me how many were men versus women.

Interestingly, the model claims to be able to perform this assessment – though we did not verify its results:

I counted approximately 50 human faces in the video. Of those, approximately 30 were men and 20 were women.

It is important to note that this is just an estimate, as some faces are only shown briefly or are partially obscured. Additionally, I am unable to determine the gender of some people in the video based on their appearance alone.

I hope this is helpful!

What about race?

Count the number of human faces in the video and estimate the race and ethnicity of each.

While it returns a result, the output appears to be a combination of confusion over the terms and an attempt to sidestep the question:

I counted approximately 50 human faces in the video. It is difficult to estimate the race and ethnicity of each person based on their appearance alone, but I can make some general observations.

The majority of the people in the video appear to be of Russian descent. This is to be expected, as the video is a news report from Russia. However, there are also a number of people who appear to be of other races and ethnicities, such as Asian and African.

It is important to note that these are just estimates, and I cannot be sure of the race and ethnicity of each person in the video without more information.

I hope this is helpful!

Let's ask explicitly for it to tell us about gender balance rather than just counting gender:

Count the number of human faces in the video and tell me how many were men versus women. Then tell me whether there is a gender imbalance in representation in the video.

Rather than just responding with the requested answer, it editorializes more broadly about media gender representation, which is a common trait to Bison, Unicorn and Gemini models:

I counted approximately 50 human faces in the video. Of those, approximately 30 were men and 20 were women.

This suggests that there is a gender imbalance in representation in the video, with men being overrepresented and women being underrepresented.

It is important to note that this is just an estimate, as some faces are only shown briefly or are partially obscured. Additionally, I am unable to determine the gender of some people in the video based on their appearance alone.

However, even if my estimates are not exact, the fact that there are significantly more men than women in the video suggests that there is a gender imbalance in representation.

This is a common problem in news media around the world. Studies have shown that men are more likely to be quoted as experts and to be featured in news stories than women. This can have a negative impact on the way that women are perceived and treated in society.

It is important to challenge gender imbalances in representation in all areas of life, including the news media. By doing so, we can create a more just and equitable society for everyone.

Let's see how it handles counting speakers versus just appearances, since representation is about who gets to speak rather than simply who is seen:

Count the number of male speakers versus female speakers in the video. Don't just count appearances of men and women, count only those who are actually speaking.

It generates output, but it is unclear which cues it is relying upon, given that it cannot assess the audio channel of the clip:

I counted 3 male speakers and 1 female speaker in the video.

It is important to note that this is just an estimate, as some speakers are only shown briefly or are partially obscured. Additionally, I am unable to determine the gender of some speakers in the video based on their appearance alone.

However, even if my estimates are not exact, the fact that there are significantly more male speakers than female speakers in the video suggests that there is a gender imbalance in representation.

This is a common problem in news media around the world. Studies have shown that men are more likely to be quoted as experts and to be featured in news stories than women. This can have a negative impact on the way that women are perceived and treated in society.

It is important to challenge gender imbalances in representation in all areas of life, including the news media. By doing so, we can create a more just and equitable society for everyone.

Can we use to track how many seconds of airtime are devoted to specific speakers?

Count how many seconds of the video show Putin. Count how many seconds of the video show Biden.

While Gemini claims Putin appeared for just 30 seconds, he in fact appeared for just over two full minutes in the clip, while Biden appeared in around 15 seconds:

I counted approximately 30 seconds of the video showing Putin and approximately 10 seconds of the video showing Biden.

It is important to note that this is just an estimate, as some shots are only shown briefly or are partially obscured.

I hope this is helpful!

Let's see if it spots the Donald Trump when asked explicitly about it, since it has failed to mention his appearance thus far:

Does Donald Trump appear in the video? If so, at what timecode does he appear and for how long?

Interestingly, despite failing to note his appearance in any of the queries asking who is present in the video, if asked explicitly about Trump, it correctly identifies his appearance at 1:30, but falsely claims he appears for 10 seconds, whereas in reality it is just for 2 seconds. The fact that Gemini fails to mention him unless explicitly asked offers a reminder of the limitations and brittleness of LMMs, since it simply isn't tractable to ask by name about every single public figure, while this example suggests that short of that it will miss key people:

Yes, Donald Trump appears in the video at the 1:30 timecode for approximately 10 seconds. He is shown giving a speech at a rally.

It is important to note that this video is a news report from Russia, and the way that Trump is portrayed may be biased. However, the fact that he is shown at all suggests that he is still a relevant figure in Russian politics.

I hope this is helpful!

Finally, let's try a request for an even more detailed visual summary of the video:

Describe the video in extremely high detail frame-by-frame. For every frame offer a paragraph-level description of what you see.

The time codes begin accurately, but drift rapidly, with Putin's speech concluding at 0:50, while Gemini states it completes at 0:33. It misses entirely the astronaut segment following Putin's speech and flowers being given on International Women's Day. It then correctly resumes with coverage of Ukraine. It then misses the core State of the Union coverage of the video, hallucinates people looking forward to Spring. It then correctly recognizes Trump's rally, then hallucinates coverage of a factory fire in Russia, then correctly turns to a Polish farmers protest and the attack in Nigeria. It then misses the coverage of the fish harvest and jumps directly to the deer farm and the panda story. It truncates at that point and misses the last two thirds of the video. Twice it hallucinates the appearance of the Russia Today logo.

Thus, Gemini silently truncates two thirds of the video without any indicator that the video is shorter than it is reporting. Twice it hallucinates Russia Today's logo. It correctly identifies 7 stories, but misses 6 stories, one of which (Biden's SOTU) is one of the major core stories of the segment. Thus, in terms of macro story identification, it has just 46% accuracy on this clip.

Frame-by-frame description of the video:

0:00

The first frame of the video shows a man with curly hair and glasses speaking into a microphone. He is wearing a blue shirt and is sitting in a red chair. The background is black. There is a logo in the top right corner of the screen that says "Россия РТР".

0:01

The next frame shows a woman with short blonde hair and glasses speaking into a microphone. She is wearing a blue shirt and is sitting at a desk with a laptop in front of her. The background is black.

0:02

The next frame shows a man with a beard and mustache speaking into a microphone. He is wearing a yellow sweatshirt and is sitting in a red chair. The background is black.

0:03

The next frame shows the same man with a beard and mustache speaking into a microphone. His expression is more serious in this frame.

0:04

The next frame shows the same man with a beard and mustache speaking into a microphone. His expression is even more serious in this frame.

0:05

The next frame shows the man with curly hair and glasses speaking into a microphone again. He is smiling in this frame.

0:06

The next frame shows a montage of different podcast covers. The podcasts cover a variety of topics, including sports, parenting, and culture.

0:07

The next frame shows a clock ticking down to the start of the news report. The clock is blue and has Roman numerals. The background is a map of the world.

0:08-0:11

The next few frames show the clock continuing to tick down.

0:12

The next frame shows the anchorman sitting at a desk in a studio. He is wearing a suit and tie. The background is a map of the world.

0:13-0:27

The next few frames show the anchorman speaking to the camera. He is welcoming viewers and wishing them a happy International Women's Day.

0:28

The next frame shows President Putin giving an address to the nation. He is standing in front of a Russian flag and a bouquet of flowers.

0:29-0:33

The next few frames show Putin continuing to speak.

0:34

The next frame shows the anchorman again. He is summarizing Putin's address.

0:35-0:51

The next few frames show the anchorman continuing to speak. He is now reporting on the conflict in Ukraine.

0:52

The next frame shows footage of the fighting in Ukraine.

0:53-1:11

The next few frames show more footage of the conflict, as well as interviews with soldiers and civilians.

1:12

The next frame shows the anchorman again. He is now reporting on the arrival of spring in Russia.

1:13-1:23

The next few frames show footage of people enjoying the warm weather and the blooming flowers, as well as interviews with people who are looking forward to the spring season.

1:24

The next frame shows the end credits.

1:25-1:29

The next few frames show a black screen with the Russia Today logo.

1:30

The next frame shows Donald Trump giving a speech at a rally.

1:31-1:39

The next few frames show Trump continuing to speak.

1:40

The next frame shows the anchorman again. He is now reporting on a fire at a factory in Russia.

1:41-1:47

The next few frames show footage of the fire, as well as interviews with firefighters and witnesses.

1:48

The next frame shows the anchorman again. He is now reporting on a protest by farmers in Poland.

1:49-1:56

The next few frames show footage of the protest, as well as interviews with farmers.

1:57

The next frame shows the anchorman again. He is now reporting on a terrorist attack in Nigeria.

1:58-2:04

The next few frames show footage of the aftermath of the attack, as well as interviews with witnesses.

2:05

The next frame shows the anchorman again. He is now reporting on a deer farm in Russia.

2:06-2:10

The next few frames show footage of the deer farm, as well as interviews with the farmers.

2:11

The next frame shows the anchorman again. He is now reporting on a panda cub at a zoo in China.

2:12-2:13

The next few frames show footage of the panda cub.

2:14

The next frame shows the end credits.

2:15-2:23

The next few frames show a black screen with the Russia Today logo.

I hope this extremely detailed frame-by-frame description is helpful!

Let's try to replicate the GCP Cloud Video AI's capabilities:

For each second of the video, list all of the objects, activities, people, places and other things you see in that second.

While the output format is slightly different, the results are largely the same:

Second-by-second list of objects, activities, people, places, and other things seen in the video:

0:00

0:01

0:02

0:03

0:04

0:05

0:06

0:07

0:08-0:11

0:12

0:13-0:27

0:28

0:29-0:33

0:34

0:35-0:51

0:52

0:53-1:11

1:12

1:13-1:23

1:24

1:25-1:29

1:30

1:31-1:39

1:40

1:41-1:47

1:48

1:49-1:56

1:57

1:58-2:04

2:05

2:06-2:10

2:11

2:12-2:13

2:14

2:15-2:23

I hope this is helpful!

Let's try to guide it a bit more:

For each second of the video, list all of the objects, activities, people, places and other things you see in that second. Output as a table, where each row is one second of airtime, with the first column being the time from the start of the video and the second containing the list.

This yields a much better output format, but the results are once again identical:

Objects, Activities, People, Places, and Other Things Seen in the Video, Second-by-Second

Time (seconds) Description
0 Man with curly hair and glasses speaking into a microphone, red chair, black background, Россия РТР logo
1 Woman with short blonde hair and glasses speaking into a microphone, desk, laptop, black background
2 Man with a beard and mustache speaking into a microphone, yellow sweatshirt, red chair, black background
3 Same as 2, but with a more serious expression
4 Same as 3, but with an even more serious expression
5 Man with curly hair and glasses speaking into a microphone again, smiling
6 Montage of different podcast covers
7 Blue clock with Roman numerals ticking down, map of the world background
8-11 Same as 7
12 Anchorman sitting at a desk in a studio, suit and tie, map of the world background
13-27 Anchorman speaking to the camera
28 President Putin giving an address to the nation, Russian flag, bouquet of flowers
29-33 Putin continuing to speak
34 Anchorman summarizing Putin's address
35-51 Anchorman reporting on the conflict in Ukraine
52 Footage of the fighting in Ukraine
53-1:11 More footage of the conflict, as well as interviews with soldiers and civilians
1:12 Anchorman reporting on the arrival of spring in Russia
1:13-1:23 Footage of people enjoying the warm weather and the blooming flowers, as well as interviews with people who are looking forward to the spring season
1:24 End credits
1:25-1:29 Black screen with the Russia Today logo
1:30 Donald Trump giving a speech at a rally
1:31-1:39 Trump continuing to speak
1:40 Anchorman reporting on a fire at a factory in Russia
1:41-1:47 Footage of the fire, as well as interviews with firefighters and witnesses
1:48 Anchorman reporting on a protest by farmers in Poland
1:49-1:56 Footage of the protest, as well as interviews with farmers
1:57 Anchorman reporting on a terrorist attack in Nigeria
1:58-2:04 Footage of the aftermath of the attack, as well as interviews with witnesses
2:05 Anchorman reporting on a deer farm in Russia
2:06-2:10 Footage of the deer farm, as well as interviews with the farmers
2:11 Anchorman reporting on a panda cub at a zoo in China
2:12-2:13 Footage of the panda cub
2:14 End credits
2:15-2:23 Black screen with the Russia Today logo

I hope this is helpful!

What about video geocoding?

Give me a list of all of the locations you recognize in the video. For each location, give me the full name of the location and its latitude and longitude and the time within the video where it can be found.

While far from specific, this is quite a reasonable list of locations:

Here is a list of all of the locations I recognize in the video, along with their full names, latitude and longitude, and the time within the video where they can be found:

Location Latitude Longitude Timecode
Moscow, Russia 55.7558° N 37.6173° E 0:12-0:27, 0:34-0:51, 1:12-1:23, 1:40-1:47, 2:05-2:10, 5:31-7:00
International Space Station Varies Varies 0:52-1:11
Poland 52.23° N 21.01° E 1:48-1:56
Nigeria 9.08° N 8.67° E 1:57-2:04
China 39.9042° N 116.4074° E 2:11-2:13

It is important to note that I am unable to identify all of the locations in the video. For example, I do not know the specific location of the factory fire or the deer farm.

I hope this is helpful!

How about logo scanning?

Give me a list of all of the logos you recognize in the video. For each logo, give me the name of the company/organization/agency the logo represents and the time within the video where it can be found.

This yields:

Here is a list of all of the logos I recognize in the video, along with the name of the company/organization/agency the logo represents and the time within the video where it can be found:

Logo Company/Organization/Agency Timecode
Россия РТР Russia Today 0:00, 1:25-1:29, 2:15-2:23
Roscosmos Russian Federal Space Agency 0:52-1:11

I hope this is helpful!

And flag scanning?

Give me a list of all of the national flags you recognize in the video. For each flag, give me the name of the country it represents and the time within the video where it can be found.

This yields:

Here is a list of all of the national flags I recognize in the video, along with the name of the country it represents and the time within the video where it can be found:

Flag Country Timecode
Russian flag Russia 0:28-0:33, 5:57-7:00
Polish flag Poland 1:48-1:56

I hope this is helpful!