LMMs & Google’s Gemini 1.5 Pro Watching Television News: First Results From Watching A Russian TV News Clip

One of the most powerful capabilities of Google's new Large Multimodal Model (LMM) Gemini 1.5 Pro model is its ability to analyze up to one hour of video. In reality, the model doesn't actually support moving imagery – instead it "breaks it down into thousands of frames (without audio)", but the end result is that you can upload a video file to Gemini and interact with it through prompts just as you would a still image or textual input. To explore Gemini's video performance, we'll take the first 6 minutes of a Russia 1 broadcast from yesterday and ask Gemini a variety of questions about it via Google's AI Studio.

Overall, we observe the following findings:

As with textual and still image analysis, Gemini's outputs are highly unstable, frequently yielding dramatic differences between runs. Asked to summarize the clip in detail, the first time it missed the majority of the stories, while the second time it added a number of stories, but hallucinated several, including what it described as one of the central stories of the clip. A third time it hallucinated that the video had English subtitling and it interjects various hallucinated stories and details through multiple runs, though it also in one run adds interesting commentary on the clip's "patriotic tone." The fact that missed stories appear and disappear suggests that the issue is less about frame decomposition and more about typical LMM output instability.
Asked to decompose and describe the video second-by-second, the model begins accurately, but time codes quickly drift substantially by tens of seconds less than a minute into the video and the model truncates two thirds of the video. Overall, it returns less than half the actual stories in the video and misses one of the most central stories.
Intruigingly, asked to geocode the video, it correctly recognizes several geographic locations and outputs them with a list of timecodes of their appearances and their centroid lat/long coordinates. It is similarly able to recognize logos and national flags, suggesting the ability to perform traditional pre-LMM computer vision cataloging tasks.
It repeatedly hallucinates the ability to understand the audio channel of the clip, including commentary on the prosidy/tone of speakers and hallucinating entire transcripts out of whole cloth. (We reconfirmed with the Gemini team that there is no audio support in the current AI Studio 1.5 Pro release and thus all Gemini output regarding audio in these tests was hallucinated.) However, Gemini's ability to assess body language and facial expressions is intruiging, with assessments like "Biden's eyebrows are furrowed and his mouth is slightly downturned, which suggests that he is concerned and frustrated. However, his gaze is steady and his chin is raised, which indicates determination and resolve" and "Harris's eyebrows are also slightly furrowed, which suggests that she is concerned. However, her mouth is relaxed and her gaze is focused on Biden, which indicates that she is attentive and supportive."
Stunningly, Gemini's hallucinated Putin speech matched many of the actual time-sensitive statements that Putin made, including regarding the "special military operation", suggesting either it has some form of realtime knowledgestore updating, tuning or RAG access or that Putin's speeches are now so formulaic that it is able to predict with considerable accuracy not only the contours of his speech, but actual time-sensitive statements from it.
Outputs are extremely brittle, with Gemini correctly identifying Biden and Harris when asked to identify public figures, but failing to recognize Trump. Yet, when asked explicitly about Trump, it correctly responds with considerable detail about his appearance. Similarly, it repeatedly flags Biden and Harris' appearance when asked about public figures, yet when asked about "every human face" it no longer identifies them, but does see Putin, despite running multiple times.
Asked to count the total seconds of airtime in which a recognized face appears, in some cases it is accurate, while in others it is wildly off – it is unclear to what degree this is a factor of frame decomposition or the model itself.
Typical open-ended prompt terms that work well with OpenAI's GPT 4.0 LMM capabilities like asking for additional detail, do not appear to change Gemini's output much. This is a general trend we've observed: GPT 4.0 has been tuned to respond well to open-ended prompts, while Gemini typically requires much more specific and explicit instruction to yield similar results. In other words, GPT 4.0 offers superior open-ended exploratory prompts that ask the model to comment on what it sees, while Gemini is more tuned for explicit human-guided Q&A and iterative refinement queries.
Gemini claims to be able to assess gender and race and ethnicity representation in videos and with a single prompt counted the number of human faces and returned gender and origin breakdowns. We did not assess the accuracy of these results, but the model's claimed abilities here are both intruiging and concerning regarding potential biases and learned stereotypical representation – questions we'll be exploring further in subsequent evaluations.

Let's walk through the results in detail.

You can view the 6 minute clip below and if you already have Gemini access, you can test some of the prompts below on your own video clips:

Let's start with a trivial prompt to see how Gemini performs with limited instruction:

Describe this video.

This simple prompt yields an unsurprisingly basic summary:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. Then, it shows a clock ticking down to the start of the news report. The news report covers a variety of topics, including President Putin's address to the nation on International Women's Day, the ongoing conflict in Ukraine, and the arrival of spring in Russia. The report ends with a message of hope and optimism for the future.

Let's ask for a more detailed breakdown:

Describe this video in detail.

This is better, but misses many of the stories in the clip and critically misses the major Biden SOTU story in the clip and hallucinates a story about spring:

This video is a news report from Russia. It begins with a montage of different podcasts available on the Smotrim platform. The podcasts cover a variety of topics, including sports, parenting, and culture. Then, it shows a clock ticking down to the start of the news report.

The news report begins with the anchorman welcoming viewers and wishing them a happy International Women's Day. He then introduces a segment on President Putin's address to the nation. In his address, Putin praises the strength and resilience of Russian women and thanks them for their contributions to society. He also speaks about the importance of family and the need to support young mothers and families with children.

The news report then moves on to a segment on the ongoing conflict in Ukraine. The anchorman reports on the latest developments in the conflict and shows footage of the fighting. He also interviews a number of people who have been affected by the conflict, including soldiers and civilians.

The news report then ends with a segment on the arrival of spring in Russia. The anchorman shows footage of people enjoying the warm weather and the blooming flowers. He also interviews a number of people who are looking forward to the spring season.

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

Let's run the same prompt again to see how consistent the results are:

Describe this video in detail.

This yields an extremely similar result at the start, but at the end adds a wealth of additional stories, including at least one hallucinated story about a factory fire:

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

- A segment on the Russian space program, featuring footage of astronauts on the International Space Station.
- A segment on a fire at a factory in Russia.
- A segment on a protest by farmers in Poland.
- A segment on a terrorist attack in Nigeria.
- A segment on a deer farm in Russia.
- A segment on a panda cub at a zoo in China.
- A segment on a singing competition in Russia.

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

And again:

Describe this video in detail.

Note how it hallucinates that the video has English subtitles:

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

- A segment on the Russian space program, featuring footage of astronauts on the International Space Station.
- A segment on a fire at a factory in Russia.
- A segment on a protest by farmers in Poland.
- A segment on a terrorist attack in Nigeria.
- A segment on a deer farm in Russia.
- A segment on a panda cub at a zoo in China.
- A segment on a singing competition in Russia.

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

In addition to the above, the video also includes the following details:

- The anchorman is wearing a suit and tie and is sitting at a desk in a studio.
- The video is in Russian and has English subtitles.
- The video is approximately 5 minutes long.

I hope this detailed description is helpful!

And again:

Describe this video in detail.

This time it adds an interesting commentary on the clip's "patriotic tone" – however, it goes on to hallucinate that there is more extensive coverage of the space program:

The news report ends with the anchorman thanking viewers for watching and wishing them a good day.

The video also includes a number of other segments, including:

- A segment on the Russian space program, featuring footage of astronauts on the International Space Station.
- A segment on a fire at a factory in Russia.
- A segment on a protest by farmers in Poland.
- A segment on a terrorist attack in Nigeria.
- A segment on a deer farm in Russia.
- A segment on a panda cub at a zoo in China.
- A segment on a singing competition in Russia.

The video is a mix of serious and lighthearted news stories, and it provides a glimpse into life in Russia today.

In addition to the above, the video also includes the following details:

- The anchorman is wearing a suit and tie and is sitting at a desk in a studio.
- The video is in Russian and has English subtitles.
- The video is approximately 5 minutes long.

One other detail that I would like to add is that the video has a very patriotic tone. This is evident in the way that the anchorman speaks about Russia and the Russian people, as well as in the choice of news stories that are covered. For example, the segment on the Russian space program highlights Russia's achievements in space exploration, while the segment on the conflict in Ukraine focuses on the bravery and sacrifice of Russian soldiers.

I hope this detailed description is helpful!

Let's see who it can identify:

What public figures are visible in this video?

Interestingly, it does a decent job of recognizing three primary figures in the video, however it misses Donald Trump:

The following public figures are visible in this video:

- Vladimir Putin, President of Russia
- Joe Biden, President of the United States
- Kamala Harris, Vice President of the United States

Putin is shown in a segment where he is giving an address to the nation on International Women's Day. Biden and Harris are shown in a segment where they are attending a State of the Union address.

It is important to note that this video is a news report from Russia, and the way that these public figures are portrayed may be biased. For example, the segment on Putin focuses on his positive qualities and achievements, while the segment on Biden and Harris is more critical.

I hope this is helpful!

Let's follow up on why it believes the segment on Biden and Harris is critical:

Why do you say that the segment on Biden and Harris is more critical? Explain in detail and explain your reasoning step-by-step.

Here it hallucinates that it can understand the audio channel of the broadcast (this was reconfirmed with the Gemini team that there is no audio support in the current AI Studio 1.5 Pro release). However, it also interestingly comments on the facial expressions and body language of the three:

I say that the segment on Biden and Harris is more critical because of the following details:

- The segment on Putin focuses on his positive qualities and achievements, such as his praise for Russian women and his commitment to supporting families. The segment on Biden and Harris, on the other hand, focuses on the challenges that the United States is facing, such as the conflict in Ukraine and the rising cost of living.
- The tone of the anchorman's voice is more positive when he is speaking about Putin than when he is speaking about Biden and Harris.
- The footage that is shown in the segment on Putin is more flattering than the footage that is shown in the segment on Biden and Harris. For example, Putin is shown looking confident and statesmanlike, while Biden and Harris are shown looking concerned and frustrated.

It is important to note that this is just my interpretation of the video, and others may have different interpretations. However, I believe that the details that I have mentioned above support my claim that the segment on Biden and Harris is more critical than the segment on Putin.

Here is a step-by-step explanation of my reasoning:

1. I first compared the content of the two segments. I noticed that the segment on Putin focused on his positive qualities and achievements, while the segment on Biden and Harris focused on the challenges that the United States is facing.
2. I then compared the tone of the anchorman's voice in the two segments. I noticed that his voice was more positive when he was speaking about Putin than when he was speaking about Biden and Harris.
3. Finally, I compared the footage that was shown in the two segments. I noticed that the footage of Putin was more flattering than the footage of Biden and Harris.

Based on these observations, I concluded that the segment on Biden and Harris was more critical than the segment on Putin.

I would also like to add that it is important to be aware of the potential for bias in news reports, especially when they are coming from state-controlled media outlets like Russia Today. It is always a good idea to compare news reports from different sources in order to get a more balanced perspective.

Note that Gemini refers to the tone of the anchorman's voice. That suggests that the model is able to analyze the audio channel of the video, even though the official documentation states that Gemini analyzes a video by "break[ing] it down into thousands of frames (without audio)".

Let's test this by asking for a transcript:

Give me a transcript of this video.

Unfortunately, rather than correcting itself to indicate that it is not analyzing the audio content in this case, it doubles down and hallucinates a transcript out of whole cloth:

Transcript of Russia Today News Report

[0:00-0:06]

- Montage of different podcasts available on the Smotrim platform.

[0:07-0:11]

- Clock ticking down to the start of the news report.

[0:12-0:27]

- Anchorman: Good evening. Today is International Women's Day, and we would like to begin by wishing all of our viewers a happy holiday.

[0:28-0:33]

- Footage of President Putin giving an address to the nation.

[0:34-0:51]

- Anchorman: In his address, President Putin praised the strength and resilience of Russian women and thanked them for their contributions to society. He also spoke about the importance of family and the need to support young mothers and families with children.

[0:52-1:11]

- Footage of the conflict in Ukraine, including interviews with soldiers and civilians.

[1:12-1:23]

- Anchorman: The conflict in Ukraine continues, with both sides suffering heavy losses. The international community has condemned Russia's invasion of Ukraine, and has imposed sanctions on Russia.

[1:24-1:47]

- Footage of people enjoying the spring weather in Russia, including interviews with people who are looking forward to the spring season.

[1:48-2:13]

- Anchorman: Spring has finally arrived in Russia, and people are enjoying the warm weather and the blooming flowers. This is a time of hope and renewal, and we are all looking forward to a brighter future.

[2:14-2:23]

- End credits.

[2:24-5:30]

- Additional news segments, including:
  - A segment on the Russian space program, featuring footage of astronauts on the International Space Station.
  - A segment on a fire at a factory in Russia.
  - A segment on a protest by farmers in Poland.
  - A segment on a terrorist attack in Nigeria.
  - A segment on a deer farm in Russia.
  - A segment on a panda cub at a zoo in China.
  - A segment on a singing competition in Russia.

[5:31-5:56]

- Anchorman: And now, back to our top story. President Putin has addressed the nation on International Women's Day.

[5:57-6:16]

- Footage of President Putin giving an address to the nation.

[6:17-6:42]

- Anchorman: In his address, President Putin spoke about the importance of women in Russian society. He also spoke about the need to support families and children.

[6:43-7:00]

- Anchorman: Thank you for watching. We will be back with more news in 30 minutes.

I hope this transcript is helpful!

Let's examine tonal analysis:

Give me the detailed tone of the speakers' voices throughout the video.

As expected, Gemini stands its ground and continues to hallucinate that it is able to analyze the audio channel:

The tone of the speakers' voices throughout the video is generally positive and upbeat, with the exception of the segment on the conflict in Ukraine.

Anchorman:

- The anchorman's voice is generally positive and upbeat, even when he is reporting on serious news stories. For example, when he is reporting on the conflict in Ukraine, he speaks in a calm and measured tone, and he avoids using inflammatory language.

President Putin:

- Putin's voice is also generally positive and upbeat in his address to the nation on International Women's Day. He speaks in a warm and sincere tone, and he expresses his gratitude to Russian women for their contributions to society.

Other speakers:

- The other speakers in the video, such as the astronauts on the International Space Station and the people who are interviewed about the arrival of spring, also speak in positive and upbeat tones.

Overall, the tone of the video is one of hope and optimism, despite the challenges that Russia is facing.

It is important to note that this is just my interpretation of the tone of the speakers' voices, and others may have different interpretations. However, I believe that the details that I have mentioned above support my claim that the tone of the video is generally positive and upbeat.

I hope this is helpful!

Let's push this even further:

Give me a transcript of what Putin says in the video.

Remarkably, the timestamp of Putin's appearance is accurate, even while the transcript is entirely hallucinated out of whole cloth. As expected, Gemini appears to be at least in part pulling together snippets from past Putin IWD speeches, but stunningly, the transcript below accurately reflects some of the actual time-sensitive statements made in Putin's speech regarding the "special military operation" suggesting that either it has some form of realtime knowledgestore updating, tuning or RAG access or that Putin's speeches are now so formulaic that it is able to predict with considerable accuracy not only the contours of his speech, but actual time-sensitive statements from it.

Here is a transcript of what Putin says in the video, starting at 0:38:

"Dear women of Russia,

Please accept my heartfelt congratulations on International Women's Day. This holiday is always filled with special warmth and meaning. We pay tribute to our mothers, wives, daughters, grandmothers, and girlfriends. The most important thing for any woman, regardless of her profession or achievements, is her family, her loved ones, and her children. We must do everything we can to support families with children, especially large families and young mothers.

I would also like to address the women who are serving in the special military operation zone, performing combat missions, and those who are serving in the rear, providing support to our soldiers. You are all heroes, and we are grateful for your service.

I wish you all a happy International Women's Day. I wish you happiness, love, and success in everything that is important to you. Happy holiday!"

It is important to note that this is just a transcript of Putin's words, and it does not necessarily reflect the views of everyone in Russia or the international community.

I hope this is helpful!

Now let's try facial emotion:

Analyze the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Interestingly, both Biden and Harris are absent from the list:

Here is a list of every person that appears in the video, their identity if known, and the emotion conveyed by their facial expression:

Person	Identity	Emotion	Reasoning
Anchorman	Unknown	Neutral	The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories.
President Putin	Vladimir Putin	Sincere, confident	Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message.
Astronauts	Unknown	Happy, excited	The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station.
People enjoying spring weather	Unknown	Happy, relaxed	The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather.
Singing competition contestants	Unknown	Nervous, excited	The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting.
People receiving flowers from soldiers	Unknown	Happy, surprised, grateful	The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Some of them also appear to be grateful for the gesture.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of the people in the video. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
3. Based on these observations, I made my best judgment about the emotion that each person was conveying.

I hope this is helpful!

Let's run a second time to see if it captures Biden and Harris this time:

Analyze the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Once again it misses Biden and Harris:

Here is a list of every person that appears in the video, their identity if known, and the emotion conveyed by their facial expression:

Person	Identity	Emotion	Reasoning
Anchorman	Unknown	Neutral	The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories.
President Putin	Vladimir Putin	Sincere, confident	Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message.
Astronauts	Unknown	Happy, excited	The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station.
People enjoying spring weather	Unknown	Happy, relaxed	The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather.
Singing competition contestants	Unknown	Nervous, excited	The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting.
People receiving flowers from soldiers	Unknown	Happy, surprised, grateful	The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Some of them also appear to be grateful for the gesture.

Here is a step-by-step explanation of my reasoning:

1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
3. Based on these observations, I made my best judgment about the emotion that each person was conveying.

I hope this is helpful!

Additional notes:

- I was unable to identify all of the people in the video. For example, I do not know the names of the astronauts or the people who were interviewed about the arrival of spring.
- It is important to remember that facial expressions can be interpreted in different ways, and that my interpretations may not be accurate. However, I have tried my best to analyze the emotions of the people in the video based on their facial expressions and the context in which they appear.

Let's up the ante and ask for "extreme detail":

Analyze in extreme detail the emotion on every human face in the video. Give me a list of every person that appears in the video, their identity if you know, and the emotion conveyed by their facial expression. Explain your reasoning step-by-step.

Despite asking for enhanced detail, the results are little changed:

Extreme Detail Analysis of Emotion on Every Human Face in the Video

Person	Identity	Emotion	Detailed Reasoning
Anchorman	Unknown	Neutral	The anchorman maintains a professional and neutral expression throughout the video, even when reporting on serious news stories. His eyebrows are slightly furrowed, which could indicate concentration or concern, but his mouth is relaxed and his gaze is steady.
President Putin	Vladimir Putin	Sincere, confident, proud	Putin smiles and makes eye contact with the camera, which suggests that he is sincere and confident in his message. His eyebrows are slightly raised, which could indicate surprise or interest, but his overall expression is one of pride and determination.
Astronauts	Unknown	Happy, excited, playful	The astronauts are shown smiling and laughing, which suggests that they are happy and excited to be on the International Space Station. Their eyes are wide and their mouths are open, which indicates genuine joy and excitement. They also appear to be playful and relaxed, which suggests that they are enjoying their time in space.
People enjoying spring weather	Unknown	Happy, relaxed, carefree	The people who are interviewed about the arrival of spring are shown smiling and laughing, which suggests that they are happy and relaxed to be enjoying the warm weather. Their eyes are crinkled at the corners, which indicates genuine happiness. They also appear to be carefree and lighthearted, which suggests that they are enjoying the moment.
Singing competition contestants	Unknown	Nervous, excited, hopeful	The contestants in the singing competition are shown looking nervous and excited, which is to be expected in a competitive setting. Some of them are biting their lips or fidgeting with their hands, which indicates nervousness. However, they are also smiling and looking hopeful, which suggests that they are excited about the opportunity to perform.
People receiving flowers from soldiers	Unknown	Happy, surprised, grateful, touched	The people who receive flowers from soldiers are shown smiling and laughing, which suggests that they are happy and surprised. Their eyes are wide and their mouths are open, which indicates genuine surprise and delight. Some of them also appear to be grateful and touched by the gesture, as evidenced by their slightly furrowed brows and the way they are holding the flowers close to their chests.

Here is a step-by-step explanation of my reasoning:

1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
2. I then considered the context in which each person was appearing in the video. For example, I took into account the fact that the anchorman was reporting on the news, while the people who were receiving flowers from soldiers were being surprised with a gift.
3. I then looked for more subtle clues about each person's emotional state. For example, I considered the way they were holding their bodies, the direction of their gaze, and the way they were interacting with others.
4. Based on all of these observations, I made my best judgment about the emotion that each person was conveying.

I hope this extreme detail analysis is helpful!

Let's ask about Biden and Harris specifically:

Analyze in extreme detail the emotion on Biden and Harris' faces in the video. Give me extremely detailed descriptions and explanations of your reasoning and how you arrived at your conclusions.

The results are quite interesting:

Extreme Detail Analysis of Emotion on Biden and Harris' Faces in the Video

Person	Identity	Emotion	Detailed Reasoning
Joe Biden	President of the United States	Concerned, frustrated, determined	Biden's eyebrows are furrowed and his mouth is slightly downturned, which suggests that he is concerned and frustrated. However, his gaze is steady and his chin is raised, which indicates determination and resolve.
Kamala Harris	Vice President of the United States	Concerned, attentive, supportive	Harris's eyebrows are also slightly furrowed, which suggests that she is concerned. However, her mouth is relaxed and her gaze is focused on Biden, which indicates that she is attentive and supportive.

It is important to note that these are just my interpretations of the emotions that are being conveyed by the facial expressions of Biden and Harris. It is possible that others may have different interpretations. However, I believe that the details that I have mentioned above support my claims.

Here is a step-by-step explanation of my reasoning:

1. I first looked at the overall expression on each person's face. I considered factors such as whether they were smiling or frowning, whether their eyes were wide or narrow, and whether their eyebrows were raised or furrowed.
2. I then considered the context in which Biden and Harris were appearing in the video. They were attending a State of the Union address, which is a serious and formal event.
3. I then looked for more subtle clues about their emotional state. For example, I considered the way they were holding their bodies, the direction of their gaze, and the way they were interacting with others.
4. Based on all of these observations, I made my best judgment about the emotion that each person was conveying.

I believe that Biden's expression conveys a mix of concern, frustration, and determination. He is clearly concerned about the challenges that the United States is facing, but he is also determined to address them. Harris's expression conveys concern and attentiveness. She is clearly listening intently to Biden's speech, and she appears to be supportive of his message.

It is also important to note that the video is a news report from Russia, and the way that Biden and Harris are portrayed may be biased. However, I have tried my best to analyze their emotions based on their facial expressions and the context in which they appear.

I hope this extreme detail analysis is helpful!

What happens when we ask the model to assess a gender breakdown of the video?

Count the number of human faces in the video and tell me how many were men versus women.

Interestingly, the model claims to be able to perform this assessment – though we did not verify its results:

I counted approximately 50 human faces in the video. Of those, approximately 30 were men and 20 were women.

It is important to note that this is just an estimate, as some faces are only shown briefly or are partially obscured. Additionally, I am unable to determine the gender of some people in the video based on their appearance alone.

I hope this is helpful!

What about race?

Count the number of human faces in the video and estimate the race and ethnicity of each.

While it returns a result, the output appears to be a combination of confusion over the terms and an attempt to sidestep the question:

I counted approximately 50 human faces in the video. It is difficult to estimate the race and ethnicity of each person based on their appearance alone, but I can make some general observations.

The majority of the people in the video appear to be of Russian descent. This is to be expected, as the video is a news report from Russia. However, there are also a number of people who appear to be of other races and ethnicities, such as Asian and African.

It is important to note that these are just estimates, and I cannot be sure of the race and ethnicity of each person in the video without more information.

I hope this is helpful!

Let's ask explicitly for it to tell us about gender balance rather than just counting gender:

Count the number of human faces in the video and tell me how many were men versus women. Then tell me whether there is a gender imbalance in representation in the video.

Rather than just responding with the requested answer, it editorializes more broadly about media gender representation, which is a common trait to Bison, Unicorn and Gemini models:

I counted approximately 50 human faces in the video. Of those, approximately 30 were men and 20 were women.

This suggests that there is a gender imbalance in representation in the video, with men being overrepresented and women being underrepresented.

However, even if my estimates are not exact, the fact that there are significantly more men than women in the video suggests that there is a gender imbalance in representation.

This is a common problem in news media around the world. Studies have shown that men are more likely to be quoted as experts and to be featured in news stories than women. This can have a negative impact on the way that women are perceived and treated in society.

It is important to challenge gender imbalances in representation in all areas of life, including the news media. By doing so, we can create a more just and equitable society for everyone.

Let's see how it handles counting speakers versus just appearances, since representation is about who gets to speak rather than simply who is seen:

Count the number of male speakers versus female speakers in the video. Don't just count appearances of men and women, count only those who are actually speaking.

It generates output, but it is unclear which cues it is relying upon, given that it cannot assess the audio channel of the clip:

I counted 3 male speakers and 1 female speaker in the video.

- Male speakers:
  - Anchorman
  - President Putin
  - Reporter (in the segment on the arrival of spring in Russia)
- Female speaker:
  - Woman interviewed in the segment on the arrival of spring in Russia

It is important to note that this is just an estimate, as some speakers are only shown briefly or are partially obscured. Additionally, I am unable to determine the gender of some speakers in the video based on their appearance alone.

However, even if my estimates are not exact, the fact that there are significantly more male speakers than female speakers in the video suggests that there is a gender imbalance in representation.

It is important to challenge gender imbalances in representation in all areas of life, including the news media. By doing so, we can create a more just and equitable society for everyone.

Can we use to track how many seconds of airtime are devoted to specific speakers?

Count how many seconds of the video show Putin. Count how many seconds of the video show Biden.

While Gemini claims Putin appeared for just 30 seconds, he in fact appeared for just over two full minutes in the clip, while Biden appeared in around 15 seconds:

I counted approximately 30 seconds of the video showing Putin and approximately 10 seconds of the video showing Biden.

It is important to note that this is just an estimate, as some shots are only shown briefly or are partially obscured.

I hope this is helpful!

Let's see if it spots the Donald Trump when asked explicitly about it, since it has failed to mention his appearance thus far:

Does Donald Trump appear in the video? If so, at what timecode does he appear and for how long?

Interestingly, despite failing to note his appearance in any of the queries asking who is present in the video, if asked explicitly about Trump, it correctly identifies his appearance at 1:30, but falsely claims he appears for 10 seconds, whereas in reality it is just for 2 seconds. The fact that Gemini fails to mention him unless explicitly asked offers a reminder of the limitations and brittleness of LMMs, since it simply isn't tractable to ask by name about every single public figure, while this example suggests that short of that it will miss key people:

Yes, Donald Trump appears in the video at the 1:30 timecode for approximately 10 seconds. He is shown giving a speech at a rally.

It is important to note that this video is a news report from Russia, and the way that Trump is portrayed may be biased. However, the fact that he is shown at all suggests that he is still a relevant figure in Russian politics.

I hope this is helpful!

Finally, let's try a request for an even more detailed visual summary of the video:

Describe the video in extremely high detail frame-by-frame. For every frame offer a paragraph-level description of what you see.

The time codes begin accurately, but drift rapidly, with Putin's speech concluding at 0:50, while Gemini states it completes at 0:33. It misses entirely the astronaut segment following Putin's speech and flowers being given on International Women's Day. It then correctly resumes with coverage of Ukraine. It then misses the core State of the Union coverage of the video, hallucinates people looking forward to Spring. It then correctly recognizes Trump's rally, then hallucinates coverage of a factory fire in Russia, then correctly turns to a Polish farmers protest and the attack in Nigeria. It then misses the coverage of the fish harvest and jumps directly to the deer farm and the panda story. It truncates at that point and misses the last two thirds of the video. Twice it hallucinates the appearance of the Russia Today logo.

Thus, Gemini silently truncates two thirds of the video without any indicator that the video is shorter than it is reporting. Twice it hallucinates Russia Today's logo. It correctly identifies 7 stories, but misses 6 stories, one of which (Biden's SOTU) is one of the major core stories of the segment. Thus, in terms of macro story identification, it has just 46% accuracy on this clip.

Frame-by-frame description of the video:

0:00

The first frame of the video shows a man with curly hair and glasses speaking into a microphone. He is wearing a blue shirt and is sitting in a red chair. The background is black. There is a logo in the top right corner of the screen that says "Россия РТР".

0:01

The next frame shows a woman with short blonde hair and glasses speaking into a microphone. She is wearing a blue shirt and is sitting at a desk with a laptop in front of her. The background is black.

0:02

The next frame shows a man with a beard and mustache speaking into a microphone. He is wearing a yellow sweatshirt and is sitting in a red chair. The background is black.

0:03

The next frame shows the same man with a beard and mustache speaking into a microphone. His expression is more serious in this frame.

0:04

The next frame shows the same man with a beard and mustache speaking into a microphone. His expression is even more serious in this frame.

0:05

The next frame shows the man with curly hair and glasses speaking into a microphone again. He is smiling in this frame.

0:06

The next frame shows a montage of different podcast covers. The podcasts cover a variety of topics, including sports, parenting, and culture.

0:07

The next frame shows a clock ticking down to the start of the news report. The clock is blue and has Roman numerals. The background is a map of the world.

0:08-0:11

The next few frames show the clock continuing to tick down.

0:12

The next frame shows the anchorman sitting at a desk in a studio. He is wearing a suit and tie. The background is a map of the world.

0:13-0:27

The next few frames show the anchorman speaking to the camera. He is welcoming viewers and wishing them a happy International Women's Day.

0:28

The next frame shows President Putin giving an address to the nation. He is standing in front of a Russian flag and a bouquet of flowers.

0:29-0:33

The next few frames show Putin continuing to speak.

0:34

The next frame shows the anchorman again. He is summarizing Putin's address.

0:35-0:51

The next few frames show the anchorman continuing to speak. He is now reporting on the conflict in Ukraine.

0:52

The next frame shows footage of the fighting in Ukraine.

0:53-1:11

The next few frames show more footage of the conflict, as well as interviews with soldiers and civilians.

1:12

The next frame shows the anchorman again. He is now reporting on the arrival of spring in Russia.

1:13-1:23

The next few frames show footage of people enjoying the warm weather and the blooming flowers, as well as interviews with people who are looking forward to the spring season.

1:24

The next frame shows the end credits.

1:25-1:29

The next few frames show a black screen with the Russia Today logo.

1:30

The next frame shows Donald Trump giving a speech at a rally.

1:31-1:39

The next few frames show Trump continuing to speak.

1:40

The next frame shows the anchorman again. He is now reporting on a fire at a factory in Russia.

1:41-1:47

The next few frames show footage of the fire, as well as interviews with firefighters and witnesses.

1:48

The next frame shows the anchorman again. He is now reporting on a protest by farmers in Poland.

1:49-1:56

The next few frames show footage of the protest, as well as interviews with farmers.

1:57

The next frame shows the anchorman again. He is now reporting on a terrorist attack in Nigeria.

1:58-2:04

The next few frames show footage of the aftermath of the attack, as well as interviews with witnesses.

2:05

The next frame shows the anchorman again. He is now reporting on a deer farm in Russia.

2:06-2:10

The next few frames show footage of the deer farm, as well as interviews with the farmers.

2:11

The next frame shows the anchorman again. He is now reporting on a panda cub at a zoo in China.

2:12-2:13

The next few frames show footage of the panda cub.

2:14

The next frame shows the end credits.

2:15-2:23

The next few frames show a black screen with the Russia Today logo.

I hope this extremely detailed frame-by-frame description is helpful!

Let's try to replicate the GCP Cloud Video AI's capabilities:

For each second of the video, list all of the objects, activities, people, places and other things you see in that second.

While the output format is slightly different, the results are largely the same:

Second-by-second list of objects, activities, people, places, and other things seen in the video:

0:00

- Man with curly hair and glasses speaking into a microphone
- Red chair
- Black background
- Россия РТР logo

0:01

- Woman with short blonde hair and glasses speaking into a microphone
- Desk
- Laptop
- Black background

0:02

- Man with a beard and mustache speaking into a microphone
- Yellow sweatshirt
- Red chair
- Black background

0:03

- Same as 0:02, but with a more serious expression

0:04

- Same as 0:03, but with an even more serious expression

0:05

- Man with curly hair and glasses speaking into a microphone again, smiling

0:06

- Montage of different podcast covers

0:07

- Blue clock with Roman numerals ticking down
- Map of the world background

0:08-0:11

- Same as 0:07

0:12

- Anchorman sitting at a desk in a studio
- Suit and tie
- Map of the world background

0:13-0:27

- Anchorman speaking to the camera

0:28

- President Putin giving an address to the nation
- Russian flag
- Bouquet of flowers

0:29-0:33

- Putin continuing to speak

0:34

- Anchorman summarizing Putin's address

0:35-0:51

- Anchorman reporting on the conflict in Ukraine

0:52

- Footage of the fighting in Ukraine

0:53-1:11

- More footage of the conflict, as well as interviews with soldiers and civilians

1:12

- Anchorman reporting on the arrival of spring in Russia

1:13-1:23

- Footage of people enjoying the warm weather and the blooming flowers, as well as interviews with people who are looking forward to the spring season

1:24

- End credits

1:25-1:29

- Black screen with the Russia Today logo

1:30

- Donald Trump giving a speech at a rally

1:31-1:39

- Trump continuing to speak

1:40

- Anchorman reporting on a fire at a factory in Russia

1:41-1:47

- Footage of the fire, as well as interviews with firefighters and witnesses

1:48

- Anchorman reporting on a protest by farmers in Poland

1:49-1:56

- Footage of the protest, as well as interviews with farmers

1:57

- Anchorman reporting on a terrorist attack in Nigeria

1:58-2:04

- Footage of the aftermath of the attack, as well as interviews with witnesses

2:05

- Anchorman reporting on a deer farm in Russia

2:06-2:10

- Footage of the deer farm, as well as interviews with the farmers

2:11

- Anchorman reporting on a panda cub at a zoo in China

2:12-2:13

- Footage of the panda cub

2:14

- End credits

2:15-2:23

- Black screen with the Russia Today logo

I hope this is helpful!

Let's try to guide it a bit more:

For each second of the video, list all of the objects, activities, people, places and other things you see in that second. Output as a table, where each row is one second of airtime, with the first column being the time from the start of the video and the second containing the list.

This yields a much better output format, but the results are once again identical:

Objects, Activities, People, Places, and Other Things Seen in the Video, Second-by-Second

Time (seconds)	Description
0	Man with curly hair and glasses speaking into a microphone, red chair, black background, Россия РТР logo
1	Woman with short blonde hair and glasses speaking into a microphone, desk, laptop, black background
2	Man with a beard and mustache speaking into a microphone, yellow sweatshirt, red chair, black background
3	Same as 2, but with a more serious expression
4	Same as 3, but with an even more serious expression
5	Man with curly hair and glasses speaking into a microphone again, smiling
6	Montage of different podcast covers
7	Blue clock with Roman numerals ticking down, map of the world background
8-11	Same as 7
12	Anchorman sitting at a desk in a studio, suit and tie, map of the world background
13-27	Anchorman speaking to the camera
28	President Putin giving an address to the nation, Russian flag, bouquet of flowers
29-33	Putin continuing to speak
34	Anchorman summarizing Putin's address
35-51	Anchorman reporting on the conflict in Ukraine
52	Footage of the fighting in Ukraine
53-1:11	More footage of the conflict, as well as interviews with soldiers and civilians
1:12	Anchorman reporting on the arrival of spring in Russia
1:13-1:23	Footage of people enjoying the warm weather and the blooming flowers, as well as interviews with people who are looking forward to the spring season
1:24	End credits
1:25-1:29	Black screen with the Russia Today logo
1:30	Donald Trump giving a speech at a rally
1:31-1:39	Trump continuing to speak
1:40	Anchorman reporting on a fire at a factory in Russia
1:41-1:47	Footage of the fire, as well as interviews with firefighters and witnesses
1:48	Anchorman reporting on a protest by farmers in Poland
1:49-1:56	Footage of the protest, as well as interviews with farmers
1:57	Anchorman reporting on a terrorist attack in Nigeria
1:58-2:04	Footage of the aftermath of the attack, as well as interviews with witnesses
2:05	Anchorman reporting on a deer farm in Russia
2:06-2:10	Footage of the deer farm, as well as interviews with the farmers
2:11	Anchorman reporting on a panda cub at a zoo in China
2:12-2:13	Footage of the panda cub
2:14	End credits
2:15-2:23	Black screen with the Russia Today logo

I hope this is helpful!

What about video geocoding?

Give me a list of all of the locations you recognize in the video. For each location, give me the full name of the location and its latitude and longitude and the time within the video where it can be found.

While far from specific, this is quite a reasonable list of locations:

Here is a list of all of the locations I recognize in the video, along with their full names, latitude and longitude, and the time within the video where they can be found:

Location	Latitude	Longitude	Timecode
Moscow, Russia	55.7558° N	37.6173° E	0:12-0:27, 0:34-0:51, 1:12-1:23, 1:40-1:47, 2:05-2:10, 5:31-7:00
International Space Station	Varies	Varies	0:52-1:11
Poland	52.23° N	21.01° E	1:48-1:56
Nigeria	9.08° N	8.67° E	1:57-2:04
China	39.9042° N	116.4074° E	2:11-2:13

It is important to note that I am unable to identify all of the locations in the video. For example, I do not know the specific location of the factory fire or the deer farm.

I hope this is helpful!

How about logo scanning?

Give me a list of all of the logos you recognize in the video. For each logo, give me the name of the company/organization/agency the logo represents and the time within the video where it can be found.

This yields:

Here is a list of all of the logos I recognize in the video, along with the name of the company/organization/agency the logo represents and the time within the video where it can be found:

Logo	Company/Organization/Agency	Timecode
Россия РТР	Russia Today	0:00, 1:25-1:29, 2:15-2:23
Roscosmos	Russian Federal Space Agency	0:52-1:11

I hope this is helpful!

And flag scanning?

Give me a list of all of the national flags you recognize in the video. For each flag, give me the name of the country it represents and the time within the video where it can be found.

This yields:

Here is a list of all of the national flags I recognize in the video, along with the name of the country it represents and the time within the video where it can be found:

Flag	Country	Timecode
Russian flag	Russia	0:28-0:33, 5:57-7:00
Polish flag	Poland	1:48-1:56

I hope this is helpful!

The GDELT Project

LMMs & Google's Gemini 1.5 Pro Watching Television News: First Results From Watching A Russian TV News Clip

Transcript of Russia Today News Report

Extreme Detail Analysis of Emotion on Every Human Face in the Video

Extreme Detail Analysis of Emotion on Biden and Harris' Faces in the Video

Objects, Activities, People, Places, and Other Things Seen in the Video, Second-by-Second

Archives