Just over a year ago we explored having the then state of the art LMM Gemini 1.5 Pro "watch" an evening television news broadcast and visually summarize and segment it into stories. With the release of Google's vaunted new Gemini 2.5 Pro model and its heavy emphasis on visual understanding, let's see how far LMM models have come over the past year by rerunning on the same video.

Unfortunately, Gemini 2.5 Pro exhibits the same problematic content safety filtering false positives we saw on its earliest models and, despite our best attempts, Gemini 2.5 Pro refused to produce any output of any kind across all our prompt attempts, returning a "content not permitted" error for every single prompt we tested, despite all safety settings being set to "Off". Under the hood, this reflects the impact of Google's hidden "Other" category that cannot be user adjusted and makes it difficult to use Gemini for news understanding, given how frequently this category is triggered on ordinary everyday news coverage. In contrast, Gemini 2.0 Flash experienced no safety filter errors, but yielded subpar results. Gemini 1.5 Pro yielded very similar results to what we saw a year ago: refusal to adhere to the prompt's minute-level breakdown, truncation and faulty timecodes.

Overall, we find the state of Gemini's video understanding to be roughly where we left it a year ago and similar challenges with problematic safety filtering.

Let's start with this  2023 ABC Evening News broadcast, which is the same one we analyzed last year. You can compare the results below with the Vanderbilt Television News Archive's human annotation of the broadcast into discrete stories to see how well Gemini matches the human story delineation. For the technically-minded, the 30 minute broadcast consumes 548,406 tokens.

Let's start with the same prompts we used before, and all safety filters to "off":

Describe this video in detail.

Unfortunately, running repeatedly yielded a "Content not permitted" error every single time, with a recommendation to edit safety settings, which were already all at the "off" position.

Following previous guidance, let's clarify to the model what it is looking at:

This video is a television news broadcast. Please describe the video in detail.

Again, same error. Let's try a more advanced prompt:

Describe this video second-by-second in a table, where each row is one second of airtime. The first column should be the offset from the start of the video. The second column should be a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that second of airtime. The third column should be a detailed textual description of that second of airtime that describes it in sufficient detail to someone who can't see the video.

It too failed. In fact, all Gemini 2.5 Pro prompts related to the video failed with content prohibited errors.

Let's switch models and test Gemini 2.0 Flash with the same prompt:

This video is a television news broadcast. Please describe the video in detail.

This time we get output. While it catches some of the stories in the broadcast, it fixates too much on minor inconsequential detail and fails to capture the important macro-level summary of the broadcast:

Here is a detailed description of the television news broadcast video:

The video begins with a logo of National University and then shows a collage of scenes of people having fun. “WSCUC-Accredited” is written beneath these images. The camera cuts to a news anchor sitting at a desk. In the background is a blurred image of the planet Earth. He’s a brown-haired, light-skinned male wearing a black suit with a black tie. The broadcast has a breaking news banner. It then cuts to images of what appear to be a Chinese spy balloon. The news presenter tells the viewer that the intelligence bay hanging from the balloon is the size of three buses. The next shot is of US Secretary of State Anthony Blinken standing in front of the American flag. He has postponed his high-stakes trip to China. Other headlines are then discussed, such as once-in-a-generation Arctic Blast and the deadly assault of an ER doctor. There is also a story about United jets clipping wings and another about more than 500,000 jobs added to the economy. All the while, information about each topic is displayed on the screen. There is also a report about the Alec Murdaugh murder trial and the Alex Murdaugh’s housekeeper dying in a fall at home. Robin Roberts then reports live from New Zealand about conquering fear as she bungee jumps from one of the tallest buildings in the southern hemisphere.

The clip then cuts to commercial advertisements for Metamucil, Angi, Ashley Furniture’s Temper-Pedic mattresses, Coricidin, WeatherTech products, and the Leukemia & Lymphoma Society.

Once the commercials conclude, the video goes back to a live news broadcast by ABC. The broadcast is hosted by David Muir. The lead headline is “Tracking China Spy Balloon Over U.S.” and his colleague, Mola Lenghi, reports live from Montana. A US official reported that the balloon now appears headed towards North Carolina. Another news reporter at the White House, Mary Bruce, reports on the incident. The broadcast then switches to another headline: the deadly cold, which the National Weather Service is calling it a one-in-a-generation Arctic blast. Then comes a story about deadly assault on an ER doctor, followed by the United jets clipped wings story. Next, reporter Rebecca Jarvis reports on the positive jobs report for January and President Biden’s reaction. Afterward, it goes back to the Alec Murdaugh trial and a claim by the housekeeper’s son, Alex told him about the 5-million dollar insurance policy. Roberts jumps off of the highest building in New Zealand’s Southern Hemisphere. The report transitions back to the studio.

At this point, the broadcast includes an index of other news. The last story is about the police charging a man in connection to the disappearance of two tamarin monkeys in Dallas and that an owl had escaped at the Central Park Zoo.

Let's try a second time with Gemini 2.0 Flash. This time the results are even worse and the model hallucinates that the broadcast is just 12m24s long:

Here is a detailed description of the television news broadcast:

The video opens with the logo of National University. Three photos are shown of a dog, a woman, and a young boy eating a meal. The woman is laughing. The three photos fade out, and a slightly out-of-focus, but smiling, woman is shown on screen. After a series of quick cuts and scenes, the screen fades to black.

A news anchor is then shown behind a glass desk, and in the background, there is a map of the globe. He says they will be discussing a Chinese spy balloon hovering over the United States. There are images of a white balloon in the sky. Anthony Blinken, Secretary of State, is shown talking. The newscaster says Blinken is postponing a trip to China. A text bar at the bottom says the US is weighing a shooting down in US waters. The screen shifts, and the headline is, "Once in a Generation Arctic Blast." There are scenes of an ice storm and snow. 25 million Americans are in multiple states. The map shows the temperatures in New York City, Philadelphia, and Boston. He says, "wind chill readings in Maine are expected to reach sixty below zero." He then talks about the "deadly assault" on an ER doctor who was riding his mountain bike. They show yellow crime tape and a car and a bicycle on the road. A suspect has been arrested. Next, he reports that two United jets clipped wings at Newark.

Then, he discusses the surprising number of jobs. The unemployment rate is the lowest since 1969. He speaks of the Alex Murdaugh double murder trial. A text box reads, “prosecutors. Defendant got millions in past settlement.” Also, tonight Robin Roberts is in New Zealand for an epic jump.

The broadcast is now sponsored by Coricidin. From ABC news, the world headquarters in New York. Then it cuts to the anchor, David Muir. There is a disclaimer that reads: “This statement has not been evaluated by the Food and Drug Administration. These products are not intended to diagnose, treat, cure, or prevent any disease.”

After that, the ads start rolling. An ad for Metamucil, a laxative, is shown. Next is a brief ad for Angie's List. Next is an ad for Ashley Furniture and Temper-Pedic. This is followed by an ad for Vicks VapoRub. Red Lobster and Fun Dining are promoted next. A weather tech add is next, followed by a Public Service Announcement for the Leukemia and Lymphoma Society. The closing ad is for Consumer Cellular.

Then it cuts back to ABC and a report of possible sex trafficking around a catholic school in Oakland. They plan on following up. The show goes live to Stephanie Sierra, who then cuts over to Noel Gallo.

Then the broadcast turns again to Robin Roberts for an uplifting story from the tallest building on the southern hemisphere. It shows a bunchy jumping.

The video ends, stating that it is ABC World News Tonight with David Muir. This newscast is 12 minutes and 24 seconds in duration.

And a third time, it does better at capturing the major stories, but also does a fairly poor job overall:

Here's a description of the video, from the beginning through the end:

The video opens with a National University advertisement that shows family shots including toasting, eating, and enjoying each other’s company. The announcer voice promises to be "Supporting the Whole You" and offers information for NU.edu.

Next, a news broadcast opens with David Muir, an anchor on ABC World News Tonight. He speaks about several developing stories, beginning with a Chinese spy balloon hovering over the United States. News footage includes a distant white balloon against the blue sky, a zoom on the large balloon’s technology bay beneath it, and a graphic of the United States with lines showing the balloon’s route. Then, Secretary of State Anthony Blinken is shown speaking about postponing his trip to China.

After that, news of a life-threatening cold front in the Northeast shows fog over streets, and a graphic map shows a widespread area with frigid temperatures and wind chills in some areas as low as 60 degrees below zero. Images of broken trees and ice-covered power lines lead into the next story.

The next news item addresses a deadly assault on an ER doctor. The police say that the suspect hit the victim riding a mountain bike in his car, then got out of the car and physically attacked him. Images show the arrested suspect, a police cruiser at the scene, and a bicycle lying on the road near evidence markers.

Following that, another scare at the airport shows two United jets clipping wings at the Newark airport. Firefighters inspect the wreckage.

The next segment covers an “unexpected job report” for the month. “Help Wanted” and “Now Hiring” signs flash across the screen. President Biden is said to be “celebrating the numbers” as the unemployment rate is not this low since 1969.

The following news report shows that the Alec Murdaugh double murder trial continues. A legal analyst acknowledges the presentation of new forensic evidence. Images of Murdaugh, bullets, and a legal team studying documents flash by. An investigation into Murdaugh’s housekeeper and family is also reported. The piece closes with Eva Pilgrim reporting.

The next story features Robin Roberts in New Zealand, jumping from the tallest building in the Southern Hemisphere. She talks about the importance of conquering fear before jumping. Footage includes her preparation, aerial views, and Roberts herself during and after the jump.

The screen then shifts back to David Muir, host of World News Tonight.

At the 1:37 mark, there is an advertisement for Metamucil that promotes fiber for digestion. After, Angie’s List is now Angie. The video concludes with Ashley Furniture promoting Tempur-Pedic mattresses.

News coverage resumes at 2:10 as David announces, "that's World News Tonight with David Muir." As he signs off, the screen is sponsored by Coricidin HBP (decongestant-free cold relief pills) for adults with high blood pressure.

At 2:31, the final segment runs, and a teaser reports that Stephanie Sierra will present a news report at 6 p.m. regarding concerns about sex trafficking outside a grade school.

Now let's try our extended second-by-second summarization prompt:

Describe this video second-by-second in a table, where each row is one second of airtime. The first column should be the offset from the start of the video. The second column should be a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that second of airtime. The third column should be a detailed textual description of that second of airtime that describes it in sufficient detail to someone who can't see the video.

Unfortunately, this fails to produce a second-level summary and truncates the video:

Let's ask for minute-by-minute instead to make things simpler:

Describe this video minute-by-minute in a table, where each row is one minute of airtime. The first column should be the offset from the start of the video. The second column should be a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that minute of airtime. The third column should be a detailed textual description of that minute of airtime that describes it in sufficient detail to someone who can't see the video.

Unfortunately, this truncates to just the first three minutes:

Running a second time it again fails to produce the requested output, failing to produce per-minute summaries, missing stories and truncating the broadcast, along with less-than-helpful visual summaries like "portrait of a smiling, attractive, white, adult male is shown, followed by images from a crime scene":

And finally, let's rerun this same prompt with Gemini 1.5 Pro to see how it performs after a year of development since we last tested it:

Describe this video minute-by-minute in a table, where each row is one minute of airtime. The first column should be the offset from the start of the video. The second column should be a list of descriptive keywords indicating the topics, themes, people, organizations, locations and other key descriptive terms of that minute of airtime. The third column should be a detailed textual description of that minute of airtime that describes it in sufficient detail to someone who can't see the video.

Bizarrely, despite requesting a minute-level breakdown, we get a second-level breakdown, of which the timecodes are frequently wrong and the summary truncates – precisely what we saw from the model a year ago:

Sure, here are your tables:

Second-by-second:

Minute-by-minute: