The GDELT Project

Story Segmentation Of TV News: Comparing Gemini 2.5 Pro/Flash, Gemini 2.0 Flash/Lite, Gemma 3 1/4/12/27B & ChatGPT 4o/o3

As part of our continuing series in using LLMs to segment television news broadcasts into their component stories, let's return to the same ABC Evening News broadcast we've been using to test story detection and this time we'll compare Gemini 2.5 Pro Thinking, Gemini 2.5 Flash Thinking, Gemini 2.5 Flash with thinking disabled, Gemini 2.0 Flash, Gemini 2.0 Flash-Lite, Gemma 3's 1B, 4B, 12B and 27B models and ChatGPT's 4o and o3 reasoning models, showing the resulting story segmentation for each below. Interestingly, all of the models produced the same overall story segmentation this time, from the smallest openly available model to the most advanced reasoning model, with the only major difference being the inclusion of the last story in the transcript, which technically was part of the following broadcast. The major difference between the models appears to be the level of detail they provide about each story.

For all of the results below we use the same prompt:

Below is a television news transcript. You are a news cataloger whose job it is to read television news transcripts and identify each of the stories described within. Read the transcript below and make a list of all of the stories it describes. Each story may be mentioned multiple times in multiple places in the transcript. Output a bulleted list of each story, with a title and summary of the main details of that story.

TELEVISION NEWS TRANSCRIPT:
[TRANSCRIPT FOLLOWS HERE]

Gemini Pro 2.5 Preview 05-06 Thinking:

Here is a list of the stories described in the television news transcript:

Running through Gemini Pro 2.5 Preview 05-06 Thinking a second time:

Here is a list of the stories described in the television news transcript:

Gemini 2.5 Flash Preview 05-20 Thinking (this time it includes the Oakland school story that was missing from the Gemini Pro summaries):

Here is a list of the stories described in the television news transcript:

Gemini 2.5 Flash Preview 05-20 with thinking mode disabled:

Here is a list of the stories described in the television news transcript:

Gemini 2.0 Flash:

Here is a list of the stories covered in the television news transcript, with a brief summary of each:

Gemini 2.0 Flash-Lite:

Here's a breakdown of the stories covered in the television news transcript:

Gemma 3 1B:

Here’s a list of the stories described in the transcript, presented in a bulleted format:

Let me know if you'd like me to elaborate on any of these stories!

Gemma 3 4B:

Here’s a list of the stories described in the television news transcript, with summaries:

This list provides a concise overview of the key stories covered in the transcript.

Gemma 3 12B:

Here's a breakdown of the stories covered in the television news transcript:

Let me know if you'd like any of these stories elaborated on!

Gemma 3 27B:

Here's a list of the stories described in the television news transcript, with titles and summaries:

And finally, ChatGPT 4o:

Here is a list of the major stories covered in the television news transcript, with titles and summaries:


Let me know if you'd like this list formatted differently (e.g., as a CSV, JSON, or timeline).

And ChatGPT o3:

And ChatGPT o4-mini: