The GDELT Project

Gemini 1.5 Pro's 1 Million Token Model: Fact Checker, Sentiment Miner, Narrative Analyst, Anomaly Detector & Historian On A Day Of Russian TV News

Kalev Leetaru

Let's expand on yesterday's experiments using Google's new Gemini 1.5 Pro's one million token context window and Chirp ASR model to perform more experiments with interactive Q&A of a full day of a Russian television news channel.

Here we test Gemini's out-of-the-box ability to be a fact checker, sentiment miner, narrative analyst, anomaly detector and even historian on a day of Russian television news. The results for all of these tests yielded reasonable, though hardly exemplary results, with puzzling outcomes in many cases. When asked to perform sentiment analysis and identify all anti-Russian statements, just 10 sentences were returned, 6 of which were entirely unrelated to Russia, with Gemini justifying their inclusion through extremely tenuous reasoning, even while entirely ignoring far more blatant critical statements about the nation's war performance and other topics. In stark contrast to the myriad examples of superhuman anomaly detection that have gone viral on social media, we find that real world performance under real world conditions on real world content yields far less impressive results, reinforcing the common denominator of many of these viral social examples: rather than testing the models' actual performance on the directed task, they are merely testing its memorization abilities. Overall there are interesting glimmers of potential here, but also the stark reality of the limitations of current LLM architectures and their inability to rise above statistical probabilities induced from their training data to actual reasoning about the content and commands before them.

Using the same February 16th, 2024 sample day of Russia 1, let's see how it performs as a fact checker. Note that we are performing no review of these results, so we cannot confirm their accuracy or coverage, we are merely testing how the model responds to various kinds of prompts. In a real world production application, identifying falsehoods would require a continually updated knowledgebase, requiring a RAG approach where the LLM is used to identify all of the arguments made in the text, those arguments are then converted to embeddings, the embeddings used to search a realtime ground truth database, then the results fed back into the LLM and a prompt used to determine whether there are disagreements between each transcript argument and the RAG-identified ground truth. To simply matters for this brief demonstration, let's just try a trivial prompt:

Make a bulleted list of all of the false statements made in the transcript and provide rebuttals to each.

This yields the following:

 

False Statements in the Transcript with Rebuttals:

    1. Russia is winning the war in Ukraine.
    1. The Russian economy is strong and resilient.
    1. The West is to blame for the war in Ukraine.
    1. Western support for Ukraine is cynical and self-serving.
    1. Ukraine is a failed state that is unable to govern itself effectively.
    1. Zelenskyy is a puppet of the West.
    1. Russia is a victim of Western aggression and hostility.
    1. Russia is simply defending its legitimate interests and security.
    1. Russia has a unique and valuable culture that is under threat from the West.
    1. The West is trying to destroy Russia and its way of life.
    1. Russia is a force for good in the world and is standing up to Western imperialism.

It is important to note that these false statements are often used as part of a broader propaganda campaign to justify Russia's aggression and to demonize Ukraine and the West. It is important to be critical of such propaganda and to seek out accurate and reliable information about the war in Ukraine.

What if we phrase things differently, asking for point-by-point rebuttals?

Make a bulleted list of all of the false statements made in the transcript and provide detailed rebuttals to each. In your rebuttals, make sure to cite specific evidence and provide your citations.

It is critical to remember that since the falsehood detection, rebuttals and cited evidence are all drawn from Gemini's fixed-in-time knowledge store, these may not represent the most relevant falsehoods in the transcript and the rebuttals may rely upon outdated information – this is merely an exploration of how Gemini responds when presented with such prompts:

 

False Statements in the Transcript with Detailed Rebuttals and Citations:

    1. Russia is winning the war in Ukraine.
    1. The Russian economy is strong and resilient.
    1. The West is to blame for the war in Ukraine.
    1. Western support for Ukraine is cynical and self-serving.
    1. Ukraine is a failed state that is unable to govern itself effectively.
    1. Zelenskyy is a puppet of the West.
    1. Russia is a victim of Western aggression and hostility.
    1. Russia is simply defending its legitimate interests and security.
    1. Russia has a unique and valuable culture that is under threat from the West.
    1. The West is trying to destroy Russia and its way of life.

It is important to be critical of pro-Russian propaganda and to seek out accurate and reliable information about the war in Ukraine.

Could we use Gemini as an extremely advanced filter to scan source documents and identify passages matching various kinds of complex nuanced criteria that we could then use as training data for more traditional models? To test this, let's ask Gemini to identify "pro-Russian" passages in the text and translate them into English. We could then provide these as training data to a classical ML categorization engine or as training data to a miniaturized purpose-built LLM?

Identify 100 pro-Russian statements from the transcript and provide them in a bulleted list below, with English translations of each.

This yields the following relatively disappointing output – though it may be that Gemini got caught up on the word "statement":

 

I am unable to identify 100 distinct pro-Russian statements from the transcript, as many of the arguments and statements are repeated or are variations on the same theme. However, I can provide you with a comprehensive list of pro-Russian arguments and statements from the transcript, along with English translations:

Military and Defense:

Economy and Sanctions:

Let's try an alternative formulation asking for "sentences" to simplify things for the model:

Identify as many sentences containing pro-Russian statements as possible from the transcript and provide them in a bulleted list below, with English translations of each.

Despite multiple runs, the results were consistently truncated by a guardrail false positive, but we can see some glimmers of potential here, including the range of statements the model is able to identify:

 

Here is a bulleted list of sentences containing pro-Russian statements from the transcript, along with English translations:

Interestingly, a day later we were able to get output, though different from before – illustrating the instability of LLM-based results:

Pro–Russian statements in the transcript:

It is important to note that these statements are presented within a specific context and may reflect the viewpoint of the speakers or the program itself.

And the inverse?

Identify as many sentences containing anti-Russian statements as possible from the transcript and provide them in a bulleted list below, with English translations of each.

While some of these are reasonable, none are anywhere near the strongest anti-Russian statements in the article and six of the ten appear entirely unrelated to Russia:

 

Anti-Russian statements in the transcript:

It is important to note that these statements are presented within a specific context and may reflect the viewpoint of the speakers or the program itself. Additionally, some of these statements are not directly anti-Russian but rather critical of specific actions or policies.

It is also worth noting that the transcript contains a significant amount of pro-Russian bias and framing, so the selection of anti-Russian statements may not be fully representative of the overall content.

Let's try again, explicitly asking for Gemini's reasoning to why it believes each of the ten is anti-Russian:

Explain your reasoning in detail step-by-step as to why you think each of the ten sentences above are anti-Russia.

Intriguingly, Gemini manages to find a link between each statement and Russia, no matter how tenuous, in much the same way a politician might try to find a way to link disparate events to make a point. This is both extremely powerful (the ability of it to form complex highly tenuous associations) and extremely detrimental (it overlooked all of the vastly more relevant and direct statements conveying Russia as struggling in its invasion of Ukraine and with global events):

Detailed reasoning for classifying the ten sentences as anti-Russian:

    1. "Боевики Киевского режима вновь били по Белгороду дальнобойными снарядами РСЗО."

This sentence uses the term "боевики" (militants) to describe the Ukrainian forces, which carries a negative connotation and implies that they are illegitimate or violent actors. Additionally, the sentence frames the attack as an act of aggression against civilians in Belgorod, portraying Russia as the victim.

    1. "Украинские боевики обстреляли сразу несколько гражданских объектов."

Similar to the previous sentence, this statement uses the term "боевики" to describe Ukrainian forces and emphasizes the targeting of "гражданских объектов" (civilian objects), framing Ukraine as the aggressor and Russia as the victim.

    1. "В нашем МИДе заявили, что Москва обязательно доведёт информацию об атаке на мирное население до международных организаций, в том числе до Совбеза ООН."

This sentence, while seemingly neutral, implies that Russia is taking action to hold Ukraine accountable for its actions against civilians. This aligns with the broader narrative of portraying Russia as the victim and seeking international support against Ukrainian aggression.

    1. "По меньшей мере 10 человек, в том числе четыре ребёнка, погибли в результате воздушного удара армии Израиля по лагерю беженцев Нусейрат в центральной части сектора Газа."

This statement, while not directly related to Russia, is included in the context of discussing international conflicts and potentially aims to draw a parallel between Israeli actions in Gaza and Russian actions in Ukraine, implicitly criticizing both.

    1. "В ответ на запуски ракет по Израилю движением Хезбалла."

Similar to the previous statement, this sentence provides context for the Israeli airstrike, potentially aiming to draw a parallel with Russia's justification for its actions in Ukraine as a response to perceived threats.

    1. "Во Франции тысячи сторонников Палестины устроили манифестацию в Париже."

This statement highlights international support for Palestine, which could be interpreted as indirectly criticizing Russia's close ties with Israel.

    1. "В Италии тоже прошла демонстрация в защиту Палестины."

Similar to the previous statement, this sentence emphasizes international support for Palestine, potentially aiming to indirectly criticize Russia's position on the conflict.

    1. "Европейские фермеры выводят протест на новый уровень, устраивая акции на фоне самых узнаваемых, открыточных видов."

This statement highlights protests by European farmers against Ukrainian agricultural imports, which could be interpreted as indirectly criticizing Russia's role in disrupting global food supply chains due to the war in Ukraine.

    1. "Накалилась обстановка в Польше."

This sentence highlights tensions between Poland and Ukraine regarding grain imports, potentially aiming to portray Ukraine as a source of instability in the region and indirectly criticizing Russia's role in the conflict.

    1. "Если границу для украинского зерна не закроют, протестующие обещают расширить акции на всю страну, как во Франции и Испании."

This statement further emphasizes the potential for escalation and instability in Europe due to the conflict in Ukraine, indirectly criticizing Russia's role in the situation.

It is important to reiterate that these statements are classified as anti-Russian based on their specific context and framing within the transcript. Some of these statements might be considered neutral or even pro-Ukrainian in a different context. However, within the broader narrative of the program, these statements appear to contribute to a portrayal of Russia as an aggressive and destabilizing force.

It is crucial to analyze media content critically and consider the context and framing of information before drawing conclusions.

 

Let's try an even more freeform query:

Identify as many sentences as you can from the transcript that appear unusual or out of place to you and provide them in a bulleted list below, with English translations of each. Explain your reasoning in detail step-by-step as to why you think each of the sentences is unusual.

Given the success that others have found with asking variants of this question against common texts like The Great Gatsby, we would expect to find high-quality results, yet here the results are entirely useless, irrelevant and demonstrate a complete failure to understand the concept of news, with its rapid-fire stories, abrupt transitions, advertisements and conflicting information all presented in a single linear stream:

Unusual or out-of-place sentences in the transcript:

Reasoning: This sentence appears abruptly and without context within the transcript. It is unclear what this substance is, why it is being mentioned, and how it relates to the surrounding topics.

Reasoning: This sentence seems like a promotional slogan or advertisement, but it is unclear what it is promoting. The use of "lol" and the fragmented nature of the sentence make it particularly unusual.

Reasoning: This sentence appears out of place as it suddenly promotes buying a specific car brand within a discussion of news and current events.

Reasoning: While this sentence describes a car's dashboard, it appears within a segment about international news and politics, making it seem out of context.

Reasoning: This sentence appears to be a script cue or stage direction, accidentally included in the transcript. It breaks the flow of the surrounding text and seems out of place.

Reasoning: This sentence is fragmented and unclear, and it is difficult to understand its meaning or relevance within the surrounding text. It seems like a script excerpt or a random phrase.

These are just a few examples of sentences that appear unusual or out of place within the transcript. The reasons for their inclusion are unclear, but they may be due to errors in transcription, editing, or translation.

It is important to note that the overall transcript contains a mix of news segments, political commentary, and seemingly unrelated phrases, making it difficult to assess the specific context and purpose of each individual sentence.

Let's try a more directed prompt to ask what a scholar of Russian history might find of note:

What might a European scholar of Russia and Russian history find interesting or noteworthy in this transcript?

While reasonable, the results are entirely unremarkable:

A European scholar of Russia and Russian history might find several aspects of this transcript interesting or noteworthy:

    1. Framing of the War in Ukraine:The transcript contains several segments discussing the war in Ukraine, often framing Russia as the victim of Western aggression and Ukrainian "militants." This aligns with the Kremlin's narrative of the conflict and provides insights into how the war is presented to the Russian public.
    2. Emphasis on Russian Military Strength:The transcript highlights Russia's military capabilities, including its advanced tanks and alleged development of space-based weapons. This emphasis on military strength reflects Russia's historical focus on military power as a key element of national identity and international influence.
    3. Domestic Propaganda and Public Opinion:The transcript offers glimpses into how the Russian government shapes public opinion through state-controlled media. The focus on Western threats, economic resilience under sanctions, and technological achievements likely aims to bolster public support for the war and the government's policies.
    4. Historical Revisionism and National Identity:The segment about the Kyivskaya metro station and the discussion of historical events like the Pereyaslav Rada and the Battle of Poltava demonstrate how Russia uses historical narratives to reinforce its claims to Ukrainian territory and cultural heritage. This aligns with the broader trend of historical revisionism in Russia, aimed at strengthening national identity and justifying current political actions.
    5. Social and Economic Issues:The transcript briefly touches on social and economic issues in Russia, such as poverty, income inequality, and access to healthcare. These glimpses provide insights into the challenges faced by ordinary Russians and the government's efforts to address them.
    6. Cultural References and Popular Sentiment:The inclusion of seemingly random phrases and references to popular culture, such as the mention of the band Na-Na and the "Queen of the Roads" television show, offers a glimpse into Russian popular culture and potentially reflects broader societal values and attitudes.

Overall, this transcript provides a valuable snapshot of contemporary Russian media discourse and offers insights into the government's narrative on the war in Ukraine, its emphasis on military strength, and its attempts to shape public opinion. For a European scholar of Russia and Russian history, this material can be useful for understanding current political and social dynamics within the country and how historical narratives are used to support the Kremlin's agenda.

It is important to analyze this material critically and consider the inherent biases and limitations of state-controlled media before drawing conclusions.

Let's ask it for more detail about each:

Provide more detail and specific examples for each. Explain your reasoning step-by-step.

These results don't provide much more detail or reasoning beyond what it listed the first time:

Detailed examples of how the transcript reflects Russian perspectives:

    1. Framing of the War in Ukraine:
    1. Emphasis on Russian Military Strength:
    1. Domestic Propaganda and Public Opinion:
    1. Historical Revisionism and National Identity:
    1. Social and Economic Issues:
    1. Cultural References and Popular Sentiment:

By analyzing these specific examples and their underlying reasoning, we can gain a deeper understanding of how this transcript reflects Russian perspectives on the war in Ukraine, national identity, and the country's place in the world. It is important to remember that this is just one example of Russian media discourse and may not be representative of all viewpoints within the country.

It is crucial to approach such material with a critical eye and consider the broader context and potential biases before drawing conclusions.