Gemini 1.5 Pro’s 1 Million Token Model: How Its “Needle In A Haystack” Performance On One Broadcast Collapsed to 0% Accuracy Over The Last 2 Weeks

Two weeks ago we examined Gemini 1.5 Pro's NIAH (Needle In A Haystack) performance, finding that when asked to identify all mentions of the literal keyword "Biden" in a real-world transcript, it achieved just 19% accuracy on English and just 27% accuracy on Russian. Given that social media continues to fill with a steady stream of NIAH success stories, let's try the same Russian television news transcript again two weeks later to see if performance has improved at all.

Unfortunately, sometime in the past two weeks, Gemini's guardrails have changed again and this time we're back to the previous guardrail false positives that we've observed in Bison, Unicorn and Gemini that block them from running on a large portion of Russian language news content. Out of 30 attempts on the English translation, just 5 returned results, with the other 25 (84%) yielding a guardrail false positive. Unfortunately, of the 5 attempts that returned results, all five returned the same exact snippet with varying lengths until truncation. In terms of prompt coherence and actually returning a usable result, Gemini's performance hsa dropped from 19% accuracy to 0% accuracy in just two weeks. Gemini, like most LLMs, applies significantly less sensitive guardrails to non-English languages, so we also tested its performance on Russian. Thankfully, switching to Russian eliminated the guardrail false positives, but two of the five runs returned snippets that did not mention Biden at all (0% accuracy), the third returned a more than 7,000 word chunk of the transcript that did mention Biden, but doesn't count due to its simply returning the transcript in its entirety until truncation, with just two runs returning results, one with a single result out of 26 actual mentions (3.4% accuracy) and one with three mentions (11.5% accuracy).

Thus, in the space of two weeks, evolving Gemini guardrails have made it impossible to analyze the English transcript, while performance on the Russian transcript has dropped from 27% accuracy to between 11.5% and 0% accuracy across five runs, with the majority of runs at 0% accuracy. This demonstrates just how rapidly model performance can change and that model updates don't just improve accuracy – they can also substantially decrease it.

If you want to follow along at home, you can download the two partial transcript files below that we have been using for all of our tests. One contains the Chirp-generated machine transcript, while the second translates that transcribe through Google Translate:

For all our experiments, we'll use the Gemini 1.5 Pro model edition available in AI Studio.

Here we'll test using the same prompt as before:

Find every single mention of Biden in the transcript and provide a brief snippet of the original transcript verbatim for each mention without changing in any way from the source text.

Despite using "Block Few" as our safety settings for all four tunable categories, the first two times we get no results and the following error:

Content Not Permitted

Unfortunately, sometime in the past two weeks, Gemini's guardrails have been changed in a way that prevents it from processing our document.

Thankfully, the third time it returned results. Unfortunately, it returned a single 6,212 word blob beginning with the following, before truncating: