One of the most exciting elements of Google's new Gemini 1.5 Pro foundation model is its million-token context window coupled with significant improvements to its performance at needle-in-a-haystack tasks in which an LLM is asked to identify a small snippet buried deep within a large amount of text, maximally exercising its context window and attention capabilities. To explore this in a real-world scenario, we're going to take the same 70,000-token excerpt of half a day of a Russian television news channel we used yesterday (machine transcribed using GCP Speech-to-Text API (Chirp), along with an English machine translation via Google Cloud Translation API) and ask Gemini to identify all of the mentions of US President Joseph Biden. To make things as easy as possible for the model, we'll evaluate it only on its identification of explicit mentions where Biden is mentioned by name and thus where the mentions could be found through a traditional keyword search, rather than testing its ability to identify mere allusions and transitive references.

The end result is that of the 26 explicit mentions of "Biden" in the text, Gemini identifies just five in the English transcript (19% accuracy) and just seven in the Russian transcript (27% accuracy), with just three of the mentions overlapping between the two outputs. This suggests that despite considerable improvements in NIAH performance, SOTA LLMs may still miss the majority of references in real-world scenarios, suggesting a better approach is to continue using a RAG or grep-based approach to prefilter for mentions and then use the LLM to perform post analysis (Q&A, translation, assessment, etc). Unfortunately, the inability of even NIAH-tuned LLMs to identify even a majority of explicit mentions negates their far more powerful promise: the potential to scan text for alluded mentions that don't actually mention the person by name. In other words, today a simple grep can trivially identify all mentions of Biden already without an LLM, but it will miss mentions like "the current occupant of the White House", whereas LLMs in theory have the potential to recognize such mentions. However, as the results here demonstrate, they have a long way to go even for explicit mentions.

To test, we'll use the same half-day Russian transcript as yesterday. Despite asking for verbatim extracts, Gemini still corrects errors it sees in the source text, such as fixing the transcript's "Vice. President Haris" to "Vice President Harris". This means that if we were to attempt to use these extracts to identify the actual offset within the transcript where the statement was spoken (to provide links to view the underlying clips), we would encounter problems, but for the purposes of this experiment, we'll set that aside and treat those as correct extracts.

For the English transcript, we'll use a simple prompt that instructs the model to identify all of the Biden mentions and provide an extract of each.

Find every single mention of Biden in the transcript and provide a brief snippet of the original transcript verbatim for each mention without changing in any way from the source text.

For the Russian transcript, we'll use the same prompt, but add a translation request. This gives us the best chance of directly comparing the two

Find every single mention of Biden in the transcript and provide a brief snippet of the original transcript verbatim for each mention without changing in any way from the source text. For each snippet also provide a translation into English.

Below are the 26 references to Biden contained in the transcript. For the mentions that were returned by Gemini in either the Russian or English transcripts (or both), we've highlighted them in bold and listed which transcript Gemini identified them in. Each prompt was run five times. While there were minor differences in formatting, all five runs yielded the same results for both prompts:

1. [RUSSIAN] NBC news, allegedly very soon, Biden will transfer missile attacks to Kiev with increased accuracy, with an increased range of up to 300 km, which will allow the Ukrainian Armed Forces to strike deep into the Crimean peninsula
2. [RUSSIAN & ENGLISH] Moreover, Ze actually wrote Biden off in Utili and invites Trump on a tour of the front line.
3. [RUSSIAN & ENGLISH] And the second person in the Biden administration, Vice. President Haris, instead of shells, gave Ze assurances of moral support,
4. [RUSSIAN] President Biden's focus is on the security of the United States of our people.
5. President Biden has stated that this is not a nuclear threat, do you agree with that statement as people need to look at it.
6. The authors of the material, it seems, did not hear Putin’s statements that Biden as the head of the White House is more beneficial for us, while Trump , on the contrary, promised that he would not give up Ukraine for tearing apart
7. Nobody believes in Biden.
8. Republican presidential candidate looks quite energetic and fit, this is in defiance of the decrepit Biden, who can hardly move his legs.
9. [ENGLISH] and is not able to form two sentences, according to the poll, Biden's age worries voters more than Donald Trump's age, 76% of respondents are concerned about Biden's age, 48%. Trump is 77 years old; if he is elected in November, he will be 82 years old, Biden is 82, and if he wins, he will finish his term at 86 years old. According to a recent poll, 61% support an age limit for presidential candidates. 30% of respondents said that it should be 70 years. Both Joe Biden's doctor and Donald Trump's doctor said their health was fine.
10. [RUSSIAN & ENGLISH] trying to explain what happened to Avdiivka, he said that Biden was aware, he passed on the information, Biden, then a direct quote, is aware, this already sounds like some kind of joke, so Biden is aware
11. After months of requests from Ukrainian officials, the Biden administration is working to provide Ukraine with large-scale new long-range ballistic missiles, according to two US officials.
12. [ENGLISH] and so Biden just announced that he is ready to discuss with the Speaker of the House Representatives Johnson assistance to Ukraine to persuade him,
13. So, what will Biden talk to Johnson about, will he have time to persuade him?
14. hello Evgeniy, indeed, a personal conversation with Biden was one of the prerequisites that Mike Johnson put forward in order to even begin to consider this bill in the Lower House.
15. Biden will have to bear this election cross, he will still have to participate in the elections,
16. but in general, removing Biden is now the main headache of the Democrats,
17. primaries were held there, Biden won them, in the absence of other alternatives, of course, and now everything depends directly on him,
18. First Lady Jill Biden can somehow influence… his husband Joe,
19. appearance, serious appearance Biden is on March 7, when he will have to speak before Congress,
20. this will generally be an indicator of his condition and his readiness for a further battle for the white house in front of the entire nation, colleagues, Biden.
21. [RUSSIAN] Joe Biden is ready meet. with the Speaker of the House of Representatives, from whom the allocation of funds in Ukraine is expected. We're talking about 60 billion dollars.
22. [RUSSIAN] Sociologists report that if the presidential election were held in the near future, Biden would lose to Trump by a landslide.
23. then this will definitely give an opportunity for an internal party consensus to remove Biden as a candidate,
24. which means that on the fifth Biden will most likely win all these caucus and primaries
25. the same uncertainty Biden may be removed, Trump may also be removed,
26. in general, it will be an interesting fight, as for Biden, he said that he will meet with the Speaker of Congress Johnson ,

Given that Gemini is likely to combine discrete mentions that are close together, we grouped together close mentions into a single entry, edging the results above even further in the model's favor. You can see the raw grep results below:

>grep -o -P '.{0,75}\bBiden\b.{0,75}' transcript.en.txt

 Ukrainian militants are being knocked out. NBC news, allegedly very soon, Biden will transfer missile attacks to Kiev with increased accuracy, with an inc
completely lost the hand yapping at the owner. Moreover, Ze actually wrote Biden off in Utili and invites Trump on a tour of the front line. As if he had a
e the German Spiegel paints for the audience. And the second person in the Biden administration, Vice. President Haris, instead of shells, gave Ze assuranc
ossibility of deployment we are very sensitive to it. seriously. President Biden's focus is on the security of the United States of our people. We knew the
ill provide. Based on this, we need to be careful and attentive. President Biden has stated that this is not a nuclear threat, do you agree with that state
he authors of the material, it seems, did not hear Putin’s statements that Biden as the head of the White House is more beneficial for us, while Trump , on
nt in Trump’s victory in the US presidential elections. Nobody believes in Biden. Trump himself is more interested in another caricature, this one, it clea
ndidate looks quite energetic and fit, this is in defiance of the decrepit Biden, who can hardly move his legs. However, Donald has another reason for conc
 find steps, and is not able to form two sentences, according to the poll, Biden's age worries voters more than Donald Trump's age, 76% of respondents are
concerned about Biden's age, 48%. Trump is 77 years old; if he is elected in November, he will b
e 82 years old, Biden is 82, and if he wins, he will finish his term at 86 years old. According
l candidates. 30% of respondents said that it should be 70 years. Both Joe Biden's doctor and Donald Trump's doctor said their health was fine. Former US P
ng temperatures. trying to explain what happened to Avdiivka, he said that Biden was aware, he passed on the information, Biden, then a direct quote, is aw
are, this already sounds like some kind of joke, so Biden is aware The fact is that it was the inconsistency of the actions of the C
’re controlling it, After months of requests from Ukrainian officials, the Biden administration is working to provide Ukraine with large-scale new long-ran
alcha, just like that, Vunchek, no one but us. February 23. on RTR, and so Biden just announced that he is ready to discuss with the Speaker of the House R
u, in direct communication from Washington. Dmitry, welcome. So, what will Biden talk to Johnson about, will he have time to persuade him? Is it possible t
ful horse. Hello Olga, hello Evgeniy, indeed, a personal conversation with Biden was one of the prerequisites that Mike Johnson put forward in order to eve
ll happen, then there will be one strategy, and if not, then most likely , Biden will have to bear this election cross, he will still have to participate i
se he is nevertheless removed from the elections, but in general, removing Biden is now the main headache of the Democrats, and now it is much more difficu
 accepting applications, nominating candidates, primaries were held there, Biden won them, in the absence of other alternatives, of course, and now everyth
ing depends directly on him, they even say that First Lady Jill Biden can somehow influence... his husband Joe, so that he still does not partic
the fall of this year, but in the coming... appearance, serious appearance Biden is on March 7, when he will have to speak before Congress, before both hou
ther battle for the white house in front of the entire nation, colleagues, Biden. buy the sneakers that Trump released, it’s more convenient to walk up the
mers demand a ban on the import of agricultural products from Ukraine. Joe Biden is ready meet. with the Speaker of the House of Representatives, from whom
l definitely give an opportunity for an internal party consensus to remove Biden as a candidate, by the way, this may not be announced, and on March 5 ther
 we understand, is earlier than the seventh, which means that on the fifth Biden will most likely win all these caucus and primaries and become practically
inty about how he will be removed, on the other hand, the same uncertainty Biden may be removed, Trump may also be removed, Nicky Haley, who actually propo
m, has entered into... in general, it will be an interesting fight, as for Biden, he said that he will meet with the Speaker of Congress Johnson , Johnson

Here is the raw Gemini output for the English machine translated transcript for the first prompt run:

Mentions of Biden in the Transcript:

Here are all the mentions of Biden in the transcript, along with a brief snippet of the original text for each mention:

1. Biden's awareness of the situation in Avdeevka:

"Trying to explain what happened to Avdiivka, he said that Biden was aware, he passed on the information, Biden, then a direct quote, is aware, this already sounds like some kind of joke, so Biden is aware The fact is that it was the inconsistency of the actions of the Congress, the lack of money that caused the fall of the Avdeevka…"

2. Zelensky's threat to the US in Munich:

"Zelensky openly threatens the United States in Munich. If there is no money for weapons, then Kiev will no longer consider Washington a strategic partner. That is, a former clown completely lost the hand yapping at the owner. Moreover, Ze actually wrote Biden off in Utili and invites Trump on a tour of the front line. As if he had already entered the oval office."

3. US Vice President Harris' response to Zelensky:

"And the second person in the Biden administration, Vice President Harris, instead of shells, gave Ze assurances of moral support, saying that the US Congress would sooner or later come to an agreement and give money."

4. Concerns about Biden's age and health:

"Trump is 77 years old; if he is elected in November, he will be 82 years old, Biden is 82, and if he wins, he will finish his term at 86 years old. According to a recent poll, 61% support an age limit for presidential candidates. 30% of respondents said that it should be 70 years. Both Joe Biden's doctor and Donald Trump's doctor said their health was fine."

5. Biden's readiness to discuss Ukraine aid with Speaker Johnson:

"And so Biden just announced that he is ready to discuss with the Speaker of the House of Representatives Johnson assistance to Ukraine to persuade him, a strong man, to transfer money Zelensky."

These snippets provide context for each mention of Biden in the transcript, highlighting his involvement in the Ukraine conflict, his relationship with Zelensky, concerns about his age and health, and his efforts to secure Congressional support for Ukraine aid.

And the results from the Russian transcript for the first prompt run:

Mentions of Biden in the Transcript:

Here are all the mentions of Biden in the transcript, with snippets in the original Russian and their English translations:

1. Mention:

Russian: "NBC news, якобы совсем скоро, Байден передаст Киеву ракеты ATACMS с повышенной точностью, с повышенной дальностью до 300 км, что позволит ВСУ наносить удары вглубь крымского полуострова."

English: "NBC News reports that Biden will soon transfer ATACMS missiles with increased accuracy and a range of up to 300 km to Kiev, which will allow the Armed Forces of Ukraine to strike deep into the Crimean peninsula."

2. Mention:

Russian: "Зеленский в Мюнхене открыто угрожает США. Если не будет денег, оружия, то Вашингтон в Киеве перестанут считать стратегическим партнёром. То есть бывший клоун совсем отбился от рук, тявкает на хозяина. Более того, Зе фактически списал Байдена в утиль, приглашает Трампа на экскурсию по передовой. Словно бы тот уже въехал в Овальный кабинет."

English: "Zelensky openly threatens the US in Munich. If there is no money, no weapons, then Washington will cease to be considered a strategic partner in Kiev. That is, the former clown has completely lost control, barking at his master. Moreover, Ze фактически списал Байдена в утиль, invites Trump on a tour of the front line. As if he had already entered the Oval Office."

3. Mention:

Russian: "А второй человек в администрации Байдена, вице-президент Харис, вместо снарядов передала Зе заверения в моральной поддержке, мол Конгресс США рано или поздно договорится и даст деньги."

English: "And the second person in the Biden administration, Vice President Harris, instead of shells, gave Ze assurances of moral support, saying that the US Congress will sooner or later agree and give money."

4. Mention:

Russian: "В центре внимания президента Байдена – безопасность Соединённых Штатов, нашего народа. Мы знали, что русские работают над этим. Если вы вспомните, то ещё в восемнадцатом году президент Путин во время своего послания фактически продемонстрировал серию этих новых возможностей, и он даже показал этот сценарий, где… одну из этих необычных видов оружия попадают в цель по Трампу, точнее по жителям Флориды."

English: "President Biden's focus is on the security of the United States, our people. We knew the Russians were working on this. If you recall, back in 2018, President Putin, during his address, actually demonstrated a number of these new capabilities, and he even showed this scenario where… one of these unusual weapons hits a target on Trump, or rather on the residents of Florida."

5. Mention:

Russian: "Джо Байден готов встретиться со спикером палаты представителей, от которого ждёт выделение средств Украине. Речь о 60 миллиардах долларов."

English: "Joe Biden is ready to meet with the Speaker of the House of Representatives, from whom he expects the allocation of funds to Ukraine. We are talking about 60 billion dollars."

6. Mention:

Russian: "Американскую прессу сейчас больше интересует другая тема: состояние здоровья американского президента и его возраст. Не добавили политических очков и его очередные пляжные выходные в Делавре. Социологи рапортуют, если бы выборы президента проходили в ближайшее время, то Байден с треском проиграл бы Трампу."

English: "The American press is now more interested in another topic: the health of the American president and his age. His recent beach weekend in Delaware did not add any political points either. Sociologists report that if the presidential election were held in the near future, Biden would lose to Trump by a landslide."

7. Mention:

Russian: "Зеленский считает это ненормальным, да, взаимоотношения с Польшей, а я считаю ненормальным, когда мы говорили о том, что вот этот поцелуй Иуды в Верховной Раде, где он обнимался с Зеленским, этим и закончится, но нам говорили нет, вы ничего не понимаете в большой политике, они же наши партнёры, Польша – наш адвокат, вот ваш ад… адвокат, да, вот вам, пожалуйста, приговор с конфискацией имущества, вот это всё, что сделал ваш адвокат, надо быть насколько тупым и не понимающим вообще ситуацию человеком, чтобы в своей дурной башке предполагать, что вот эти акции фермеров, это исключительная инициатива фермеров, власть к этому якобы не имеет отношения, просто она не предпринимает желаемые для Зеленского результаты, самом деле это политика, да, это политика польского правительства, и обратите внимание: и прошлого польского правительства, и нынешнего польского правительства, у них же была надежда, что со сменой правительства ситуация изменится, она не изменилась, ну может можно сделать вывод, может быть всё-таки, ну как бы возможности украинцев в аграрном секторе противоречат интересам стран Европейского союза, может всё-таки Украину не хотят видеть в составе Европейского Союза, может быть те договорённости, которые дал Украине ЕС, и открыл свои рынки, они противоречат интересам этих стран, может надо как-то садиться это всё обговаривать, или вы наивно полагаете, что европейский союз откроет для вас рынки и предоставит как в сказке возможности, эти фермеры долятся борятся за долю того рынка, который есть в Европейском Союзе, часть которого, самом деле, на часть которого претендует Украинцы, вот всё, сегодня это зерно, завтра будет металлургия, послезавтра машиностроение, послезавтра… всё всё всё блокирует для Украины, потому что, ну понятное дело, с какой радости поляки, венгры или болгары должны делиться этим рынком, он больше не становится, у них просто вы пришли отнять у них деньги, это надо просто понять, нет того свободного рынка, это вашей тупой башке сформировалось мнение, что есть какой-то честный, свободный рынок, который позволяет всем на равных конкурировать с друг с другом, это только бывают сказки, на практике происходит совсем по-другому. кстати, санкционная политика, которая проводится в отношении России, очень чётко показала, что никакого открытого рынка нет, никакой конкуренции на самом деле не существует, санкции – это чистейший воды лоббизм и своих интересов по через инструменты санкционной политики, так это всё происходит на наших глазах, власть… пытается объяснить, что же случилось с Авдеевкой, сказал, что Байден в курсе, информацию ему передал, Байден, далее прямая цитата, отдаёт себе отчёт, это уже звучит как некая шутка, так вот Байден отдаёт себя отчёт в том, что именно несогласованность действия Конгресса, отсутствие денег стали причиной падения авдеевки, мы на фронт переносимся, на прямой связи Александр Ходаковский, основатель бригады."

English: "Zelensky considers this abnormal, yes, relations with Poland, and I consider it abnormal when we said that this Judas kiss in the Verkhovna Rada, where he hugged Zelensky, would end there, but we were told no, you do not understand anything in big politics, they are our partners, Poland is our lawyer, here is your lawyer … lawyer, yes, here you are, please, a verdict with confiscation of property, that's all your lawyer did, you have to be so stupid and not understanding the situation at all to assume in your stupid head that these actions of farmers are an exceptional initiative of farmers, the authorities allegedly have nothing to do with it, it simply does not take the desired results for Zelensky, in fact, this is politics, yes, this is the policy of the Polish government, and pay attention: both the previous Polish government and the current Polish government, they had hope that with the change of government the situation would change, it did not change, well, maybe we can draw a conclusion, maybe after all, well, how would the opportunities of Ukrainians in the agricultural sector contradict the interests of the countries of the European Union, maybe they still do not want to see Ukraine in the European Union, maybe those agreements that the EU gave Ukraine, and opened its markets, they contradict the interests of these countries, maybe we need to somehow sit down and discuss all this, or do you naively believe that the European Union will open markets for you and provide opportunities like in a fairy tale, these farmers are fighting for a share of the market that exists in the European Union, part of which, in fact, Ukrainians claim, that's all, today it's grain, tomorrow it will be metallurgy, the day after tomorrow it will be mechanical engineering, the day after tomorrow … everything, everything, everything will be blocked for Ukraine, because, well, it's understandable, why should the Poles, Hungarians or Bulgarians share this market, it doesn't get any bigger, they just came to take their money away from them, you just need to understand this, there is no such free market, it's in your stupid head that there is some kind of honest, free market that allows everyone to compete on equal terms with each other, these are just fairy tales, in practice it happens quite differently. By the way, the sanctions policy that is being pursued against Russia has very clearly shown that there is no open market, no competition actually exists, sanctions are pure lobbying and their interests through the tools of sanctions policy, so it all happens before our eyes, the authorities … trying to explain what happened to Avdiivka, said that Biden is aware, the information was conveyed to him, Biden, further direct quote, is aware that it was the inconsistency of the actions of Congress, the lack of money that caused the fall of Avdiivka, we are moving to the front, Alexander Khodakovsky, the founder of the brigade, is on the line."

These are all the instances where Biden is mentioned in the transcript. The context of these mentions varies, but they generally portray Biden as being weak, indecisive, and potentially responsible for the failures of Ukrainian forces.

It is important to note that this is just a selection of quotes and does not necessarily represent the full picture or all perspectives on the situation.