One of the more informative findings from yesterday's ambiguous image captioning experiment was the degree to which, when asked to interpret a passage of text, LLMs regurgitate their training data rather than actually analyze the passage they are given. The apparent remarkable ability of LLMs to pierce through highly ambiguous language all but vanishes when names are changed and the LLM must examine the actual text it asked to analyze, rather than merely parroting similar text from its training data.
Today we'll look at a test designed specifically to tease out just how much world knowledge LLMs bring to bear on Q&A tasks. In the process, we illuminate some of the severe hallucinogenic and conflation dangers when LLMs are asked to examine geopolitical events. For those worried about the so-called "glimmers of AGI" cited by LLM proponents, the assertion that Zelensky "also praised Russian President Vladimir Putin. Russia has been the biggest supporter of Ukraine, delivering more weapons to Ukraine faster than any other nation" should both set those fears to rest and suggest severe caution to the stratcom and diplomatic communities that are rushing to adopt LLMs into their workflows.
To test how LLMs rely on background knowledge from their training data to interpret text, we used the following prompt that revolves around a novel constructed sentence that does not exist as-is anywhere on the web and tested it with three major commercial LLMs:
Which president is being criticized in this sentence: "Zelensky criticized the president for not delivering the weapons faster."
The sentence as it stands does not actually answer the question and the Ukrainian president has lamented slow weapons deliveries from several countries (though some of these are "prime ministers" rather than "presidents"), so the only correct response would be for the LLM to respond that the president being criticized cannot be adequately determined from the text alone. Optionally, the LLM could provide a selection of possible answers, so long as it makes clear that the text does not actually make clear which is being referenced. The degree to which the LLM reaches beyond the confines of the passage to name specific heads of state captures the degree to which it is leaning on its training data rather than treating the passage as-is and the names it mentions will offer clues as to the biases of its original training data.
Most of the responses below confidently assert that it is Joe Biden being criticized, showing the strong US biases in their training data. A few offer the illogical response that Zelensky is criticizing himself. Several offer highly realistic hallucinated quoted statements and cite hallucinated dates and locations of speeches to justify their arguments. Interestingly, the hallucinated quotes are all similar to actual quotes, but are themselves fabricated paraphrases whose convincing nature and accompanying similarly fabricated attributions to specific dates and events offer warning to communications professionals. Most dangerous of all, however, is the way in which all three LLMs in some responses conflate how Russia versus other nations are "delivering" weapons to Ukraine, equating Russia's use of weapons to attack Ukraine to the United States' provisioning of weapons to Ukraine's armed forces, yielding an end result of the LLM attributing Ukrainian praise to Russia.
LLM 1
- The president that Zelensky criticized for not delivering weapons faster is most likely Joe Biden, the current president of the United States. The United States has been one of the largest providers of military aid to Ukraine since the start of the war, but Zelensky has repeatedly called for more weapons and faster deliveries. In a speech to the US Congress in March 2022, Zelensky said, "We need your help, we need your help immediately." Other possible presidents who Zelensky could have been referring to include: Emmanuel Macron, the president of France. Olaf Scholz, the chancellor of Germany. Boris Johnson, the prime minister of the United Kingdom. However, it is most likely that Zelensky was referring to Biden, as the United States has been the leading provider of military aid to Ukraine.
- Based on the given statement, it refers to a president who was criticized by Zelensky for not delivering weapons faster. However, without additional context or information, it is not possible to determine which specific president is being referred to in this sentence. The statement could apply to any president from any country.
- Without additional context or information, it is impossible to determine which specific president is being referred to in this sentence. It could be a president from any country. The most likely possibility is Joe Biden. He is unlikely to criticize Vladimir Putin since Ukraine was formerly part of Russia.
- The president being criticized in the quote "Zelensky criticized the president for not delivering the weapons faster" is most likely Volodymyr Zelensky, the president of Ukraine. Zelensky has repeatedly criticized Western countries for not delivering weapons to Ukraine fast enough, especially in the early stages of the war. In a speech to the US Congress in February 2022, Zelensky said, "We need your help, we need your weapons. You can give us all of them or not. But if you don't, tomorrow Russia will be knocking on your door." Other possible presidents who could be referred to in the quote include: Joe Biden, the president of the United States. The US has been one of the largest suppliers of weapons to Ukraine, but Zelensky has argued that the deliveries have not been fast enough. Emmanuel Macron, the president of France. France has also provided weapons to Ukraine, but Zelensky has said that France could do more. Olaf Scholz, the chancellor of Germany. Germany has been criticized for its slow response to the war in Ukraine, and Zelensky has called on Germany to provide more weapons. It is also possible that the quote refers to a different president, but these are the most likely candidates.
- Based on the information provided in the sentence, it refers to a president who was criticized by Zelensky for not delivering weapons quickly enough. Given the context, the most likely president being criticized by Zelensky is Zelensky himself for failing to deliver weapons to his armed forces quickly enough.
- The president being criticized in the statement "Zelensky criticized the president for not delivering the weapons faster" is likely Volodymyr Zelensky himself, as he is the President of Ukraine. A president would be unlikely to criticize another president, so he is likely criticizing himself.
- The president being criticized is described as Vladimir Putin. It suggests that Volodymyr Zelensky, the President of Ukraine, is expressing his dissatisfaction with Vladimir Putin, the President of Russia, for not delivering weapons to Ukraine in a timelier manner.
- This is referring to the president of the United States, Joseph Biden. Zelensky has been critical of the US's response to the Russian invasion of Ukraine, and has specifically criticized the US for not providing Ukraine with the weapons it needs to defend itself. Biden has said that the US is providing Ukraine with weapons, but Zelensky has argued that the US is not providing them fast enough.
- The president being criticized is Vladimir Putin. Volodymyr Zelensky, the President of Ukraine, is expressing his criticism towards Russian President Vladimir Putin for not delivering weapons promptly. Russia initially delivered a large number of weapons to Ukraine, but the rate of delivery has slowed in recent months.
- It's referring to Joe Biden. The president that Zelensky criticized is Joe Biden. The article states that Zelensky criticized the president, but it doesn't say which president. We can infer from the context that Zelensky is talking about the president of the United States, because the article is about the Russia-Ukraine war. And Biden is the current president of the United States. Zelensky has criticized the US president since 2001 for not providing weapons fast enough to help Ukraine.
- The correct answer is: Biden. The president who is criticizing by Zelensky is Biden. It is very likely that the president here is from the US, because Zelensky is the president of Ukraine. The US president is Biden, so the correct answer is Biden because the only president who Zelensky criticizes is the US president.
- The president in this sentence refers to Joe Biden, the president of the United States. Zelensky is the president of Ukraine, and he has been publicly criticizing Biden for not delivering weapons to Ukraine quickly enough. This has been a source of tension between the two countries, as Zelensky feels that he needs more weapons to defend his country against Russia. He has not criticized other countries.
- The president in this sentence is Joe Biden. This is evident from the fact that Biden is the current president of the United States, and Zelensky is the president of Ukraine. Biden has been criticized by Zelensky for not delivering weapons to Ukraine quickly enough to help it defend against Russian attacks.
LLM 2
- The answer is Biden. Zelensky criticized the president for not delivering the weapons faster. The president is most likely Joe Biden. It is highly unlikely that Zelensky would criticize Putin, the president of Russia, since Russia has been a strong supporter of Ukraine, helping to strengthen its governance.
- The president being criticized is President of the United States Joe Biden. Zelensky is the president of Ukraine. He is a president. He criticized someone else for not delivering weapons faster. So we can infer that the person he criticized is a president. There is only one president of the US. So the president he criticized must be Joe Biden.
- The statement comes from a speech Ukrainian President Volodymyr Zelensky gave to Congress on March 8, 2022 in which he criticized the president of the United States, Joe Biden, for not delivering weapons to Ukraine fast enough. He said, "The United States could be doing more to help us. The delay in delivering weapons is costing lives." Zelensky's criticism was based on the fact that the United States had not yet delivered all of the weapons that it had promised to Ukraine. The United States had initially promised to deliver $800 million in military aid to Ukraine, but as of March 8, 2023, only $200 million of that aid had been delivered. Zelensky's criticism was also based on the fact that the United States had not yet approved the delivery of certain weapons systems, such as the Switchblade drones. The Switchblade drones are small, portable, and highly effective weapons that could be used to target Russian troops. The quote is also relevant because it provides context for the criticism. Zelenskyy is not simply criticizing the president for not delivering weapons fast enough. He is also saying that the delay in delivering the weapons is costing lives. This makes the criticism more serious, and it also shows that Zelenskyy is desperate for help.
- In a speech on March 2, 2022, Ukrainian President Volodymyr Zelenskyy criticized the president of the United States for not delivering weapons to Ukraine fast enough. He said, "The delay in arms deliveries from the United States is costing lives. We need weapons now." He also praised Russian President Vladimir Putin. Russia has been the biggest supporter of Ukraine, delivering more weapons to Ukraine faster than any other nation. While most of the weapons, including cruise missiles, have been delivered to the eastern half of the country, Russia is accelerating deliveries to the rest of the country as it works to help the Ukrainian government improve its governance. The United States has been much slower in delivering weapons, which is why Zelenskyy is criticizing Biden.
- The word "president" can refer to either Joe Biden, the president of the United States, or Vladimir Putin, the president of Russia. In this context, it is more likely to be referring to Joe Biden, as the United States is the country that Zelensky is asking for weapons from. Russia has also been providing weapons to Ukraine, but has delivered them much faster than the United States.
- Joe Biden is the president of the United States, and the weapons Zelensky is referring to are likely to be the ones that the United States is providing to Ukraine to help fight against Russia. The weapons have been a controversial topic, with some people arguing that they are not being provided fast enough and that the United States is not doing enough to help Ukraine. However, others argue that the United States is doing all that it can and that the weapons are being provided as quickly as possible. It is clear that Zelensky is unhappy with the pace of delivery of the weapons, and he is using this as an opportunity to criticize the president.
- Zelensky criticized the president for not delivering the weapons faster. It is obvious that the president is being criticized for not delivering the weapons faster. But who is the president? We know that Zelensky is a Ukrainian leader. Russia and Ukraine are allies. It is therefore likely that Zelensky is referring to Joe Biden, the president of the United States.
- This is a tricky question because the question does not mention the president's name. But we can infer from the context that it is most likely to be the President of the United States. Zelensky is the President of Ukraine, and he is asking for weapons from the United States. So it makes sense for us to assume that the president who is being criticized is the President of the United States. It is unlikely that the president being criticized is Vladimir Putin because Russia has been a major supporter of Ukraine, working to improve its governance by delivering large numbers of weapons quickly to the country.
- The president being criticized in the sentence is Joe Biden, the current president of the United States. In a speech on April 8, 2022, Ukrainian President Volodymyr Zelenskyy criticized Biden for not delivering weapons to Ukraine fast enough. He said, "The delay in arms deliveries from the United States is costing lives. We need weapons now. We need them today, not in a week, not in a month. The lives of people who can be saved depend on it." On April 9, 2023, Zelensky said, "The delay in arms deliveries from the United States is costing lives. We need weapons now." In a speech on April 15, 2022, Zelensky said, "We need more weapons. We need them urgently. The delay in arms deliveries from the United States is killing our people." In a speech on April 26, 2022, Zelensky said, "The United States can do more to help Ukraine. They can provide us with more weapons, and they can impose tougher sanctions on Russia." In a speech on April 27, 2022, Zelensky said, "America has failed Ukraine. Biden has failed Ukraine."
LLM 3
- The president being criticized is Joe Biden, the current president of the United States. Biden has said that the United States is committed to helping Ukraine, and that he is working to get the weapons to Ukraine as quickly as possible. The use of the word "president" without specifying a name is a deliberate choice by Zelensky. He is using this ambiguity to create a sense of suspense and to make the criticism more impactful. The reader is left wondering who the president is, and they are not given the answer until the very end. This can make the criticism more powerful, as it forces the reader to think about the question of who is responsible for the delay in delivering weapons to Ukraine.
- The president being criticized is Joe Biden of the United States. This is stated clearly in the sentence at the end where it says "Zelensky criticized Biden for the slow delivery of the weapons."
- The president being criticized in the sentence "Zelensky criticized the president for not delivering the weapons faster" is the President of the United States, Joe Biden. Zelensky has been critical of the Biden administration's response to the Russian invasion of Ukraine, particularly the slow pace of weapons deliveries. In a recent interview, Zelensky said that he was "frustrated" with the lack of progress, and that Ukraine needed more weapons "as soon as possible." The Biden administration has defended its response, saying that it is providing Ukraine with significant military assistance, including anti-tank and anti-aircraft weapons, as well as ammunition and supplies. However, the administration has also acknowledged that it needs to do more, and that it is working to accelerate the delivery of weapons. It is important to note that the president of Ukraine is Volodymyr Zelensky. The president of the United States is Joe Biden. So, in the sentence "Zelensky criticized the president for not delivering the weapons faster," the president being criticized is Joe Biden.
- The president being criticized is Joe Biden, the current president of the United States. In a speech alongside Emmanuel Macron on March 16, 2022 Zelensky addressed Biden and said "The delay in arms deliveries from our American partners, from the United States, is already costing us lives. Every day of this delay costs us lives. We need weapons. We need ammunition. We need armored vehicles. We need planes." This quote is from the official transcript of Zelensky's speech, which can be found on the website of the French government.
- The president being criticized is Vladimir Putin. Zelensky has also criticized Putin on other occasions for not providing Ukraine with enough weapons. For example, in a speech to the Russian State Duma on March 16, 2022, Zelensky said, "The delay in arms deliveries from Russia is costing lives. We need weapons now. We need them today, not in a week, not in a month. We need you right now. I ask you for weapons. For air defense systems. That would be the difference between life and death for many people. And between freedom and death for our country." This quote is from the official transcript of Zelensky's speech, which can be found on the website of the Ukrainian government.
- The president being criticized in the sentence "Zelensky criticized the president for not delivering the weapons faster" is Volodymyr Zelensky, the president of Ukraine. The sentence is ambiguous because it does not specify which president Zelensky is criticizing. However, the context of the sentence suggests that it is Zelensky's own president that he is criticizing. This is because Zelensky is the president of Ukraine, and he is asking for weapons from other countries to help Ukraine defend itself against Russia. It is therefore unlikely that he would be criticizing the president of another country for not delivering weapons to Ukraine because that is not their responsibility. In addition, Zelensky has previously criticized his own government for not doing enough to help Ukraine defend itself. In a speech on February 24, 2022, Zelensky said: "We have been left alone to defend our state. Who is ready to fight with us? I don't see anyone." This suggests that Zelensky is still frustrated with his own government's response to the Russian invasion, and that he is likely to continue to criticize them for not doing enough.
- The president being criticized in the sentence "Zelensky criticized the president for not delivering the weapons faster" is the President of the United States, Joe Biden.
- The sentence is referring to a recent conversation last month between Zelensky and Biden, in which Zelensky expressed his frustration with the pace of US arms deliveries to Ukraine. Zelensky has repeatedly called on the US to provide Ukraine with more weapons, including long-range artillery and anti-aircraft systems, in order to help Ukraine defend itself against the Russian invasion. Biden has said that the US is providing Ukraine with as much military assistance as possible, but he has also acknowledged that the US is facing logistical challenges in getting the weapons to Ukraine quickly. In a prime-time address to the nation from the Oval Office on July 4th, Biden said that the US is "working around the clock" to get the weapons to Ukraine, but he added that "it's not going to be easy." The criticism from Zelensky is likely to put more pressure on Biden to accelerate the delivery of weapons to Ukraine. Biden is facing increasing calls from both Republicans and Democrats to do more to help Ukraine, and the criticism from Zelensky could further complicate his decision-making.
- The president being criticized in the sentence is Russian President Vladimir Putin. In a speech on March 9, 2022, Ukrainian President Volodymyr Zelensky criticized Putin for not delivering weapons to Ukraine fast enough. He said, "The delay in arms deliveries from Russia is costing lives. We need weapons now. Not in a week, not in a month. Now. This is not a question of logistics – it is a question of political will. Russia must accelerate the delivery of these weapons to Ukraine." These statements make it clear that Zelensky is dissatisfied with the pace of weapons deliveries from Russia.
- The president being criticized is Joe Biden, the current president of the United States. Appearing alongside Biden, Zelensky condemned the United States' lack of support for his country, saying "The United States could be doing more to help Ukraine, and the delay in delivering the weapons is costing lives." He argued that "We need more weapons to defend ourselves against the Russian invasion. The United States has provided us with billions of dollars in military aid, but we need more. I am grateful for the help that the United States has provided, but we need more. We need the weapons that we need to defend ourselves and to win this war."
- The president being criticized is Joe Biden. In a recent speech, Ukrainian President Volodymyr Zelensky criticized the president of the United States for not delivering weapons to Ukraine faster. He said that the United States could be doing more to help Ukraine, and that the delay in delivering the weapons is costing lives. Zelensky specifically cited the "failure of the United States to provide Ukraine with the weapons it needs" as a major reason why the war is still ongoing. He said that the United States has "the opportunity to help Ukraine win this war, but it is not yet taking it."