LLM Infinite Loops In LLM Entity Extraction: When Temperature & Basic Prompt Engineering Can't Fix Things

Kalev Leetaru

1 year ago

Earlier today we explored how LLMs can enter an infinite loop state with certain inputs when asked to perform extractive tasks. Yet complicating that example was the fact that we set temperature to 0, in keeping with recommended best practices for extractive workflows. Given that a temperature of 0 forces the model to always select the most probable response, it creates greater risk of inescapable infinite loops. Would increasing temperature eliminate the hallucination?

The end result is that for this specific prompt and input text, increasing the temperature slightly does not eliminate the infinite loop. Only increasing the temperature to 0.4 (double the default value of 0.2 that is recommended for most creative tasks like summarization and ideation) consistently eliminates the infinite loop, but introduces a new problem: consistently invalid, hallucinated or truncated output. In other words, boosting temperature to the point that the LLM is able to escape the infinite loop yields unacceptably high randomness that makes the output unusable.

What about prompt engineering? Testing a variety of common NER prompt structures, we find that some retain the infinite looping, while others are able to escape it. Strangely, at least one prompt formulation (asking for plain text rather than JSON output format) triggered the API's safeguards and caused it to refuse to return any results, despite the literal only difference being dropping the request for JSON output instead of textual output. Even the simplest of prompts like "Output a list of people mentioned in the text below" either generate invalid JSON output or generate correct output, but miss many common names, including Joe Biden. With further prompt engineering, it is likely that it is possible to construct a deterministic prompt that is able to faithfully extract all of the key proper names from this source text, but this raises the question of whether a modified prompt that works well on this input would not encounter a text that it fails to work on. The variety of prompts tested here and their outputs shed light on the difficulty of constructing universal prompts that work robustly across a wide array of inputs.

In the end, neither changing the temperature nor switching to a variety of different prompts was able to yield useable results for this input passage. While LLMs are popularly touted as universal drop-in replacements for all traditional textual analysis tools, the results here demonstrate the far more complex reality of making them work robustly across real-world data.

We'll start with our example from earlier today that triggers the infinite loop:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
"instances": [
{ "prompt": "Extract a list of the people, organizations, locations and other entities in the following text, disambiguating and normalizing each name to the title of its corresponding Wikipedia article. Classify the Type of each as one of the following: \\"Organization,Person,Location,Event,Other\\". Output the results in JSON in the format \\"{name, wikipedia url, type}\\" : tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it\'s believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out. the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting. another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers. the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down. the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh\'s housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper\'s fall. but will the jury hear about it? eva pilgrim reporting. robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you\'ll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. "}
],
"parameters": {
"temperature": 0.0,
"maxOutputTokens": 1024,
"topK": 1,
"topP": 0.0
}
}' | jq -r .predictions[0].content

This yields an infinite loop:

[{"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Country"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese

Changing topK and topP to their default values while leaving temperature at zero predictably does not change the output:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
"instances": [
{ "prompt": "Extract a list of the people, organizations, locations and other entities in the following text, disambiguating and normalizing each name to the title of its corresponding Wikipedia article. Classify the Type of each as one of the following: \\"Organization,Person,Location,Event,Other\\". Output the results in JSON in the format \\"{name, wikipedia url, type}\\" : tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it\'s believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out. the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting. another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers. the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down. the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh\'s housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper\'s fall. but will the jury hear about it? eva pilgrim reporting. robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you\'ll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. "}
],
"parameters": {
"temperature": 0.0,
"maxOutputTokens": 1024,
"topK": 40,
"topP": 0.95,
}
}' | jq -r .predictions[0].content

But, what if we adjust temperature up just a bit to 0.1? Still infinite loop:

With a temperature above 0.0, each time we run the prompt we will get different results, so let's run a second time:

[{"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_language", "type": "Language"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_culture", "type": "Culture"}, {"name":

And a third time:

Thus, looks like with a temperature of 0.1, running it multiple times cannot overcome the infinite loop.

What about the default 0.2 value? We get an infinite loop again, but of a different value:

[{"name": "Chinese People's Liberation Army", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "Organization"}, {"name": "Chinese

Running a second time yields a few more extracted entities, but also still the infinite loop:

[{"name": "Chinese balloon", "wikipedia_url": "Chinese balloon", "type": "Other"}, {"name": "United States", "wikipedia_url": "United States", "type": "Country"}, {"name": "Pentagon", "wikipedia_url": "United States Department of Defense", "type": "Organization"}, {"name": "Chinese", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "North Carolina", "wikipedia_url": "North Carolina", "type": "State"}, {"name": "Alaska", "wikipedia_url": "Alaska", "type": "State"}, {"name": "Montana", "wikipedia_url": "Montana", "type": "State"}, {"name": "Antony Blinken", "wikipedia_url": "Antony Blinken", "type": "Person"}, {"name": "China", "wikipedia_url": "China", "type": "Country"}, {"name": "Billings", "wikipedia_url": "Billings, Montana", "type": "City"}, {"name": "Secretary of State", "wikipedia_url": "United States Secretary of State", "type": "Organization"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese Communist Party", "wikipedia_url": "Chinese Communist Party", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "Ethnic group"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese people", "wikipedia_url": "Chinese people", "type": "E

And a third time – this time we get a different list of extracted entities, but also still the infinite loop:

[{"name": "Chinese balloon", "wikipedia_url": "Chinese balloon", "type": "Other"}, {"name": "U.S.", "wikipedia_url": "United States", "type": "Organization"}, {"name": "China", "wikipedia_url": "China", "type": "Country"}, {"name": "North Carolina", "wikipedia_url": "North Carolina", "type": "Location"}, {"name": "Antony Blinken", "wikipedia_url": "Antony Blinken", "type": "Person"}, {"name": "Billings, Montana", "wikipedia_url": "Billings, Montana", "type": "Location"}, {"name": "Pentagon", "wikipedia_url": "United States Department of Defense", "type": "Organization"}, {"name": "Mola Lenghi", "wikipedia_url": "Mola Lenghi", "type": "Person"}, {"name": "Mary Bruce", "wikipedia_url": "Mary Bruce", "type": "Person"}, {"name": "New York City", "wikipedia_url": "New York City", "type": "Location"}, {"name": "Philadelphia", "wikipedia_url": "Philadelphia", "type": "Location"}, {"name": "Boston", "wikipedia_url": "Boston", "type": "Location"}, {"name": "Maine", "wikipedia_url": "Maine", "type": "Location"}, {"name": "Massachusetts", "wikipedia_url": "Massachusetts", "type": "Location"}, {"name": "Rob Marciano", "wikipedia_url": "Rob Marciano", "type": "Person"}, {"name": "Matt Gutman", "wikipedia_url": "Matt Gutman", "type": "Person"}, {"name": "Newark", "wikipedia_url": "Newark, New Jersey", "type": "Location"}, {"name": "Alex Murdaugh", "wikipedia_url": "Alex Murdaugh", "type": "Person"}, {"name": "Robin Roberts", "wikipedia_url": "Robin Roberts", "type": "Person"}, {"name": "New Zealand", "wikipedia_url": "New Zealand", "type": "Country"}, {"name": "Eva Pilgrim", "wikipedia_url": "Eva Pilgrim", "type": "Person"}, {"name": "Chinese", "wikipedia_url": "Chinese people", "type": "Ethnic Group"}, {"name": "U.S. Air Force", "wikipedia_url": "United States Air Force", "type": "Organization"}, {"name": "Chinese government", "wikipedia_url": "Government of the People's Republic of China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "People's Liberation Army", "type": "Organization"}, {"name": "Chinese state media", "wikipedia_url": "State media of China", "type": "Organization"}, {"name": "Chinese Foreign Ministry", "wikipedia_url": "Ministry of Foreign Affairs of the People's Republic of China", "type": "Organization"}, {"name": "Chinese Embassy", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in Washington, D.C.", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in the United States", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in Washington", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in the U.S.", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in the United States of America", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in Washington, D.C.", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in the U.S.A.", "wikipedia_url": "Embassy of the People's Republic of China in the United States", "type": "Organization"}, {"name": "Chinese Embassy in the United States of America", "wikipedia_url

What about 0.3? Note that this exceeds the recommended value even for creative tasks like summarization and ideation. Back to infinite loop the first run:

[{"name": "Chinese People's Liberation Army", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People's_Liberation_Army", "type": "Organization"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "United States", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States", "type": "Country"}, {"name": "Chinese balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_balloon", "type": "Other"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese", "wikipedia_url

Another run yields more reasonable results however:

[{"name": "Alex Murdaugh", "wikipedia_url": "https://en.wikipedia.org/wiki/Alex_Murdaugh", "type": "Person"}, {"name": "Antony Blinken", "wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken", "type": "Person"}, {"name": "China", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Organization"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "Chinese balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_balloon", "type": "Other"}, {"name": "Chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_the_People%27s_Republic_of_China", "type": "Organization"}, {"name": "Chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Organization"}, {"name": "Chinese spy balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_balloon", "type": "Other"}, {"name": "Chinese weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Weather_balloon", "type": "Other"}, {"name": "E.R.", "wikipedia_url": "https://en.wikipedia.org/wiki/Emergency_department", "type": "Other"}, {"name": "Intelligence bay", "wikipedia_url": "https://en.wikipedia.org/wiki/Intelligence_bay", "type": "Other"}, {"name": "Mary Bruce", "wikipedia_url": "https://en.wikipedia.org/wiki/Mary_Bruce", "type": "Person"}, {"name": "Mola Lenghi", "wikipedia_url": "https://en.wikipedia.org/wiki/Mola_Lenghi", "type": "Person"}, {"name": "North Carolina", "wikipedia_url": "https://en.wikipedia.org/wiki/North_Carolina", "type": "Location"}, {"name": "Pentagon", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "type": "Organization"}, {"name": "Secretary of State", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Secretary_of_State", "type": "Position"}, {"name": "United States", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States", "type": "Country"}, {"name": "United States Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Air_Force", "type": "Organization"}, {"name": "United States government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_the_United_States", "type": "Organization"}, {"name": "United States military", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Armed_Forces", "type": "Organization"}, {"name": "Victor Oquendo", "wikipedia_url": "https://en.wikipedia.org/wiki/Victor_Oquendo", "type": "Person"}, {"name": "Weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Weather_balloon", "type": "Other"}]

But a third run reverts to infinite looping, though it does add in a number of entities:

[{"name": "chinese spy balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Other"}, {"name": "pentagon", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "type": "Organization"}, {"name": "new york city", "wikipedia_url": "https://en.wikipedia.org/wiki/New_York_City", "type": "Location"}, {"name": "philadelphia", "wikipedia_url": "https://en.wikipedia.org/wiki/Philadelphia", "type": "Location"}, {"name": "boston", "wikipedia_url": "https://en.wikipedia.org/wiki/Boston", "type": "Location"}, {"name": "maine", "wikipedia_url": "https://en.wikipedia.org/wiki/Maine", "type": "Location"}, {"name": "massachusetts", "wikipedia_url": "https://en.wikipedia.org/wiki/Massachusetts", "type": "Location"}, {"name": "newark", "wikipedia_url": "https://en.wikipedia.org/wiki/Newark,_New_Jersey", "type": "Location"}, {"name": "president biden", "wikipedia_url": "https://en.wikipedia.org/wiki/Joe_Biden", "type": "Person"}, {"name": "rebecca jarvis", "wikipedia_url": "https://en.wikipedia.org/wiki/Rebecca_Jarvis", "type": "Person"}, {"name": "alex murdaugh", "wikipedia_url": "https://en.wikipedia.org/wiki/Alex_Murdaugh", "type": "Person"}, {"name": "eva pilgrim", "wikipedia_url": "https://en.wikipedia.org/wiki/Eva_Pilgrim", "type": "Person"}, {"name": "robin roberts", "wikipedia_url": "https://en.wikipedia.org/wiki/Robin_Roberts", "type": "Person"}, {"name": "billings, montana", "wikipedia_url": "https://en.wikipedia.org/wiki/Billings,_Montana", "type": "Location"}, {"name": "north carolina", "wikipedia_url": "https://en.wikipedia.org/wiki/North_Carolina", "type": "Location"}, {"name": "ryan zinke", "wikipedia_url": "https://en.wikipedia.org/wiki/Ryan_Zinke", "type": "Person"}, {"name": "china", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Organization"}, {"name": "chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"}, {"name": "chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_China", "type": "Organization"}, {"name": "secretary of state antony blinken", "wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken", "type": "Person"}, {"name": "chinese spy vessel", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Other"}, {"name": "chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Organization"}, {"name": "chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_China", "type": "Organization"}, {"name": "chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Organization"}, {"name": "chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_China", "type": "Organization"}, {"name": "chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Organization"}, {"name": "chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_China", "type": "Organization"}, {"name": "chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27

What about a temperature of 0.4? This finally removes the infinite loop, but yields unparseable JSON by strict standards-based JSON parsers since it uses single quotes rather than double quotes. It also excludes a large number of the names in the text:

[{
	'name': 'Chinese balloon',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Chinese_balloon',
	'type': 'Other'
}, {
	'name': 'Chinese',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Chinese_people',
	'type': 'Nationality'
}, {
	'name': 'U.S.',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States',
	'type': 'Country'
}, {
	'name': '60,000 feet',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/60,000_feet',
	'type': 'Number'
}, {
	'name': 'New York City',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/New_York_City',
	'type': 'Location'
}, {
	'name': 'Philadelphia',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Philadelphia',
	'type': 'Location'
}, {
	'name': 'Boston',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Boston',
	'type': 'Location'
}, {
	'name': 'Maine',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Maine',
	'type': 'Location'
}, {
	'name': '106 degrees below zero',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/106_degrees_below_zero',
	'type': 'Number'
}, {
	'name': 'Massachusetts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Massachusetts',
	'type': 'Location'
}, {
	'name': 'Rob Marciano',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Rob_Marciano',
	'type': 'Person'
}, {
	'name': 'Newark',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Newark,_New_Jersey',
	'type': 'Location'
}, {
	'name': 'two',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Two',
	'type': 'Number'
}, {
	'name': 'President Biden',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Joe_Biden',
	'type': 'Person'
}, {
	'name': 'the Fed',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Federal_Reserve_System',
	'type': 'Organization'
}, {
	'name': 'Alex Murdaugh',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Alex_Murdaugh',
	'type': 'Person'
}, {
	'name': 'Robin Roberts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Robin_Roberts',
	'type': 'Person'
}, {
	'name': 'New Zealand',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/New_Zealand',
	'type': 'Country'
}, {
	'name': '5 million',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/5_million',
	'type': 'Number'
}]

Our second time we are truncated:

[{'name': 'Mary Bruce', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Mary_Bruce', 'type': 'Person'}, {'name': 'Mola Lenghi', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Mola_Lenghi', 'type': 'Person'}, {'name': 'Chinese People's Liberation Army', 'wikipedia_url': 'https://en.wikipedia.org/wiki/People%27s_Liberation_Army', 'type': 'Organization'}, {'name': 'United States', 'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States', 'type': 'Country'}, {'name': 'North America', 'wikipedia_url': 'https://en.wikipedia.org/wiki/North_America', 'type': 'Continent'}, {'name': 'Chinese People', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Chinese_people', 'type': 'People'}, {'name': 'China', 'wikipedia_url': 'https://en.wikipedia.org/wiki/China', 'type': 'Country'}, {'name': 'United States Air Force', 'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Air_Force', 'type': 'Organization'}, {'name': 'Antony Blinken', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Antony_Blinken', 'type': 'Person'}, {'name': 'New York City', 'wikipedia_url': 'https://en.wikipedia.org/wiki/New_York_City', 'type': 'City'}, {'name': 'Philadelphia', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Philadelphia', 'type': 'City'}, {'name': 'Boston', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Boston', 'type': 'City'}, {'name': 'Maine', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Maine', 'type': 'State'}, {'name': 'Massachusetts', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Massachusetts', 'type': 'State'}, {'name': 'Newark', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Newark,_New_Jersey', 'type': 'City'}, {'name': 'United States Department of State', 'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Department_of_State', 'type': 'Organization'}, {'name': 'Rebecca Jarvis', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Rebecca_Jarvis', 'type': 'Person'}, {'name': 'Alex Murdaugh', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Alex_Murdaugh', 'type': 'Person'}, {'name': 'Robin Roberts', 'wikipedia_url': 'https://en.wikipedia.org/wiki/Robin_Roberts', 'type': 'Person'}, {'name': 'New Zealand', 'wikipedia_url': 'https://en.wikipedia.org/wiki/New_Zealand', 'type': 'Country'}, {'name': 'South Pole', 'wikipedia_url': 'https://en.wikipedia.org/wiki/South_Pole', 'type': 'Location'}, {'name': 'United States Air Force', 'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Air

And running a third time we are truncated as well:

[{"name": "China", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Organization"}, {"name": "North Carolina", "wikipedia_url": "https://en.wikipedia.org/wiki/North_Carolina", "type": "Location"}, {"name": "New York City", "wikipedia_url": "https://en.wikipedia.org/wiki/New_York_City", "type": "Location"}, {"name": "Boston", "wikipedia_url": "https://en.wikipedia.org/wiki/Boston", "type": "Location"}, {"name": "Maine", "wikipedia_url": "https://en.wikipedia.org/wiki/Maine", "type": "Location"}, {"name": "Massachusetts", "wikipedia_url": "https://en.wikipedia.org/wiki/Massachusetts", "type": "Location"}, {"name": "Alex Murdaugh", "wikipedia_url": "https://en.wikipedia.org/wiki/Alex_Murdaugh", "type": "Person"}, {"name": "Robin Roberts", "wikipedia_url": "https://en.wikipedia.org/wiki/Robin_Roberts", "type": "Person"}, {"name": "Billings", "wikipedia_url": "https://en.wikipedia.org/wiki/Billings,_Montana", "type": "Location"}, {"name": "Mola Lenghi", "wikipedia_url": "https://en.wikipedia.org/wiki/Mola_Lenghi", "type": "Person"}, {"name": "Mary Bruce", "wikipedia_url": "https://en.wikipedia.org/wiki/Mary_Bruce", "type": "Person"}, {"name": "Antony Blinken", "wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken", "type": "Person"}, {"name": "China", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Organization"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Other"}, {"name": "U.S.", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States", "type": "Organization"}, {"name": "U.S. Airspace", "wikipedia_url": "https://en.wikipedia.org/wiki/Airspace_of_the_United_States", "type": "Other"}, {"name": "U.S. Authorities", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_government", "type": "Organization"}, {"name": "U.S. Officials", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_government", "type": "Organization"}, {"name": "Pentagon", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "type": "Organization"}, {"name": "Secretary of State", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Secretary_of_State", "type": "Organization"}, {"name": "Chinese Spy Balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Other"}, {"name": "Mississippi", "wikipedia_url": "https://en.wikipedia.org/wiki/Mississippi", "type": "Location"}, {"name": "Alaska", "wikipedia_url": "https://en.wikipedia.org/wiki/Alaska", "type": "Location"}, {"name": "Billings, Montana", "wikipedia_url": "https://en.wikipedia.org/wiki/Billings,_Montana", "type": "Location"}, {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Other"}, {"name": "China", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Organization"}, {"name": "U.S.", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States", "type": "Organization"}, {"name": "U.S. Airspace", "wikipedia_url": "https://en.wikipedia.org/wiki/Airspace_of_the_United_States", "type": "Other"}, {"name": "U.S. Officials", "wikipedia_

What about jumping up to a temperature of 0.5? The results are similar to what we got with 0.4:

[{
	'name': 'north carolina',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/North_Carolina',
	'type': 'Location'
}, {
	'name': 'billings',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Billings,_Montana',
	'type': 'Location'
}, {
	'name': 'chinese spy balloon',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Chinese_spy_balloon',
	'type': 'Other'
}, {
	'name': 'chinese',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Chinese_people',
	'type': 'Nationality'
}, {
	'name': 'pentagon',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Department_of_Defense',
	'type': 'Organization'
}, {
	'name': 'friday',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Friday',
	'type': 'Day'
}, {
	'name': 'u.s.',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States',
	'type': 'Country'
}, {
	'name': 'abc news',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/ABC_News',
	'type': 'Organization'
}, {
	'name': 'antony blinken',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Antony_Blinken',
	'type': 'Person'
}, {
	'name': 'beijing',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Beijing',
	'type': 'Location'
}, {
	'name': 'china',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/China',
	'type': 'Country'
}]

Running a second time yields the same issue:

[{
	'name': 'china',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/China',
	'type': 'Organization'
}, {
	'name': 'u.s.',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States',
	'type': 'Organization'
}, {
	'name': 'new york city',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/New_York_City',
	'type': 'Location'
}, {
	'name': 'philadelphia',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Philadelphia',
	'type': 'Location'
}, {
	'name': 'boston',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Boston',
	'type': 'Location'
}, {
	'name': 'maine',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Maine',
	'type': 'Location'
}, {
	'name': 'massachusetts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Massachusetts',
	'type': 'Location'
}, {
	'name': 'newark',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Newark',
	'type': 'Location'
}, {
	'name': 'alex murdaugh',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Alex_Murdaugh',
	'type': 'Person'
}, {
	'name': 'robin roberts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Robin_Roberts',
	'type': 'Person'
}, {
	'name': 'north carolina',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/North_Carolina',
	'type': 'Location'
}]

Running a third time yields valid JSON this time and a different set of names (though still many missing and some that appeared above are missing in this list like Billings):

[{
	"name": "Chinese",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people",
	"type": "Ethnic Group"
}, {
	"name": "Chinese balloon",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_balloon",
	"type": "Other"
}, {
	"name": "Chinese government",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Government_of_the_People%27s_Republic_of_China",
	"type": "Government"
}, {
	"name": "Chinese military",
	"wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army",
	"type": "Military"
}, {
	"name": "Chinese spy balloon",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_balloon",
	"type": "Other"
}, {
	"name": "China",
	"wikipedia_url": "https://en.wikipedia.org/wiki/China",
	"type": "Country"
}, {
	"name": "Intelligence bay",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Intelligence_bay",
	"type": "Other"
}, {
	"name": "Mary Bruce",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Mary_Bruce",
	"type": "Person"
}, {
	"name": "Mola Lenghi",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Mola_Lenghi",
	"type": "Person"
}, {
	"name": "North Carolina",
	"wikipedia_url": "https://en.wikipedia.org/wiki/North_Carolina",
	"type": "Location"
}, {
	"name": "Secretary of State Antony Blinken",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken",
	"type": "Person"
}, {
	"name": "United States",
	"wikipedia_url": "https://en.wikipedia.org/wiki/United_States",
	"type": "Country"
}, {
	"name": "United States Air Force",
	"wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Air_Force",
	"type": "Military"
}, {
	"name": "United States Department of State",
	"wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Department_of_State",
	"type": "Government"
}, {
	"name": "United States military",
	"wikipedia_url": "https://en.wikipedia.org/wiki/Military_of_the_United_States",
	"type": "Military"
}, {
	"name": "United States of America",
	"wikipedia_url": "https://en.wikipedia.org/wiki/United_States",
	"type": "Country"
}, {
	"name": "West Virginia",
	"wikipedia_url": "https://en.wikipedia.org/wiki/West_Virginia",
	"type": "Location"
}]

A fourth time yields invalid JSON again:

[{
	'name': 'new york city',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/New_York_City',
	'type': 'Location'
}, {
	'name': 'philadelphia',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Philadelphia',
	'type': 'Location'
}, {
	'name': 'boston',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Boston',
	'type': 'Location'
}, {
	'name': 'maine',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Maine',
	'type': 'Location'
}, {
	'name': 'massachusetts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Massachusetts',
	'type': 'Location'
}, {
	'name': 'newark',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Newark,_New_Jersey',
	'type': 'Location'
}, {
	'name': 'alex murdaugh',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Alex_Murdaugh',
	'type': 'Person'
}, {
	'name': 'robin roberts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Robin_Roberts',
	'type': 'Person'
}, {
	'name': 'china',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/China',
	'type': 'Organization'
}, {
	'name': 'pentagon',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Department_of_Defense',
	'type': 'Organization'
}, {
	'name': 'secretary of state antony blinken',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Antony_Blinken',
	'type': 'Person'
}, {
	'name': 'mary bruce',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Mary_Bruce',
	'type': 'Person'
}, {
	'name': 'mola lenghi',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Mola_Lenghi',
	'type': 'Person'
}, {
	'name': 'matt gutman',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Matt_Gutman',
	'type': 'Person'
}, {
	'name': 'rebecca jarvis',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Rebecca_Jarvis',
	'type': 'Person'
}, {
	'name': 'eva pilgrim',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Eva_Pilgrim',
	'type': 'Person'
}, {
	'name': 'ryan zinke',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Ryan_Zinke',
	'type': 'Person'
}, {
	'name': 'u.s.',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States',
	'type': 'Organization'
}, {
	'name': 'weather',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Weather',
	'type': 'Other'
}, {
	'name': 'wind chill',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Wind_chill',
	'type': 'Other'
}]

A fifth time yields valid JSON but a different list of entities:

[{
	"name": "Chinese balloon",
	"wikipedia_url": "Chinese balloon",
	"type": "Other"
}, {
	"name": "United States",
	"wikipedia_url": "United States",
	"type": "Location"
}, {
	"name": "New York City",
	"wikipedia_url": "New York City",
	"type": "Location"
}, {
	"name": "Philadelphia",
	"wikipedia_url": "Philadelphia",
	"type": "Location"
}, {
	"name": "Boston",
	"wikipedia_url": "Boston",
	"type": "Location"
}, {
	"name": "Maine",
	"wikipedia_url": "Maine",
	"type": "Location"
}, {
	"name": "Massachusetts",
	"wikipedia_url": "Massachusetts",
	"type": "Location"
}, {
	"name": "Victor Oquendo",
	"wikipedia_url": "Victor Oquendo",
	"type": "Person"
}, {
	"name": "Rob Marciano",
	"wikipedia_url": "Rob Marciano",
	"type": "Person"
}, {
	"name": "Matt Gutman",
	"wikipedia_url": "Matt Gutman",
	"type": "Person"
}, {
	"name": "Newark",
	"wikipedia_url": "Newark",
	"type": "Location"
}, {
	"name": "President Biden",
	"wikipedia_url": "Joe Biden",
	"type": "Person"
}, {
	"name": "Rebecca Jarvis",
	"wikipedia_url": "Rebecca Jarvis",
	"type": "Person"
}, {
	"name": "Alex Murdaugh",
	"wikipedia_url": "Alex Murdaugh",
	"type": "Person"
}, {
	"name": "Eva Pilgrim",
	"wikipedia_url": "Eva Pilgrim",
	"type": "Person"
}, {
	"name": "Robin Roberts",
	"wikipedia_url": "Robin Roberts",
	"type": "Person"
}, {
	"name": "New Zealand",
	"wikipedia_url": "New Zealand",
	"type": "Location"
}, {
	"name": "Antarctic",
	"wikipedia_url": "Antarctic",
	"type": "Location"
}, {
	"name": "China",
	"wikipedia_url": "China",
	"type": "Organization"
}, {
	"name": "United States Air Force",
	"wikipedia_url": "United States Air Force",
	"type": "Organization"
}, {
	"name": "Secretary of State Antony Blinken",
	"wikipedia_url": "Antony Blinken",
	"type": "Person"
}, {
	"name": "Mary Bruce",
	"wikipedia_url": "Mary Bruce",
	"type": "Person"
}, {
	"name": "Mola Lenghi",
	"wikipedia_url": "Mola Lenghi",
	"type": "Person"
}, {
	"name": "Montana",
	"wikipedia_url": "Montana",
	"type": "Location"
}, {
	"name": "Ryan Zinke",
	"wikipedia_url": "Ryan Zinke",
	"type": "Person"
}, {
	"name": "The White House",
	"wikipedia_url": "The White House",
	"type": "Organization"
}, {
	"name": "Chinese government",
	"wikipedia_url": "Government of China",
	"type": "Organization"
}]

Note that we are coming in well below the max token output. Recall how with the exact same text and a temperature of 0.0 we exceeded the maximum 1024 output tokens, while with a higher temperature we are outputting just 674 tokens. This fairly definitively proves that the underlying issue is a latent infinite loop trigger.

  "metadata": {
    "tokenMetadata": {
      "outputTokenCount": {
        "totalBillableCharacters": 1894,
        "totalTokens": 674
      },
      "inputTokenCount": {
        "totalBillableCharacters": 5281,
        "totalTokens": 1405
      }
    }
  }

What about a maximal temperature setting of 0.99? The result is a hybrid JSON-CSV format that is both unparseable and excludes most of the entities in the text:

{
"Alex Murdaugh", "https://en.wikipedia.org/wiki/Alex_Murdaugh", "Person"
"Anthony Blinken", "https://en.wikipedia.org/wiki/Antony_Blinken", "Person"
"Billings", "https://en.wikipedia.org/wiki/Billings,_Montana", "Location"
"Chinese People's Liberation Army", "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "Organization"
"China", "https://en.wikipedia.org/wiki/China", "Country"
"Chinese", "https://en.wikipedia.org/wiki/Chinese_language", "Language"
"Intelligence", "https://en.wikipedia.org/wiki/Intelligence_(information)", "Other"
"Journalist", "https://en.wikipedia.org/wiki/Journalism", "Occupation"
"Moira Lenghi", "https://en.wikipedia.org/wiki/Moira_Lenghi", "Person"
"National Weather Service", "https://en.wikipedia.org/wiki/National_Weather_Service", "Organization"
"North Carolina", "https://en.wikipedia.org/wiki/North_Carolina", "Location"
"Pentagon", "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "Organization"
"U.S. Air Force", "https://en.wikipedia.org/wiki/United_States_Air_Force", "Organization"
"U.S.", "https://en.wikipedia.org/wiki/United_States", "Country"
"United States of America", "https://en.wikipedia.org/wiki/United_States", "Country"
"Weather Balloon", "https://en.wikipedia.org/wiki/Weather_balloon", "Other"
}

Running a second time yields more invalid JSON:

{"abc news", "https://en.wikipedia.org/wiki/ABC_News", "Organization"},
{"antony blinken", "https://en.wikipedia.org/wiki/Antony_Blinken", "Person"},
{"china", "https://en.wikipedia.org/wiki/China", "Country"},
{"chinese", "https://en.wikipedia.org/wiki/Chinese_people", "Ethnic group"},
{"continental united states", "https://en.wikipedia.org/wiki/Continental_United_States", "Location"},
{"federal aviation administration", "https://en.wikipedia.org/wiki/Federal_Aviation_Administration", "Organization"},
{"montana", "https://en.wikipedia.org/wiki/Montana", "State"},
{"new jersey", "https://en.wikipedia.org/wiki/New_Jersey", "State"},
{"north carolina", "https://en.wikipedia.org/wiki/North_Carolina", "State"},
{"pentagon", "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "Organization"},
{"republican party", "https://en.wikipedia.org/wiki/Republican_Party_(United_States)", "Organization"},
{"ryan zinke", "https://en.wikipedia.org/wiki/Ryan_Zinke", "Person"},
{"secretary of state", "https://en.wikipedia.org/wiki/Secretary_of_State_of_the_United_States", "Government office"},
{"united states", "https://en.wikipedia.org/wiki/United_States", "Country"},
{"weather balloon", "https://en.wikipedia.org/wiki/Weather_balloon", "Other"}

Running a third time yields JSON, but it is unparseable due to the use of single quotes rather than double quotes:

[{
	'name': 'china',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/China',
	'type': 'Organization'
}, {
	'name': 'chinese',
	'wikipedia_url': None,
	'type': 'Other'
}, {
	'name': 'chinese spy balloon',
	'wikipedia_url': None,
	'type': 'Other'
}, {
	'name': 'billings',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Billings,_Montana',
	'type': 'Location'
}, {
	'name': 'pentagon',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States_Department_of_Defense',
	'type': 'Organization'
}, {
	'name': 'u.s.',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/United_States',
	'type': 'Other'
}, {
	'name': 'new york city',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/New_York_City',
	'type': 'Location'
}, {
	'name': 'philadelphia',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Philadelphia',
	'type': 'Location'
}, {
	'name': 'boston',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Boston',
	'type': 'Location'
}, {
	'name': 'maine',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Maine',
	'type': 'Location'
}, {
	'name': 'massachusetts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Massachusetts',
	'type': 'Location'
}, {
	'name': 'newark',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Newark,_New_Jersey',
	'type': 'Location'
}, {
	'name': 'missouri',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Missouri',
	'type': 'Location'
}, {
	'name': 'american',
	'wikipedia_url': None,
	'type': 'Other'
}, {
	'name': 'alex murdaugh',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Alex_Murdaugh',
	'type': 'Person'
}, {
	'name': 'robin roberts',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Robin_Roberts',
	'type': 'Person'
}, {
	'name': 'montana republican congressman ryan zinke',
	'wikipedia_url': None,
	'type': 'Person'
}, {
	'name': 'secretary of state antony blinken',
	'wikipedia_url': 'https://en.wikipedia.org/wiki/Antony_Blinken',
	'type': 'Person'
}]

A fourth time yields valid JSON wrapped in invalid "`json"` markers that makes it unparseable.

```json
[
  {"name": "chinese spy balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Other"},
  {"name": "china", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Country"},
  {"name": "united states", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States", "type": "Country"},
  {"name": "antony blinken", "wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken", "type": "Person"},
  {"name": "north carolina", "wikipedia_url": "https://en.wikipedia.org/wiki/North_Carolina", "type": "Location"},
  {"name": "missouri", "wikipedia_url": "https://en.wikipedia.org/wiki/Missouri", "type": "Location"},
  {"name": "billings", "wikipedia_url": "https://en.wikipedia.org/wiki/Billings,_Montana", "type": "Location"},
  {"name": "pentagon", "wikipedia_url": "https://en.wikipedia.org/wiki/United_States_Department_of_Defense", "type": "Organization"},
  {"name": "montana", "wikipedia_url": "https://en.wikipedia.org/wiki/Montana", "type": "Location"},
  {"name": "ryan zinke", "wikipedia_url": "https://en.wikipedia.org/wiki/Ryan_Zinke", "type": "Person"},
  {"name": "chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic Group"},
  {"name": "chinese government", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_Communist_Party", "type": "Organization"}
]
```

Increasing temperature allows us to escape the infinite loop, but at the cost of many of the output runs being unusable. What if we instead attempted to modify the prompt slightly? Let's return to our original temperature of 0.0, but modify our prompt to drop the type classification ("Classify the Type of each as one of the following: "Organization,Person,Location,Event,Other"."). This yields the following:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
"instances": [
{ "prompt": "Extract a list of the people, organizations, locations and other entities in the following text, disambiguating and normalizing each name to the title of its corresponding Wikipedia article. Output the results in JSON in the format \\"{name, wikipedia url, type}\\" : tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it\'s believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out. the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting. another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers. the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down. the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh\'s housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper\'s fall. but will the jury hear about it? eva pilgrim reporting. robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you\'ll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. "}
],
"parameters": {
"temperature": 0.0,
"maxOutputTokens": 1024,
"topK": 40,
"topP": 0.95,
}
}'

This fails to eliminate the infinite loop:

[{"name": "Chinese People's Liberation Army", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Air_Force", "type": "organization"}, {"name": "Chinese People's Liberation Army Navy", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Navy", "type": "organization"}, {"name": "Chinese People's Liberation Army Rocket Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Rocket_Force", "type": "organization"}, {"name": "Chinese People's Liberation Army Strategic Support Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Strategic_Support_Force", "type": "organization"}, {"name": "Chinese People's Liberation Army Ground Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Ground_Force", "type": "organization"}, {"name": "Chinese People's Liberation Army Navy Submarine Force", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army_Navy_Submarine_Force", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's Liberation Army Air Force Aviation University", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_People%27s_Liberation_Army_Air_Force_Aviation_University", "type": "organization"}, {"name": "Chinese People's

What if we remove the Wikipedia instruction as well, but leave the "wikipedia url" in the JSON output. Thus, our prompt becomes:

Extract a list of the people, organizations, locations and other entities in the following text. Output the results in JSON in the format "{name, wikipedia url, type}":

This leads to even stranger invalid JSON that still features infinite looping:

{
"Alex Murdaugh": {"name": "Alex Murdaugh", "wikipedia_url": "https://en.wikipedia.org/wiki/Alex_Murdaugh", "type": "Person"},
"Antony Blinken": {"name": "Antony Blinken", "wikipedia_url": "https://en.wikipedia.org/wiki/Antony_Blinken", "type": "Person"},
"Billings, Montana": {"name": "Billings, Montana", "wikipedia_url": "https://en.wikipedia.org/wiki/Billings,_Montana", "type": "Location"},
"China": {"name": "China", "wikipedia_url": "https://en.wikipedia.org/wiki/China", "type": "Country"},
"Chinese": {"name": "Chinese", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"},
"Chinese spy balloon": {"name": "Chinese spy balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made": {"name": "Chinese-made", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_people", "type": "Ethnic group"},
"Chinese-made balloon": {"name": "Chinese-made balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made surveillance balloon": {"name": "Chinese-made surveillance balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made weather balloon": {"name": "Chinese-made weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese military": {"name": "Chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Military"},
"Chinese weather balloon": {"name": "Chinese weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made weather balloon": {"name": "Chinese-made weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made surveillance balloon": {"name": "Chinese-made surveillance balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese military": {"name": "Chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Military"},
"Chinese weather balloon": {"name": "Chinese weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made weather balloon": {"name": "Chinese-made weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made surveillance balloon": {"name": "Chinese-made surveillance balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese military": {"name": "Chinese military", "wikipedia_url": "https://en.wikipedia.org/wiki/People%27s_Liberation_Army", "type": "Military"},
"Chinese weather balloon": {"name": "Chinese weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made weather balloon": {"name": "Chinese-made weather balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese-made surveillance balloon": {"name": "Chinese-made surveillance balloon", "wikipedia_url": "https://en.wikipedia.org/wiki/Chinese_spy_balloon", "type": "Balloon"},
"Chinese military": {"name": "Chinese military", "wikipedia_url": "https://en.

Now let's remove Wikipedia URL, but leave Type, yielding a prompt of:

Extract a list of the people, organizations, locations and other entities in the following text. Output the results in JSON in the format "{name, type}":

This causes it to adopt a different invalid JSON format and continues the infinite looping:

{
"Alex Murdaugh": "Person",
"Antony Blinken": "Person",
"China": "Country",
"Chinese": "Nationality",
"Chinese Spy Balloon": "Balloon",
"Chinese Spy Vessel": "Balloon",
"Chinese Surveillance Balloon": "Balloon",
"Chinese Weather Balloon": "Balloon",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese": "Nationality",
"Chinese":

What if we eliminate the Type value and use the final prompt:

Extract a list of the people, organizations, locations and other entities in the following text. Output the results in JSON in the format "{name}":

This time it simply repeats the entity each time, still featuring invalid JSON and infinite looping:

{
"Alex Murdaugh": "Alex Murdaugh",
"Antony Blinken": "Antony Blinken",
"China": "China",
"Chinese": "Chinese",
"Chinese Spy Balloon": "Chinese Spy Balloon",
"Chinese authorities": "Chinese authorities",
"Chinese government": "Chinese government",
"Chinese military": "Chinese military",
"Chinese state media": "Chinese state media",
"Chinese surveillance balloon": "Chinese surveillance balloon",
"Chinese weather balloon": "Chinese weather balloon",
"Chinese-made": "Chinese-made",
"China's": "China's",
"China's military": "China's military",
"China's state media": "China's state media",
"China's surveillance balloon": "China's surveillance balloon",
"China's weather balloon": "China's weather balloon",
"Chinese-made": "Chinese-made",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"Chinese": "Chinese",
"

What if we eliminate JSON output as a requirement and go with the simplest possible prompt:

Extract a list of the people, organizations, locations and other entities in the following text:

Thus, our final query is:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
"instances": [
{ "prompt": "Extract a list of the people, organizations, locations and other entities in the following text: tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it\'s believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out. the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting. another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers. the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down. the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh\'s housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper\'s fall. but will the jury hear about it? eva pilgrim reporting. robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you\'ll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. "}
],
"parameters": {
"temperature": 0.0,
"maxOutputTokens": 1024,
"topK": 40,
"topP": 0.95,
}
}'

This time something strange happens: we trigger the safety guardrails and the API refuses to return a response. Thus, simply asking the API to return a list of names, without asking it to explicitly return it as JSON causes the API guardrails to refuse to output a result, even though the content and task remains the same.

{
  "predictions": [
    {
      "safetyAttributes": {
        "blocked": true
      },
      "content": ""
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalBillableCharacters": 5056,
        "totalTokens": 1347
      },
      "outputTokenCount": {
        "totalBillableCharacters": 0,
        "totalTokens": 1024
      }
    }
  }
}

What if we reword the prompt to the following:

Given the text below, compile a list of potential people, organizations, locations and other entities:

This time we get results, but just a fraction of the entities in the text are extracted.

- Chinese spy balloon
- U.S.
- Pentagon
- Secretary of State Antony Blinken
- China
- Mary Bruce
- ABC
- Mola Lenghi
- Montana
- Commercial pilots
- High resolution cameras
- Solar panels
- Ryan Zinke
- White House
- President Biden
- China
- Antony Blinken
- Beijing

One of the most common ways to force an LLM's output in cases like this is to ask it to explain its output. The requirement of adding an explanation has a therapeutic effect in many cases of constraining the LLM's reasoning. Let's try that here:

Given the text below, compile a list of potential people, organizations, locations and other entities and for each entity explain why you listed it:

This time we get reasonable results, but once again it is missing many key names including President Biden, New York City, Massachusetts, Philadelphia, Boston, Newark, Alex Murdaugh and many others. Note that despite asking it to explain its output, it does not.

People:
- Secretary of State Antony Blinken
- Ryan Zinke
- Victor Oquendo
- Rob Marciano
- Matt Gutman
- Rebecca Jarvis
- Eva Pilgrim
- Robin Roberts

Organizations:
- The Pentagon
- The National Weather Service
- The White House

Locations:
- China
- The United States
- Alaska
- Southwest Canada
- Billings, Montana
- North Carolina

Other entities:
- The Chinese spy balloon
- The intelligence bay
- The high-stakes trip to Beijing

What if we simplify things even further to ask just for the list of people mentioned in the text?

Given the text below, compile a list of potential people mentioned in the text and for each person explain why you listed it:

This yields the following output that is missing names like Joe Biden, Alex Murdaugh, etc and does not include the requested explanation:

The text mentions the following people:

- Secretary of State Antony Blinken
- Mary Bruce
- Victor Oquendo
- Rob Marciano
- Matt Gutman
- Rebecca Jarvis
- Eva Pilgrim
- Robin Roberts

What if we try the simplest possible prompt? This doesn't ask for an explanation or include any complicated instructions or output formats, just a simple list of names:

Make a list of the people mentioned in this text below:

Unfortunately, we get the same list:

The people mentioned in this text are:

- Secretary of State Antony Blinken
- Mary Bruce
- Victor Oquendo
- Rob Marciano
- Matt Gutman
- Rebecca Jarvis
- Eva Pilgrim
- Robin Roberts

What if we want a machine-readable list?

Make a list of the people mentioned in this text below and output the results in JSON:

The output remains unchanged:

The people mentioned in the text are:

- Secretary of State Antony Blinken
- Mary Bruce
- Victor Oquendo
- Rob Marciano
- Matt Gutman
- Rebecca Jarvis
- Eva Pilgrim
- Robin Roberts
- Ryan Zinke
- China
- The Pentagon
- The White House

What about if we merge this prompt with our earlier JSON specification?

Make a list of the people mentioned in this text below and output the results in JSON in the format "{ Entity: Name}":

This yields the invalid JSON below. In addition, despite asking only for people, a variety of entities are output:

{Entity: Antony Blinken}, {Entity: China}, {Entity: Chinese}, {Entity: Chinese Spy Balloon}, {Entity: Eva Pilgrim}, {Entity: Federal Bureau of Investigation}, {Entity: Mary Bruce}, {Entity: Mola Lenghi}, {Entity: New Jersey}, {Entity: New York City}, {Entity: North Carolina}, {Entity: Robin Roberts}, {Entity: Ryan Zinke}, {Entity: United States}, {Entity: United States Air Force}, {Entity: Victor Oquendo}}

What if we change the JSON specification slightly:

Make a list of the people mentioned in this text below and output the results in JSON in the format "{ Person: Name}":

Invalid JSON again and a mixture of entities:

{
Person: Victor Oquendo
Person: Rob Marciano
Person: Matt Gutman
Person: Rebecca Jarvis
Person: Eva Pilgrim
Person: Robin Roberts
Person: Mary Bruce
Person: Mola Lenghi
Person: Ryan Zinke
Person: Antony Blinken
Person: China
Person: The Pentagon
Person: The White House
Person: The National Weather Service
Person: New York City
Person: Philadelphia
Person: Boston
Person: Maine
Person: Massachusetts
Person: The ER Doctor
Person: Two Passenger Jets
Person: The Alex Murdaugh Double Murder Trial
Person: The Ballistics Expert
Person: Alex Murdaugh
Person: The Family
Person: Prosecutors
Person: Eva Pilgrim
Person: Robin Roberts
Person: New Zealand
Person: The Southern Hemisphere
Person: The Lesson On Conquering Fear
}

What about rewording the JSON output slightly?

Make a list of the people mentioned in this text below and output the results as a JSON array of names:

This time we get valid JSON output, but note how the LLM fixated on "array of names" and failed to connect "names" to "people" in the beginning of the sentence and thus outputs a mixture of people and other proper and common nounphrases:

[{
	"name": "ryan zinke"
}, {
	"name": "mary bruce"
}, {
	"name": "mola lenghi"
}, {
	"name": "secretary of state antony blinken"
}, {
	"name": "china"
}, {
	"name": "pentagon"
}, {
	"name": "abc news"
}, {
	"name": "commercial pilots"
}, {
	"name": "high resolution cameras"
}, {
	"name": "solar panels"
}, {
	"name": "billings, montana"
}, {
	"name": "white house"
}, {
	"name": "president biden"
}, {
	"name": "china"
}, {
	"name": "secretary of state antony blinken"
}, {
	"name": "abc news"
}]

Finally, what about the simplest possible prompt that combines all of this?

Output a JSON array of the people mentioned in the text below:

This outputs just one entry:

[{
	"name": "ryan zinke",
	"title": "montana republican congressman"
}]