One of the greatest strengths of ChatGPT is its ability to generate fluent prose that differs each time it is run. At the same time, this variability can complicate downstream analyses, though it can be guided in part through prompt engineering. One key question is the degree to which substantive questions like whether ChatGPT believes a statement is fact or opinion, varies from run to run. Here we repeated the same text (the final summarized version of an ABC evening news broadcast multiple times, testing several small variations in the prompt and sometimes just seeing how the model output varies between runs.
One finding is that using unstructured phrases like "tell me" that don't circumscribe the model's output cause it to be more flexible in its output, often with quotations around specific words and can yield more informative results, but with the downside that the structure can vary dramatically between runs. This suggests that there may be benefit in using relatively unstructured prompts early in the development process to allow the model to be more expressive and demonstrate a fuller range of its possible responses, before successively narrowing the prompt to request a specific output format. For example, "make a list" tends to yield more rigid results. Importantly, in some cases the model does differ in how it classifies certain statements as fact versus opinion between runs, suggesting that substantive determinations are non-deterministic, not just the output wording.
Prompt:
You are an AI assistant that summarizes and answers questions about television news transcripts.
Response:
Yes, that's correct! I have been trained on a large corpus of text data and I can assist you with summarizing and answering questions about television news transcripts, as well as other types of text. Just let me know what you need help with!
Prompt:
Tell me the statements of fact and the opinions in this article: "The news covers multiple stories tonight, including a suspected Chinese spy balloon hovering over the US, dangerous cold affecting the Northeast, an attack on an ER doctor, a scare at Newark airport, a positive jobs report, and the trial of Alex Murdaugh. The President has been presented with options for taking action against the Chinese balloon, including military action, and is being regularly briefed on the situation. A brutal arctic front is causing concern for hypothermia, frostbite, and power outages in the Northeast. Meanwhile, in California, an ER doctor was murdered by a driver. The economy is showing strength, with widespread hiring across nearly every industry. The trial of Alex Murdaugh for double murder continues, with the defense pushing for the murder weapon to be ruled out. There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo. The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school. Finally, Robin Roberts took a sky jump in Auckland, New Zealand, conquering her fear."
Response:
In this article, the following statements can be considered as facts:
-
- A suspected Chinese spy balloon is hovering over the US.
- Dangerous cold is affecting the Northeast.
- An ER doctor was attacked in California.
- There was a scare at Newark airport.
- The jobs report is positive and shows widespread hiring across nearly every industry.
- The trial of Alex Murdaugh for double murder continues.
- The President has been presented with options for taking action against the Chinese balloon.
- The President is being regularly briefed on the situation.
- The arctic front is causing concern for hypothermia, frostbite, and power outages in the Northeast.
- The ER doctor was murdered by a driver in California.
- The defense is pushing for the murder weapon to be ruled out in the trial of Alex Murdaugh.
- There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo.
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school.
- Robin Roberts took a sky jump in Auckland, New Zealand.
The following statements can be considered opinions:
-
- The suspected Chinese spy balloon is a "dangerous cold."
- The economy is "showing strength."
- Robin Roberts "conquered her fear" with the sky jump.
Prompt:
Make a table of the facts and options in this article: "[]"
Response:
Prompt:
Make a table of the facts and opinions in this article and why it is a fact or opinion: "[]"
Response:
Prompt:
Make a list of the facts and opinions in this article: "[]"
Response:
Facts:
-
- A suspected Chinese spy balloon is hovering over the US.
- Dangerous cold is affecting the Northeast, causing concern for hypothermia, frostbite, and power outages.
- An ER doctor was murdered by a driver in California.
- The President has been presented with options for taking action against the Chinese balloon, including military action, and is being regularly briefed on the situation.
- The economy is showing strength, with widespread hiring across nearly every industry.
- The trial of Alex Murdaugh for double murder continues.
- There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo.
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school.
- Robin Roberts took a sky jump in Auckland, New Zealand.
Opinions:
-
- The cold affecting the Northeast is brutal.
- The economy is showing strength.
- The defense in the trial of Alex Murdaugh is pushing for the murder weapon to be ruled out.
- Robin Roberts conquered her fear by taking a sky jump in Auckland, New Zealand.
Prompt:
Tell me the statements of fact and the opinions in this article: "[]"
Response:
Facts:
-
- A suspected Chinese spy balloon is hovering over the US.
- The Northeast is experiencing dangerous cold weather.
- An ER doctor was attacked in California.
- There was a scare at Newark airport.
- There is a positive jobs report.
- Alex Murdaugh is on trial for double murder.
- The President has been presented with options for taking action against the Chinese balloon, including military action.
- The President is being regularly briefed on the situation.
- The brutal arctic front is causing concern for hypothermia, frostbite, and power outages in the Northeast.
- An ER doctor was murdered by a driver in California.
- The economy is showing strength with widespread hiring across nearly every industry.
- The defense in the trial of Alex Murdaugh is pushing for the murder weapon to be ruled out.
- There are investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo.
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school.
- Robin Roberts took a sky jump in Auckland, New Zealand.
Opinions:
-
- The news covers multiple stories. (Not a statement of fact or opinion)The President is taking action against the Chinese balloon. (Not a statement of fact or opinion)
- The brutal arctic front is dangerous. (Opinion – the word "brutal" implies danger)
- The economy is showing strength. (Opinion – the word "strength" implies a positive connotation)
- Robin Roberts conquered her fear. (Opinion – the word "conquered" implies she succeeded)
Prompt:
Tell me the statements of fact and the opinions in this article: "[]"
Response:
Facts:
-
- A suspected Chinese spy balloon is hovering over the US
- The Northeast is experiencing dangerous cold temperatures
- An ER doctor was murdered in California by a driver
- There was a scare at Newark airport
- The economy is showing strength with widespread hiring across industries
- Alex Murdaugh is on trial for double murder
- The President has been presented with options for taking action against the Chinese balloon
- The President is being regularly briefed on the situation
- The arctic front is causing concern for hypothermia, frostbite, and power outages in the Northeast
- There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo
- The city of Oakland is taking action to address possible prostitution and human trafficking outside a school
- Robin Roberts took a sky jump in Auckland, New Zealand
Opinions:
-
- The suspected Chinese spy balloon is described as "dangerous"
- The brutal arctic front is described as "concern"
- The economy is described as "showing strength"
- The defense is pushing for the murder weapon to be ruled out in the trial of Alex Murdaugh
- Robin Roberts "conquered her fear" by taking a sky jump.
Prompt:
Tell me the statements of fact and the opinions in this article: "[]"
Response:
Facts:
-
- A suspected Chinese spy balloon is hovering over the US.
- The Northeast is facing dangerous cold weather conditions.
- An ER doctor was murdered in California by a driver.
- Newark airport had a scare.
- There was a positive jobs report.
- Alex Murdaugh is on trial for double murder.
- The President has been presented with options for taking action against the Chinese balloon, including military action.
- The President is being regularly briefed on the situation with the Chinese balloon.
- The arctic front is causing concern for hypothermia, frostbite, and power outages in the Northeast.
- There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo.
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school.
- Robin Roberts took a sky jump in Auckland, New Zealand.
Opinions:
-
- The news is "brutal."
- The economy is "showing strength."
- Robin Roberts "conquered her fear."
- The murder weapon may be "ruled out."
- The defense is "pushing" for the murder weapon to be ruled out.
Prompt:
Tell me the statements of fact and the opinions in this article: "[]"
Response:
Facts:
-
- There is a suspected Chinese spy balloon hovering over the US
- Dangerous cold is affecting the Northeast causing concern for hypothermia, frostbite, and power outages
- An ER doctor was murdered by a driver in California
- There is a positive jobs report with widespread hiring across nearly every industry
- The trial of Alex Murdaugh for double murder continues
- There are several investigations happening including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school
- Robin Roberts took a sky jump in Auckland, New Zealand
Opinions:
-
- The President being regularly briefed on the situation regarding the suspected Chinese spy balloon
- The defense pushing for the murder weapon to be ruled out in the trial of Alex Murdaugh
- The economy showing strength is described as a positive jobs report
- Robin Roberts conquering her fear by taking a sky jump in Auckland, New Zealand
Prompt:
Make a list of the statements of fact and the opinions in this article: "[]"
Response:
Facts:
-
- A suspected Chinese spy balloon is hovering over the US.
- There is dangerous cold affecting the Northeast.
- An ER doctor was attacked in California and murdered by a driver.
- There was a scare at Newark airport.
- The economy is showing strength with widespread hiring across nearly every industry.
- The trial of Alex Murdaugh for double murder is ongoing.
- The defense is pushing for the murder weapon to be ruled out.
- There are several investigations happening, including firings and suspensions of police officers and charges against a man for crimes at the Dallas Zoo.
- The city of Oakland is taking action to address the issue of possible prostitution and human trafficking outside a school.
- Robin Roberts took a sky jump in Auckland, New Zealand.
Opinions:
-
- The President is being regularly briefed on the situation with the suspected Chinese spy balloon.
- A brutal arctic front is causing concern.
- The economy is showing strength.
- Robin Roberts conquered her fear with the sky jump.