This past February we explored the use of LLMs for extracting and codifying disease outbreaks, finding potential but significant limitations that rendered them ill-suited for production use at the time. We followed up that initial exploration with a deeper dive this August to examine how well the rapid LLM developments of the preceding months had addressed the earlier limitations, finding that the same underlying issues remained: hallucination, coherence and exclusion chief among them. In other words, when asked to compile a list of disease-related illnesses and deaths discussed in a news article, LLMs will deviate from their assigned task (coherence), will exclude mentions explicitly described in the text (exclusion) or will falsely claim details or even entire incidents not mentioned in the text (hallucination). With the general availability of maximal-capability models like GPT 4 and a broader range of model sizes like Bison, Unicorn and now Gemini, how do the latest generation models fare and have we reached a point where automated LLM disease extraction is now sufficiently stable and robust to deploy in production in fully automated workflows?
To test this question, we'll explore several disease outbreak articles through the eyes of five major commercial LLMs: Google's Bison, Unicorn and Gemini Pro models and OpenAI's GPT 3.5 and GPT 4.0 models. We will explicitly avoid model-specific or multishot prompt tuning, instead utilizing the same example-less generic prompt across all five models in order to test their out-of-the-box capabilities. Fewshot prompting would likely improve the results below but can introduce more complex and brittle behaviors when applied to the highly divergent nature of global news content, especially under multilingual pipelines where they can lead to unexpected interactions. Model fine tuning can also significantly improve results, but not all hosted models support tuning and this can counterintuitively yield lower performance on edge cases. At the same time, they can dramatically improve performance in under circumstances and we'll explore these complexities in our next experiment, but here we will focus purely on a generic directive prompt.
The results here reinforce the basic truth that generative model performance does not scale with model size. While so-called "AI influencers" and large swaths of the AI consultancy routinely advise companies and governments that merely scaling model size will yield exponential increases in output quality, the truth is far different. In all three examples below, the results are a wash across the five models, rather than scaling with model size. In fact, the smallest and least capable of the five (Bison) produces the best results overall across the three articles, offering equal or superior results to GPT 4.0. While companies are rushing to deploy the most expensive and advanced models in their workflows, the results here offer a reminder that vastly smaller models can actually outperform the largest and most advanced models in real-world applications (rather than the artificial laboratory benchmarks typically employed in the community) when deployed in the kinds of ad-hoc instant workflows that tend to dominate LLM workflows in real world organizations.
In other words, far from providing vastly superior results due to its size, GPT 4.0's results are largely indistinguishable from the others and the smallest model (Bison) matches or outperforms its much larger peers below.
Let's start with this article about vibriosis infections from seafood.
Reading through the article, we see the following details which we would expect the models to capture and codify:
- 1 person died on New York's Suffolk County (Long Island). No information on when they died.
- 3 people in Connecticut were hospitalized in July-August each aged 60-80, of which two died in July.
- CDC says 80,000 in the US are sickened annually from vibriosis, of which 52,000 are due to contaminated food and 100 die, most between May-October.
- In June a 54-year-old Missouri man died
- Last year two people in Florida died after eating oysters from Louisiana.
For all of the experiments below, we leave all of their settings to their defaults except where noted, other than maxOutputTokens, which we set to 1,000 for all runs across all models.
BISON
Let's start with Bison's default deterministic 0.0 temperature. This perfectly captures the details of the source document.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Victim came into contact with Vibrio vulnificus bacteria, likely through seawater or raw/undercooked seafood. | The cause of the infection is still under investigation. | | Death | 2 | Vibriosis | Connecticut | July 2022 | Two residents, aged 60 to 80, died from vibriosis. | The source of the infection is unknown, but it is not believed to be linked to Connecticut shellfish. | | Illness | 3 | Vibriosis | Connecticut | Since July 1, 2022 | Three cases of vibriosis reported, all hospitalized. | Two cases were wound infections not associated with seafood, one was from consuming raw oysters not harvested from Long Island Sound. | | Illness | 80,000 (estimated) | Vibriosis | United States | Annually | About 52,000 cases are estimated to be the result of eating contaminated food, including oysters. | Most infections occur from May to October when water is warmer. | | Death | 100 (estimated) | Vibriosis | United States | Annually | About 1 in 5 of those with serious infections die, sometimes within a day or two of illness. | People with liver disease, cancer, weakened immune systems, or those taking medicine to decrease stomach acid levels are particularly susceptible. | | Death | 1 | Vibriosis | Missouri | June 2022 | 54-year-old man died after eating raw oysters bearing the bacteria. | The oysters were likely contaminated with Vibrio vulnificus. | | Death | 2 | Vibriosis | Florida | Last year | Two people died after eating raw oysters from Louisiana. | The oysters were likely contaminated with Vibrio vulnificus. |
Let's try it with the formerly-default 0.2 temperature. This misses the second two sets of deaths:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Victim came into contact with Vibrio vulnificus bacteria, likely through seawater or raw/undercooked seafood. | The cause of the infection is still being investigated. | | Death | 2 | Vibriosis | Connecticut | July 2022 | Both victims were ages 60 to 80. One consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. The other two cases were wound infections not associated with seafood. | This is the first time Connecticut has seen a Vibrio case in three years. | | Illness | 3 | Vibriosis | Connecticut | Since July 1, 2022 | All three victims were hospitalized. Two of the cases were wound infections not associated with seafood, and the third infection was of a Connecticut resident who consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. | The state has added requirements designed to cool oysters to the point where the bacteria cannot survive. | | Illness | 80,000 (estimated) | Vibriosis | United States | Annually | About 52,000 of these illnesses are estimated to be the result of eating contaminated food, including oysters. | Most infections occur from May to October, when water is warmer. | | Death | 100 (estimated) | Vibriosis | United States | Annually | About 1 in 5 of those with serious infections die, sometimes within a day or two of illness. | People with liver disease, cancer or weakened immune systems or those taking medicine to decrease stomach acid levels are particularly susceptible to vibriosis infections or complications. |
And again with 0.2 temperature. Again, this yields the same results, missing the deaths in Missouri and Florida:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Victim came into contact with Vibrio vulnificus bacteria, likely through seawater or raw/undercooked seafood. | The cause of the infection is still being investigated. | | Death | 2 | Vibriosis | Connecticut | July 2022 | Both victims were ages 60 to 80. One consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. The other two cases were wound infections not associated with seafood. | This is the first time Connecticut has seen a Vibrio case in three years. | | Illness | 3 | Vibriosis | Connecticut | Since July 1, 2022 | All three victims were hospitalized. Two of the cases were wound infections not associated with seafood, and the third infection was of a Connecticut resident who consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. | The state has added requirements designed to cool oysters to the point where the bacteria cannot survive. | | Illness | 80,000 (estimated) | Vibriosis | United States | Annually | About 52,000 of these illnesses are estimated to be the result of eating contaminated food, including oysters. | Most infections occur from May to October, when water is warmer. | | Death | 100 (estimated) | Vibriosis | United States | Annually | About 1 in 5 of those with serious infections die, sometimes within a day or two of illness. | People with liver disease, cancer or weakened immune systems or those taking medicine to decrease stomach acid levels are particularly susceptible to vibriosis infections or complications. |
UNICORN
How about Unicorn with its default 0.0 temperature? This captures the explicit deaths, but not the estimated annual illnesses and infections. Despite being the more advanced model, it actually performs worse:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | July | Under investigation | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Connecticut | July | Wound infections not associated with seafood | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Illness | 3 | Vibriosis | Connecticut | July | All three people were hospitalized. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 1 | Vibriosis | Missouri | June | Ate raw oysters bearing the bacteria. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Florida | Last year | Ate raw oysters from Louisiana. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. |
Let's try it with the formerly-recommended 0.2 temperature. This time not only does it miss the two annual counts, but it incorrectly lists 1 illness for Connecticut instead of 3:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | July | Under investigation | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Connecticut | July | Wound infections not associated with seafood | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Illness | 1 | Vibriosis | Connecticut | July | Consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 1 | Vibriosis | Missouri | June | Ate raw oysters bearing the bacteria | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Florida | 2021 | Ate raw oysters from Louisiana | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. |
Let's run again with 0.2 temperature. This time we get the correct illness count, but still not the annual counts:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | July | Under investigation | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Connecticut | July | Wound infections not associated with seafood | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Illness | 3 | Vibriosis | Connecticut | July | All three people were hospitalized. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 1 | Vibriosis | Missouri | June | Ate raw oysters bearing the bacteria. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Florida | Last year | Ate raw oysters from Louisiana. | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. |
And one more time with 0.2 temperature – again it records the wrong illness count for Connecticut and lacks the annual counts. In fact, running repeatedly we could not get it to output the annual counts under any circumstances:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Vibriosis | Suffolk County, New York | July | Under investigation | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Connecticut | July | Wound infections not associated with seafood | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Illness | 1 | Vibriosis | Connecticut | July | Consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 1 | Vibriosis | Missouri | June | Ate raw oysters bearing the bacteria | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. | | Death | 2 | Vibriosis | Florida | 2021 | Ate raw oysters from Louisiana | Caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. |
GEMINI PRO
How about Google's most advanced model, Gemini Pro?
The first run misses the annual counts, misclassifies the Suffolk death as a second illness and excludes the Missouri and Florida deaths:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation| | --------- | --------- | --------- | --------- | --------- | --------- | ---------| | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the victim's death are not available. | The Centers for Disease Control and Prevention has estimated that about 1 in 5 vibriosis patients will die - sometimes within a day or two ofillness. | | Death | 2 | Vibriosis | Connecticut | July | Occurred this summer. | The victims were between 60 and 80 years old. | According to the CDC as of July 1st 2021, it had 78 reported cases of vibriosis out of which 37 were hospitalizations and 7 deaths. | | Illness | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the illness are not available. | The CDC estimates that the incidence of vibriosis in general might be ten times the reported rate as many of the cases are not severe and do not require medical attention. | | Illness | 3 | Vibriosis | Connecticut | Since July 1 | Including two wound infections and one an oyster-associated infection. | Last month, the Department of Public Health had reported three vibriosis cases and all three were hospitalized. |
Let's try a second time. Again the annual counts are missing, it doubles the Suffolk death as an illness and excludes the annual counts:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | | --------- | --------- | --------- | --------- | --------- | --------- | --------- | | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the victim's death are not available. | The Centers for Disease Control and Prevention has estimated that about 1 in 5 vibriosis patients will die - sometimes within a day or two of illness. | | Death | 2 | Vibriosis | Connecticut | July | Occurred this summer. | The victims were between 60 and 80 years old. | According to the CDC as of July 1st 2021, it had 78 reported cases of vibriosis out of which 37 were hospitalizations and 7 deaths. | | Illness | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the illness are not available. | The CDC estimates that the incidence of vibriosis in general might be ten times the reported rate as many of the cases are not severe and do not require medical attention. | | Illness | 3 | Vibriosis | Connecticut | Since July 1 | Including two wound infections and one an oyster-associated infection. | Last month, the Department of Public Health had reported three vibriosis cases and all three were hospitalized. | Vibriosis is caused by eating contaminated food, primarily oysters, and swimming in or having a wound exposed to warm seawater. | | Death | 1 | Vibriosis | Missouri | June | The victim ate raw oysters bearing the bacteria. | Vibriosis is found in raw and undercooked seafood and is more common in the summer months when the water is warmer. | | Death | 2 | Vibriosis | Florida | Last year | The victims ate raw oysters from Louisiana. | | So far in 2021, Florida has had 35 reported cases of Vibriosis as of July 1 2021 including 6 hospitalizations,and 1 death. |
Running again we get the same results:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | | --------- | --------- | --------- | --------- | --------- | --------- | --------- | | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the victim's death are not available. | The Centers for Disease Control and Prevention has estimated that about 1 in 5 vibriosis patients will die - sometimes within a day or two of illness. | | Death | 2 | Vibriosis | Connecticut | July | Occurred this summer. | The victims were between 60 and 80 years old. | According to the CDC as of July 1st 2021, it had 78 reported cases of vibriosis out of which 37 were hospitalizations and 7 deaths. | | Illness | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the illness are not available. | The CDC estimates that the incidence of vibriosis in general might be ten times the reported rate as many of the cases are not severe and do not require medical attention. | | Illness | 3 | Vibriosis | Connecticut | Since July 1 | Including two wound infections and one an oyster-associated infection. | Last month, the Department of Public Health had reported three vibriosis cases and all three were hospitalized. | Vibriosis is caused by eating contaminated food, primarily oysters, and swimming in or having a wound exposed to warm seawater. | | Death | 1 | Vibriosis | Missouri | June | The victim ate raw oysters bearing the bacteria. | Vibriosis is found in raw and undercooked seafood and is more common in the summer months when water is warmer. | | Death | 2 | Vibriosis | Florida | Last year | The victims ate oysters from Louisiana. | The CDC has estimated that around 52,000 of the 80,000 cases of vibriosis are a result of eating contaminated food, including oysters. |
Finally on our fourth run we get the annual counts, but the Connecticut is double-counted:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | | --------- | --------- | --------- | --------- | --------- | --------- | --------- | | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the victim's death are not available. | The Centers for Disease Control and Prevention (CDC) estimates that about 100 people die annually from vibriosis in the U.S. | | Death | 2 | Vibriosis | Connecticut | July | Both victims were ages 60 to 80. | Vibrio vulnificus has never been found in state waters, and most infections are linked to shellfish from much warmer waters where it can thrive. | | Illness | 1 | Vibriosis | Connecticut | July | A Connecticut resident who consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. | Vibriosis can cause a variety of symptoms, including diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. | | Illness | 2 | Vibriosis | Connecticut | July | Two wound infections not associated with seafood. | Wound infections can occur when the bacteria enters the body through a break in the skin. | | Illness | 80,000 | Vibriosis | U.S. | Annually | An estimated 80,000 illnesses occur every year in the U.S., about 52,000 of which are linked to eating contaminated food. | Most infections occur during the summer months when the water is warmer. | | Death | 100 | Vibriosis | U.S. | Annually | About 100 deaths occur annually in the U.S. | People with liver disease, cancer, or weakened immune systems are particularly susceptible to severe infections or complications from vibriosis. | | Death | 1 | Vibriosis | Missouri | June | A 54-year-old man died after eating raw oysters bearing the bacteria. | Vibriosis is a rare but potentially fatal infection. | | Death | 2 | Vibriosis | Florida | Last year | Two people in Florida died of vibriosis after eating raw oysters from Louisiana. | Vibriosis is a serious infection that can lead to death if not treated promptly. |
Finally on the fifth run we get the correct results, but it adds an extra illness without count for "Eastern U.S.":
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation| | --------- | --------- | --------- | --------- | --------- | --------- | ---------| | Death | 1 | Vibriosis | Suffolk County, New York | N/A | Details about the victim's death are not available. | The victim died after contracting Vibriosis, an illness caused by Vibrio vulnificus bacteria found in seawater and raw or undercooked seafood.| | Death | 2 | Vibriosis | Connecticut | July | Both victims were ages 60 to 80 and consumed raw oysters. | Two Connecticut residents died from Vibriosis in July as a result of consumingraw oysters. | | Illness | 3 | Vibriosis | Connecticut | Since July 1 | All three people were hospitalized, and two of the three cases were wound infections not associated with seafood. | Three cases of Vibriosis were reported in Connecticut since July 1, including two wound infections unrelated to seafood consumption. | | Illness | 80,000 (est.) | Vibriosis | United States | Annually | 52,000 of which are estimated to be the result of eating contaminated food, including oysters. | Vibriosis causes an estimated 80,000 illnesses in the United States annually, with most infections occurring during warmer months. | | Death | 100 (est.) | Vibriosis | United States | Annually | About 1 in 5 of those with serious infections die, sometimes within a day or two of illness. | Vibriosis causes an estimated 100 deaths annually in the United States, with a high mortality rate among those with severe infections. | | Illness | N/A | Vibriosis | Eastern U.S. | Since the late 1990s | Could continue to increase in the years to come due to climate change and warming waters. | Vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s, and the trend is expected to continue due to climate change. | | Death | 1 | Vibriosis | Missouri | June | The victimdied after eating raw oysters bearing the bacteria. | A 54-year-old Missouri man died in June after consuming raw oysters with Vibrio vulnificus bacteria. | | Death | 2 | Vibriosis | Florida | Last year | Both victims died after eating raw oysters from Louisiana. | Two people in Florida died from Vibriosis last year after they ate raw oysters from Louisiana. |
CHATGPT 3.5
ChatGPT (GPT-3.5) performs fairly poorly on its first try, failing even to extract the two illness counts and the annual counts. Interestingly, these results are far different from four months ago:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |--------|------------------|--------------|----------------------------|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, New York | Not available | The death is still under investigation, and details about when the victim died are not immediately available. New York Gov. Kathy Hochul's office issued new guidance and urged residents to take precautions. | The cause of death is linked to Vibrio vulnificus bacteria, which is found in seawater and raw or undercooked seafood. The investigation is ongoing to determine where the victim came into contact with the bacteria. | Death | 2 | Vibriosis | Connecticut | July | Two residents in Connecticut died from vibriosis this summer. Both deaths occurred in July, and the victims were ages 60 to 80. Three cases were reported since July 1, with two being wound infections not associated with seafood. The third infection occurred in a Connecticut resident who consumed raw oysters from an out-of-state establishment not harvested from Long Island Sound. | The Vibrio vulnificus bacteria causing the deaths is typically associated with seawater and raw or undercooked seafood. Connecticut, despite being a hub for oysters, has not seen a Vibrio case in three years. There is no evidence linking the infections to Connecticut shellfish, as regular tests for the bacteria are conducted. | | Illness| Not specified | Vibriosis | Not specified (Connecticut)| Since July 1 | Three cases of vibriosis were reported in Connecticut since July 1. Two cases were wound infections unrelated to seafood, and the third case involved a resident consuming raw oysters from an out-of-state establishment. All three cases resulted in hospitalization, with two deaths occurring in July. | Vibriosis is a bacterial infection caused by Vibrio vulnificus found in seawater and seafood. The infections reported in Connecticut are not linked to local shellfish, as the state conducts regular tests and has added requirements to reduce bacterial survival. | Illness| Not specified | Vibriosis | Not specified (New York) | Not available | No specific details about the illness in New York are provided in the article. New guidance was issued by Gov. Kathy Hochul's office, urging residents to take precautions to avoid Vibrio vulnificus infections. | Vibriosis is a bacterial infection with symptoms like skin breakdowns, ulcers, diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. The bacteria is found in seawater and raw or undercooked seafood. Precautions include covering wounds, avoiding raw shellfish with a weakened immune system, wearing gloves when handling shellfish, and washing hands after contact. || Illness| Not specified | Vibriosis | Not specified (General) | Not specified | Vibriosis causes an estimated 80,000 illnesses annually in the U.S., with about 52,000 resulting from eating contaminated food, including oysters. Most infections occur from May to October in warmer water. Symptoms include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious cases may require intensive care or limb amputation, with a 20% fatality rate. | Vibriosis is more common in warmer waters, and the likelihood of infection is higher for individuals with liver disease, cancer, weakened immune systems, or those on medication to decrease stomach acid. Recommendations include covering wounds in warm seawater, avoiding raw shellfish with a weakened immune system, wearing gloves when handling raw shellfish, and washing hands after contact. | Death | 1 | Vibriosis | Missouri | June | A 54-year-old Missouri man died after eating raw oysters bearing the Vibrio vulnificus bacteria. | Another case of death due to Vibrio vulnificus infection after consuming raw oysters. | Death | 2 | Vibriosis | Florida | Not specified | Two people in Florida died of vibriosis after consuming raw oysters from Louisiana. | Vibrio vulnificus infections are not limited to a specific region, as seen in cases in Missouri and Florida. Consuming raw oysters from various sources can lead to fatal outcomes.
How about a second time? This yields the same results, missing all of the same counts:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |--------|------------------|-----------------------------------------------|---------------------------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Death | 1 | Vibrio vulnificus infection | Suffolk County, New York | Not specified | One death in Suffolk County, New York, being investigated. Details regarding when the victim died and contact with bacteria not immediately available. New guidance issued. | Investigation ongoing; details yet to be determined. | | Death | 2 | Vibrio vulnificus infection | Connecticut | July | Two deaths in Connecticut this summer from vibriosis. Three cases reported since July 1, all hospitalized. Two wound infections not associated with seafood. Third infection from raw oysters consumed out-of-state. Victims aged 60 to 80. First Vibrio case in Connecticut in three years. | Vibrio cases in Connecticut linked to wound infections and raw oyster consumption. Victims aged 60 to 80. | | Illness| Not specified | Vibrio vulnificus infection | Not specified | May-Oct | Vibriosis causes an estimated 80,000 illnesses annually in the U.S. About 52,000 result from eating contaminated food, including oysters. Most infections occur from May to October when water is warmer. Symptoms include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections may require intensive care or limb amputation. 1 in 5 serious cases results in death. | Vibrio vulnificus infections can lead to various symptoms, with serious cases having severe complications and a 20% mortality rate. May occur from contaminated food consumption, including oysters, during warmer months. | Illness| Not specified | Vibrio vulnificus infection | Not specified | Not specified | Individuals with liver disease, cancer, weakened immune systems, or taking stomach acid-lowering medicine are more susceptible to vibriosis infections. Precautions include covering wounds, recent piercings, or tattoos exposed to warm seawater. Avoiding oysters and raw shellfish with weakened immune systems. Wearing gloves when handling raw shellfish. Washing wounds if exposed to saltwater or raw seafood. | Certain individuals at higher risk; precautions advised to reduce the likelihood of vibriosis infections. | | Death | 1 | Vibrio vulnificus infection | Missouri | June | A 54-year-old Missouri man died after eating raw oysters bearing the bacteria. | Fatal case in Missouri after consuming raw oysters contaminated with Vibrio vulnificus. | | Death | 2 | Vibrio vulnificus infection | Florida | Not specified | Two people in Florida died of vibriosis last year after eating raw oysters from Louisiana. | Fatal cases in Florida linked to consumption of raw oysters from Louisiana. |
CHATGPT 4.0
How about OpenAI's most advanced LLM model, ChatGPT 4.0? Shockingly, despite being the most advanced model in this list, it produces among the worst results, on par with Gemini Pro's first output. It it misses the 100 annual deaths and the Missouri and Florida deaths. This is extremely surprising:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |------------|-----------------|--------------|------------------|-------------|------------------------------------------|--------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, NY | Not specified | Victim's contact with bacteria under investigation | Caused by Vibrio vulnificus bacteria | | Death | 2 | Vibriosis | Connecticut | July | Two deaths among three hospitalized cases; two cases were wound infections, one was due to raw oysters consumption | Linked to Vibrio vulnificus, not associated with Connecticut shellfish | | Illness | 3 | Vibriosis | Connecticut | Since July 1 | Three cases reported; all hospitalized | Two cases from wound infections, one from consuming raw oysters | | Illness | Estimated 80,000 annually | Vibriosis | U.S. | Annually | Approx. 52,000 from eating contaminated food, including oysters | Common from May to October due to warmer water |
Let's try a second time. This time it breaks the Connecticut illnesses into two rows and drops the annual illness estimate, along with continuing to exclude the 100 annual deaths and the Missouri and Florida deaths. This is even more surprising even how widely touted GPT 4.0 is. Immediately clear is that GPT 4.0 requires further prompt engineering and/or one or multi-shot prompting to yield results on par with the latest Bison, Unicorn and Gemini Pro models for this particular article.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|-----------------|--------------------------------|-----------|-------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------| | Death | 1 | Vibrio vulnificus (Vibriosis) | Suffolk County, New York | Not specified | A person died from a rare flesh-eating bacterial infection. Death still under investigation. | Vibrio vulnificus bacteria found in seawater and undercooked seafood, causing severe symptoms. | | Death | 2 | Vibrio vulnificus (Vibriosis) | Connecticut | July | Two residents died. One from wound infection, another after consuming raw oysters. | Infections not linked to Connecticut shellfish. Victims aged 60-80. | | Illness | 1 | Vibrio vulnificus (Vibriosis) | Connecticut | Since July 1 | One case reported of a resident hospitalized after consuming raw oysters. | Infection from oysters not harvested from Long Island Sound. | | Illness | 2 | Vibrio vulnificus (Vibriosis) | Connecticut | Since July 1 | Two cases of wound infections resulting in hospitalization. | Wound infections not associated with seafood. |
Let's try an article that mentions multiple Powassan cases within close proximity in the text with one mention earlier in the article.
BISON
Bison correctly extracts all of the counts, though it double-counts "New England" with the actual breakdown of individual states:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1 | Powassan virus disease | Washington County, Rhode Island | Mid-July 2023 | Female over the age of 80 developed neurological symptoms | The patient died after contracting the disease. | | Death | 1 | Powassan virus disease | Washington County, Rhode Island | Mid-July 2023 | Female over the age of 80 | The patient died after contracting the disease. | | Illness | 239 | Powassan virus disease | United States | 2013-2022 | 239 cases reported in the past 10 years | The number of cases has increased in recent years. | | Illness | 93 | Powassan virus disease | New England | 2013-2022 | 93 cases reported in New England | The majority of cases in the United States have been reported in the Northeast and the Great Lakes region. | | Illness | 49 | Powassan virus disease | Massachusetts | 2013-2022 | 49 cases reported in Massachusetts | Massachusetts has the highest number of cases in New England. | | Illness | 18 | Powassan virus disease | Connecticut | 2013-2022 | 18 cases reported in Connecticut | Connecticut has the second highest number of cases in New England. | | Illness | 16 | Powassan virus disease | Maine | 2013-2022 | 16 cases reported in Maine | Maine has the third highest number of cases in New England. | | Illness | 5 | Powassan virus disease | New Hampshire | 2013-2022 | 5 cases reported in New Hampshire | New Hampshire has the fourth highest number of cases in New England. | | Illness | 5 | Powassan virus disease | Rhode Island | 2013-2022 | 5 cases reported in Rhode Island | Rhode Island has the fifth highest number of cases in New England. |
UNICORN
Unicorn yields the same results both times, but fixates on the New England total and refuses to provide the state-by-state breakdowns:
| Type | Number Affected | Disease Name | Location | Date | Details and Explanation | |---|---|---|---|---|---| | Death | 1 | Powassan virus disease | Rhode Island | July 2023 | A female over the age of 80 from Washington County, who developed neurological symptoms and died in mid-July. | | Illness | 239 | Powassan virus disease | United States | 2013-2022 | Over 239 cases of Powassan have been reported in the United States in the past 10 years. | | Illness | 93 | Powassan virus disease | New England | 2013-2022 | Between 2013 and 2022, there were 93 cases of Powassan reported in New England: 49 cases in Massachusetts, 18 cases in Connecticut, 16 cases in Maine, five cases in New Hampshire, and five cases in Rhode Island. |
GEMINI PRO
Gemini Pro yields identical overall results to Bison.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation| |---|---|---|---|---|---|---| | Illness | 1 | Powassan virus disease | Rhode Island | Mid-July 2023 | Female over 80 from Washington County developed neurological symptoms | Transmitted by the bite of an infected tick | | Death | 1 | Powassan virus disease | Rhode Island | Mid-July 2023 | Female over 80 from Washington County died after contracting the disease | Powassan can be fatal, especially in older adults | | Illness | 239 | Powassan virus disease | United States | 2013-2022 | Over 239 cases reported in the past 10 years | Cases are rare, but the number has increased in recent years | | Illness | 93 | Powassan virus disease | New England | 2013-2022 | 93 cases reported | Cases have been reported in Massachusetts, Connecticut, Maine, New Hampshire, and Rhode Island | | Illness | 49 | Powassan virus disease | Massachusetts | 2013-2022 | 49 cases reported | Most cases in New England have occurred in Massachusetts | | Illness | 18 | Powassan virus disease | Connecticut | 2013-2022 | 18 cases reported | Cases have been reported in Connecticut as well | | Illness | 16 | Powassan virus disease | Maine | 2013-2022 | 16 cases reported | Cases have been reported in Maine as well | | Illness | 5 | Powassan virus disease | New Hampshire | 2013-2022 | 5 cases reported | Cases have been reported in New Hampshire as well | | Illness | 5 | Powassan virus disease | Rhode Island | 2013-2022 | 5 cases reported | Cases have been reported in Rhode Island as well |
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation| |---|---|---|---|---|---|---| | Illness | 1 | Powassan virus disease | Rhode Island | Mid-July 2023 | Female over 80 from Washington County developed neurological symptoms | Transmitted by the bite of an infected tick | | Death | 1 | Powassan virus disease | Rhode Island | Mid-July 2023 | Female over80 from Washington County died after contracting the disease | Powassan can be fatal, especially in older adults | | Illness | 239 | Powassan virus disease | United States | 2013-2022 | Over 239 cases reported in the past 10 years | Cases are rare, but the number has increased in recent years | | Illness | 93 | Powassan virus disease | New England | 2013-2022 | 93 cases reported | Cases have been reported in Massachusetts, Connecticut, Maine, New Hampshire, and Rhode Island | | Illness | 49 | Powassan virus disease | Massachusetts | 2013-2022 | 49 cases reported | Most cases in New England have occurred in Massachusetts | | Illness | 18 | Powassan virus disease | Connecticut | 2013-2022 | 18 cases reported | Cases have been reported in Connecticut as well | | Illness | 16 | Powassan virus disease | Maine | 2013-2022 | 16 cases reported | Cases have been reported in Maine as well | | Illness | 5 | Powassan virus disease | New Hampshire | 2013-2022 | 5 cases reported | Cases have been reported in New Hampshire as well | | Illness | 5 | Powassan virus disease | Rhode Island | 2013-2022 | 5 cases reported | Cases have been reported in Rhode Island as well |
GPT 3.5
ChatGPT 3.5 performs extremely poorly on this example, extracting just a single case the first time and grouping together various passages into incorrect extractions the second time:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|-------------------|------------------|-------------|-------------------------------------------------------------------|---------------------------------------------------------------------------------------------| | Illness | 1 | Powassan Disease | Washington County, Rhode Island | Mid-July | A female over the age of 80 from Washington County developed neurological symptoms and died after contracting Powassan. | Powassan is a tick-borne disease found in the Northeast and Great Lakes region.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|---------------------|--------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | Illness | 1 | Powassan Virus | Rhode Island | Mid-July | A female over the age of 80 from Washington County, Rhode Island, developed neurological symptoms and died in mid-July after contracting the tick-borne Powassan virus disease. | Powassan is a tick-borne disease found in the Northeast and the Great Lakes region of the U.S. and eastern Canada. It can cause neurological symptoms and, in severe cases, may lead to death. | | Illness | Unknown | Powassan Virus | United States | 2013-2022 | Over 239 cases of Powassan have been reported in the United States in the past 10 years (2013-2022). | Powassan cases are rare but have been increasing in recent years. | | Illness | 93 | Powassan Virus | New England | 2013-2022 | Between 2013 and 2022, there were 93 cases of Powassan reported in New England: 49 cases in Massachusetts, 18 cases in Connecticut, 16 cases in Maine, five cases in New Hampshire, and five cases in Rhode Island. | New England had several reported cases of Powassan during this period. | | Illness | Various | Powassan Virus | Various | Various | Initial symptoms of Powassan include fever, headache, vomiting, and generalized weakness. The disease usually progresses to meningoencephalitis, which may include meningeal signs, altered mental status, seizures, aphasia, paresis (muscular weakness or paralysis), movement disorders, or cranial nerve palsies. | Powassan can cause a range of neurological symptoms and may require hospitalization. | | Illness | Various | Powassan Virus | Various | Various | There is no vaccine or treatment for Powassan, so preventing exposure to ticks is the best strategy to avoid this disease. | Prevention is essential since there is no specific treatment or vaccine. | | Illness | Unknown | Lyme Disease | Rhode Island | Not specified | RIDOH and the Rhode Island Department of Environmental Management (DEM) remind Rhode Islanders to take steps to prevent tick-borne diseases, including Powassan and Lyme Disease, when spending time outdoors. | The article mentions Lyme Disease as another tick-borne disease to prevent. |
GPT 4.0
Surprisingly, GPT 4.0 also performs extremely poorly the first time it is run.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |------------|-----------------|--------------|-------------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------| | Death | 1 | Powassan | Washington County, Rhode Island | Mid-July | Female over 80, developed neurological symptoms and died. | Powassan is a rare, tick-borne disease. The patient's age and the severity of symptoms led to her death. | | Illness | 239 (over 10 years) | Powassan | United States (Northeast and Great Lakes region), Eastern Canada | 2013-2022 | 239 cases reported, with increasing frequency in recent years. | Powassan is rare but increasing in frequency. Initial symptoms include fever, headache, vomiting, and weakness. | | Illness | 93 | Powassan | New England (Massachusetts, Connecticut, Maine, New Hampshire, Rhode Island) | 2013-2022 | 49 cases in Massachusetts, 18 in Connecticut, 16 in Maine, 5 in New Hampshire, 5 in Rhode Island.
The second time it is run it experiences coherence failure and outputs a lengthy and verbose writeup:
To organize the information from the provided news article into a table, I will extract the relevant details regarding the illnesses and deaths associated with the Powassan virus as described in the article:
1. **Individual Case in Rhode Island**
– Type: Death
– Number Affected: 1
– Disease Name: Powassan Virus Disease
– Location: Washington County, Rhode Island
– Date: Mid-July (year not specified)
– Details: Female over 80, developed neurological symptoms
– Explanation: Confirmed case of tick-borne disease, progressed to severe symptoms
2. **Powassan Virus Disease in the United States (2013-2022)**
– Type: Illness
– Number Affected: 239
– Disease Name: Powassan Virus Disease
– Location: United States
– Date: 2013-2022
– Details: Cases reported over 10 years
– Explanation: Tick-borne disease, increased cases in recent years
3. **Powassan Virus Disease in New England (2013-2022)**
– Type: Illness
– Number Affected: 93 (49 in Massachusetts, 18 in Connecticut, 16 in Maine, 5 in New Hampshire, 5 in Rhode Island)
– Disease Name: Powassan Virus Disease
– Location: New England
– Date: 2013-2022
– Details: Regional distribution of cases
– Explanation: Part of the overall U.S. cases, highlighting regional prevalence
I will now create a table to clearly present this information.
Here is the table summarizing the illnesses and deaths related to the Powassan virus disease as described in the news article:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|———|————————————|————————-|———————————–|———————————-|———————————————–|———————————————————————|
| Death | 1 | Powassan Virus Disease | Washington County, Rhode Island | Mid-July (year not specified) | Female over 80, developed neurological symptoms | Confirmed case of tick-borne disease, progressed to severe symptoms |
| Illness | 239 | Powassan Virus Disease | United States | 2013-2022 | Cases reported over 10 years | Tick-borne disease, increased cases in recent years |
| Illness | 93 (49 MA, 18 CT, 16 ME, 5 NH, 5 RI) | Powassan Virus Disease | New England | 2013-2022 | Regional distribution of cases | Part of the overall U.S. cases, highlighting regional prevalence |
This table provides a concise overview of the impact of Powassan virus disease in the specified regions and time frames.
It even includes a structured machine-friendly representation of the extracted table:
import pandas as pd
# Data extracted from the article
data = [
{
"Type": "Death",
"Number Affected": 1,
"Disease Name": "Powassan Virus Disease",
"Location": "Washington County, Rhode Island",
"Date": "Mid-July (year not specified)",
"Details": "Female over 80, developed neurological symptoms",
"Explanation": "Confirmed case of tick-borne disease, progressed to severe symptoms"
},
{
"Type": "Illness",
"Number Affected": 239,
"Disease Name": "Powassan Virus Disease",
"Location": "United States",
"Date": "2013-2022",
"Details": "Cases reported over 10 years",
"Explanation": "Tick-borne disease, increased cases in recent years"
},
{
"Type": "Illness",
"Number Affected": "93 (49 MA, 18 CT, 16 ME, 5 NH, 5 RI)",
"Disease Name": "Powassan Virus Disease",
"Location": "New England",
"Date": "2013-2022",
"Details": "Regional distribution of cases",
"Explanation": "Part of the overall U.S. cases, highlighting regional prevalence"
}
]
# Creating a DataFrame
df = pd.DataFrame(data)
df
Run a third time it yields the following table that is accurate, though fixates on New England rather than grouping by individual state:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|---------------------|-----------------|-----------------|----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Powassan virus | Washington County, Rhode Island | Mid-July | Female over 80, developed neurological symptoms and died | A confirmed case of Powassan in an elderly female leading to death, after displaying neurological symptoms | | Illness | 239 (in 10 years) | Powassan virus | Northeast U.S., Great Lakes region, Eastern Canada | 2013-2022 | Over 239 cases reported in the U.S. | Increased reporting of Powassan cases in the past decade, mostly in Northeast U.S. and Great Lakes region | | Illness | 93 (in 10 years) | Powassan virus | New England | 2013-2022 | 49 in Massachusetts, 18 in Connecticut, 16 in Maine, 5 in New Hampshire, 5 in Rhode Island | Increased reporting of Powassan in New England over the past decade, with Massachusetts having the highest number of cases | | Illness | - | Powassan virus | - | - | Symptoms: fever, headache, vomiting, weakness, progresses to meningoencephalitis | Description of Powassan symptoms, including fever, headache, and potentially severe neurological complications like meningoencephalitis, seizures, and muscular weakness |
Finally, how about a more complex example about Monkeypox that includes a range of numbers in close proximity and places more pressure on the parsing and associative capabilities of the model?
BISON
Bison lumps the county-level breakouts (493 cases in Miami-Dade / 436 cases in Broward County) under the state-level total in the "details" field making them not directly accessible for machine processing. However, it does properly break the others down by county:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1,266 | Monkeypox | Florida | August 2022 | 493 cases in Miami-Dade, 436 cases in Broward County | Largest number of cases in South Florida | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2022 | Age group of 15 to 19 | New cases in younger age group | | Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 2022 | Age group of 20 to 24 | Increase in cases in this age group | | Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 2022 | Age group of 25 to 29 | Increase in cases in this age group | | Illness | 1 | Monkeypox | Martin County | August 2022 | Age group of 0 to 4 | Rare case in younger age group |
UNICORN
Unicorn produces nearly identical results, however it fails to report the county-level breakdowns under the state-level total like Bison did:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1,266 | Monkeypox | Florida | August 2022 | Most cases occur in men who have sex with men, but anyone can contract the virus. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2022 | Cases in the age group of 15 to 19. | N/A | | Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 2022 | Cases in the age group of 20 to 24. | N/A | | Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 2022 | Cases in the age group of 25 to 29. | N/A | | Illness | 1 | Monkeypox | Martin County | August 2022 | Case in the age group of 0 to 4. | N/A |
GEMINI PRO
Gemini Pro truncates its first output mid-response with a "finishReason" of "OTHER", despite usageMetadata showing it is far below the max token length. Unfortunately, the Gemini documentation does not provide any hints as to what may cause an "OTHER" response:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation| |---|---|---|---|---|---|---| | Illness | 1,266 | Monkeypox | Florida | August 2023 | Largest number of cases in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2023 | Age group of 15 to 19. | New cases have come at a time when K-12 schools have opened, and thousands of students and staff will begin classes this month at Florida’s universities and community colleges. | | Illness | 64 | Monkeypox | Broward,
Interestingly, it provides citations to syndicated versions of the article, showing it recognizes them:
- https://www.city-data.com/forum/florida/3376010-monkey-pox-general-thread.html
- https://www.newsbreak.com/news/2706779843044/monkeypox-outbreak-infections-rising-in-teens-and-young-adults-nearly-1-300-cases-in-fl
Running a second time it similarly truncates the output:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1,266 | Monkeypox | Florida | August 2023 | Largest number of cases in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2023 | Age group of 15 to 19. | New cases have come at a time when K-12 schools have opened, and thousands of students and staff will begin classes this month at Florida’s universities and community colleges. | | Illness | 64 | Monkeypox | Broward,
GPT 3.5
GPT 3.5 provides the most detailed extraction, but fails to break out by county and includes some extraneous duplication. Unlike Bison and Unicorn, it displays numeric summaries like "14 counties in Florida" instead of listing the actual counties like Bison and Unicorn:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|--------------------------------------|-------------|----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | As of Monday | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected. | Monkeypox outbreak in Florida, affecting teens and young adults. | | Illness | 493 | Monkeypox | Miami-Dade County, Florida | As of Monday | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade County. | Highest concentration of monkeypox cases in Miami-Dade County. | | Illness | 436 | Monkeypox | Broward County, Florida | As of Monday | Broward County has 436 monkeypox cases. | Significant monkeypox cases in Broward County. | | Illness | Various | Monkeypox | 28 counties in Florida | As of Monday | Monkeypox cases reported in 28 counties in Florida. | Widespread monkeypox cases across multiple counties in Florida. | | Illness | Various | Monkeypox | Multiple counties in Florida | As of Monday | Monkeypox cases have increased in multiple age groups. | Monkeypox cases affecting different age groups in various counties. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | As of Monday | Eight monkeypox cases reported in the age group of 15 to 19 in Broward, Lee, Leon, and Miami-Dade. | Monkeypox cases in the age group 15 to 19 in specific counties. | | Illness | 64 | Monkeypox | 14 counties in Florida | As of Monday | In the 20 to 24 age range, there are now 64 monkeypox cases. | Monkeypox cases in the age group 20 to 24 across 14 counties. | | Illness | 159 | Monkeypox | 19 counties in Florida | As of Monday | In the 25 to 29 age range, there are 159 monkeypox cases. | Monkeypox cases in the age group 25 to 29 across 19 counties. | | Illness | 1 | Monkeypox | Martin County, Florida | As of Monday | There is one case in Martin County, in the 0 to 4 age range. | A single monkeypox case in the age group 0 to 4 in Martin County. | | Death | Not mentioned | Not mentioned| Not mentioned | Not mentioned| Not mentioned | Not mentioned
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|---------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | As of Monday | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected in several counties in the outbreak. | Monkeypox outbreak in Florida affecting a significant number of individuals. | | Illness | 493 | Monkeypox | Miami-Dade County, Florida | As of Monday | Largest number of cases remain in South Florida, with 493 cases in Miami-Dade. | High concentration of Monkeypox cases in Miami-Dade County. | | Illness | 436 | Monkeypox | Broward County, Florida | As of Monday | 436 cases in Broward County. | Significant number of Monkeypox cases in Broward County. | | Illness | Various | Monkeypox | 28 counties in Florida | As of Monday | 28 counties have one or more monkeypox cases. | Widespread distribution of Monkeypox cases across Florida. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade, Florida | As of Monday | Eight monkeypox cases in the age group of 15 to 19 in Broward, Lee, Leon, and Miami-Dade. | Monkeypox affecting teenagers and young adults in multiple counties. | | Illness | 64 | Monkeypox | 14 counties in Florida | As of Monday | In the 20 to 24 age range, there are 64 monkeypox cases in 14 counties. | Significant increase in Monkeypox cases among individuals aged 20 to 24. | | Illness | 159 | Monkeypox | 19 counties in Florida | As of Monday | In the 25 to 29 age range, there are 159 monkeypox cases in 19 counties. | Increase in Monkeypox cases among individuals aged 25 to 29 in multiple counties. | | Illness | 1 | Monkeypox | Martin County, Florida | As of Monday | There is one case in Martin County, in the 0 to 4 age range. | Monkeypox case in a young child in Martin County. | | Death | Not mentioned | Monkeypox | Not mentioned (Federal and global context) | August 4, 2023 | Federal government declared a public health emergency over monkeypox. | Declaration of a public health emergency at the federal level due to Monkeypox. | | Illness | Not mentioned | Monkeypox | Florida | Not mentioned | Republican Gov. Ron DeSantis shut down the idea of declaring a state of emergency in Florida. | Florida's response to Monkeypox outbreak differs from other states. | | Illness | Not mentioned | Monkeypox | New York, California, Illinois | Not mentioned | Other states like New York, California, and Illinois declared a state of emergency due to the rise in cases. | Contrasting responses to Monkeypox outbreak in different states.
GPT 4.0
GPT 4.0 places into machine-readable format the two county-level counts that Bison and Unicorn placed in the human-readable Details field (the 493 and 436 counts), but like GPT 3.5 lumped the multi-county breakdowns under the numeric headings of "14 counties" and "19 counties". From a mapping perspective, its results are actually poorer than those of the much smaller and less capable Bison. Far from providing vastly superior results due to its size, GPT 4.0's results are largely indistinguishable from the others.
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|-----------------|--------------|-------------------------------------|------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | Not specified| Statewide total cases | Outbreak in Florida | | Illness | 493 | Monkeypox | Miami-Dade | Not specified| County-specific cases | Part of Florida outbreak | | Illness | 436 | Monkeypox | Broward County | Not specified| County-specific cases | Part of Florida outbreak | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | Not specified| Age group 15-19 cases | Part of Florida outbreak, noted in specific counties | | Illness | 64 | Monkeypox | 14 counties | Not specified| Age group 20-24 cases | Increased from 43, spread across multiple counties | | Illness | 159 | Monkeypox | 19 counties | Not specified| Age group 25-29 cases | Increased from 124, spread across multiple counties | | Illness | 1 | Monkeypox | Martin County | Not specified| Age group 0-4 case | Rare case in a very young individual |
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|-----------------------|--------------|-------------------------------------------------------------------------|------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | As of Monday | Total cases in Florida | Statewide outbreak | | Illness | 493 | Monkeypox | Miami-Dade, FL | As of Monday | Cases in Miami-Dade County | Part of the statewide outbreak | | Illness | 436 | Monkeypox | Broward County, FL | As of Monday | Cases in Broward County | Part of the statewide outbreak | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | As of Monday | Cases in 15-19 age group in specified counties | Part of the statewide outbreak, noted in teenagers | | Illness | 64 | Monkeypox | 14 counties | As of Monday | Cases in 20-24 age group across 14 counties | Increase from 43 cases last week, young adult impact | | Illness | 159 | Monkeypox | 19 counties | As of Monday | Cases in 25-29 age group across 19 counties | Increase from 124 cases last week, young adult impact| | Illness | 1 | Monkeypox | Martin County, FL | As of Monday | Case in the 0-4 age range | Rare case in a very young child |