Earlier this year we explored the use of LLMs for disease extraction and codification tasks from news coverage, testing how well they could be used to autonomously codify illness and death information from realtime news coverage. Given the immense advances that have occurred in LLMs over the 6 months since that analysis, let's revisit this topic and expand to GCP's Bison, OpenAI's GPT-3.5 and Anthropic's Claude across a range of example texts from trivial to complex.
The end result is that even for highly complex texts, LLMs are capable of substantial codification of disease information. The biggest limitation is the instability of those results: running a model multiple times (each from scratch in a fresh session) can yield very different results, with some runs producing quite impressive codifications, while others yield far more modest results. In a production application, this means model performance will vary dramatically over input text, producing high quality results for some articles and far less accurate results for others that it actually is capable of producing high quality results for. As with all of our LLM work, we recommend running a model multiple times and ranking the results. While these examples do not feature hallucinations, they do exhibit a variety of failure states in which they improperly codify unrelated content, though in this case the missing extracted fields could be used as a filter. The models do struggle across the board with the sheer diversity of ways case and death counts are expressed in the media and can improperly double count cases. Overall, these results suggest substantial promise but also recommend substantial caution given their extreme instability. In other words, with any given run, the difference between high quality codification and missing key parts of the article is down to mere random chance.
It is important to recognize that prompt optimization and especially model-specific prompt tuning designed for each model's idiosyncrasies, could improve these results, but instability and failure states are endemic to current LLM architectures, meaning prompt engineering can yield improvements, but not existential stability beyond the results seen here.
Let's start with this article about vibriosis infections from seafood.
Let's start with Bison. We'll use the following prompt with temperature of 0.2:
Make a table that extracts all of the illnesses and deaths described in the following article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location, Date, Details and Explanation.
This yields the following table that misses the majority of the text, including Connecticut, Missouri and Florida:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1 | Vibriosis | Suffolk County, New York | July | Not yet known | Still under investigation | | Death | 2 | Vibriosis | Connecticut | July | Two of the three cases were wound infections not associated with seafood | Connecticut has not seen a Vibrio case in three years | | Death | 1 | Vibriosis | Connecticut | July | Consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment | Connecticut is home to a thriving oyster industry, and conducts regular tests for the bacteria |
Lets run a second time:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1 | Vibriosis | Suffolk County, New York | July | Not yet known | Still under investigation | | Death | 2 | Vibriosis | Connecticut | July | Two of the three cases were wound infections not associated with seafood, and the third infection was a Connecticut resident that consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. | All three victims were between the ages of 60 to 80 and the two deaths occurred in July, the department said, adding that it’s first time Connecticut has seen a Vibrio case in three years. |
Third:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 1 | Vibriosis | Suffolk County, New York | July | Not yet known | Still under investigation | | Death | 2 | Vibriosis | Connecticut | July | Two of the three cases were wound infections not associated with seafood, the health department said, and the third infection was a Connecticut resident that consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. | All three victims were between the ages of 60 to 80 and the two deaths occurred in July, the department said, adding that it’s first time Connecticut has seen a Vibrio case in three years. |
Fourth:
| Type | Number Affected | Disease Name | Location | Date | Details | Explaination | |---|---|---|---|---|---|---| | Illness | 1 | Vibriosis | Suffolk County, New York | July | Not yet known | Still under investigation | | Death | 2 | Vibriosis | Connecticut | July | Two deaths occurred | First time Connecticut has seen a Vibrio case in three years |
Overall, Bison misses nearly all of the cases mentioned in the text and does a fairly poor job of codifying its contents.
What about the same prompt and text with Claude? It yields fairly similar results:
Here is the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Death | 1 | Vibriosis | Suffolk County, NY | July/August 2023 | Victim contracted bacteria from unknown source | Rare bacterial infection that can come from seawater or raw seafood | | Death | 2 | Vibriosis | Connecticut | July 2023 | Victims were 60-80 years old and ate raw oysters from out of state | Rare bacterial infection that can come from seawater or raw seafood | | Illness | 3 | Vibriosis | Connecticut | July 2023 | Hospitalized after contracting from raw oysters (1 case), wound infections (2 cases) | Rare bacterial infection that can come from seawater or raw seafood |
Running a second time doesn't change the results:
Here is the table summarizing the illnesses and deaths described in the article: |Type|Number Affected|Disease Name|Location|Date|Details|Explanation| |-|-|-|-|-|-|-| |Death|1|Vibriosis|Suffolk County, New York|July/August 2022|Victim contracted bacteria from unknown source|Rare flesh-eating bacterial infection| |Death|2|Vibriosis|Connecticut|July 2022|Victims consumed raw oysters from out-of-state|Rare flesh-eating bacterial infection| |Illness|3|Vibriosis|Connecticut|July 2022|2 cases from wound infections, 1 case from consuming raw oysters|Rare flesh-eating bacterial infection|
And a third time:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Death | 1 | Vibriosis | Suffolk County, NY | July/August 2022 | Victim contracted bacteria from unknown source | Rare bacterial infection found in seawater and raw seafood | | Death | 2 | Vibriosis | Connecticut | July 2022 | Victims were 60-80 years old and ate raw oysters from out of state | Rare bacterial infection found in seawater and raw seafood | | Illness | 3 | Vibriosis | Connecticut | July 2022 | Hospitalized after 2 got wound infections, 1 ate raw oysters | Rare bacterial infection found in seawater and raw seafood |
Claude yields results extremely similar to Bison and again fails to extract most results.
What about the same prompt and text with OpenAI's ChatGPT (GPT-3.5)? This yields results that are far more comprehensive than Bison, but still excludes the macro-level 100 deaths and 80,000 illnesses per year:
Here's a table summarizing the illnesses and deaths described in the provided article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|------------------|-----------------------------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, New York | Not stated | The victim's death is being investigated to determine the source of bacterial contact. New guidance is issued, and precautions are advised for residents. | Further details about the date and circumstances of the death are not immediately available. | | Death | 2 | Vibriosis | Connecticut | July | Two Connecticut residents, aged 60 to 80, died in July from vibriosis. One case was linked to raw oysters consumption, while two were wound infections not associated with seafood. The Department of Public Health noted that it was the first Vibrio case in Connecticut in three years. | The infection sources include seafood consumption and wound infections. Connecticut typically conducts tests for the bacteria and has taken measures to mitigate its presence in harvested oysters. | | Illness | 3 | Vibriosis | Various locations | July | Three cases of vibriosis were reported in Connecticut since July 1. All three individuals were hospitalized. Symptoms included diarrhea, stomach cramps, vomiting, fever, and chills. | The infections in these cases were linked to seafood consumption and wound infections. | | Illness | Unknown | Vibriosis | United States | May-Oct | Vibriosis causes an estimated 80,000 illnesses annually in the U.S., with about 52,000 cases resulting from contaminated food consumption, including oysters. Symptoms range from mild (diarrhea, stomach cramps) to severe (intensive care or limb amputation), with a 20% fatality rate for serious cases. High-risk individuals are more susceptible. | Vibriosis is a well-known infection caused by vibrio vulnificus bacteria, commonly found in seawater and raw/undercooked seafood. It can lead to various symptoms and outcomes, particularly affecting individuals with weakened immune systems or underlying health conditions. | | Death | 1 | Vibriosis | Missouri | June | A 54-year-old man died in Missouri after consuming raw oysters contaminated with the bacteria. | Consumption of contaminated raw oysters led to a fatal case of vibriosis. | | Death | 2 | Vibriosis | Florida | Not stated | Two people in Florida died from vibriosis after eating raw oysters from Louisiana. | Consumption of contaminated raw oysters from Louisiana resulted in fatal cases of vibriosis. | Please note that some details such as specific dates and certain information may not be provided in the article. The table summarizes the available information from the provided text.
Running a second time yields much more comprehensive results. This time it includes the annual figures. But it also includes a number of false positives including codifying "Washing wounds with soap and water if exposed to saltwater, raw seafood, or its juices is recommended" and "Healthcare providers are reminded to consider vibriosis as a diagnosis, especially in relevant cases" as illness references.
Here's a table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|----------------------------|---------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, NY | July | One person in New York's Suffolk County died from an infection linked to vibriosis, caused by vibrio vulnificus bacteria found in seawater and raw seafood. The exact source of exposure is being investigated. | Vibrio vulnificus bacteria, present in seawater and raw seafood, caused the death. | | Death | 2 | Vibriosis | Connecticut | July | Two people in Connecticut died from infections linked to vibriosis. Two of the three cases were wound infections not associated with seafood. The third infection was a Connecticut resident who consumed raw oysters not harvested from Long Island Sound at an out-of-state establishment. Victims were between 60 to 80 years old. | Vibriosis infections, including wound infections and oyster consumption, resulted in deaths. | | Illness | 3 | Vibriosis | Connecticut | July | Three cases of vibriosis were reported in Connecticut since July 1, all leading to hospitalizations. Two cases were wound infections not associated with seafood, and one case was linked to consuming raw oysters from an out-of-state establishment. Victims were between 60 to 80 years old. | Vibrio vulnificus infections caused illnesses, including wound infections and oyster consumption. | | Illness | Est. 80,000 | Vibriosis | United States | Yearly | Vibriosis causes an estimated 80,000 illnesses in the U.S. annually. About 52,000 cases are related to eating contaminated food, including oysters. Most infections occur between May and October. | Vibriosis is a common illness caused by vibrio vulnificus bacteria, often linked to contaminated food. | | Death | 100 | Vibriosis | United States | Yearly | Vibriosis results in approximately 100 deaths in the U.S. each year. | Serious cases of vibriosis can lead to death. | | Illness | Varied | Vibriosis | United States | Varied | Vibriosis symptoms include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious cases may require intensive care or limb amputation. People with certain health conditions are more susceptible. | Vibriosis can cause a range of symptoms, from mild to severe, and can lead to serious complications. | | Illness | Varied | Vibriosis | United States | Varied | People with liver disease, cancer, a weakened immune system, or taking certain medications are particularly susceptible to vibriosis. | Individuals with specific health conditions are at higher risk of vibriosis infections. | | Illness | Varied | Vibriosis | United States | Varied | Precautions to reduce vibriosis risk include covering wounds, piercings, or tattoos when exposed to warm seawater, avoiding raw shellfish with a weakened immune system, wearing gloves when handling raw shellfish, and washing hands afterward. | Certain precautions can help reduce the risk of vibriosis infection. | | Illness | Varied | Vibriosis | United States | Varied | Washing wounds with soap and water if exposed to saltwater, raw seafood, or its juices is recommended. | Proper wound care can prevent infection. | | Illness | Varied | Vibriosis | United States | Varied | Providers are reminded to consider vibriosis as a diagnosis, especially in relevant cases. | Vibriosis may not always be the first diagnosis considered by healthcare providers. | | Death | 1 | Vibriosis | Missouri, United States | June | A 54-year-old man died after consuming raw oysters containing vibrio vulnificus bacteria. | Consumption of raw oysters led to a fatal infection. | | Death | 2 | Vibriosis | Florida, United States | Yearly | Two people died of vibriosis in Florida after consuming raw oysters from Louisiana. | Raw oyster consumption led to fatal vibriosis infections. | Please note that some information in the article is not specified or fully detailed, so the table provides a summary based on the available information. Additionally, the "Number Affected" in the table is based on the provided information and may not include all reported cases.
A third time yields identical results:
Sure, here's a table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|--------------------------|-------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, NY | July | One person in Suffolk County, NY, died from an infection linked to vibriosis caused by vibrio vulnificus bacteria found in seawater and raw seafood. The source of exposure is under investigation. | Vibrio vulnificus, present in seawater and raw seafood, caused the death. | | Death | 2 | Vibriosis | Connecticut | July | Two people in Connecticut died from vibriosis. Two cases were wound infections unrelated to seafood. The third case was a Connecticut resident who ate raw oysters from an out-of-state establishment. All victims were aged 60 to 80. | Vibriosis infections, including wound infections and oyster consumption, resulted in deaths. | | Illness | 3 | Vibriosis | Connecticut | July | Connecticut reported three vibriosis cases since July 1. Two were wound infections not tied to seafood. The third case involved a Connecticut resident consuming raw oysters from an out-of-state establishment. All affected individuals were between 60 and 80 years old. | Vibrio vulnificus infections led to illnesses, including wound infections and oyster consumption. | | Illness | Est. 80,000 | Vibriosis | United States | Yearly| Vibriosis causes about 80,000 illnesses in the U.S. annually. Approximately 52,000 cases result from eating contaminated food, including oysters. Infections mainly occur between May and October. | Vibriosis is a common illness often linked to contaminated food. | | Death | 100 | Vibriosis | United States | Yearly| Vibriosis leads to around 100 deaths in the U.S. every year. | Serious cases of vibriosis can result in death. | | Illness | Varied | Vibriosis | United States | Varied| Vibriosis symptoms include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Severe cases may require intensive care or even limb amputation. People with certain health conditions are more susceptible. | Vibriosis can range from mild to severe, potentially leading to serious complications. | | Illness | Varied | Vibriosis | United States | Varied| Individuals with liver disease, cancer, weakened immune systems, or taking specific medications are particularly vulnerable to vibriosis. | People with certain health conditions are at higher risk of vibriosis infections. | | Illness | Varied | Vibriosis | United States | Varied| Precautions to reduce vibriosis risk include covering wounds, piercings, or tattoos when exposed to warm seawater, avoiding raw shellfish with a weakened immune system, wearing gloves when handling raw shellfish, and washing hands afterward. | Taking precautions can lower the risk of vibriosis infection. | | Illness | Varied | Vibriosis | United States | Varied| Washing wounds with soap and water after exposure to saltwater, raw seafood, or their juices is advised. | Proper wound care can prevent infection. | | Illness | Varied | Vibriosis | United States | Varied| Healthcare providers are reminded to consider vibriosis as a diagnosis, especially in relevant cases. | Vibriosis may not always be the first diagnosis considered by healthcare providers. | | Death | 1 | Vibriosis | Missouri, United States | June | A 54-year-old man died after eating raw oysters containing vibrio vulnificus bacteria. | Consuming raw oysters led to a fatal infection. | | Death | 2 | Vibriosis | Florida, United States | Yearly| Two people in Florida died of vibriosis after consuming raw oysters from Louisiana. | Consumption of raw oysters led to fatal vibriosis infections. | Please note that some information might not be explicitly stated in the article, and certain details are inferred based on the context provided.
A fourth time eliminates the annual estimates once again:
Here's the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, New York | July | One person in Suffolk County, New York, died from vibriosis linked to vibrio vulnificus bacteria. | The victim's contact with the bacteria is being investigated. Vibriosis is caused by vibrio vulnificus bacteria found in seawater and seafood. Serious cases can lead to death, particularly in individuals with certain medical conditions. | | Death | 2 | Vibriosis | Connecticut | July | Two people in Connecticut died from vibriosis linked to vibrio vulnificus bacteria. | Both deaths occurred in July, with the victims aged 60 to 80. One case was associated with raw oyster consumption, and the other cases were wound infections not related to seafood. Connecticut rarely sees Vibrio cases. | | Illness | 3 | Vibriosis | Connecticut | July-Sept | Three people in Connecticut were affected by vibriosis. All three cases resulted in hospitalization. | Two cases were wound infections not related to seafood. The third case involved consuming raw oysters from out-of-state. Vibriosis symptoms range from mild to severe, with serious cases potentially leading to death. | | Death | Unknown | Vibriosis | Not specified | Not known | More information about the Suffolk County victim's death was not immediately available. | Information about this case is limited in the article. | | Illness | Unknown | Vibriosis | United States (annual estimate) | Ongoing | About 80,000 illnesses and 100 deaths annually in the U.S. due to vibriosis, with around 52,000 cases from contaminated food. | Vibriosis is a common illness caused by vibrio vulnificus bacteria. Infections primarily occur between May and October in warmer waters. Vulnerable individuals are at higher risk of severe outcomes. | | Illness | Multiple | Vibriosis | Eastern U.S. | Since 1990s | Vibriosis infections have been increasing in the eastern U.S. due to climate change and warming waters. | Changing environmental conditions contribute to the rise in vibriosis cases. | | Death | 1 | Vibriosis | Missouri | June | A 54-year-old man in Missouri died from vibriosis after consuming raw oysters with vibrio vulnificus bacteria. | Consumption of contaminated raw oysters led to the fatal vibriosis infection. | | Death | 2 | Vibriosis | Florida (Louisiana oysters) | Not known | Two people in Florida died from vibriosis after eating raw oysters harvested from Louisiana. | Consuming raw oysters contaminated with vibrio vulnificus bacteria resulted in fatal infections. | Please note that some information, such as specific dates and details, is missing or unspecified in the article. The table is based on the available information from the provided article.
Let's modify our prompt to request only entries with counts:
Make a table that extracts all of the illnesses and deaths described in the following article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location, Date, Details and Explanation. Only include illnesses and deaths that specify a specific disease and include the number affected, even if it is an estimate.
This fails to filter out the entries without specific counts and fails to place the 80,000 estimate under the Number Affected column. It also fixates on the location of the oysters to offer "Florida (Louisiana)" as a location.
Sure, here's the table focusing specifically on the illnesses and deaths with specified diseases, numbers affected, and locations from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|-------------------------|-------|---------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Vibriosis | Suffolk County, NY | July | One person in Suffolk County, New York, died from vibriosis linked to vibrio vulnificus bacteria. | The victim's contact with the bacteria is being investigated. Vibriosis is caused by vibrio vulnificus bacteria found in seawater and seafood. Serious cases can lead to death, particularly in individuals with certain medical conditions. | | Death | 2 | Vibriosis | Connecticut | July | Two people in Connecticut died from vibriosis linked to vibrio vulnificus bacteria. | Both deaths occurred in July, with the victims aged 60 to 80. One case was associated with raw oyster consumption, and the other cases were wound infections not related to seafood. Connecticut rarely sees Vibrio cases. | | Illness | 3 | Vibriosis | Connecticut | Jul-Sep | Three people in Connecticut were affected by vibriosis. All three cases resulted in hospitalization. | Two cases were wound infections not related to seafood. The third case involved consuming raw oysters from out-of-state. Vibriosis symptoms range from mild to severe, with serious cases potentially leading to death. | | Illness | Unknown | Vibriosis | United States (annual) | Ongoing | About 80,000 illnesses and 100 deaths annually in the U.S. due to vibriosis, with around 52,000 cases from contaminated food. | Vibriosis is a common illness caused by vibrio vulnificus bacteria. Infections primarily occur between May and October in warmer waters. Vulnerable individuals are at higher risk of severe outcomes. | | Illness | Multiple | Vibriosis | Eastern U.S. | Since 1990s | Vibriosis infections have been increasing in the eastern U.S. due to climate change and warming waters. | Changing environmental conditions contribute to the rise in vibriosis cases. | | Death | 1 | Vibriosis | Missouri | June | A 54-year-old man in Missouri died from vibriosis after consuming raw oysters with vibrio vulnificus bacteria. | Consumption of contaminated raw oysters led to the fatal vibriosis infection. | | Death | 2 | Vibriosis | Florida (Louisiana) | Unknown | Two people in Florida died from vibriosis after eating raw oysters harvested from Louisiana. | Consuming raw oysters contaminated with vibrio vulnificus bacteria resulted in fatal infections. | Please note that some information, such as specific dates and details, is missing or unspecified in the article. The table is based on the available information from the provided article.
How about this article about dengue?
Again, we'll use our prompt with temperature 0.2 with Bison:
Make a table that extracts all of the illnesses and deaths described in the following article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location, Date, Details and Explanation. Only include illnesses and deaths that specify a specific disease and include the number affected, even if it is an estimate.
Unfortunately it yields only the following:
Illness: Dengue Number Affected: 78,000 Disease Name: Dengue Location: Bangladesh Date: January - August 2023 Details: More than 78,000 people have been infected since January this year Explanation: Dengue causes flu-like symptoms that most people recover from, but also can rapidly lead to internal bleeding, organ failure and even death. There generally is no specific treatment for the disease, but vaccines are available. Illness: Dengue Number Affected: 364 Disease Name: Dengue Location: Bangladesh Date: 2023 Details: The government’s Directorate General of Health Services reports the 2023 death toll from the mosquito-borne illness has reached 364 in the south Asian country. Explanation: Dengue causes flu-like symptoms that most people recover from, but also can rapidly lead to internal bleeding, organ failure and even death. There generally is no specific treatment for the disease, but vaccines are available.
Running a second time yields no better results:
Illness: Dengue Number Affected: 78,000 Disease Name: Dengue Location: Bangladesh Date: January - August 2023 Details: More than 78,000 people have been infected since January this year Explanation: Dengue causes flu-like symptoms that most people recover from, but also can rapidly lead to internal bleeding, organ failure and even death. There generally is no specific treatment for the disease, but vaccines are available. Illness: Dengue Number Affected: 364 Disease Name: Dengue Location: Bangladesh Date: 2023 Details: The government’s Directorate General of Health Services reports the 2023 death toll from the mosquito-borne illness has reached 364 in the south Asian country. Explanation: Dengue causes flu-like symptoms that most people recover from, but also can rapidly lead to internal bleeding, organ failure and even death. There generally is no specific treatment for the disease, but vaccines are available.
What about Claude? While slightly better, it is still missing several of the counts expressed in the article:
Here is the table summarizing the illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 78,000+ | Dengue | Bangladesh | Jan - Aug 2023 | Over 78,000 dengue cases reported since January | Outbreak driven by monsoon rains | | Death | 364+ | Dengue | Bangladesh | Jan - Aug 2023 | Over 364 dengue deaths reported since January | Many patients have co-morbidities that increase risk | | Illness | 23,000+ | Dengue | Bangladesh | Aug 1-10, 2023 | Over 23,000 dengue cases in first 10 days of August | Rate much higher than July | | Death | 100+ | Dengue | Bangladesh | Aug 1-10, 2023 | Over 100 dengue deaths in first 10 days of August | Rate much higher than July |
Running a second time doesn't improve the results and in fact drops the 100 deaths:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 78,000+ | Dengue | Bangladesh | January - August 2023 | Over 78,000 infections since January | Mosquito-borne viral disease outbreak | | Death | 364+ | Dengue | Bangladesh | January - August 10, 2023 | Over 364 deaths since January, over 100 in August | High mortality rate likely due to overwhelmed hospitals | | Illness | 23,000+ | Dengue | Bangladesh | August 1-10, 2023 | Over 23,000 cases in first 10 days of August | Shows sharp increase in cases this month | | Death | 281 | Dengue | Bangladesh | 2022 | 281 dengue deaths last year | Comparison to show dramatic increase in 2023 |
Running a third time does expand our results but still misses some counts:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 78,000+ | Dengue | Bangladesh | Jan - Aug 2023 | Over 78,000 infected since January | Mosquito-borne illness outbreak | | Death | 364+ | Dengue | Bangladesh | Jan - Aug 2023 | Over 364 deaths since January | Mosquito-borne illness outbreak | | Illness | 23,000+ | Dengue | Bangladesh | Aug 1-10, 2023 | Over 23,000 cases in first 10 days of Aug | Worse than July total of 43,854 | | Death | 100+ | Dengue | Bangladesh | Aug 1-10, 2023 | Over 100 deaths in first 10 days of Aug | Over a third of 281 total deaths in 2022 | | Illness | 62,382 | Dengue | Bangladesh | 2022 | 62,382 infected in 2022 | Record high cases | | Death | 281 | Dengue | Bangladesh | 2022 | 281 deaths in 2022 | Record high deaths |
What about ChatGPT? It yields the most comprehensive results of the group, but still has a number of problems. It misses the 43,854 cases reported in July and the 62,382 cases in 2022, though it cases the second half of that sentence to record 281 deaths. Interestingly, while Claude captured the 62,382 cases, ChatGPT misses them:
Sure, here's a table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details and Explanation | |----------|-----------------|--------------|-------------------|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 78,000+ | Dengue | Bangladesh | 2023 | Dengue cases have been reported in Bangladesh, with over 78,000 people infected since January. Dengue causes flu-like symptoms and can lead to severe complications including internal bleeding, organ failure, and death. | | Death | 364 | Dengue | Bangladesh | 2023 | The death toll from dengue in Bangladesh has reached 364 this year. Dengue can lead to fatal complications such as internal bleeding and organ failure. | | Illness | 23,000+ | Dengue | Bangladesh | August 2023 | In the first 10 days of August, more than 23,000 people were diagnosed with dengue, indicating a rapid spread of the disease. | | Death | 100+ | Dengue | Bangladesh | August 2023 | Over 100 people have died from dengue in August, raising concerns about the severity of the outbreak. | | Death | 281 | Dengue | Bangladesh | 2022 | In 2022, 281 people died of dengue in Bangladesh, highlighting the impact of the disease on public health. | | Illness | Unknown | Dengue | Cyprus | 2023 | Cyprus has also seen dengue outbreaks this year, although the exact number of affected individuals is not specified. | | Illness | Unknown | Dengue | Bolivia | 2023 | Bolivia has experienced dengue outbreaks in 2023, but the article doesn't provide specific figures for the number of affected individuals. | Please note that some information, such as specific dates and exact numbers of affected individuals, may not be provided in the article. Also, the article mentions that dengue can cause various symptoms and complications, which are not fully detailed in the table.
Running a second time the model becomes oversensitive and extracts many unrelated counts and also conflates "Cyprus, Bolivia" (two different countries) with "Dhaka, Bangladesh". It still misses several counts.
Here's a table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|------------------|---------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| | Death | 364 | Dengue | Bangladesh | 2023 | Death toll from mosquito-borne illness, 364 deaths reported. | Dengue causes flu-like symptoms and can lead to internal bleeding, organ failure, and death. | | Illness | 78,000 (est.) | Dengue | Bangladesh | January 2023 | Over 78,000 people infected with dengue since January 2023. | Dengue causes flu-like symptoms. | | Illness | 23,000 (est.) | Dengue | Bangladesh | August 2023 | Over 23,000 diagnosed with dengue in the first 10 days of August. | Dengue causes flu-like symptoms. | | Death | 100 (est.) | Dengue | Bangladesh | August 2023 | Over 100 deaths reported in August, more than a third of total deaths in the previous year. | Dengue causes flu-like symptoms and can lead to internal bleeding, organ failure, and death. | | Death | 281 | Dengue | Bangladesh | 2022 | In 2022, 281 deaths and 62,382 infections reported from dengue. | Dengue causes flu-like symptoms and can lead to internal bleeding, organ failure, and death. | | Illness | - | Dengue | Cyprus, Bolivia | - | Other countries, like Cyprus and Bolivia, also experienced dengue outbreaks. | Dengue causes flu-like symptoms. | | Illness | - | Unknown | Bangladesh | - | Infected people overwhelming hospitals, some doctors/nurses lack experience in treating dengue. | Lack of expertise in treating dengue cases in some medical personnel. | | Death | - | Unknown | Bangladesh | - | Critical patients with other ailments struggling to receive support, leading to deaths. | Lack of critical support for patients with other ailments. | | Illness | - | Unknown | Bangladesh | - | People with immune suppression or co-morbidities (like cancer, diabetes) are more vulnerable to dengue. | Immune-suppressed individuals at higher risk for severe dengue. | | Illness | - | Unknown | Bangladesh | - | Authorities arranging extra beds and IV saline in state-run hospitals to address the outbreak. | Overwhelming demand for medical resources due to dengue outbreak. | | Illness | - | Unknown | Bangladesh | - | Ministry of Health urging collective effort for dengue prevention. | Emphasizing the need for individual and collective actions to prevent dengue. | | Illness | - | Unknown | Dhaka, Bangladesh| - | Anti-mosquito drive and awareness campaigns initiated by city corporations. | Efforts to control mosquito breeding and raise public awareness. | Please note that some information is not provided explicitly in the article (e.g., specific disease names for "Unknown" illnesses), and estimates are marked as "(est.)". The table provides a summary based on the available information in the article.
And a third time it still misses several counts:
Here's a table summarizing the illnesses and deaths described in the provided article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|-------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| | Death | 364 | Dengue | Bangladesh | 2023 | Death toll from mosquito-borne illness reached 364; more than 100 deaths in August; warning of potential record-breaking deaths and infections this month. | Dengue causes flu-like symptoms, can lead to internal bleeding, organ failure, and death; vulnerable individuals at higher risk. | | Illness | 78,000+ | Dengue | Bangladesh | January 2023 | Over 78,000 people infected with dengue since January 2023. | Dengue causes flu-like symptoms, some cases can be severe. | | Illness | 23,000+ | Dengue | Bangladesh | August 2023 | Over 23,000 diagnosed with dengue in first 10 days of August. | Outbreak exacerbated by monsoon season and intermittent rains. | | Death | >100 | Dengue | Bangladesh | August 2023 | Over 100 deaths from dengue in August; more deaths than the entire previous year. | Higher death toll due to severe cases and potentially virulent dengue variant. | | Death | 281 | Dengue | Bangladesh | 2022 | In 2022, 281 people died of dengue; 62,382 people infected. | Dengue caused fatalities and infections in the previous year. | Please note that the "Explanation" column includes general information about dengue and its effects based on the article's context. For specific details about each case, you should refer to the original article.
And a fungal outbreak with the same prompt and temperature in Bison:
Illness: Suspected cases of blastomycosis Number Affected: 19 confirmed cases, 74 probable cases Disease Name: Blastomycosis Location: Escanaba, Michigan Date: March 2023 Details: Employees of the Escanaba Billerud Paper Mill appeared to have "atypical" pneumonia cases. Testing revealed that the cases were not pneumonia but a fungal infection called blastomycosis. Explanation: Blastomycosis is caused by a fungus that primarily lives in moist soil and decomposing organic matter, such as wood, and it spreads through the inhalation of spores. Symptoms can include fever and cough, and people with more weakened immune systems may have more severe cases. The infection can spread to the lungs and other organs.
Claude yields similar results, though it incorrectly dates it as March 2022. It does use the paper mill as the location, which is more accurate than the city-level results returned by Bison:
Here is the table with illnesses and deaths extracted from the article: |Type|Number Affected|Disease Name|Location|Date|Details|Explanation| |-|-|-|-|-|-|-| |Illness|19 confirmed, 74 probable|Blastomycosis|Escanaba Billerud Paper Mill, Michigan|March 2022 - present|Fungal infection linked to paper mill. Causes fever, cough. Can spread to lungs, other organs.|Caused by fungus found in soil and wood. Likely spread through spores at the mill.|
Running a second time doesn't change the results much, although this time it gets the year right:
Here is the table summarizing the illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 19 confirmed, 74 probable cases | Blastomycosis | Escanaba Billerud Paper Mill, Michigan | Investigation began March 2023 | Fungal infection affecting lungs and other organs, symptoms include fever and cough | Caused by Blastomyces fungus found in soil and decomposing matter, spreads through inhalation of spores |
What about ChatGPT? Here overcounts, reporting both the 15 cases that started the investigation with the 19 and 74 counts:
I've extracted the relevant information from the article to create a table as requested: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|-------------------|-----------------------------------|-------------------|--------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 15 | Blastomycosis | Escanaba Billerud Paper Mill | Early in the year | Initially appeared as "atypical" pneumonia cases, later identified as blastomycosis infection. | Blastomycosis is a fungal infection caused by inhaling spores of the Blastomyces fungus. It primarily lives in moist soil and decomposing organic matter. Symptoms include fever and cough, with more severe cases in weakened immune systems. | | Illness | 19 (confirmed) | Blastomycosis | Escanaba Billerud Paper Mill | As of Friday | Confirmed cases of blastomycosis at the paper mill. | Blastomycosis is a fungal infection caused by inhaling spores of the Blastomyces fungus. It primarily lives in moist soil and decomposing organic matter. Symptoms include fever and cough, with more severe cases in weakened immune systems. | | Illness | 74 (probable) | Blastomycosis | Escanaba Billerud Paper Mill | As of last week | Probable cases of blastomycosis at the paper mill. | Blastomycosis is a fungal infection caused by inhaling spores of the Blastomyces fungus. It primarily lives in moist soil and decomposing organic matter. Symptoms include fever and cough, with more severe cases in weakened immune systems. | | Death | N/A | N/A | N/A | N/A | N/A | The article does not mention any deaths related to the fungal infection at the paper mill. | Please note that the article does not specify any deaths related to the fungal infection at the paper mill, so the "Death" section in the table is left as "N/A."
A second time yields similar results:
I've extracted the relevant information from the article to create a table listing the illnesses and deaths: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------------|---------------------------------------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 15 (initially) | Blastomycosis | Escanaba Billerud Paper Mill, Michigan | March | Employees appeared sick, later confirmed as blastomycosis fungal infection. | | Illness | At least 19 | Blastomycosis | Escanaba Billerud Paper Mill, Michigan | Ongoing | Confirmed cases at the paper mill. | | Illness | 74 (probable) | Blastomycosis | Escanaba Billerud Paper Mill, Michigan | Ongoing | Probable cases at the paper mill. | | Death | N/A | N/A | N/A | N/A | N/A | No specific deaths mentioned in the article. | Note: The article does not specify any deaths or provide detailed information about individual cases beyond the mentioned numbers. The focus of the article is primarily on the investigation into the fungal infection outbreak at the paper mill.
A third time continues to improperly double count:
I'll summarize the information from the article and create a table as per your request: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|------------------|-------------------------------|----------------|-------------------------------------------------------|-------------------------------------------------------------------------------------------------------| | Illness | 15 confirmed | Blastomycosis | Escanaba Billerud Paper Mill | March | Employees showed symptoms of "atypical" pneumonia, later diagnosed as blastomycosis fungal infection. | Blastomycosis is caused by inhaling fungal spores found in moist soil and organic matter. | | Illness | 19 confirmed | Blastomycosis | Escanaba Billerud Paper Mill | Ongoing | Confirmed cases of blastomycosis among employees. | Blastomycosis is a fungal infection affecting the lungs and other organs. | | Illness | 74 probable | Blastomycosis | Escanaba Billerud Paper Mill | Ongoing | Additional probable cases of blastomycosis. | Probable cases refer to individuals with symptoms consistent with blastomycosis. | Please note that the article does not provide specific details about individual deaths associated with this outbreak. If you require further information, it's recommended to refer to the original source or official health reports for more accurate and up-to-date information.
What about a measles outbreak?
This time Bison yields reasonable results:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Illness | 2,056 | Measles | Zimbabwe | August 15, 2019 | Most victims were not vaccinated. | The measles virus attacks mainly children with the most serious complications including blindness, brain swelling, diarrhea and severe respiratory infections. | | Death | 157 | Measles | Zimbabwe | August 15, 2019 | Most victims were not vaccinated. | The measles virus attacks mainly children with the most serious complications including blindness, brain swelling, diarrhea and severe respiratory infections. |
Claude fixates on the 2000+ estimate and fails to recognize that the precise number later in the text:
Here is the table with the requested information from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 2000+ | Measles | Zimbabwe | August 2023 | Over 2000 measles infections reported across Zimbabwe | Measles outbreak across Zimbabwe, cases growing rapidly since first infection logged earlier in August | | Death | 157 | Measles | Zimbabwe | August 15, 2022 | 157 children killed by measles outbreak | Deaths from measles outbreak in Zimbabwe, almost doubled in less than a week |
Running a second and third time doesn't help:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 2000+ | Measles | Zimbabwe | August 2023 | Cases growing rapidly, 157 deaths | Outbreak blamed on lack of vaccination, affecting mainly unvaccinated children | | Death | 157 | Measles | Zimbabwe | August 2023 | Almost doubled in less than a week | Most victims were unvaccinated |
ChatGPT correctly extracts the counts:
Sure, here's the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|----------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------| | Death | 157 | Measles | Zimbabwe | August 15, 2023 | Measles outbreak in Zimbabwe resulted in 157 deaths and 2,056 infections. Deaths doubled in less than a week. | Measles virus causing serious complications including blindness, brain swelling, diarrhea, and respiratory infections. | | Illness | 2,056 | Measles | Zimbabwe | August 15, 2023 | Measles outbreak in Zimbabwe resulted in 157 deaths and 2,056 infections. Deaths doubled in less than a week. | Measles virus causing symptoms like a red rash, starting on the face and spreading, with serious complications possible. | Please note that the information is based on the provided article and includes the specific details mentioned in the text.
Second time:
Here's the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-----------|-----------------|--------------|--------------|------------|---------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 2,056 cases | Measles | Zimbabwe | August 15 | Measles outbreak with 2,056 reported cases and 157 deaths in Zimbabwe. Cases rapidly growing since earlier this month. | Measles virus outbreak causing severe complications, mainly affecting children, including blindness, brain swelling, and respiratory infections. | | Death | 157 deaths | Measles | Zimbabwe | August 15 | Measles outbreak with 2,056 reported cases and 157 deaths in Zimbabwe. Cases rapidly growing since earlier this month. | Measles virus outbreak causing severe complications, mainly affecting children, including blindness, brain swelling, and respiratory infections. | Please note that the table only includes specific illnesses and deaths mentioned in the article with corresponding details and explanations. It excludes general information or references to preventable diseases in Africa mentioned in the article.
Running a third time it loses focus and incorrectly flags all sentences containing medical terms:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|---------------|------------|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 157 | Measles | Zimbabwe | August 15 | 157 deaths reported | Measles outbreak in Zimbabwe with reported deaths and infections. | | Illness | 2,056 | Measles | Zimbabwe | August 15 | 2,056 infections reported | Rapidly growing measles cases in Zimbabwe with over 2,000 infections reported. | | Illness | - | Measles | Zimbabwe | August 15 | Cases growing rapidly | Measles cases growing rapidly in Zimbabwe since the first infection was reported earlier this month. | | Illness | - | Measles | Zimbabwe | August 15 | Reported deaths almost doubled | The reported deaths due to measles almost doubled in less than a week in Zimbabwe. | | Illness | - | Measles | Zimbabwe | August 15 | Most victims were not vaccinated | Most victims of the outbreak were not vaccinated against measles. | | Illness | - | Measles | Zimbabwe | August 15 | Blamed on church sect gatherings | The health ministry blamed the measles outbreak on church sect gatherings. | | Illness | - | Measles | Zimbabwe | August 15 | Severe complications and symptoms | Measles virus can cause serious complications like blindness, brain swelling, diarrhea, and severe respiratory infections. | | Illness | - | Measles | Zimbabwe | August 15 | Symptoms include red rash | Measles symptoms include a red rash that starts on the face and spreads to the body. | | Illness | - | Measles | Zimbabwe | August 15 | Preventable with a vaccine | Measles is now preventable with a vaccine. | | Illness | - | Measles | Africa | April | Measles cases jumped 400% | The World Health Organization (WHO) stated that Africa faced an explosion of preventable diseases due to delays in vaccinating children, with measles cases jumping 400%. | Please note that some information is missing or not specified in the provided article, such as the exact dates of the initial infection and the number of affected individuals for some points. These missing details have been marked with dashes (-).
What about a particularly complex Monkeypox article that includes age range breakdowns?
Bison fixates on the top-line number of 1,266 infections and leaves it at that:
Illness: Monkeypox Number Affected: 1,266 Disease Name: Monkeypox Location: Florida Date: August 16, 2022 Details: Most cases occur in men who have sex with men, but anyone can contract the virus. The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills.
Claude includes a few breakdowns:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 1,266 | Monkeypox | Florida | August 15, 2022 | - 493 cases in Miami-Dade <br>- 436 cases in Broward County <br>- 28 counties have at least 1 case | Cases have risen in Florida, with more teens and young adults infected recently. Largest numbers still in South Florida. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 15, 2022 | Ages 15-19 | New cases in teens as schools start. | | Illness | 64 | Monkeypox | 14 counties including Broward, Miami-Dade | August 15, 2022 | Ages 20-24 | Cases up from 43 last week. | | Illness | 159 | Monkeypox | 19 counties including Broward, Miami-Dade | August 15, 2022 | Ages 25-29 | Cases up from 124 last week. | | Illness | 1 | Monkeypox | Martin | August 15, 2022 | Ages 0-4 | One case in young child. |
ChatGPT captures all of the counts, but fails to see them in terms of age breakdowns, resulting in overcounting:
Here's the table summarizing the illnesses and deaths described in the provided article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|--------------|-----------------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | Ongoing | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected in several counties in the outbreak. | Monkeypox is a rare viral disease that causes a characteristic rash and various symptoms, including muscle aches, respiratory symptoms, and chills. It is not limited to a specific demographic, although certain groups may be more susceptible. | | Illness | 493 | Monkeypox | Miami-Dade County, Florida | Ongoing | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade County. | Monkeypox cases are concentrated in South Florida, particularly in Miami-Dade County. | | Illness | 436 | Monkeypox | Broward County, Florida | Ongoing | There are 436 cases in Broward County. | Broward County has a significant number of monkeypox cases. | | Illness | Various | Monkeypox | 28 counties in Florida | Ongoing | In total, 28 counties in Florida have reported one or more monkeypox cases. | The outbreak has spread to multiple counties in Florida. | | Illness | 8 | Monkeypox | Various counties in Florida | Ongoing | There are eight monkeypox cases in the age group of 15 to 19, in Broward, Lee, Leon, and Miami-Dade. | Monkeypox cases have been reported among teenagers and young adults in multiple counties. | | Illness | 64 | Monkeypox | 14 counties in Florida | Ongoing | In the age range of 20 to 24, there are 64 monkeypox cases in 14 counties: Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, and Seminole. | Cases of monkeypox have increased among individuals aged 20 to 24 in various counties. | | Illness | 159 | Monkeypox | 19 counties in Florida | Ongoing | In the age range of 25 to 29, there are 159 monkeypox cases in 19 counties: Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, and more. | Cases of monkeypox have increased among individuals aged 25 to 29 in numerous counties. | | Illness | 1 | Monkeypox | Martin County, Florida | Ongoing | There is one case in Martin County, in the 0 to 4 age range. | A single monkeypox case has been reported in a very young child in Martin County. | | Illness | Various | Monkeypox | Florida | Aug 4 | The federal government declared a public health emergency over monkeypox in Florida on August 4. | The federal government recognized the severity of the outbreak and declared a public health emergency. | | Illness | Various | Monkeypox | Global | July | The World Health Organization declared monkeypox a public health emergency of international concern in July. | The global health community acknowledged the international significance of the monkeypox outbreak and declared it a public health emergency. | | Illness | Various | Monkeypox | Various states including NY, CA, IL | Ongoing | Some states like New York, California, and Illinois declared a state of emergency due to the rise in monkeypox cases. | Certain states recognized the seriousness of the outbreak and declared states of emergency. | Please note that the provided article does not mention any deaths or specific numbers related to deaths. Therefore, the table only includes information related to illnesses, specifically monkeypox cases.
Running a second time yields similar results, but drops Florida from the location field:
Here's a table that extracts the relevant information from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|--------------|---------------------------|-----------|-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| | Illness | 1,266 | Monkeypox | Florida | - | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected... | Outbreak of monkeypox in Florida, affecting multiple counties. | | Illness | 493 | Monkeypox | Miami-Dade County | - | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade County. | High concentration of monkeypox cases in Miami-Dade County. | | Illness | 436 | Monkeypox | Broward County | - | There are 436 cases in Broward County. | High concentration of monkeypox cases in Broward County. | | Illness | Varies | Monkeypox | 28 counties | As of Monday | In all, 28 counties have one or more monkeypox cases, as of Monday. | Monkeypox cases reported in 28 counties across Florida. | | Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | - | There are now eight monkeypox cases in the age group of 15 to 19, in Broward, Lee, Leon and Miami-Dade. | Monkeypox cases reported in specific age group and counties. | | Illness | 64 | Monkeypox | 14 counties | - | In the 20 to 24 age range, there are now 64 monkeypox cases, up from 43 cases just last week... | Increase in monkeypox cases in the 20 to 24 age range across 14 counties. | | Illness | 159 | Monkeypox | 19 counties | - | In the 25 to 29 age range, there are 159 monkeypox cases, up from 124 cases last week... | Increase in monkeypox cases in the 25 to 29 age range across 19 counties. | | Illness | 1 | Monkeypox | Martin County | - | There is one case in Martin County, in the 0 to 4 age range. | Single monkeypox case reported in the 0 to 4 age range in Martin County. | | Illness | - | Monkeypox | - | Aug. 4 | The federal government on Aug. 4 declared a public health emergency over monkeypox... | Declaration of a public health emergency by the federal government due to monkeypox. | | Illness | - | Monkeypox | - | July | In July, the World Health Organization declared it a public health emergency of international concern. | Global public health emergency declaration by the World Health Organization due to monkeypox. | | Illness | - | Monkeypox | New York, California, Illinois | - | Other states such as New York, California, and Illinois had declared a state of emergency due to the rise in cases. | Declaration of state of emergency in multiple states due to the rise in monkeypox cases. | Please note that some details might be inferred from the text, and the table might not capture nuances or changes in the situation beyond the provided article.
What about Salmonella with country-level breakdowns? (Using only the summary and executive summary text, not the full report).
Bison takes the easy way out and reports only the top-line number and a list of countries:
Illness: 92 cases of Salmonella Senftenberg Disease Name: Salmonella Senftenberg Location: Austria, Belgium, Czechia, Estonia, Finland, France, Germany, Ireland, the Netherlands, Norway, Sweden, the United Kingdom, and the United States Date: August 2022 - July 2023 Details: Most cases were reported between October 2022 and March 2023, with a decline in the number of countries with exposures after December. In Austria, Germany, and France, cherry-like tomatoes were identified as the food exposure most reported by interviewed cases. Explanation: The outbreak strain was detected in France from a mixed salad dish, containing cherry tomatoes and green leafy vegetables, prepared on 17 August 2022 but not served. Tomatoes from the salad in France and tomatoes in Austria were suspected as the vehicle of infections by national authorities and were traced back to wholesalers in Germany, the Netherlands and Spain, and to growers in the Netherlands, Spain and Morocco. In the absence of microbiological evidence from the tomatoes, the source of the infections could not be established. Death: 1 patient has died of the infection.
Claude does the same:
Here is the table summarizing the illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Illness | 92 | Salmonella Senftenberg | Austria, Belgium, Czechia, Estonia, Finland, France, Germany, Ireland, Netherlands, Norway, Sweden, UK, US | August 2022 - June 2023 | 69.6% cases were female. Cherries tomatoes were a suspected source. | Outbreak linked to contaminated tomatoes originating from farms in Netherlands, Spain and Morocco. Cases reported over 10 months across 13 countries. Risk has decreased. | | Death | 1 | Salmonella Senftenberg | Not specified | Not specified | One patient died | Part of the Salmonella outbreak linked to contaminated tomatoes. |
ChatGPT does the same for its first output, but unlike the others incorrectly locates all the illnesses in Austria:
Here's a table summarizing the illnesses and death described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|----------------------|--------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 92 | Salmonella Senftenberg | Austria | Aug 2022 - Jul 2023 | 92 cases reported in Austria (5), Belgium (4), Czechia (4), Estonia (1), Finland (12), France (16), Germany (26), Ireland (1), the Netherlands (5), Norway (1), Sweden (11), the UK (4), and the US (2). | Outbreak strain detected in France from a mixed salad dish containing cherry tomatoes and green leafy vegetables, prepared on 17 Aug 2022. Tomatoes suspected as the vehicle of infections traced back to wholesalers and growers. Common source(s) causing a cross-border outbreak reported in 11 EU/EEA countries, the UK, and the US. Risk of new infections decreased since Dec 2022. | | Death | 1 | Salmonella Senftenberg | Unknown | Unknown | One patient died of the infection. | Not specified in the article. | Please note that the article doesn't provide a detailed explanation for the death, including the specific location or date of the death. Additionally, while the article discusses the outbreak of Salmonella Senftenberg, it doesn't provide extensive details about the nature of the illness itself.
Its second run provides country-level breakdowns, but incorrectly ascribes the top-level count to Austria again. Belgium and Czechia get "various" instead of the counts listed.
Here is the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|---------------------|---------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| | Illness | 92 | Salmonella Senftenberg | Austria | Aug 2022 - Jul 2023 | 92 cases reported. 69.6% female cases. Cherry-like tomatoes identified as common food exposure in Austria. | Outbreak strain detected in France from mixed salad dish. Tomatoes traced to wholesalers in Germany, Netherlands, and Spain, and to growers in Netherlands, Spain, and Morocco. Source of infections not established. | | Death | 1 | Salmonella Senftenberg | Not specified | Not specified | 1 patient died of the infection. | - | | Illness | Various | Salmonella Senftenberg | Belgium | Aug 2022 - Jul 2023 | 4 cases reported. | - | | Illness | Various | Salmonella Senftenberg | Czechia | Aug 2022 - Jul 2023 | 4 cases reported. | - | | Illness | 1 | Salmonella Senftenberg | Estonia | Aug 2022 - Jul 2023 | 1 case reported. | - | | Illness | 12 | Salmonella Senftenberg | Finland | Aug 2022 - Jul 2023 | 12 cases reported. | - | | Illness | 16 | Salmonella Senftenberg | France | Aug 2022 - Jul 2023 | 16 cases reported. First case reported on 22 Aug 2022. Outbreak strain detected from mixed salad dish. | Genetic similarity of strains suggests common source for 10-month outbreak. | | Illness | 26 | Salmonella Senftenberg | Germany | Aug 2022 - Jul 2023 | 26 cases reported. Cherry-like tomatoes identified as common food exposure. | - | | Illness | 1 | Salmonella Senftenberg | Ireland | Aug 2022 - Jul 2023 | 1 case reported. | - | | Illness | 5 | Salmonella Senftenberg | Netherlands | Aug 2022 - Jul 2023 | 5 cases reported. Cherry-like tomatoes identified as common food exposure. | Tomatoes traced to wholesalers in Germany, Netherlands, and Spain, and to growers in Netherlands, Spain, and Morocco. | | Illness | 1 | Salmonella Senftenberg | Norway | Aug 2022 - Jul 2023 | 1 case reported. | - | | Illness | 11 | Salmonella Senftenberg | Sweden | Aug 2022 - Jul 2023 | 11 cases reported. Most recent case reported on 24 Jun 2023. | Most cases reported between Oct 2022 and Mar 2023. Cases intermittently reported in multiple countries. | | Illness | 4 | Salmonella Senftenberg | United Kingdom | Aug 2022 - Jul 2023 | 4 cases reported. | - | | Illness | 2 | Salmonella Senftenberg | United States | Aug 2022 - Jul 2023 | 2 cases reported. | - | Please note that the "Explanation" column provides a brief summary of the details and context for each entry.
In its third run it once again reverts to just the top-level count:
Here's a table summarizing the illnesses and death described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|----------------------|------------------|-------------------|------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | Illness | 92 | Salmonella Senftenberg | Austria | August 2022 - July 2023 | Outbreak cases reported in multiple countries, with highest numbers in Germany and France. | Cross-border foodborne outbreak with likely source from contaminated tomatoes, possibly originating from farms. | | Death | 1 | Salmonella Senftenberg | Unspecified Country | Unspecified Date | One patient died of Salmonella Senftenberg infection. | A fatal outcome from the outbreak, with no specific details provided. | Please note that the article did not provide detailed information on each individual case, and the table is based on the information provided. Additionally, some details such as "Details" and "Explanation" might be general due to the lack of specific information in the article.
What about Powassan?
Bison reports only the death despite running multiple times:
| Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---|---|---|---|---|---|---| | Death | 1 | Powassan virus disease | Rhode Island | Mid-July | Female over the age of 80 from Washington County developed neurological symptoms and died. | Laboratory testing confirmed the finding earlier this month. |
Claude's first run is actually perfect:
Here is the table extracting illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Death | 1 | Powassan virus | Washington County, RI | mid-July 2023 | Female over 80 developed neurological symptoms and died | Tick-borne disease found in Northeast and Great Lakes region of US and Canada. 239 cases reported in US in past 10 years. | | Illness | 239 | Powassan virus | United States | 2013-2022 | | Number of Powassan cases reported in US in past 10 years | | Illness | 49 | Powassan virus | Massachusetts | 2013-2022 | | Number of Powassan cases reported in New England in past 10 years | | Illness | 18 | Powassan virus | Connecticut | 2013-2022 | | Number of Powassan cases reported in New England in past 10 years | | Illness | 16 | Powassan virus | Maine | 2013-2022 | | Number of Powassan cases reported in New England in past 10 years | | Illness | 5 | Powassan virus | New Hampshire | 2013-2022 | | Number of Powassan cases reported in New England in past 10 years | | Illness | 5 | Powassan virus | Rhode Island | 2013-2022 | | Number of Powassan cases reported in New England in past 10 years |
A second run exhibits the same behavior as Bison:
Here is the table with the requested information extracted from the article: |Type|Number Affected|Disease Name|Location|Date|Details|Explanation| |-|-|-|-|-|-|-| |Death|1|Powassan virus disease|Washington County, RI|Mid-July 2023|Female over 80 developed neurological symptoms and died|Tick-borne disease found in northeastern US and Canada. 239 cases in US in past 10 years. Cases increasing in recent years. |
A third run repeats its initial results:
Here is the table with illnesses and deaths from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Death | 1 | Powassan virus | Washington County, RI | Mid-July 2023 | Female over 80 developed neurological symptoms and died | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 239 | Powassan virus | United States | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 49 | Powassan virus | Massachusetts | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 18 | Powassan virus | Connecticut | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 16 | Powassan virus | Maine | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 5 | Powassan virus | New Hampshire | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 5 | Powassan virus | Rhode Island | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years |
A fourth run forgoes the state-by-state breakdown with the top-level New England count:
Here is the table with the requested information extracted from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-|-|-|-|-|-|-| | Death | 1 | Powassan virus | Washington County, RI | mid-July 2023 | Female over 80 developed neurological symptoms and died | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 239 | Powassan virus | United States | 2013-2022 | | Rare tick-borne disease, number of cases increasing in recent years | | Illness | 93 | Powassan virus | New England | 2013-2022 | 49 cases in MA, 18 in CT, 16 in ME, 5 in NH, 5 in RI | Rare tick-borne disease, number of cases increasing in recent years |
ChatGPT's first run mimics Claude's fourth run:
Sure, here's the information extracted from the article and organized into a table: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|---------------------|------------------|-------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Powassan virus | Rhode Island | Mid-July | A female over the age of 80 from Washington County developed neurological symptoms and died after contracting the Powassan virus disease. | Powassan is a tick-borne disease found in the Northeast and Great Lakes region of the U.S. and eastern Canada. Laboratory testing by CDC confirmed the Powassan virus as the cause of death. | | Illness | 239 | Powassan virus | United States | 2013-2022 | Over the past 10 years, there have been 239 reported cases of Powassan virus disease in the United States. | Powassan virus disease is a rare tick-borne illness primarily found in the Northeast and Great Lakes region of the U.S. The reported number of cases has increased in recent years. | | Illness | 93 | Powassan virus | New England | 2013-2022 | Between 2013 and 2022, there were 93 cases of Powassan virus disease reported in New England: 49 cases in Massachusetts, 18 cases in Connecticut, 16 cases in Maine, five cases in New Hampshire, and five cases in Rhode Island. | Powassan virus disease is rare but has been reported in various states within New England, including Massachusetts, Connecticut, Maine, New Hampshire, and Rhode Island. | Please note that the information provided is based on the given article and may not be up-to-date with the most recent data or developments.
Its second run uses the top-level count and then tries to parse the state-level counts, but fails:
Here's the table extracting the relevant information from the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |-----------|-----------------|-----------------|----------------|-----------------|------------------------------------------------------------------|-----------------------------------------------------------------| | Death | 1 | Powassan virus | Rhode Island | Mid-July | A female over the age of 80 from Washington County | Contracted Powassan disease and died | | Illness | 239 | Powassan virus | United States | 2013-2022 | Cases reported in the Northeast, Great Lakes, and eastern Canada| Tick-borne disease with increasing reported cases | | Illness | 93 | Powassan virus | New England | 2013-2022 | 49 cases in Massachusetts, 18 cases in Connecticut, | Cases reported in New England states | | | | | | | 16 cases in Maine, 5 cases in New Hampshire, | | | | | | | | 5 cases in Rhode Island | | Please note that the article provides specific information about only one death and one illness case related to the Powassan virus disease. The other numbers mentioned are estimates of reported cases for specific time periods and regions.
Its third run misses almost all of the cases:
Here's the table summarizing the illnesses and deaths described in the article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |---------|-----------------|-----------------------|----------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Illness | 1 | Powassan virus disease| Rhode Island | Mid-July | A female over the age of 80 from Washington County, Rhode Island, contracted the tick-borne Powassan virus disease. She developed neurological symptoms and died in mid-July. | Powassan virus disease is a tick-borne illness found in the Northeast and the Great Lakes region of the U.S. and in eastern Canada. It can cause neurological symptoms and in severe cases, can be fatal. The disease is transmitted through tick bites, and cases are rare but have been increasing in recent years. | | Death | 1 | Powassan virus disease| Rhode Island | Mid-July | A female over the age of 80 from Washington County, Rhode Island, developed neurological symptoms and died in mid-July after contracting the Powassan virus disease. The disease was confirmed through laboratory testing by CDC. | The individual's death was a result of complications from Powassan virus disease. Powassan virus disease is a tick-borne illness that can lead to severe neurological issues and even death in some cases. Laboratory testing by CDC confirmed the presence of the disease in the deceased individual. This case underscores the potential severity of tick-borne diseases, particularly among vulnerable populations. |
And its fourth run again just reports the top-level count:
Here is the table summarizing the illnesses and deaths described in the provided article: | Type | Number Affected | Disease Name | Location | Date | Details | Explanation | |----------|-----------------|------------------|-------------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Death | 1 | Powassan virus | Washington County | Mid-July | A female over the age of 80 from Washington County contracted Powassan virus disease and developed neurological symptoms. She subsequently died in mid-July. | The individual was infected with Powassan virus, a tick-borne disease found in the Northeast and Great Lakes region of the U.S. The disease can cause neurological symptoms and, in some cases, be fatal. Laboratory testing by the CDC confirmed the presence of the virus. | | Illness | 239 | Powassan virus | Various | 2013-2022 | Over the past 10 years (2013-2022), there have been 239 cases of Powassan virus disease reported in the United States. The disease is mostly found in the Northeast and Great Lakes region, with rare cases reported. | Powassan virus disease is a tick-borne illness that can lead to various neurological symptoms. It has been reported in the U.S. and Canada, with the highest concentration of cases in the Northeast and Great Lakes regions. The number of reported cases has increased in recent years, but the disease remains relatively rare. | | Illness | 93 | Powassan virus | New England | 2013-2022 | Between 2013 and 2022, there were 93 reported cases of Powassan virus disease in New England: 49 cases in Massachusetts, 18 cases in Connecticut, 16 cases in Maine, five cases in New Hampshire, and five cases in Rhode Island. | Powassan virus disease has been reported in New England states, with varying numbers of cases in different states. These cases are part of the larger trend of increased Powassan virus disease reports in recent years. | Please note that the information in the table is extracted from the provided article and is accurate based on the content available.