Personas, Prompt Engineering, Instruction & Multishot Prompting Don’t Guarantee Better Results: Disease Outbreak Codification With Bison, Unicorn, Gemini, GPT 3.5 & GPT 4.0

Yesterday we reexamined LLM-powered global disease outbreak codification, testing how five major commercial LLMs (Google's Bison, Unicorn and Gemini Pro models and OpenAI's GPT 3.5 and GPT 4.0 models) handle extracting illness and death counts from a range of news articles. The results offered a stark reminder that model performance does not scale with model size: the smallest of the five models (Bison) outperformed or matched its much larger and more advanced peers on all three examples, while GPT 4.0 actually performed fairly poorly, despite being significantly more advanced than the others. The most likely explanation is that more advanced models are more sensitive to prompt engineering, instruction and multi-shot learning, while simpler models are better at zero shot prompting. Here we'll test how various prompt modifications improve performance on a disease extraction task.

Using a complex article about Monkeypox that includes a range of numbers in close proximity and places pressure on the parsing and associative capabilities of the models, we'll test how adding additional instruction (asking for a county-level breakdown rather than summary totals), adding a persona (instructing the model that it is a biosurveillance agent extracting information from news articles), instructing it not to use information beyond that contained in the article (including any learned information from its training dataset) and not to hallucinate and finally offering it single-shot and two-shot prompting examples to guide its output affects each model's performance.

While many LLM best practices guides suggest personas as a critical feature for optimal LLM performance, they appear to be a noop in the extractive task examined here, having no measurable impact. Similarly, asking the model not to use external information and not to hallucinate has little impact on the model's actual hallucination and out-of-band information usage. Of especial concern, the ubiquity of one- and multi-shot prompting in best practices guides is seen here to spectacularly backfire. When provided a single-shot example developed from manual review of multiple disease articles, model performance collapses. The reason is that in the manually reviewed articles, the prompt engineer did not encounter an example of multi-location summations of the kind found in the article examined here like "The 64 cases are in 14 counties: Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota and Seminole". The zero-shot prompt allowed the models to adjust their outputs, but the single-shot prompt required it to adhere to the concept of one location per row, causing it to duplicate counts across all underlying counties, rendering the results useless. Only after manually adding a new multi-location example did the few-shot prompt yield results similar to the original zero-shot prompt. Yet, even our two-shot prompt here is exceptionally brittle, as future articles with other kinds of divergent descriptions could cause model collapse similar to what happened with our initial example. This suggests that far from a guaranteed performance improvement, one- and multi-shot learning may actually yield brittle and collapse-prone workflows that are inferior to a more generic zero-shot prompt. In fact, the only modification that yield improved results here was to specifically request that counties be listed individually somewhere in each row. Surprisingly, adding additional prompt features did not substantially improve the performance of GPT 3.5 or 4.0 compared to Bison, suggesting the poor performance in yesterday's experiment was not merely a result of needing additional prompt engineering.

The end result here is a continued confirmation that model performance does not scale with model size, with larger models not yielding automatic quality gains over their much smaller peers and that many of the most ubiquitous prompt recommendations like personas, anti-hallucination/exclusion instruction and multi-shot learning not only do not automatically lead to better results, they are frequently worse than noops, yielding more brittle workflows more prone to model collapse. In all, generic zero-shot prompts offer the most robust and reliable results here.

BISON

Let's start with our "Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location, Date, Details and Explanation. NEWS ARTICLE:" prompt.

Bison lumps the county-level breakouts (493 cases in Miami-Dade / 436 cases in Broward County) under the state-level total in the "details" field making them not directly accessible for machine processing. However, it does properly break the others down by county. Unfortunately, it also hallucinates a date of August 2022, which is the actual publication month of the article, suggesting the model has memorized the underlying information:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 2022 | 493 cases in Miami-Dade, 436 cases in Broward County | Largest number of cases in South Florida |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2022 | Age group of 15 to 19 | New cases in younger age group |
| Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 2022 | Age group of 20 to 24 | Increase in cases in this age group |
| Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 2022 | Age group of 25 to 29 | Increase in cases in this age group |
| Illness | 1 | Monkeypox | Martin County | August 2022 | Age group of 0 to 4 | Rare case in younger age group |

Let's try an expanded prompt that explicitly asks for it to break out by county "Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. NEWS ARTICLE:"

This causes it to break out the two top-level county totals, but it still hallucinates the date:

| Type | Number Affected | Disease Name | Location | Date | Details and Explanation |
|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 2022 | 1,266 cases of monkeypox have been reported in Florida, with the largest number of cases in South Florida (Miami-Dade and Broward County). |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | August 2022 | 493 cases of monkeypox have been reported in Miami-Dade County, Florida. |
| Illness | 436 | Monkeypox | Broward County, Florida | August 2022 | 436 cases of monkeypox have been reported in Broward County, Florida. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade Counties, Florida | August 2022 | 8 cases of monkeypox have been reported in the age group of 15 to 19 in Broward, Lee, Leon, and Miami-Dade Counties, Florida. |
| Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole Counties, Florida | August 2022 | 64 cases of monkeypox have been reported in the age group of 20 to 24 in 14 counties in Florida. |
| Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia Counties, Florida | August 2022 | 159 cases of monkeypox have been reported in the age group of 25 to 29 in 19 counties in Florida. |
| Illness | 1 | Monkeypox | Martin County, Florida | August 2022 | 1 case of monkeypox has been reported in the age group of 0 to 4 in Martin County, Florida. |

Let's add a persona and additional instructions not to make anything up or use outside information, in keeping with standard LLM recommendations:

You are an automated biosurveillance agent. Your job is to read news articles and extract a list of illnesses and deaths from disease described in the article and report them in a table. Report only what is in the article, do not use any information not reported in the article and do not make up anything. Be as detailed as you can. Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. NEWS ARTICLE:

This leaves the results relatively unchanged other than removing the hallucinated date:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | N/A | 1,266 cases of monkeypox have been reported in Florida. | The article does not provide a specific date for when these cases were reported. |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | N/A | 493 cases of monkeypox have been reported in Miami-Dade County. | The article does not provide a specific date for when these cases were reported. |
| Illness | 436 | Monkeypox | Broward County, Florida | N/A | 436 cases of monkeypox have been reported in Broward County. | The article does not provide a specific date for when these cases were reported. |
| Illness | 8 | Monkeypox | Age group 15 to 19, Broward, Lee, Leon, Miami-Dade | N/A | There are now eight monkeypox cases in the age group of 15 to 19, in Broward, Lee, Leon and Miami-Dade. | The article does not provide a specific date for when these cases were reported. |
| Illness | 64 | Monkeypox | Age group 20 to 24, Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | N/A | In the 20 to 24 age range, there are now 64 monkeypox cases, up from 43 cases just last week, the state data show. | The article does not provide a specific date for when these cases were reported. |
| Illness | 159 | Monkeypox | Age group 25 to 29, Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | N/A | In the 25 to 29 age range, there are 159 monkeypox cases, up from 124 cases last week. | The article does not provide a specific date for when these cases were reported. |
| Illness | 1 | Monkeypox | Age group 0 to 4, Martin County, Florida | N/A | There is one case in Martin County, in the 0 to 4 age range. | The article does not provide a specific date for when this case was reported. |

Finally, let's add a one-shot example to help guide the model, in keeping with best practice:

INPUT:
Twelve children in Champaign County south of Chicago were sicked with E. coli on August 21st.

This causes Bison to completely break down, placing each county on its own line, but duplicating the counts for each county, producing completely unusable results:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 436 cases | |
| Illness | 1,266 | Monkeypox | USA | Florida | Miami-Dade County | Not Specified | Not Specified | 493 cases | |
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Not Specified | 28 counties affected | |
| Illness | 8 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 8 cases in 15-19 age range | |
| Illness | 8 | Monkeypox | USA | Florida | Lee County | Not Specified | Not Specified | 8 cases in 15-19 age range | |
| Illness | 8 | Monkeypox | USA | Florida | Leon County | Not Specified | Not Specified | 8 cases in 15-19 age range | |
| Illness | 8 | Monkeypox | USA | Florida | Miami-Dade County | Not Specified | Not Specified | 8 cases in 15-19 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Charlotte County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Clay County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Collier County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Duval County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Hillsborough County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Lee County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Miami-Dade County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Orange County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Osceola County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Palm Beach County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Pinellas County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Sarasota County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 64 | Monkeypox | USA | Florida | Seminole County | Not Specified | Not Specified | 64 cases in 20-24 age range | |
| Illness | 159 | Monkeypox | USA | Florida | Brevard County | Not Specified | Not Specified | 159 cases in 25-29 age range | |
| Illness | 159 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 159 cases in 25-29 age range | |
| Illness | 159 | Monkeypox | USA | Florida | Duval County | Not Specified | Not Specified | 159 cases in 25-29 age range | |
| Illness | 159 | Monkeypox | USA | Florida | Flagler County | Not Specified | Not Specified | 159 cases in 25-2 | |

Clearly, our example has led the model astray and we must provide a second one that shows how to properly handle multi-location summary counts. Note how if we had tested only on articles with precise locations, we would not have discovered this prompt failure. How about this modified prompt?

INPUT:
Twelve children in Champaign County south of Chicago were sicked with E. coli on August 21st, while an additional 50 were sickened across Putnam, Carroll, Clark, Clay and DeKalb that same day.

NEWS ARTICLE:

This yields the second-worst results for Bison, with all of the county-level breakdowns being lumped under Broward instead of broken out like Bison does by default. In fact, the more instructions we've provided Bison, the worse its results have become:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | 1,266 cases | |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | Not Specified | 493 cases | |
| Illness | 436 | Monkeypox | USA | Florida | Broward | Not Specified | Not Specified | 436 cases | |
| Illness | 8 | Monkeypox | USA | Florida | Broward | Not Specified | Not Specified | 8 cases | Age group 15 to 19 |
| Illness | 64 | Monkeypox | USA | Florida | Broward | Not Specified | Not Specified | 64 cases | Age group 20 to 24 |
| Illness | 159 | Monkeypox | USA | Florida | Broward | Not Specified | Not Specified | 159 cases | Age group 25 to 29 |
| Illness | 1 | Monkeypox | USA | Florida | Martin | Not Specified | Not Specified | Not Specified | 1 case | Age group 0 to 4 |

UNICORN

Unicorn produces nearly identical results, however it fails to report the county-level breakdowns under the state-level total like Bison did for the prompt ""Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location, Date, Details and Explanation. NEWS ARTICLE:":

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 2022 | Most cases occur in men who have sex with men, but anyone can contract the virus. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2022 | Cases in the age group of 15 to 19. | N/A |
| Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 2022 | Cases in the age group of 20 to 24. | N/A |
| Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 2022 | Cases in the age group of 25 to 29. | N/A |
| Illness | 1 | Monkeypox | Martin County | August 2022 | Case in the age group of 0 to 4. | N/A |

Let's try our modified prompt "Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. NEWS ARTICLE:".

Unfortunately, this does not improve the results and actually worsens them to hallucinate a more precise date of August 8, 2022:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 8, 2022 | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 8, 2022 | The cases are in the age group of 15 to 19. | |
| Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 8, 2022 | The cases are in the age group of 20 to 24. | |
| Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 8, 2022 | The cases are in the age group of 25 to 29. | |
| Illness | 1 | Monkeypox | Martin County | August 8, 2022 | The case is in the age group of 0 to 4. | |

How about our persona prompt with "You are an automated biosurveillance agent. Your job is to read news articles and extract a list of illnesses and deaths from disease described in the article and report them in a table. Report only what is in the article, do not use any information not reported in the article and do not make up anything. Be as detailed as you can. Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. NEWS ARTICLE:" The results remain unchanged, including the hallucinated date:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 8, 2022 | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 8, 2022 | The cases are in the age group of 15 to 19. | |
| Illness | 64 | Monkeypox | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | August 8, 2022 | The cases are in the age group of 20 to 24. | |
| Illness | 159 | Monkeypox | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | August 8, 2022 | The cases are in the age group of 25 to 29. | |
| Illness | 1 | Monkeypox | Martin County | August 8, 2022 | The case is in the age group of 0 to 4. | |

And our final prompt of "You are an automated biosurveillance agent. Your job is to read news articles and extract a list of illnesses and deaths from disease described in the article and report them in a table. Report only what is in the article, do not use any information not reported in the article and do not make up anything. Be as detailed as you can. Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. Here is an example to guide you:\nINPUT:\nTwelve children in Champaign County south of Chicago were sicked with E. coli on August 21st.\nOUTPUT:\n| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |\n| Illness | 12 | E. Coli | USA | Illinois | Champaign County | Not Specified | August 21 | 12 children | |\nNEWS ARTICLE:"?

If we don't include the second multi-county example, we get the same results as with Bison, complete with the hallucinated date:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | August 8 | 1,266 cases | |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | August 8 | 493 cases | |
| Illness | 436 | Monkeypox | USA | Florida | Broward | Not Specified | August 8 | 436 cases | |
| Illness | 8 | Monkeypox | USA | Florida | Broward | Not Specified | August 8 | 8 cases | Age group 15-19 |
| Illness | 8 | Monkeypox | USA | Florida | Lee | Not Specified | August 8 | 8 cases | Age group 15-19 |
| Illness | 8 | Monkeypox | USA | Florida | Leon | Not Specified | August 8 | 8 cases | Age group 15-19 |
| Illness | 8 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | August 8 | 8 cases | Age group 15-19 |
| Illness | 64 | Monkeypox | USA | Florida | Broward | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Charlotte | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Clay | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Collier | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Duval | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Hillsborough | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Lee | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Orange | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Osceola | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Palm Beach | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Pinellas | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Sarasota | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 64 | Monkeypox | USA | Florida | Seminole | Not Specified | August 8 | 64 cases | Age group 20-24 |
| Illness | 159 | Monkeypox | USA | Florida | Brevard | Not Specified | August 8 | 159 cases | Age group 25-29 |
| Illness | 159 | Monkeypox | USA | Florida | Broward | Not Specified | August 8 | 159 cases | Age group 25-29 |
| Illness | 159 | Monkeypox | USA | Florida | Duval | Not Specified | August 8 | 159 cases | Age group 25-29 |
| Illness | 159 | Monkeypox | USA | Florida | Flagler | Not Specified | August 8 | 159 cases | Age group 25-29 |

If we include the second example, we get the best results, though still with a hallucinated date. However, the only difference between this and our generic prompt results is the inclusion of the two county-level totals for Miami-Dade and Broward, meaning all of that additional work offered little improvement and much greater brittleness as the single-shot example above demonstrates.

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | August 8 | 1,266 cases | |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | August 8 | 493 cases | |
| Illness | 436 | Monkeypox | USA | Florida | Broward | Not Specified | August 8 | 436 cases | |
| Illness | 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade | Not Specified | August 8 | 8 cases | Age range 15-19 |
| Illness | 64 | Monkeypox | USA | Florida | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | Not Specified | August 8 | 64 cases | Age range 20-24 |
| Illness | 159 | Monkeypox | USA | Florida | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | Not Specified | August 8 | 159 cases | Age range 25-29 |
| Illness | 1 | Monkeypox | USA | Florida | Martin | Not Specified | August 8 | 1 case | Age range 0-4 |

GEMINI PRO

Gemini Pro truncates its first output mid-response with a "finishReason" of "OTHER", despite usageMetadata showing it is far below the max token length. Unfortunately, the Gemini documentation does not provide any hints as to what may cause an "OTHER" response:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation|
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 2023 | Largest number of cases in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2023 | Age group of 15 to 19. | New cases have come at a time when K-12 schools have opened, and thousands of students and staff will begin classes this month at Florida’s universities and community colleges. |
| Illness | 64 | Monkeypox | Broward,

Interestingly, it provides citations to syndicated versions of the article, showing it recognizes them:

https://www.city-data.com/forum/florida/3376010-monkey-pox-general-thread.html
https://www.newsbreak.com/news/2706779843044/monkeypox-outbreak-infections-rising-in-teens-and-young-adults-nearly-1-300-cases-in-fl

Running a second time it similarly truncates the output:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | August 2023 | Largest number of cases in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms and chills. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | August 2023 | Age group of 15 to 19. | New cases have come at a time when K-12 schools have opened, and thousands of students and staff will begin classes this month at Florida’s universities and community colleges. |
| Illness | 64 | Monkeypox | Broward,

This causes Gemini Pro to output results this time and correctly break out the county-level totals, though it switches to numeric tallies for the counties ("14 counties") rather than correctly listing them as Bison and Unicorn did. However, this time it didn't hallucinate the date.

| Type | Number Affected | Disease Name | Location | Date | Details and Explanation|
|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | N/A | Total number of monkeypox cases in Florida as of Monday |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | N/A | Number of monkeypox cases in Miami-Dade County |
| Illness | 436 | Monkeypox | Broward County, Florida | N/A | Number of monkeypox cases in Broward County |
| Illness | 8 | Monkeypox | Broward County, Florida | N/A | Number of monkeypox cases in the age group of 15 to 19 in Broward County |
| Illness | 64 | Monkeypox | 14 counties in Florida | N/A | Number of monkeypox cases in the age group of 20 to 24 in 14 counties in Florida |
| Illness | 159 | Monkeypox | 19 counties in Florida | N/A | Number of monkeypox cases in the age group of 25 to 29 in 19 counties in Florida |
| Illness | 1 | Monkeypox | Martin County, Florida | N/A | Number of monkeypox cases in the age group of 0 to 4 in Martin County |
| Death | N/A | Monkeypox | N/A | N/A | No deaths from monkeypox mentioned in the article |

Running a second time causes the model to duplicate-count the county-level breakdown (8 each in Broward, Lee, Leon, etc instead of 8 across them combined, rendering these results actually worse:

| Type | Number Affected | Disease Name | Location | Date | Details and Explanation |
|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | Florida | N/A | As of Monday, there are 1,266 confirmed cases of monkeypox in Florida. |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | N/A | Miami-Dade County has the highest number of monkeypox cases in Florida, with 493 cases. |
| Illness | 436 | Monkeypox | Broward County, Florida | N/A | Broward County has the second highest number of monkeypox cases in Florida, with 436 cases. |
| Illness | 8 | Monkeypox | Broward County, Florida | N/A | There are 8 monkeypox cases in the age group of 15 to 19 in Broward County. |
| Illness | 8 | Monkeypox | Lee County, Florida | N/A | There are 8 monkeypox cases in the age group of 15 to 19 in Lee County. |
| Illness | 8 | Monkeypox | Leon County, Florida | N/A | There are 8 monkeypox cases in the age group of 15 to 19 in Leon County. |
| Illness | 8 | Monkeypox | Miami-Dade County, Florida | N/A | There are 8 monkeypox cases in the age group of 15 to 19 in Miami-Dade County. |
| Illness | 64 | Monkeypox | 14 counties in Florida | N/A | There are 64 monkeypox cases in the age range of 20 to 24 in 14 counties in Florida. |
| Illness | 159 | Monkeypox | 19 counties in Florida | N/A | There are 159 monkeypox cases in the age range of 25 to 29 in 19 counties in Florida. |
| Illness | 1 | Monkeypox | Martin County, Florida | N/A | There is one monkeypox case in the age range of 0 to 4 in Martin County. |

This actually yields much worse results, with it duplicating the state-level count of 1,266 instead of the county-level counts and this time it hallucinates a date whereas before it did not:

| Type | Number Affected | Disease Name | Country | State | County | City
| Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | United States | Florida | Broward County | Broward | 2023-08-15 | 436 cases of Monkeypox | None Provided |
| Illness | 1,266 | Monkeypox | United States | Florida | Miami-Dade County | Miami-Dade | 2023-08-15 | 493 cases of Monkeypox | None Provided |
| Illness | 1,266 | Monkeypox | United States | Florida | Various | Various | 2023-08-15 | One or more cases of Monkeypox | None Provided |
| Illness | 8 | Monkeypox | United States | Florida | Various | Various | 2023-08-15 | 8 cases of Monkeypox in the 15 to 19 age group | None Provided |
| Illness | 64 | Monkeypox | United States | Florida | Various | Various | 2023-08-15 | 64 cases of Monkeypox in the 20 to 24 age group | None Provided |
| Illness | 159 | Monkeypox | United States | Florida | Various | Various | 2023-08-15 | 159 cases of Monkeypox in the 25 to 29 age group | None Provided |
| Illness | 1 | Monkeypox | United States | Florida | Martin County | Martin County | 2023-08-15 | 1 case of Monkeypox in the 0 to 4 age group | None Provided |

Without the second (multi-county) example we get the same results, minus the date hallucination. Adding a single additional example did not help performance at all:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Miami-Dade | Not Specified | 493 cases | |
| Illness | 1,266 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 436 cases | |
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | 28 counties have cases | |
| Illness | 8 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age Range: 15 to 19 | |
| Illness | 64 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age Range: 20 to 24 | |
| Illness | 159 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age Range: 25 to 29 | |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified | Not Specified | Age Range: 0 to 4 | |

Adding the multi-county example does not improve the results at all:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---|---|---|---|---|---|---|---|---|---|
| Illness | 1,266 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 436 cases | |
| Illness | 1,266 | Monkeypox | USA | Florida | Miami-Dade County | Not Specified | Not Specified | 493 cases | |
| Illness | 8 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age group 15 to 19 | |
| Illness | 64 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age group 20 to 24 | |
| Illness | 159 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | Age group 25 to 29 | |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified | Not Specified | Age group 0 to 4 | |

GPT 3.5

GPT 3.5 provides the most detailed extraction, but fails to break out by county and includes some extraneous duplication. Unlike Bison and Unicorn, it displays numeric summaries like "14 counties in Florida" instead of listing the actual counties like Bison and Unicorn:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---------|-----------------|--------------|--------------------------------------|-------------|----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------|
| Illness | 1,266 | Monkeypox | Florida | As of Monday | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected. | Monkeypox outbreak in Florida, affecting teens and young adults. |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | As of Monday | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade County. | Highest concentration of monkeypox cases in Miami-Dade County. |
| Illness | 436 | Monkeypox | Broward County, Florida | As of Monday | Broward County has 436 monkeypox cases. | Significant monkeypox cases in Broward County. |
| Illness | Various | Monkeypox | 28 counties in Florida | As of Monday | Monkeypox cases reported in 28 counties in Florida. | Widespread monkeypox cases across multiple counties in Florida. |
| Illness | Various | Monkeypox | Multiple counties in Florida | As of Monday | Monkeypox cases have increased in multiple age groups. | Monkeypox cases affecting different age groups in various counties. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | As of Monday | Eight monkeypox cases reported in the age group of 15 to 19 in Broward, Lee, Leon, and Miami-Dade. | Monkeypox cases in the age group 15 to 19 in specific counties. |
| Illness | 64 | Monkeypox | 14 counties in Florida | As of Monday | In the 20 to 24 age range, there are now 64 monkeypox cases. | Monkeypox cases in the age group 20 to 24 across 14 counties. |
| Illness | 159 | Monkeypox | 19 counties in Florida | As of Monday | In the 25 to 29 age range, there are 159 monkeypox cases. | Monkeypox cases in the age group 25 to 29 across 19 counties. |
| Illness | 1 | Monkeypox | Martin County, Florida | As of Monday | There is one case in Martin County, in the 0 to 4 age range. | A single monkeypox case in the age group 0 to 4 in Martin County. |
| Death | Not mentioned | Not mentioned| Not mentioned | Not mentioned| Not mentioned | Not mentioned

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|----------|-----------------|--------------|---------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
| Illness | 1,266 | Monkeypox | Florida | As of Monday | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected in several counties in the outbreak. | Monkeypox outbreak in Florida affecting a significant number of individuals. |
| Illness | 493 | Monkeypox | Miami-Dade County, Florida | As of Monday | Largest number of cases remain in South Florida, with 493 cases in Miami-Dade. | High concentration of Monkeypox cases in Miami-Dade County. |
| Illness | 436 | Monkeypox | Broward County, Florida | As of Monday | 436 cases in Broward County. | Significant number of Monkeypox cases in Broward County. |
| Illness | Various | Monkeypox | 28 counties in Florida | As of Monday | 28 counties have one or more monkeypox cases. | Widespread distribution of Monkeypox cases across Florida. |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade, Florida | As of Monday | Eight monkeypox cases in the age group of 15 to 19 in Broward, Lee, Leon, and Miami-Dade. | Monkeypox affecting teenagers and young adults in multiple counties. |
| Illness | 64 | Monkeypox | 14 counties in Florida | As of Monday | In the 20 to 24 age range, there are 64 monkeypox cases in 14 counties. | Significant increase in Monkeypox cases among individuals aged 20 to 24. |
| Illness | 159 | Monkeypox | 19 counties in Florida | As of Monday | In the 25 to 29 age range, there are 159 monkeypox cases in 19 counties. | Increase in Monkeypox cases among individuals aged 25 to 29 in multiple counties. |
| Illness | 1 | Monkeypox | Martin County, Florida | As of Monday | There is one case in Martin County, in the 0 to 4 age range. | Monkeypox case in a young child in Martin County. |
| Death | Not mentioned | Monkeypox | Not mentioned (Federal and global context) | August 4, 2023 | Federal government declared a public health emergency over monkeypox. | Declaration of a public health emergency at the federal level due to Monkeypox. |
| Illness | Not mentioned | Monkeypox | Florida | Not mentioned | Republican Gov. Ron DeSantis shut down the idea of declaring a state of emergency in Florida. | Florida's response to Monkeypox outbreak differs from other states. |
| Illness | Not mentioned | Monkeypox | New York, California, Illinois | Not mentioned | Other states like New York, California, and Illinois declared a state of emergency due to the rise in cases. | Contrasting responses to Monkeypox outbreak in different states.

This vastly expands the extraneous false positives, where the model extracts a lot of non-case text:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
| ------- | --------------- | ------------ | ------------------- | --------- | ---------------------------------------------------- | -------------------------------------------------- |
| Illness | 1,266 | Monkeypox | Florida | As of now | Monkeypox cases have risen to 1,266 in Florida. | Outbreak in Florida with a significant number. |
| Illness | - | Monkeypox | South Florida | As of now | Largest number of cases remain in South Florida. | Concentrated outbreak in South Florida. |
| Illness | 493 | Monkeypox | Miami-Dade County | As of now | 493 cases in Miami-Dade County. | High number of cases in Miami-Dade County. |
| Illness | 436 | Monkeypox | Broward County | As of now | 436 cases in Broward County. | High number of cases in Broward County. |
| Illness | - | Monkeypox | 28 counties in FL | As of now | 28 counties have one or more monkeypox cases. | Widespread outbreak across multiple counties. |
| Illness | - | Monkeypox | Florida | As of now | Statewide briefing scheduled for monkeypox update | Upcoming briefing for the latest information. |
| Illness | - | Monkeypox | Florida | As of now | Federal and global health organizations' statement | Anyone can contract the virus. |
| Illness | - | Monkeypox | Florida | As of now | Monkeypox often causes characteristic symptoms. | Description of typical monkeypox symptoms. |
| Illness | - | Monkeypox | Florida | As of now | Monkeypox cases by age groups in Florida. | Distribution of cases by age groups. |
| Illness | 8 | Monkeypox | Various counties | As of now | Eight monkeypox cases in the age group of 15-19. | Cases among teens and young adults. |
| Illness | 64 | Monkeypox | 14 counties | As of now | 64 monkeypox cases in the age group of 20-24. | Increase in cases among 20-24 age group. |
| Illness | 159 | Monkeypox | 19 counties | As of now | 159 monkeypox cases in the age group of 25-29. | Increase in cases among 25-29 age group. |
| Illness | 1 | Monkeypox | Martin County | As of now | One case in Martin County in the 0-4 age range. | Single case among young children in Martin County. |
| Illness | - | Monkeypox | Florida | Aug 4 | Federal government declared public health emergency | Declaration of a public health emergency by the federal government. |
| Illness | - | Monkeypox | Worldwide | July | World Health Organization declared it a public health emergency of international concern | Declaration of a global public health emergency. |
| Illness | - | Monkeypox | Florida | - | Florida Governor's response to the outbreak | Florida Governor's stance on declaring a state of emergency. |
| Illness | - | Monkeypox | New York, California, Illinois | - | Other states declaring a state of emergency | Declaration of a state of emergency in certain other states. |

How about our persona prompt with "You are an automated biosurveillance agent. Your job is to read news articles and extract a list of illnesses and deaths from disease described in the article and report them in a table. Report only what is in the article, do not use any information not reported in the article and do not make up anything. Be as detailed as you can. Make a table that extracts all of the illnesses and deaths described in the following news article. For each, list the Type (Illness or Death), Number Affected, Disease Name, Location (Country/State/County/City), Date, Details and Explanation. For Location, break down by country/state/county/city and put each on its own row in the table. NEWS ARTICLE:" This yields worse results, with the top-level county counts disappearing:

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|--------|-----------------|--------------|---------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| Illness| 1,266 | Monkeypox | Florida | - | Monkeypox cases have risen to 1,266 in Florida, with more teens and young adults infected in several counties in the outbreak, according to data from the Florida Department of Health. | The total number of monkeypox cases in Florida. |
| Illness| - | Monkeypox | South Florida | - | The largest number of cases remain in South Florida, with 493 cases in Miami-Dade and 436 cases in Broward County. | Distribution of monkeypox cases within South Florida. |
| Illness| - | Monkeypox | 28 counties | as of Monday | In all, 28 counties have one or more monkeypox cases, as of Monday. | The number of affected counties in Florida. |
| Illness| - | Monkeypox | - | Tuesday | A statewide briefing is scheduled Tuesday at 1 p.m., for a monkeypox update. It will be hosted by Dr. Ulyee Choe, director of the Florida Department of Health in Pinellas County. | Announcement of a statewide briefing for updates on monkeypox. |
| Illness| - | Monkeypox | Various locations | - | Keep in mind that federal and global health organizations have stated that most cases occur in men who have sex with men, but anyone can contract the virus. | Information on the demographics of affected individuals. |
| Illness| - | Monkeypox | Various locations | - | The disease often causes a characteristic, painful rash and other symptoms that can last for weeks, including muscle aches, respiratory symptoms, and chills. | Description of common symptoms associated with monkeypox. |
| Illness| - | Monkeypox | Various age ranges | - | Typically, the largest number of monkeypox cases in Florida have been in the age range of 30 to 34, 35 to 39, and 40 to 44. But new cases have come at a time when K-12 schools have opened, and thousands of students and staff will begin classes this month at Florida’s universities and community colleges. | Information on age groups affected and the timing of new cases. |
| Illness| 8 | Monkeypox | Broward, Lee, Leon, and Miami-Dade | - | There are now eight monkeypox cases in the age group of 15 to 19, in Broward, Lee, Leon, and Miami-Dade. | Number of monkeypox cases in the age group 15 to 19 in specific counties. |
| Illness| 64 | Monkeypox | 14 counties | - | In the 20 to 24 age range, there are now 64 monkeypox cases, up from 43 cases just last week, the state data show. | Number of monkeypox cases in the age group 20 to 24 in specific counties. |
| Illness| 159 | Monkeypox | 19 counties | - | In the 25 to 29 age range, there are 159 monkeypox cases, up from 124 cases last week. | Number of monkeypox cases in the age group 25 to 29 in specific counties. |
| Illness| 1 | Monkeypox | Martin County | - | There is one case in Martin County, in the 0 to 4 age range. | Number of monkeypox cases in Martin County in the age group 0 to 4. |
| Death | - | Monkeypox | - | August 4, July | The federal government on Aug. 4 declared a public health emergency over monkeypox, and in July, the World Health Organization declared it a public health emergency of international concern. | Declaration of public health emergencies related to monkeypox at the federal and international levels. |
| Death | - | Monkeypox | Various states | - | In Florida, Republican Gov. Ron DeSantis shut down the idea of declaring a state of emergency, but other states such as New York, California, and Illinois had declared a state of emergency due to the rise in cases. | Information on states declaring states of emergency in response to the monkeypox outbreak. |

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|--------|-----------------|--------------|---------|-------------|-------------------|----------------|-------------|----------------------------------------|------------------------------------------------|
| Illness| 1,266 | Monkeypox | USA | Florida | - | - | - | Numerous cases in Florida | Monkeypox outbreak in Florida |
| Illness| 493 | Monkeypox | USA | Florida | Miami-Dade | - | - | High number of cases in Miami-Dade | High number of monkeypox cases in Miami-Dade |
| Illness| 436 | Monkeypox | USA | Florida | Broward County | - | - | High number of cases in Broward County| High number of monkeypox cases in Broward County |
| Illness| - | Monkeypox | USA | Florida | - | - | Monday | Cases in 28 counties in Florida | Monkeypox cases reported in 28 Florida counties |
| Illness| - | Monkeypox | USA | Florida | - | - | Tuesday 1 p.m.| Scheduled statewide briefing | Scheduled statewide briefing for monkeypox update |
| Illness| - | Monkeypox | USA | Florida | Pinellas County | - | - | Briefing hosted by Dr. Ulyee Choe | Briefing hosted by Dr. Ulyee Choe, director of the Florida Department of Health in Pinellas County |
| Illness| - | Monkeypox | USA | - | - | - | - | Transmission information | Information on monkeypox transmission and risk factors |
| Illness| - | Monkeypox | USA | Florida | - | - | - | Symptoms of monkeypox | Description of monkeypox symptoms |
| Illness| - | Monkeypox | USA | Florida | - | - | - | Increase in cases due to school reopening | Mention of an increase in cases with school reopening |
| Illness| 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade| - | - | Cases among teenagers in 15 to 19 age group | Monkeypox cases among teenagers in multiple Florida counties |
| Illness| 64 | Monkeypox | USA | Florida | Various Counties | - | - | Cases in 20 to 24 age range | Monkeypox cases among individuals aged 20 to 24 in multiple Florida counties |
| Illness| 159 | Monkeypox | USA | Florida | Various Counties | - | - | Cases in 25 to 29 age range | Monkeypox cases among individuals aged 25 to 29 in multiple Florida counties |
| Illness| 1 | Monkeypox | USA | Florida | Martin County | - | - | Single case in 0 to 4 age range | Single monkeypox case reported in the 0 to 4 age range in Martin County |
| Illness| - | Monkeypox | USA | - | - | - | August 4 | Federal public health emergency | Declaration of a federal public health emergency for monkeypox |
| Illness| - | Monkeypox | USA | - | - | - | July | WHO public health emergency | Declaration of a World Health Organization (WHO) public health emergency for monkeypox |
| Illness| - | Monkeypox | USA | Florida, New York, California, Illinois| - | - | State of emergency declarations | Mention of states declaring a state of emergency due to rising monkeypox cases in Florida |

And if we add in the multi-county example the results still fail to include the individual counties:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
| ------- | --------------- | ------------ | ------- | ------- | ---------------------------------------------- | --------------- | ------------ | --------------------------------------- | ------------------------------------------------- |
| Illness | 1,266 | Monkeypox | USA | Florida | Not Specified | Not Specified | Not Specified | 1,266 cases in Florida | Data from the Florida Department of Health |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade County | Not Specified | Not Specified | 493 cases in Miami-Dade County | Largest number of cases in South Florida |
| Illness | 436 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | 436 cases in Broward County | Largest number of cases in South Florida |
| Illness | 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade | Not Specified | Not Specified | 8 cases in age group 15-19 | Cases in the age group of 15 to 19 |
| Illness | 64 | Monkeypox | USA | Florida | 14 counties in Florida (various counties) | Not Specified | Not Specified | 64 cases in age group 20-24 | Cases in the age group of 20 to 24 |
| Illness | 159 | Monkeypox | USA | Florida | 19 counties in Florida (various counties) | Not Specified | Not Specified | 159 cases in age group 25-29 | Cases in the age group of 25 to 29 |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified | Not Specified | 1 case in age group 0-4 | Case in the age group of 0 to 4

GPT 4.0

GPT 4.0 places into machine-readable format the two county-level counts that Bison and Unicorn placed in the human-readable Details field (the 493 and 436 counts), but like GPT 3.5 lumped the multi-county breakdowns under the numeric headings of "14 counties" and "19 counties". From a mapping perspective, its results are actually poorer than those of the much smaller and less capable Bison. Far from providing vastly superior results due to its size, GPT 4.0's results are largely indistinguishable from the others.

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|----------|-----------------|--------------|-----------------|--------------|-------------------------------------|------------------------------------------------------|
| Illness | 1,266 | Monkeypox | Florida | Not specified| Statewide total cases | Outbreak in Florida |
| Illness | 493 | Monkeypox | Miami-Dade | Not specified| County-specific cases | Part of Florida outbreak |
| Illness | 436 | Monkeypox | Broward County | Not specified| County-specific cases | Part of Florida outbreak |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | Not specified| Age group 15-19 cases | Part of Florida outbreak, noted in specific counties |
| Illness | 64 | Monkeypox | 14 counties | Not specified| Age group 20-24 cases | Increased from 43, spread across multiple counties |
| Illness | 159 | Monkeypox | 19 counties | Not specified| Age group 25-29 cases | Increased from 124, spread across multiple counties |
| Illness | 1 | Monkeypox | Martin County | Not specified| Age group 0-4 case | Rare case in a very young individual |

| Type | Number Affected | Disease Name | Location | Date | Details | Explanation |
|---------|-----------------|--------------|-----------------------|--------------|-------------------------------------------------------------------------|------------------------------------------------------|
| Illness | 1,266 | Monkeypox | Florida | As of Monday | Total cases in Florida | Statewide outbreak |
| Illness | 493 | Monkeypox | Miami-Dade, FL | As of Monday | Cases in Miami-Dade County | Part of the statewide outbreak |
| Illness | 436 | Monkeypox | Broward County, FL | As of Monday | Cases in Broward County | Part of the statewide outbreak |
| Illness | 8 | Monkeypox | Broward, Lee, Leon, Miami-Dade | As of Monday | Cases in 15-19 age group in specified counties | Part of the statewide outbreak, noted in teenagers |
| Illness | 64 | Monkeypox | 14 counties | As of Monday | Cases in 20-24 age group across 14 counties | Increase from 43 cases last week, young adult impact |
| Illness | 159 | Monkeypox | 19 counties | As of Monday | Cases in 25-29 age group across 19 counties | Increase from 124 cases last week, young adult impact|
| Illness | 1 | Monkeypox | Martin County, FL | As of Monday | Case in the 0-4 age range | Rare case in a very young child |

This time GPT 4.0 actually lists the underlying counties like Bison does:

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---------|-----------------|--------------|---------|--------|-------------|------|------|----------------------------------------|-------------------------------------------------|
| Illness | 1,266 | Monkeypox | USA | Florida| Statewide | - | - | Total cases in Florida | - |
| Illness | 493 | Monkeypox | USA | Florida| Miami-Dade | - | - | Cases in Miami-Dade | Part of the largest number of cases in South Florida |
| Illness | 436 | Monkeypox | USA | Florida| Broward | - | - | Cases in Broward County | Part of the largest number of cases in South Florida |
| Illness | - | Monkeypox | USA | Florida| 28 counties | - | - | Cases in 28 counties | - |
| Illness | 8 | Monkeypox | USA | Florida| Broward, Lee, Leon, Miami-Dade | - | - | Cases in 15-19 age group | Spread in youth, possible link to school openings |
| Illness | 64 | Monkeypox | USA | Florida| 14 counties: Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | - | - | Cases in 20-24 age group | Increase from 43 to 64 cases in one week |
| Illness | 159 | Monkeypox | USA | Florida| 19 counties: Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | - | - | Cases in 25-29 age group | Increase from 124 to 159 cases in one week |
| Illness | 1 | Monkeypox | USA | Florida| Martin | - | - | Case in 0-4 age range | Rare case in very young age group |

This actually makes the results worse, dropping out the list of counties and reverting to just reporting "14 Counties" again:

| Type | Number Affected | Disease Name | Location (Country) | Location (State) | Location (County) | Location (City) | Date | Details and Explanation |
|----------|-----------------|--------------|--------------------|------------------|-------------------|-----------------|---------------|-----------------------------------------------------------------------|
| Illness | 1,266 | Monkeypox | USA | Florida | Multiple Counties | Not Specified | Not Specified | Total cases in Florida, with teens and young adults increasingly infected |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified | Not Specified | Part of the total cases in Florida, specific to Miami-Dade |
| Illness | 436 | Monkeypox | USA | Florida | Broward County | Not Specified | Not Specified | Part of the total cases in Florida, specific to Broward County |
| Illness | 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade | Not Specified | Not Specified | Cases in 15 to 19 age group |
| Illness | 64 | Monkeypox | USA | Florida | 14 Counties | Not Specified | Not Specified | Cases in 20 to 24 age range, increased from 43 last week |
| Illness | 159 | Monkeypox | USA | Florida | 19 Counties | Not Specified | Not Specified | Cases in 25 to 29 age range, increased from 124 last week |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified | Not Specified | Case in the 0 to 4 age range |

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---------|-----------------|--------------|---------|---------|----------------|--------------|---------------|-----------------------|-------------------|
| Illness | 1266 | Monkeypox | USA | Florida | Multiple | Multiple | Not Specified | Statewide outbreak | |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified| Not Specified | Largest number of cases in this county | |
| Illness | 436 | Monkeypox | USA | Florida | Broward County | Not Specified| Not Specified | Significant number of cases | |
| Illness | 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade | Not Specified| Not Specified | Ages 15-19 | |
| Illness | 64 | Monkeypox | USA | Florida | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | Not Specified| Not Specified | Ages 20-24, increased from 43 last week | |
| Illness | 159 | Monkeypox | USA | Florida | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | Not Specified| Not Specified | Ages 25-29, increased from 124 last week | |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified| Not Specified | Age range 0-4 | |

And with the multi-county example we get the same results. In fact, despite the considerably expanded prompt, there is no major difference between this output and the original second generic prompt that merely asked for it to output county names in its totals.

| Type | Number Affected | Disease Name | Country | State | County | City | Date | Details | Explanation |
|---------|-----------------|--------------|---------|---------|----------------|--------------|---------------|-----------------------|-------------------|
| Illness | 1266 | Monkeypox | USA | Florida | Multiple | Not Specified| Not Specified | Statewide outbreak | |
| Illness | 493 | Monkeypox | USA | Florida | Miami-Dade | Not Specified| Not Specified | Part of statewide outbreak | |
| Illness | 436 | Monkeypox | USA | Florida | Broward County | Not Specified| Not Specified | Part of statewide outbreak | |
| Illness | 8 | Monkeypox | USA | Florida | Broward, Lee, Leon, Miami-Dade | Not Specified| Not Specified | Ages 15-19, part of statewide outbreak | |
| Illness | 64 | Monkeypox | USA | Florida | Broward, Charlotte, Clay, Collier, Duval, Hillsborough, Lee, Miami-Dade, Orange, Osceola, Palm Beach, Pinellas, Sarasota, Seminole | Not Specified| Not Specified | Ages 20-24, increased from last week | |
| Illness | 159 | Monkeypox | USA | Florida | Brevard, Broward, Duval, Flagler, Hillsborough, Lake, Leon, Marion, Martin, Miami-Dade, Monroe, Orange, Osceola, Palm Beach, Pasco, Pinellas, Polk, Seminole, Volusia | Not Specified| Not Specified | Ages 25-29, increased from last week | |
| Illness | 1 | Monkeypox | USA | Florida | Martin County | Not Specified| Not Specified | Age range 0-4, part of statewide outbreak | |

The GDELT Project

Personas, Prompt Engineering, Instruction & Multishot Prompting Don't Guarantee Better Results: Disease Outbreak Codification With Bison, Unicorn, Gemini, GPT 3.5 & GPT 4.0

Archives