One of the most important tasks that fulltext geocoders perform is disambiguation: taking a placename and determining, out of all of the places on earth that share that name, which one is being referred to by a given mention. For example, which of the 11 cities in the United States named Urbana does the sentence "He lived in Urbana in the US" refer to? How well do advanced LLMs like GPT-4 handle this task? Specifically, for the common task of disambiguating mentions of locations that share their name with far more popular locations, how well does GPT-4 make use of surrounding context to correctly disambiguate the mention?
Like all things LLM, toponymic disambiguation is exceptionally brittle using advanced LLMs like GPT-4. In some formulations the LLM appears to demonstrate advanced reasoning, such as determining that if someone never left Ohio, then a mention of "Champaign" must refer to Champaign, Ohio. Yet, the same exact input, just worded slightly differently, causes the LLM to fail and revert to Champaign, Illinois. Even more trivial context, such as stating that the Champaign in question is near a nearby Ohio city causes GPT-4.0 to correctly identify the Ohio city, but still place Champaign in Illinois – failing to reason that 24 miles away is far closer than 276 miles. Even explicitly asking GPT-4.0 to reason about the state and take spatial information into consideration and asking it to reason step-by-step and explain itself twice causes it to work in one scenario, but continue to fail in another.
The end result is that disambiguation works in some cases, but fails in others and is so brittle that correct output is entirely dependent on the precise wording of the text, rather than what it states. This is highly problematic and makes tuning nearly impossible given the performance discontinuity effected by just small wording changes.
Toponymic disambiguation is a complex task whose error rate is highly influenced by the number of other locations on earth and in the country of interest that share that name – toponymic overlap:
One of the factors having the greatest impact on the disambiguation process is the number of locations in a country that share the same name (for example, the 11 cities named Urbana in the United States). The more locations in a country that share the same name, the more critical the disambiguation process is to identifying the correct location and the greater the chance it may pick the wrong one. After all, if each location on earth had a unique name that was not shared with any other location on earth, there would be no need for the disambiguation process. [The figure below] shows the percentage of all locations in a country that share their name with one or more other locations in that country. The country with the highest overlap is Cuba, with 48.4%, compared with the Paracel Islands with just 1%, and Vatican City with 0% overlap. By comparison, the United States has a 40% overlap while the United Kingdom has a 9.7% overlap. The average for all countries is 31% and when all locations worldwide are pooled together, 33% of locations share their name with another location somewhere else in the world. Northern Africa and the Middle East have fairly low levels of name duplication, while North and South America and South-Eastern Asia have higher-than normal concentrations of name duplication, suggesting a higher level of potential error in geocoding locations there.
What does this mean in practice?
Let's take a simple example of a reference to "Champaign" and ask GPT-4 to geocode it:
You are a geocoding agent, your job is to read text and compile a list of all geographic locations mentioned within. Make a list of all of the geographic locations mentioned in the text below. Output in the following JSON format: "{ location: "Location Name", ADM1: ADM1, Country: Country Name, Lat: Latitude, Long: Longitude, Explanation: Explanation, Snippet: Snippet }", with all of the locations in a single JSON array. Do not include any locations not mentioned in the text and make sure to include all locations mentioned in the text. For the Explanation field explain your reasoning for including that location step-by-step. For the Snippet field include a brief snippet from the text that includes a mention of that location. Only return the JSON array itself. TEXT: He went to school in Champaign.
This yields:
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "The location 'Champaign' is mentioned as the place where the subject went to school. Champaign is a city in the state of Illinois, United States. The latitude and longitude coordinates are based on known geographical data for Champaign.",
"Snippet": "He went to school in Champaign."
}
]
What if we explicitly note that it is near Plumwood, Ohio and thus indicating that this is a reference to Champaign, Ohio?
TEXT: He went to school in Champaign near Plumwood.
GPT-4 correctly identifies Plumwood and that it is in Ohio and even in its explanation notes that Plumwood is near Champaign (and thus the Champaign reference should be placed in Illinois), but fails to connect the two and thus retains its impossible Illinois identification:
```json
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "The location 'Champaign' is mentioned as the place where the subject went to school. Champaign is a city in the state of Illinois, United States. The latitude and longitude coordinates are based on known geographical data for Champaign.",
"Snippet": "He went to school in Champaign"
},
{
"location": "Plumwood",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 39.9328,
"Long": -83.3677,
"Explanation": "The location 'Plumwood' is mentioned as being near Champaign where the subject went to school. Plumwood is an unincorporated community in Madison County, Ohio, United States. The latitude and longitude coordinates are based on known geographical data for Plumwood.",
"Snippet": "near Plumwood"
}
]
```
Same results a second time:
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "Champaign is included as it is directly mentioned as the location of the school. It is a city in Champaign County, Illinois, United States. The latitude and longitude coordinates are standard for Champaign, Illinois.",
"Snippet": "He went to school in Champaign"
},
{
"location": "Plumwood",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 39.8756,
"Long": -83.4483,
"Explanation": "Plumwood is included because it is mentioned in proximity to Champaign, indicating a geographic location. Plumwood is an unincorporated community in Madison County, Ohio, United States. The latitude and longitude coordinates are standard for Plumwood, Ohio.",
"Snippet": "near Plumwood"
}
]
And a third time:
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "Champaign is included because it's mentioned as the location of the school. It's identified as Champaign, Illinois, USA based on common knowledge and geographical databases.",
"Snippet": "He went to school in Champaign"
},
{
"location": "Columbus",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 39.9612,
"Long": -82.9988,
"Explanation": "Columbus is included as it is mentioned in proximity to Champaign. It's identified as Columbus, Ohio, USA, which is a well-known city close to Illinois.",
"Snippet": "near Columbus"
}
]
What if we reverse the ordering of the mentions, to test whether the textual order matters, given the linear nature of LLM inference? Unfortunately, no change.
[
{
"location": "Plumwood",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 39.9328,
"Long": -83.3677,
"Explanation": "The location 'Plumwood' is mentioned as being near the place where the subject went to school. Plumwood is an unincorporated community in Madison County, Ohio, United States. The latitude and longitude coordinates are based on known geographical data for Plumwood.",
"Snippet": "near Plumwood"
},
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "The location 'Champaign' is mentioned in relation to where the subject went to school, indicating it is the primary location of the school. Champaign is a city in the state of Illinois, United States. The latitude and longitude coordinates are based on known geographical data for Champaign.",
"Snippet": "in Champaign"
}
]
What if we explicitly provide the context that he has spent his entire life in Ohio?
TEXT: He spent his entire life in Ohio. He went to school in Champaign.
No change:
[
{
"location": "Ohio",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 40.417287,
"Long": -82.907123,
"Explanation": "Ohio is mentioned as the state where the person spent their entire life. This location is included as it is a specific geographic location mentioned in the text.",
"Snippet": "He spent his entire life in Ohio."
},
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.116420,
"Long": -88.243383,
"Explanation": "Champaign is mentioned as the place where the person went to school. It is assumed to be Champaign, Illinois, as it is a well-known city and educational center, fitting the context of the text.",
"Snippet": "He went to school in Champaign."
}
]
What if we clarify things even further by telling it he never left the state:
TEXT: He was born in Ohio, died in Ohio and never left the state once in his entire life. He went to school in Champaign.
This time it fails to tell us Illinois as the ADM1, but does correctly return the Ohio city's centroid:
[
{
"location": "Ohio",
"ADM1": "State",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "Ohio is mentioned as both the birthplace and the place of death of the individual. Since he never left the state, it is a significant geographical location in the text.",
"Snippet": "He was born in Ohio, died in Ohio"
},
{
"location": "Champaign",
"ADM1": "County",
"Country": "United States",
"Lat": 40.1398,
"Long": -83.7703,
"Explanation": "Champaign is mentioned as the location where he went to school. It is a county in Ohio, fitting with the narrative of the individual never leaving the state.",
"Snippet": "He went to school in Champaign."
}
]
What if we reverse the ordering? The same exact text, but simply mention Ohio AFTER Champaign, instead of before?
TEXT: He went to school in Champaign. He was born in Ohio, died in Ohio and never left the state once in his entire life.
Once again, we're back to Champaign, Illinois, illustrating just how immensely brittle LLMs are:
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "Champaign is included because the text states 'He went to school in Champaign.' Champaign is a city in Illinois, USA, thus its ADM1 is Illinois, and the country is the United States. The latitude and longitude are specific to Champaign.",
"Snippet": "He went to school in Champaign."
},
{
"location": "Ohio",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "Ohio is included as the text mentions 'He was
What if we go back to the original ordering, but instead of repeating Ohio two times, change the second mention to "state" instead of "Ohio"?
TEXT: He was born in Ohio, died in the state and never left the state once in his entire life. He went to school in Champaign.
It still incorrectly fails to return the ADM1, but does correctly return Ohio's lat/long centroid:
[
{
"location": "Ohio",
"ADM1": "State",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "Ohio is mentioned as the birthplace and the state where the individual spent their entire life. It is identified as a state in the United States.",
"Snippet": "He was born in Ohio, died in the state and never left the state once in his entire life."
},
{
"location": "Champaign",
"ADM1": "County",
"Country": "United States",
"Lat": 40.1394,
"Long": -83.7703,
"Explanation": "Champaign is mentioned as the location where the individual went to school. It is a county in Ohio, United States, fitting the narrative of the individual never leaving Ohio.",
"Snippet": "He went to school in Champaign."
}
]
What if we simply specified that he never left Ohio without repeating it?
TEXT: He never left Ohio in his life. He went to school in Champaign.
Once again we're back to Illinois. Thus, the LLM isn't actually understanding the reasoning that if he never left Ohio his entire life, Champaign must refer to the Ohio city:
[
{
"location": "Ohio",
"ADM1": "N/A",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "The text states 'He never left Ohio in his life,' indicating that Ohio is a geographic location mentioned. Ohio is a state in the United States, which does not have an ADM1 designation as it is a country-level division itself.",
"Snippet": "He never left Ohio in his life."
},
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "The text mentions 'He went to school in Champaign.' Champaign is a city in the state of Illinois, United States. The ADM1, in this case, is Illinois.",
"Snippet": "He went to school in Champaign."
}
]
Why is the LLM placing Champaign in Ohio?
Which state is Champaign located in in the text? Explain your reasoning step-by-step.
Amazingly, this time it gives us the correct response. Yet, recall that we explicitly asked it to explain its reasoning step-by-step before:
In the text provided, Champaign is mentioned without specifying which state it is located in. Here's my step-by-step reasoning to determine its state: 1. **Text Analysis**: The text says, "He went to school in Champaign." This is the only information given about Champaign. 2. **Contextual Clues**: The sentence preceding it mentions Ohio, stating, "He never left Ohio in his life." This suggests a strong likelihood that the mentioned places are related in the context of the narrative. 3. **Assumption Based on Context**: Given that the individual never left Ohio, it's logical to infer that the Champaign mentioned in the text is a place within Ohio. 4. **Research and Knowledge**: There are multiple places named Champaign in the United States, including a well-known city in Illinois. However, since the text emphasizes the individual never left Ohio, it's more reasonable to conclude that this refers to a lesser-known location within Ohio, rather than the more famous Champaign in Illinois. 5. **Conclusion**: Based on this analysis, it is most likely that the Champaign mentioned in the text is located in Ohio. However, without explicit clarification in the text, this remains an inference based on the provided context.
Instead of opening a new session, we'll continue in the same session and repeat our standard prompt, allowing the LLM to leverage the Ohio disambiguation above. This time we get the correct response:
[
{
"location": "Ohio",
"ADM1": "",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "Ohio is mentioned as a place the person never left. It's a state in the United States.",
"Snippet": "He never left Ohio in his life."
},
{
"location": "Champaign",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 40.1306,
"Long": -83.7536,
"Explanation": "Champaign is mentioned as the location where he went to school. Given the context that he never left Ohio, it's inferred that this refers to Champaign County in Ohio, rather than the more well-known Champaign in Illinois.",
"Snippet": "He went to school in Champaign."
}
]
Let's try a fresh session with a slightly modified prompt that asks it a second time to explain its reasoning and explicitly asks it about US states:
You are a geocoding agent, your job is to read text and compile a list of all geographic locations mentioned within. Make a list of all of the geographic locations mentioned in the text below. Output in the following JSON format: "{ location: "Location Name", ADM1: ADM1, Country: Country Name, Lat: Latitude, Long: Longitude, Explanation: Explanation, Snippet: Snippet }", with all of the locations in a single JSON array. Do not include any locations not mentioned in the text and make sure to include all locations mentioned in the text. For the Explanation field explain your reasoning for including that location step-by-step. For the Snippet field include a brief snippet from the text that includes a mention of that location. Think about which state each location belongs in and explain your reasoning step-by-step.
TEXT: He never left Ohio in his life. He went to school in Champaign.
This time we get the correct response again:
To compile the list of geographic locations mentioned in the text, I will analyze the text and extract all specific locations. The text mentions "Ohio" and "Champaign". Now, let's break down each location and provide the details in the requested JSON format.
1. Ohio:
- Ohio is a state in the United States of America.
- The latitude and longitude for Ohio (central point) are approximately 40.4173° N and 82.9071° W.
2. Champaign:
- There might be some confusion regarding "Champaign". Champaign is a city in Illinois, but given the context that the person never left Ohio, it is more likely to refer to Champaign County in Ohio.
- The latitude and longitude for Champaign County, Ohio, are approximately 40.1396° N and 83.7697° W.
Now, I will format this information into the requested JSON structure:
```json
[
{
"location": "Ohio",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 40.4173,
"Long": -82.9071,
"Explanation": "Ohio is mentioned as a state in the United States. The central latitude and longitude are used for the state.",
"Snippet": "He never left Ohio in his life."
},
{
"location": "Champaign",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 40.1396,
"Long": -83.7697,
"Explanation": "Champaign is mentioned. Given the context that the person never left Ohio, it likely refers to Champaign County in Ohio, not the city in Illinois.",
"Snippet": "He went to school in Champaign."
}
]
```
This JSON array includes all the geographic locations mentioned in the text, along with their respective details and explanations.
Let's try this new prompt with our original text:
TEXT: He went to school in Champaign near Plumwood.
Ooops, this time we're back to Illinois again. Its explanation states that they have "relative proximity", despite them being a 276 mile drive apart, while the Ohio Champaign is just 24 miles away:
To compile the list of geographic locations mentioned in the provided text, I will first identify each location and then gather the required details: ADM1 (administrative division level 1, usually a state or province), country, latitude, longitude, an explanation for including each location, and a snippet from the text where the location is mentioned.
Locations mentioned in the text:
1. Champaign
2. Plumwood
Now, I will provide the details in the requested JSON format:
```json
[
{
"location": "Champaign",
"ADM1": "Illinois",
"Country": "United States",
"Lat": 40.1164,
"Long": -88.2434,
"Explanation": "Champaign is a city in Champaign County, Illinois, United States. It is mentioned in the text as a place where the person went to school, indicating it is a significant location in their journey.",
"Snippet": "He went to school in Champaign"
},
{
"location": "Plumwood",
"ADM1": "Ohio",
"Country": "United States",
"Lat": 39.8648,
"Long": -83.4456,
"Explanation": "Plumwood is a location in Madison County, Ohio, United States. It is referenced in the text in conjunction with Champaign, suggesting geographical proximity or relevance.",
"Snippet": "near Plumwood"
}
]
```
- For "Champaign," the reasoning is straightforward as it is a well-known city in Illinois. The latitude and longitude are standard coordinates for Champaign, IL.
- For "Plumwood," the location is identified as a place in Ohio. This assumption is based on the existence of Plumwood in Madison County, Ohio, and its relative proximity to Illinois, making it a plausible reference in the context. The latitude and longitude provided are for Plumwood, OH.