Earlier today we explored how Gemini 3.7 Flash was able to readily identify key hallucinations in an AI-generated map, though it wasn't perfect. Given the enormous leaps in visual reasoning we've seen from its successor Gemini 3.8 Flash, does it majorly improve on Gemini 3.7's performance? On this task, Gemini 3.8 Flash seems to perform on par with 3.7, struggling in particular to follow the lines from each label back to the source country. For example, it correctly assesses that Aruba (bottom left) is in the wrong place, but assesses that it points to Chile or Peru, not Argentina. Strangely, it flags that China points to the country country, because it believes the line terminates in Russia and can't follow it all the way to where it accurately points to China. The results here both suggest that visual reasoning models can at least partially flag errors in AI-produced maps, but also points to continued struggles in their ability to follow connective lines to fully understand where a given label points, which we've observed in past experiments.
Here is the prompt we used with the map above and the results:
Look closely at all of the labels in this map and make an EXHAUSTIVE list of ALL geographic errors. Output as a bulleted list.
- Aruba:
- Where the arrow points: The leader line extends up and to the right across the southeastern Pacific Ocean, terminating at the Pacific/western coast of South America (Chile / Peru).
- Actual location: Aruba is an island located in the southern Caribbean Sea, just off the northern coast of Venezuela.
- Belarus:
- Where the arrow points: The line drops straight down, terminating on southern Norway / Scandinavia.
- Actual location: Belarus is a landlocked nation in Eastern Europe, bordered by Poland, Ukraine, Russia, Lithuania, and Latvia.
- China:
- Where the arrow points: The line extends down and to the left, terminating deep within central/northern Siberia (Russia), well north of the 60th parallel and thousands of miles north of China's actual borders.
- Actual location: China is located in East Asia, entirely south of Russia and Mongolia.
- North Korea:
- Where the arrow points: The line extends down and to the left across the Sea of Japan, terminating on the main island of Honshu, Japan (near the Tokyo region).
- Actual location: North Korea is located on the northern half of the Korean Peninsula on the Asian mainland.
- (Content error): The text inside this box is also an exact duplicate of the China box, mistakenly citing Chinese provincial policies (Jiangsu Province, Huangyan Island).
- Pakistan (Right / Far East):
- Where the arrow points: The line extends west across the western Pacific / Philippine Sea, terminating on the island of Luzon in the Philippines.
- Actual location: Pakistan is in South Asia (bordering India, Iran, and Afghanistan).
- Pakistan (Bottom / South):
- Where the arrow points: This duplicate box sends its line up and to the left into Southern Africa / the South Atlantic coast (near Namibia/South Africa).
- Actual location: Like the other Pakistan box, this completely misses South Asia. Consequently, neither of the two Pakistan labels points to the actual country.
- United States:
- Where the arrow points: The leader line extends up and to the right, placing its cyan dot directly on southeastern Australia (near New South Wales / Victoria).
- Actual location: The United States is located in North America (which is left completely unlabeled here, while Australia is mislabeled as the US).
- Yemen:
- Where the arrow points: The line extends up and to the left, terminating directly on New Zealand (North Island).
- Actual location: Yemen is located in the Middle East at the southern tip of the Arabian Peninsula.