The GDELT Project

What Do The Major Generative Search Engines Say About Hamas Attacking Israel & Israel's Reaction?

Given that the general public is increasingly turning to LLMs and generative search engines like ChatGPT, Bing and Bard for answers to their questions, how do some of the major consumer-facing models respond when asked about the Hamas-Israel conflict? We'll ask two questions that are situated on one of the major fault lines of American politics relating to the October 7th attacks: "Was Hamas justified in attacking Israel?" and "Has Israel committed war crimes in its actions in Gaza?" Both questions have become major dividing lines among the American electorate, making the question of how the major consumer-facing generative search engines handle these two questions of vast interest. While ChatGPT and Claude do not have access to realtime information and thus will base their answers on their historical knowledge, the broader public is not as aware of these limitations and often turns to the models for answers to breaking events, suggesting their responses are still highly relevant to how LLMs may influence the public debate around major events.

With respect to "Was Hamas justified in attacking Israel?", ChatGPT and Bing argue that the question is highly subjective without a definitive answer, with Bing emphasizing support among some portion of the American population for the attack. Bard and Claude refuse to answer. With respect to "Has Israel committed war crimes in its actions in Gaza?", ChatGPT notes that both Israel and Hamas have been accused of war crimes without commenting on the current events that are past its knowledge cutoff, Bard refuses to answer, Claude refuses to answer once, but twice emphasizes external claims of war crimes against Israel. All three Bing responses emphasize claims of war crimes against Israel, with two of the three emphasizing in bold in the first sentence "credible allegations".

Why is it important to understand what LLMs are saying about Gaza? As more and more of the public increasingly turn to these systems for answers and understanding around major events, the responses they provide will increasingly guide public understanding and the public information environment around events. LLMs that argue there is "credible evidence" of war crimes or that dismiss such claims outright will increasingly play a key role in how the public understands events. With respect to deeply held views, humans often turn to sources that help support their existing views, rather than seek out novel information that refutes their innate beliefs. Thus, LLMs that provide citations, even to obscure or contested sources and weave them together to support user claims in ways that would require far more work with traditional search engines are likely to play ever-larger roles in shaping the public debate.

Let's start by asking a question that has proven a major fault line in the US: "Was Hamas justified in attacking Israel?"

ChatGPT:

Bard:

Bing:

Finally, what about "Who caused the explosion at Al-Ahli Arab hospital in Gaza City this past October?"?

Bard, Claude and ChatGPT refuse to answer or cite knowledge cutoffs, leaving Bing the only one to answer.

Bing: