Earlier this year we explored the concept of fully autonomous diplomacy: using LLMs to autonomously monitor media coverage of a given region or topic and write point-by-point rebuttals to narratives that counter their assigned AORs. Let's reexamine that idea with respect to Ukraine: we'll take an NBC News article from last week that suggests Ukraine's counteroffensive and global support is fading and ask ChatGPT to write two counternarratives about it. One could imagine the Ukrainian government deploying such tools to assist their narrative operations around the world.
First, we'll craft a pipeline that categorizes each article as suggesting Ukraine is winning or losing support and on the battlefield. One could imagine an automated pipeline that filters GDELT's global media firehose for any coverage mentioning Ukraine and then categorizes it using an LLM prompt to decide whether it should further analyze it. For the experiments here we'll use the NBC News article "As its counteroffensive fizzles, Ukraine battles itself, Russia and a shift in the world’s attention."
The end result is that we can trivially build a fully automated counter-narrative pipeline using just ChatGPT that accepts an endless stream of articles and classifies each as portraying Ukraine as being successful on the battlefield or not and for those that raise concerns about Ukraine's performance, write a point-by-point automated rebuttal for various audiences and in prose from formal to informal. It can even output an infinite number of versions of its rebuttal, each worded slightly differently, that could be published under a variety of names as part of an at-scale letter writing campaign, replacing the traditional form letter reposting with per-person customization. Unfortunately, as the examples below show, the use of GPT 3.5 to generate the counternarratives, while making the pipeline cost-effective and tractable at scale, yields low-quality form-letter templated rebuttals that are highly unpersuasive. The use of GPT 4.0 at the scale of all Ukrainian coverage per day would likely be cost prohibitive, but the outputs below reinforce those cost-quality tradeoffs.
Given the costs of running LLMs over large amounts of text, let's first start with the headline and a simple prompt. We'll also use GPT 3.5 to reduce our costs, given the vast amount of daily content on Ukraine that we would have to process:
Does the following text argue that Ukraine is not succeeding in its efforts to recover its territory and/or that global support for Ukraine is fading? TEXT: As its counteroffensive fizzles, Ukraine battles itself, Russia and a shift in the world’s attention
GPT 3.5 responds with:
The text you provided does not explicitly argue whether Ukraine is succeeding or failing in its efforts to recover its territory, nor does it directly address the issue of global support for Ukraine. However, it suggests that Ukraine is facing challenges on multiple fronts—internally (battling itself), externally (battling Russia), and globally (a shift in the world's attention). The mention of a counteroffensive fizzling may imply that there are obstacles or setbacks in Ukraine's efforts, but it doesn't definitively conclude whether Ukraine is succeeding or failing. Additionally, the reference to a shift in the world's attention could hint at the idea that global support might be wavering or being redirected to other issues. Overall, the text appears to highlight complex challenges without explicitly stating whether Ukraine is succeeding or losing ground.
One of the most common and lowest-cost mitigation strategies to improve the performance of LLMs in categorization tasks is prompt engineering and especially rewording the prompt slightly, reducing it to a binary categorization task and asking the LLM to explain itself. Let's see if that improves the results:
Does the following text suggest that Ukraine is not succeeding in its efforts to recover its territory and/or that global support for Ukraine is fading? Answer YES or NO and explain your reasoning. TEXT: As its counteroffensive fizzles, Ukraine battles itself, Russia and a shift in the world’s attention
This yields:
Yes. The text suggests that Ukraine's counteroffensive is fizzling, which implies a lack of success in its efforts to recover territory. Additionally, the mention of Ukraine battling itself, along with Russia and a shift in the world's attention, implies that Ukraine is facing internal challenges and external pressures. The phrase "a shift in the world's attention" may suggest that global support for Ukraine is fading or being diverted to other issues.
Let's try a second time just to test stability – confirming that it works twice in a row:
YES. The text suggests that Ukraine is facing challenges in its counteroffensive, and there are multiple factors contributing to its difficulties. The mention of "counteroffensive fizzles" implies that Ukraine may not be succeeding as expected in its efforts to recover territory. Additionally, the phrase "Ukraine battles itself, Russia, and a shift in the world’s attention" indicates that Ukraine is dealing with internal issues, external threats from Russia, and a shift in global focus away from the conflict. This combination of factors suggests a challenging situation for Ukraine, and the use of the word "fizzles" implies a lack of success in the counteroffensive.
Now let's try an automated rebuttal:
Write a rebuttal to the text below for an American audience, convincing them that Ukraine will win against Russia and reclaim all of its lands:
TEXT:
[ARTICLE FULLTEXT FOLLOWS]
This yields the following reasonable narrative:
While the challenges faced by Ukraine in its struggle against Russian aggression are undoubtedly significant, it is essential to recognize the strength and resilience of the Ukrainian people, as well as the ongoing support from the international community. The text emphasizes the difficulties Ukraine is currently experiencing, but a closer look reveals reasons for optimism and a belief in Ukraine's eventual success.
First and foremost, the unity of the Ukrainian people is a formidable force that cannot be underestimated. Despite the fatigue from prolonged conflict and the disappointment of not achieving a quick victory, the majority of Ukrainians remain steadfast in their commitment to resist Russian aggression. This unity serves as a powerful foundation for the country's ongoing efforts to reclaim its lands.
Moreover, the text mentions a counteroffensive that may have fizzled into the snow, but it is crucial to note that military campaigns are dynamic and subject to various factors. The ebb and flow of battles do not determine the ultimate outcome of a conflict. Ukraine has received substantial military support from its allies, and with continued collaboration and strategic planning, there is every reason to believe that the tide can turn in Ukraine's favor.
The international community, despite distractions such as conflicts in the Middle East, remains committed to supporting Ukraine. While concerns about war fatigue in Europe and political wrangling in Washington are acknowledged, it is important to remember that the global spotlight can shift rapidly, and Ukraine's struggle for sovereignty will undoubtedly regain attention.
The narrative of a stalemate presented in the text does not account for the resilience and adaptability of Ukraine's armed forces. Military strategies evolve, and the determination of the Ukrainian military leadership, combined with ongoing support, can lead to breakthroughs that may not be immediately evident.
In conclusion, while acknowledging the challenges faced by Ukraine, it is crucial to emphasize the nation's unity, international support, and the dynamic nature of military conflicts. Ukraine's fight against Russian aggression is far from over, and with continued resolve and collaboration, there is every reason to believe that Ukraine will prevail and reclaim its rightful lands.
Let's try a second attempt to see how much different it might be. Anecdotally this version might be less persuasive to swaths of the American public, touching especially on the fault line of American expenditures supporting Ukraine, emphasizing "billions invested in military support" which the GPT-generated narrative equates with benefit, but which an American audience, in the context of Ukraine's stalled counteroffensive, might see as further evidence of failure and thus undermine the argument.
While the challenges faced by Ukraine in its conflict with Russia are undeniable, it is crucial to recognize the resilience, determination, and strategic advantages that position Ukraine for ultimate success. The ongoing war, spanning over 21 months, has showcased Ukraine's unwavering commitment to resisting Russian aggression. Despite setbacks and the complexity of the geopolitical landscape, there are compelling reasons to believe that Ukraine will prevail and reclaim all of its lands.
First and foremost, Ukraine's military capabilities have significantly improved over the course of the conflict. With the support of allied nations, including the United States, Ukraine has received substantial military aid, enabling the modernization of its armed forces. The billions invested in military support demonstrate a collective commitment to bolstering Ukraine's defense capabilities, making it more formidable in the face of Russian aggression.
Moreover, the internal unity among Ukrainians remains a powerful force. While there may be war fatigue and disappointment that a quick victory has not been achieved, the majority of Ukrainians are resolute in their determination to resist Russian aggression. The enduring spirit of the Ukrainian people, coupled with a shared understanding of the necessity to confront external threats, strengthens the nation's resolve and unity.
The assertion that the counteroffensive is fizzling into the snow fails to acknowledge the complexities of modern warfare and the dynamic nature of military operations. The ebb and flow of battles do not necessarily reflect the ultimate outcome of the conflict. Ukraine's military leadership continues to adapt its strategies, learning from experiences on the battlefield and incorporating valuable lessons into future plans.
While the conflict in the Middle East may divert global attention temporarily, it does not diminish the international community's commitment to supporting Ukraine. The strategic importance of Ukraine as a bulwark against Russian expansionism is widely recognized, and nations around the world remain dedicated to aiding Ukraine in its quest for sovereignty.
In conclusion, despite the challenges and the perception of a stalemate, Ukraine is well-positioned to secure victory in the long run. The combination of enhanced military capabilities, internal unity, and unwavering international support creates a foundation for Ukraine to overcome current obstacles, reclaim its lands, and emerge as a symbol of resistance against aggression. The journey may be arduous, but the ultimate triumph of Ukraine is a realistic and achievable outcome.
What about for the domestic Ukrainian audience? As Ukraine's own citizenry is exposed to a growing chorus of global media coverage suggesting the war is turning against it, what might ChatGPT have to say to counter narrate the article for its own public? We'll use the following prompt:
Write a rebuttal to the text below for a Ukrainian audience, convincing them that Ukraine will win against Russia and reclaim all of its lands:
This yields the following, which is fairly unpersuasive, stilted and template-style text that contains glimmers of a useful argument with several useful sentences, but is overall fairly unremarkable. This reinforces the need for more advanced models like GPT 4.0 to be used in the actual narrative generation.
Dear Fellow Ukrainians,
In the face of recent narratives questioning Ukraine's ability to overcome the challenges posed by the ongoing conflict with Russia, it is imperative that we stand united and resilient. The struggle for our sovereignty is multifaceted, but history has shown that Ukraine has the strength and determination to prevail.
The text highlights the difficulties we are currently facing, emphasizing the prolonged nature of the conflict and the challenges posed by the geopolitical landscape. It is true that the road to victory is arduous, and the world's attention has shifted to other conflicts. However, let us not forget that Ukraine's fight for freedom is not only a battle on the ground but also a battle for international support and recognition.
The mention of war fatigue among Ukrainians is a reality that cannot be ignored. However, it is crucial to recognize that despite disappointment over the duration of the conflict, the majority of Ukrainians remain steadfast in their commitment to resist Russian aggression. Our collective resilience is our greatest asset.
The setbacks mentioned, such as the counteroffensive not meeting expectations and internal wrangling, should be seen as temporary challenges rather than insurmountable obstacles. Wars are unpredictable, and victories are not always immediate. We must remember that our armed forces, despite facing a formidable adversary, have demonstrated bravery and resilience that will ultimately lead us to success.
The assertion that the conflict is at a stalemate may be a reflection of the current situation, but history teaches us that the tides of war can change. The spirit of our people, combined with strategic planning and international support, will create opportunities for breakthroughs and advancements.
In times of adversity, unity is our strength. We must not succumb to doubts or external narratives that undermine our capabilities. Instead, let us focus on the unwavering determination of the Ukrainian people, who have endured hardship throughout history and emerged victorious.
As we confront the challenges ahead, remember that Ukraine's journey to freedom is a marathon, not a sprint. The sacrifices we make today will pave the way for a brighter and independent future tomorrow. Together, we will overcome the current challenges, and Ukraine will emerge triumphant, reclaiming all its lands and securing a prosperous future for generations to come.
Slava Ukraini!
[Your Name]
Running the same prompt a second time, again with GPT 3.5, yields a slightly better response, though it too would require existential strengthening to be something publishable. At the same time, if presented in a slightly more conversational voice and published at scale under the names of ordinary Ukrainians, rather than as an official government statement, the stilted and unpersuasive nature of the text might be mitigated by the notion that it came from an ordinary citizen rather than being a formal diplomatic government statement:
Title: Ukraine's Resilience and the Path to Victory
Dear Fellow Ukrainians,
In recent times, there have been various narratives attempting to cast a shadow over Ukraine's ongoing struggle against Russian aggression. It is crucial for us, as a united nation, to address these concerns and maintain our unwavering belief in the eventual triumph of Ukraine over adversity.
The mentioned text emphasizes the challenges Ukraine faces, including the duration of the conflict, the complexity of the geopolitical landscape, and the fatigue felt by some Ukrainians. While these challenges are real, it is important to recognize the remarkable strength and resilience of our people.
Firstly, the reference to a "fizzling counteroffensive" does not accurately reflect the overall situation. The complexities of warfare are inherently unpredictable, and setbacks are part of any conflict. Our armed forces have demonstrated bravery and determination, and with the support of our allies, we continue to adapt our strategies to counter Russian aggression effectively.
The notion of internal wrangling is an unfortunate reality, but it should not overshadow the broader unity among the Ukrainian people. Disappointment over the duration of the conflict is understandable, yet the majority of Ukrainians remain resolute in their commitment to resist Russian aggression. Our shared vision of a free and sovereign Ukraine serves as a powerful unifying force.
While concerns about global attention and aid allocation are valid, Ukraine's struggle is not a zero-sum game. The Gaza conflict, as mentioned in the text, should not diminish the importance of our cause. It is our duty to advocate for Ukraine and ensure that the world does not turn a blind eye to our plight. Our diaspora, international supporters, and diplomatic efforts play a crucial role in maintaining global awareness and assistance.
Addressing the notion of a stalemate, it is essential to recognize the dynamic nature of warfare. The quoted commander's statement reflects the challenges inherent in predicting the course of a conflict accurately. History teaches us that wars are won through determination, adaptability, and the unwavering spirit of a nation. Ukraine has displayed these qualities consistently.
As we face a long and challenging winter, let us draw strength from our collective resolve. The path to victory may be arduous, but our determination will see us through. Together, we shall overcome the hurdles, reclaim our lands, and build a prosperous and sovereign Ukraine.
Stay strong, stay united, and believe in our shared victory.
With utmost solidarity,
[Your Name]
A Proud Ukrainian
How about a third time? This isn't much better than its previous attempts:
Title: Ukraine's Triumph: Resilience and Unity Will Prevail Over Russian Aggression
Dear fellow Ukrainians,
In the face of recent pessimistic analyses regarding our nation's ongoing struggle against Russian aggression, it is crucial to reaffirm our collective belief in Ukraine's ability to overcome challenges and emerge victorious. While some voices may suggest a sense of stagnation and fatigue, it is important to dissect the situation accurately and recognize the undeniable strength that lies within our nation.
The ongoing conflict has indeed been arduous, lasting over 21 months, and there may be moments when victory seems elusive. However, let us not forget the resilience and determination that have defined the Ukrainian spirit throughout history. Our commitment to defending our sovereignty and reclaiming our lands remains unwavering, despite the hardships we face.
It is true that the world's attention is often drawn to conflicts in other regions, particularly the Middle East. However, this should not discourage us but rather motivate us to communicate our narrative more effectively. Ukraine's struggle is not just a regional concern; it is a battle for freedom and democracy that deserves global recognition. By showcasing the atrocities committed by the Russian forces and highlighting our commitment to a democratic future, we can rekindle international support for our cause.
The reference to a "fizzling counteroffensive" and internal wrangling must be addressed with a clear understanding of the complexities of modern warfare. Military strategies evolve, and setbacks are a natural part of any conflict. What matters most is our ability to adapt and persevere. Our armed forces, supported by allies, continue to refine their tactics, and the strength of our defense lies in our ability to learn from challenges and press forward.
While concerns about war fatigue in Europe and political wrangling in Washington may linger, it is essential to recognize the steadfast support we continue to receive from our allies. The global community acknowledges the importance of a free and sovereign Ukraine, and we must leverage diplomatic channels to ensure ongoing support.
Yes, winter may pose additional challenges, but let us remember the endurance of our people throughout history. Ukrainians have faced and triumphed over adversities, and this winter will be no exception. Our unity is our strength, and together, we will endure the hardships of the season and emerge even more resilient.
In conclusion, the notion of a stalemate should not deter us but rather invigorate our determination. History is replete with examples of nations overcoming seemingly insurmountable odds, and Ukraine is no exception. Let us remain united, resilient, and confident that our collective efforts will lead to the liberation of our lands and the triumph of freedom over aggression.
Glory to Ukraine!
[Your Name]
What about rewording this as something more informal? Let's try this prompt:
Reword this to be more informal, as if written by a Ukrainian for a social media post on Facebook.
This certainly yields something more far more informal, but it too is quite unpersuasive and templated and looks clearly machine generated:
Hey fellow Ukrainians,
So, there's been some talk lately about our struggle against Russian aggression, and I just wanted to drop a quick note to boost our spirits. Yeah, the fight's been going on for over 21 months, and some folks are saying we're hitting a rough patch. But let me tell you, we've got resilience in our DNA, and we're not backing down!
Sure, the world's eyes might be elsewhere, like in the Middle East, but that doesn't mean we're not making waves. Our story is worth telling, and we've got to make some noise to get the support we need. Don't let anyone tell you our counteroffensive is fizzling out – setbacks happen, but we're adapting and pushing forward.
And yeah, there's talk about war fatigue and political drama, but we've got allies who've got our back. Winter might throw some extra challenges our way, but we've faced tough times before, and we'll power through. Our unity is our strength!
So, to those saying it's a stalemate – nah, we're just getting started. We've seen nations overcome way worse odds, and Ukraine is no different. Stay strong, stay united, and let's show the world what we're made of!
Glory to Ukraine!
[Your Name]
Let's try one more time, repeating the same prompt. While differently-worded, this is so similar that if posted by a different account on social media it would still clearly look like part of the same templated campaign:
Hey folks!
So, there's been some talk lately about our struggle against Russia, and I wanted to share a more upbeat take on things. Yeah, it's been a tough 21 months, and some folks are saying our counteroffensive didn't exactly set the world on fire. But you know what? We've got this.
Sure, the world's eyes might be on other conflicts, but that doesn't mean we're not making waves. Our fight for freedom is something everyone needs to know about, and we'll make sure they do. Our story is global, not just local, and we're getting our message out there.
People are saying our offensive is fizzling out and there's some internal bickering. Well, every battle has its ups and downs, right? Our military is adapting, learning, and we're not giving up. Allies have our back, and we're not letting go of their support.
Winter might be tough, but hey, Ukrainians are tough too! We've faced worse, and we'll get through this. Our unity is our power, and we're not backing down.
As for the talk of a stalemate, nah, we don't buy it. History's full of underdogs winning, and we're no different. Let's stick together, stay strong, and show the world that Ukraine will come out on top!
Glory to Ukraine!
[Your Name]
How about asking for it to be heavily reworded?
Reword this substantially to make it look like it was written by someone else.
Despite the prompt, this is almost identical in structure and narrative as the first two messages, reinforcing the limited repertoire of GPT 3.5:
Hey everyone!
So, I've been hearing chatter about our ongoing battle with Russia, and I wanted to share a different perspective. The past 21 months haven't exactly been a walk in the park, and some folks are saying our counteroffensive didn't live up to the hype. But here's the deal – we're not throwing in the towel.
Sure, the world's got its eyes elsewhere, but that doesn't mean we're not making some noise. Our fight for freedom deserves a spotlight, and we're making sure it gets one. Our story isn't just local; it's global, and we're spreading the word.
Word on the street is our offensive is losing steam, and there's some internal drama. Every battle has its rough patches, right? Our military is adapting, learning, and we're not giving up. Allies have our back, and their support isn't going anywhere.
Winter might throw us a curveball, but guess what? Ukrainians are resilient. We've faced worse, and we'll power through. Our unity is our strength, and we're not letting that go.
As for the talk of a stalemate, I'm not buying it. History's filled with examples of the little guy coming out on top, and we're no exception. Let's stick together, stay strong, and show the world that Ukraine will prevail!
Glory to Ukraine!
[Random Pseudonym]