Experiments With Anthropic's Claude 2 For Summarization, Event & Relation Extraction, NER & Q&A

Continuing our explorations of how various commercial LLMs perform on the chaotic conflicting cacophony that is global news, what does it look like to summarize and explore a television evening news broadcast using Anthropic's Claude 2?

We'll start with a CBC Evening News broadcast from July 5th that includes this synopsis of the story:

jericka: good evening to our viewers in the west, and thank you for joining us this wednesday night. i'm jericka duncan in for norah. we begin with the secret service leading an investigation into who brought cocaine into the white house. the surprising discovery of a bag of suspicious white powder initially caused a panic leading to the temporary closure of the building on sunday. the illegal drug was found near a visitors entrance for those taking a tour of the west wing. today lab tests came back positive for cocaine. president biden and the first family were at camp david at the time. cbs's weijia jiang starts us off from the white house. good evening. >> reporter: good evening, jericka. the white house says it's very important to president biden to get to the bottom of who brough% that illegal drug inside. but officials cautioned this could be a challenging investigation. tonight the leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. according to senior law enforcement officials. the discovery sunday nights put the white house on temporary lockdown until the substance was deemed 'not a threat.' the bag was discovered on the ground floor in a common area near the entry to the west wing. not far from the situation room. it was close to a set of storage cubbies where visitors leave their cell p phones duringng a . the secret service is leading the investigation looking at visitor logs and camera surveillance. they are camera systems they will go back as a part of the investigation and research. not every camera is trained on every body at every minute. so there will always be a possibility that this cocould go unanswered. reporteter: white house visitorsrs undergo background checks and go through magnetometers on site. but the secret service does not have k-9 units that target illegal drugs. instead, the dog search for explosives, biohazardous materials, and weapons. >> this episode shines a light on the fact that you can bring in illegal substances to the white house, so what is preventing a visitor from bringing an anthrax or something that is not magnetic into the white house? i totally understand the question, but it is under investigation. we will let the secret service do their job. i'm not going to get ahead of if or whens or changes. reporter: senator tom cotton, a republican from arkansas sent a letter to the director of the secret service to say that congress needs to know if the white house complex is not secure. and he asked for a plan to correct any security flaws. jericka: weijia jiang at the white house, thank you.

Let's try summarization using the prompt "Summarize this television news transcript":

Here is a summary of the key points from the news transcript:

– Cocaine was discovered in a bag near the visitors entrance in the West Wing of the White House on Sunday, leading to a temporary lockdown.

– The White House was empty at the time as President Biden was away at Camp David.

– Lab tests confirmed it was cocaine. The Secret Service is leading an investigation into who brought the illegal drugs inside.

– The main theory is that a visitor on a tour left the bag. Visitors undergo background checks but the Secret Service does not screen for drugs.

– The discovery raises concerns about security flaws as visitors could potentially bring other dangerous substances inside.

– Republican Senator Tom Cotton criticized the lack of White House security and asked the Secret Service for a plan to correct any issues.

How about a single paragraph ("Summarize this television news transcript in a single paragraph")?

• Cocaine was discovered in a bag near the visitor entrance in the West Wing of the White House on Sunday while President Biden was away, leading to a temporary lockdown. Lab tests confirmed it was cocaine, likely left by a visitor on tour as the Secret Service does not screen for drugs. The incident raises concerns about security vulnerabilities as visitors could potentially bring dangerous substances inside. An investigation is underway to determine who brought the illegal drugs inside, but Republican Senator Tom Cotton criticized the lack of security and demanded a plan from the Secret Service to correct any flaws exposed by this breach.
• A bag of cocaine was discovered near a visitors' entrance in the White House West Wing on Sunday, July 16th, leading to a temporary lockdown. Lab tests confirmed it was cocaine. The Secret Service is investigating the security breach, believing a visitor on a tour likely left the bag. The incident raises concerns that visitors could bring other dangerous substances into the White House. Republican Senator Tom Cotton criticized the lapse and asked the Secret Service for improved security measures.
• Cocaine was discovered in a bag near the visitors entrance to the West Wing of the White House on Sunday while President Biden was away, leading to a temporary lockdown. Lab tests confirmed it was cocaine, and the Secret Service has launched an investigation, with the leading theory being that a visitor on a tour left the bag. The incident raises concerns about security flaws allowing visitors to potentially bring dangerous substances inside. Republican Senator Tom Cotton criticized the lack of security and asked the Secret Service for a plan to correct any issues exposed by the discovery of drugs inside the White House.

And using our abstracting prompt ("write an abstract for this television news transcript"):

• A bag of cocaine was discovered near the visitors entrance in the White House West Wing on Sunday, July 16, 2023, triggering a temporary lockdown. With President Biden away at Camp David, the White House stood empty during the security breach. Lab results confirmed the suspicious white powder was cocaine, prompting an investigation by the Secret Service into how the illegal drugs entered the premises. The leading theory suggests a visitor on a public White House tour may have left the bag in a common area. While visitors undergo background checks, the Secret Service does not screen for drugs. The security lapse raises concerns that dangerous substances could be brought inside. Senator Tom Cotton criticized the flaws exposed by the drug discovery and requested the Secret Service outline a plan to correct any vulnerabilities in White House security.

And headlines ("Write a title for this television news transcript"):

Here are some potential title ideas for the television news transcript:

– Cocaine Found in White House Sparks Security Concerns
– White House Drug Discovery Leads to Lockdown and Investigation
– Secret Service Probes Security Lapse After Cocaine Found in White House
– Senator Calls for Security Review After Cocaine Discovered in White House
– White House Cocaine Discovery Raises Red Flags About Security Flaws
– Drugs and Security Breach at the White House

And a second time:

Here are some possible title ideas for the television news transcript:

– Cocaine Found in White House Sparks Security Concerns
– White House Lockdown After Cocaine Discovery During Biden Visit
– Secret Service Investigating Cocaine Found in White House West Wing
– Security Flaws Exposed After Cocaine Found in White House
– Senator Calls for Security Review After Drugs Found in White House
– White House Cocaine Discovery Raises Security Questions

And hashtags ("What are potential hashtags for this television news transcript"):

Here are some potential hashtags that could be used when sharing this television news transcript on social media:

#WhiteHouseCocaine
#CocaineInWhiteHouse
#WHLockdown
#SecretServiceInvestigation
#WhiteHouseSecurityFlaw
#WhiteHouseTours
#TomCotton
#BidenCampDavid
#SecurityBreach
#DrugsInWH
#WestWingCocaine
#WHDrugs
#WhiteHouseDrugs
#SecurityReview
#WhiteHouseVulnerability
#SSInvestigation
#BidenAway
#WHSecurity
#SecurityLapses
#WHCocaineDiscovery
#CocaineDiscovery
#DrugsInWhiteHouse

And again:

Here are some potential hashtags that could be used for this television news transcript:

#WhiteHouseCocaine
#WHLockdown
#SecretServiceInvestigation
#WhiteHouseSecurityFlaw
#CocaineInTheWhiteHouse
#WhiteHouseDrugs
#BidenOutOfTown
#SecurityBreach
#SenatorCotton
#DrugsInWH
#WhiteHouseTours
#SecurityReview
#WHSecurity
#DrugsFoundInWH
#WHCocaineDiscovery
#SecurityVulnerabilities
#WHVisitorCocaine
#WestWingCocaine
#WhiteHouseVisitorSecurity

And let's try sentiment ("What is the sentiment of this television news transcript"):

Based on the language and tone of the news transcript, the overall sentiment seems to be concern and alarm. Key phrases that indicate this include:

– "surprising discovery"
– "panic leading to the temporary closure"
– "illegal drug"
– "challenging investigation"
– "concerning security flaw"
– "shine a light on the fact that you can bring in illegal substances"
– "not secure"
– "security flaws"

The discovery of cocaine and the security breach are treated seriously, with words like "illegal", "challenging", "concerning", and "not secure" conveying worry. The Republican senator's criticism adds to the general sentiment of alarm. While the tone is serious, it is mostly factual rather than sensational. But the implications for White House security are presented as troubling and urgent. Overall, the transcript expresses apprehension about the breach and urgency to address the security weaknesses it revealed.

And again:

Based on the language and content of the news transcript, the overall sentiment seems to be:

Concerned and alarmed

The discovery of cocaine and the security breach at the White House is treated seriously, with words like "surprising", "panic", and "illegal" indicating the severity. The fact that lab tests were done and the Secret Service launched an investigation shows the level of concern.

There are also tones of criticism and urgency, with Senator Cotton calling out the "security flaws" and pushing for corrections. Phrases like "security lapse" and "vulnerabilities" reinforce the belief that the situation exposes problems that need to be addressed urgently.

While the news is delivered in an objective, factual manner, the choice of details highlights the alarming and problematic nature of drugs making their way into the White House. So the predominant sentiment is alarm and urgency around securing the White House.

And constrain the tone response to a categorical statement ("Classify as either negative or positive the tone of this television news transcript"):

I would classify the tone of this television news transcript as negative. Here are the reasons why:

– Words like "suspicious", "panic", and "illegal" used to describe the cocaine discovery convey a negative tone.

– The incident is referred to as a "security breach" and a "security lapse" which implies a failure in security.

– The transcript raises concerns about flaws and vulnerabilities in White House security, which highlights weaknesses.

– Senator Cotton's criticism of the lack of security adds a disapproving perspective.

– The discovery is treated as a serious matter requiring investigation, rather than a neutral or positive event.

– While the news delivery is objective, the choice of alarming details creates an overall negative tone regarding the implications of the drug discovery.

So in summary, the tone comes across as negative, conveying concern, alarm and urgency rather than neutrality or praise around the cocaine found in the White House. The language and details focus on the problematic aspects of the incident.

And again:

I would classify the tone of this television news transcript as negative.

Some reasons why it has a negative tone:

– Words like "surprising", "panic", "illegal", and "security breach" used to describe the incident imply it was a troubling and concerning event.

– The incident is referred to as a "security lapse" and said to expose "vulnerabilities", framing it as a failure.

– The discovery of drugs is described as "shining a light" on flaws and gaps in White House security. This suggests shortcomings were revealed.

– Senator Cotton's letter sharply criticizes the lack of security, using phrases like "not secure" which paint the situation negatively.

– The story raises questions about risks of other dangerous substances being brought in, worrying implications.

– The urgent call for the Secret Service to correct issues gives a sense things are not as they should be and need fixing.

Overall, the tone focuses on problematic aspects of the incident, security failures, and need for change rather than any positives, making it decidedly negative.

Let's try a machine-readable structured response ("For this television news transcript return a JSON object that has fields sentiment and explanation. Acceptable values for sentiment are Very Positive, Positive, Neutral, Negative or Very Negative. The explanation field contains text that explains the sentiment.") (Results were reformatted to JSON minified):

• {"sentiment":"Negative","explanation":"The transcript has an overall negative tone based on words like 'surprising', 'panic', and 'illegal' used to describe the security breach incident. Concerns are raised about vulnerabilities in White House security, with criticism directed at the lack of proper security screening. The urgent call for the Secret Service to address flaws also contributes to the negative sentiment."}
• {"sentiment":"Negative","explanation":"The transcript has an overall negative tone based on words like 'surprising', 'panic', 'illegal', and 'security breach' used to describe the troubling incident of drugs being found in the White House. Phrases such as 'security lapse', 'vulnerabilities', and a 'lack of security' frame the event as a failure revealing gaps and shortcomings. The urgent call for the Secret Service to correct issues also gives the transcript a negative sense that things need fixing."}

What about a machine-readable categorical example ("Return a JSON object with Category and Explanation. For Category classify the text as one of the following categories: Political, Crime, War, Fashion, Tourism. The Explanation field contains text that explains the category selection."):

• {"Category":"Crime","Explanation":"The transcript primarily covers an incident involving illegal drugs (cocaine) being discovered in the White House. The focus is on the Secret Service investigation into how the drugs got into the secure government building, indicating laws were broken. Discussion of security concerns and flaws also relates to criminal activity and preventing crime. While politics are touched on with mention of the president and a senator, the main thrust examines a criminal act and investigation, so it is best categorized as a Crime news story."}
• {"Category":"Crime","Explanation":"The news transcript primarily covers an incident involving illegal drugs being discovered at the White House. It focuses on the investigation into how cocaine got into the building, theories on who brought it inside, and the security flaws that allowed criminal activity and prohibited substances to breach White House security. With the main subject being an illegal act and the law enforcement response, the clear category is Crime."}
• {"Category":"Crime","Explanation":"The news transcript primarily covers an incident involving illegal drugs being discovered at the White House. Details are provided about cocaine being found, the Secret Service investigating, and concerns over security flaws that allowed criminal activity to occur. The focus is on the illegal nature of the drugs and the implications for security procedures and law enforcement at the presidential residence. Given the centrality of the illegal drug possession and procedures related to criminal investigations, the category of 'Crime' best fits this news transcript."}

Note how it consistently retains its "Crime" categorization.

What about another categorical example, this time with three custom categories where we provide a specific definition that partially overlaps and rely on the LLM to decide the topical boundaries ("Return a JSON object with Category and Explanation. For Category classify the text as one of the following categories: Topic1 or Topic2 or Topic3. The Topic1 category is defined as relating to US domestic politics. The Topic2 category is defined as relating to the White House or the current presidential administration. The Topic3 category relates to any criminal activity in the US. The Explanation field contains text that explains the category selection."):

• {"Category":"Topic2","Explanation":"The news transcript is most closely aligned with Topic2, relating to the White House and current presidential administration. While there are elements of US domestic politics and criminal activity, the primary focus is on an incident occurring at the White House involving illegal drugs being found. The bulk of the transcript covers the discovery of cocaine at the White House visitors entrance, the investigation by the Secret Service, concerns about security flaws at the White House, and criticism of White House security procedures. As the main subject relates to a happening at the current presidential residence and administration, categorizing this transcript as Topic2 – White House/current presidential administration – is most appropriate."}
• {"Category":"Topic2","Explanation":"The news transcript is primarily focused on an incident relating to the White House, including details about cocaine being discovered in the West Wing, the Secret Service investigation, and concerns about security procedures specifically at the presidential residence. While there are connections to US domestic politics and criminal activity, the main subject is the breach of security and protocols at the White House itself during the current Biden administration. Therefore, this transcript is best categorized as 'Topic2' which is defined as relating to the White House or current presidential administration."}
• {"Category":"Topic2","Explanation":"The news transcript focuses on an incident relating to illegal drugs being found at the White House during the current Biden administration presidency. Details are provided about the discovery occurring in the West Wing, the investigation by the Secret Service which protects the president, and concerns over security flaws at the White House complex. Since the main subject is an event centered around the White House under the current administration, rather than broader US politics or criminal activity, the category 'Topic2' concerning the White House and current presidential administration is the best fit."}

How about a NER task ("What are the locations mentioned in the text?"):

The locations mentioned in the text are:

– White House (mentioned multiple times)
– West Wing of the White House (mentioned multiple times)
– Visitors entrance of the West Wing
– Near the situation room in the West Wing
– Storage cubbies where visitors store phones during tours
– Camp David (presidential retreat where Biden was visiting)

So the main locations discussed are the White House itself, specifically the West Wing area, as well as Camp David. Other minor details provide specificity about where in the West Wing the drugs were found. But the White House and its West Wing are the clearly dominant locations referenced.

And again:

The locations mentioned in the text are:

– White House (mentioned multiple times)
– West Wing of the White House (mentioned multiple times)
– Visitors entrance of the West Wing
– Near the situation room in the West Wing
– Camp David

The key locations emphasized are the White House complex, specifically the West Wing, where the cocaine was discovered near the visitors entrance. Camp David is also mentioned as the location where President Biden was during the incident.

Third time:

The locations mentioned in the text are:

– White House (mentioned several times)
– West Wing of the White House (mentioned several times)
– Visitors entrance for the West Wing
– Storage cubbies where visitors leave phones during tours
– Situation Room in the White House
– Camp David (presidential retreat where Biden was visiting)

So the main locations discussed are the White House itself, specifically the West Wing area, as well as Camp David where President Biden was visiting at the time of the incident. Other minor details provide locations within the White House like the visitors entrance, storage cubbies, and Situation Room. But the focus is on the White House in general.

Fourth time:

The locations mentioned in the text are:

– White House
– West Wing (of the White House)
– Situation Room (in the White House)
– Camp David

Fifth time:

The locations mentioned in the text are:

– White House (mentioned multiple times)
– West Wing of the White House (mentioned multiple times)
– Visitors entrance of the West Wing
– Near the situation room in the West Wing
– Camp David

Similar to other LLMs, entity extraction is more unstable across runs.

What about structured entity extraction to machine readable format, which replicates a real-world entity extraction use case ("Extract all of the entities, their types and descriptions in the following text into a JSON object.")?

• {"Entities":[{"Entity":"Jericka Duncan","Type":"Person","Description":"News anchor for the show"},{"Entity":"Norah","Type":"Person","Description":"News anchor Jericka Duncan is filling in for"},{"Entity":"Secret Service","Type":"Organization","Description":"Leading investigation into cocaine found in White House"},{"Entity":"White House","Type":"Location","Description":"Where cocaine was found near visitors entrance"},{"Entity":"West Wing","Type":"Location","Description":"Part of White House where cocaine found"},{"Entity":"President Biden","Type":"Person","Description":"President who was away at Camp David when incident occurred"},{"Entity":"Camp David","Type":"Location","Description":"Presidential retreat where Biden was during White House incident"},{"Entity":"Weijia Jiang","Type":"Person","Description":"CBS News reporter covering story from White House"},{"Entity":"Tom Cotton","Type":"Person","Description":"Republican Senator who sent letter about security concerns"}]}
• {"Entities":[{"Text":"White House","Type":"Location","Description":"The official residence and workplace of the President of the United States"},{"Text":"West Wing","Type":"Location","Description":"The western part of the White House complex where the Oval Office and offices of the President's senior staff are located"},{"Text":"visitors entrance","Type":"Location","Description":"An entrance to the White House for public tours and visitors"},{"Text":"situation room","Type":"Location","Description":"A conference room complex in the West Wing basement where the President and advisors can monitor and deal with crises"},{"Text":"Camp David","Type":"Location","Description":"The country retreat for the President of the United States located in rural Maryland"},{"Text":"Sunday","Type":"Date","Description":"The day of the week"},{"Text":"President Biden","Type":"Person","Description":"The current President of the United States"},{"Text":"cocaine","Type":"Substance","Description":"An illegal stimulant drug derived from coca plants"},{"Text":"Secret Service","Type":"Organization","Description":"The US federal law enforcement agency responsible for the protection of senior political leaders"},{"Text":"Senator Tom Cotton","Type":"Person","Description":"A Republican United States Senator from Arkansas"}]}
• json"`{"entities":[{"text":"White House","type":"Location","description":"The official residence and workplace of the President of the United States"},{"text":"West Wing","type":"Location","description":"The western part of the White House complex where the Oval Office and offices of the president's senior staff are located"},{"text":"visitors entrance","type":"Facility","description":"An entrance to the White House used by non-staff visitors taking tours"},{"text":"situation room","type":"Facility","description":"A conference room complex in the West Wing basement where the president and advisors can monitor and deal with crises"},{"text":"Camp David","type":"Location","description":"The country retreat for the president located in rural Maryland"},{"text":"Sunday","type":"Date","description":"The day the cocaine was discovered at the White House"},{"text":"president biden","type":"Person","description":"The current and 46th president of the United States"},{"text":"first family","type":"Person","description":"Informal term referring to the immediate family members of the president"},{"text":"weijia jiang","type":"Person","description":"CBS News reporter covering the White House"},{"text":"tom cotton","type":"Person","description":"Republican Senator from Arkansas who commented on the incident"},{"text":"director of the secret service","type":"Position","description":"The head of the U.S. Secret Service"}]}"`
• {"Entities":[{"Text":"jericka","Type":"Person","Description":"Name of news anchor"},{"Text":"norah","Type":"Person","Description":"Name of another news anchor"},{"Text":"white house","Type":"Location","Description":"Official residence of the US President"},{"Text":"west wing","Type":"Location","Description":"Western part of the White House where Oval Office is located"},{"Text":"president biden","Type":"Person","Description":"Current President of the United States"},{"Text":"first family","Type":"Person","Description":"President's immediate family members"},{"Text":"camp david","Type":"Location","Description":"Country retreat for the US President"},{"Text":"weijia jiang","Type":"Person","Description":"CBS News reporter"},{"Text":"tom cotton","Type":"Person","Description":"Republican Senator from Arkansas"},{"Text":"secret service","Type":"Organization","Description":"US federal law enforcement agency"}]}
• {"Entities":[{"Entity":"White House","Type":"Location","Description":"The official residence and workplace of the President of the United States"},{"Entity":"West Wing","Type":"Location","Description":"The western part of the White House complex where the Oval Office and offices of the President's senior staff are located"},{"Entity":"visitors entrance","Type":"Location","Description":"The entrance used by visitors taking tours of the White House"},{"Entity":"situation room","Type":"Location","Description":"Secure conference room in the West Wing where the President and advisors can monitor and deal with crises domestically and abroad"},{"Entity":"Camp David","Type":"Location","Description":"Country retreat for the President located in Maryland."},{"Entity":"cocaine","Type":"Drug","Description":"Illegal stimulant drug derived from coca leaves"},{"Entity":"Secret Service","Type":"Organization","Description":"U.S. federal law enforcement agency responsible for the protection of senior politicians and officials"},{"Entity":"Senator Tom Cotton","Type":"Person","Description":"Republican Senator from Arkansas"}]}
• json"`{"Entities":[{"Entity":"Cocaine","Type":"Drug","Description":"An illegal stimulant drug that was found in the White House."},{"Entity":"White House","Type":"Location","Description":"The official residence and workplace of the President of the United States, where the cocaine was found."},{"Entity":"West Wing","Type":"Location","Description":"The western part of the White House complex, where the cocaine was specifically found near a visitors entrance."},{"Entity":"Secret Service","Type":"Organization","Description":"The federal law enforcement agency leading the investigation into the cocaine found in the White House."},{"Entity":"President Biden","Type":"Person","Description":"The current President of the United States who was away at Camp David when the cocaine was found at the White House."},{"Entity":"Camp David","Type":"Location","Description":"The country retreat for the President located in Maryland, where President Biden was visiting when the cocaine was discovered."},{"Entity":"Tom Cotton","Type":"Person","Description":"The Republican Senator from Arkansas who sent a letter criticizing White House security in light of the cocaine discovery."}]}"`
• {"entities":[{"text":"White House","type":"Location","description":"Official residence and workplace of the President of the United States"},{"text":"West Wing","type":"Location","description":"Western part of the White House complex where Oval Office is located"},{"text":"Oval Office","type":"Location","description":"Official office of the President located in the West Wing"},{"text":"President","type":"Person","description":"Head of state of the United States"},{"text":"United States","type":"Location","description":"Country in North America consisting of 50 states"},{"text":"North America","type":"Location","description":"Continent bordering the Arctic Ocean and Atlantic Ocean"},{"text":"50","type":"Number","description":"Total number of U.S. states"}]}
• {"entities":[{"text":"New York Times","type":"Organization","description":"American daily newspaper based in New York City"},{"text":"New York City","type":"Location","description":"Most populous city in the United States"},{"text":"United States","type":"Location","description":"Country in North America consisting of 50 states"},{"text":"North America","type":"Location","description":"Continent bordering the Arctic Ocean and Atlantic Ocean"},{"text":"50","type":"Number","description":"Total number of U.S. states"},{"text":"Pulitzer Prize","type":"Award","description":"Prestigious award for achievements in newspaper journalism, literature and musical composition"},{"text":"The New York Times Company","type":"Organization","description":"Publicly traded American media company which publishes The New York Times"}]}
• {"entities":[{"text":"NASA","type":"Organization","description":"U.S. space agency"},{"text":"U.S.","type":"Location","description":"United States of America"},{"text":"Mars","type":"Location","description":"Planet in the solar system"},{"text":"solar system","type":"Location","description":"Gravitational system that includes the sun and the objects orbiting it"},{"text":"sun","type":"Celestial","description":"The star at the center of the solar system"},{"text":"Perseverance","type":"Vehicle","description":"Mars rover designed by NASA"},{"text":"Jezero Crater","type":"Location","description":"Crater on Mars where the Perseverance rover landed"}]}
• {"entities":[{"text":"White House","type":"Location","description":"Official residence and workplace of the President of the United States"},{"text":"West Wing","type":"Location","description":"Western part of the White House complex where Oval Office is located"},{"text":"visitors entrance","type":"Facility","description":"Entrance to the White House used by non-staff visitors taking tours"},{"text":"situation room","type":"Location","description":"Conference room complex in the West Wing basement for monitoring crises"},{"text":"Sunday","type":"Date","description":"The day the cocaine was discovered at the White House"},{"text":"Secret Service","type":"Organization","description":"U.S. federal law enforcement agency responsible for protecting the President"},{"text":"cocaine","type":"Drug","description":"Illegal stimulant drug derived from coca leaves"}]}

The results here show the same instability of other LLMs in which results vary considerably between runs. Two outputs include the triple-tick formatting indicator we've seen produced by other LLMs, though this could simply reflect a failure of the UI to translate formatting codes on display. While the majority of names were converted to upper-case, one output contains all lower-case and several outputs contain a mixture of upper and lower case names, which would require downstream reprocessing.

Recall that PaLM 2 identified the names of Jericka Duncan and Weijia Jiang only a single time out of eight runs and actually respelled Duncan's name as "Jerrika". Here Claude identifies Duncan's name twice out of ten runs and Jiang's name three out of ten times.

It is unclear what happened with runs 8 and 9 – the LLM appears to have lost attention and hallucinated entirely different stories from whole cloth. Yet, in the tenth run it returned again.

What about a Q&A ("Where was the cocaine discovered?"):

Based on the details in the news transcript, the cocaine was discovered in the West Wing of the White House, specifically near the visitors entrance and close to a set of storage cubbies where visitors leave their cell phones during tours.

The relevant lines from the transcript are:

– "i'm jericka duncan in for norah. we begin with the secret service leading an investigation into who brought cocaine into the white house."

– "the illegal drug was found near a visitors entrance for those taking a tour of the west wing."

– "according to senior law enforcement officials. the discovery sunday nights put the white house on temporary lockdown until the substance was deemed 'not a threat.' the bag was discovered on the ground floor in a common area near the entry to the west wing."

– "not far from the situation room. it was close to a set of storage cubbies where visitors leave their cell p phones duringng a tour."

So in summary, the cocaine was discovered in the West Wing of the White House near where visitors enter and store belongings during tours.

Or another question ("Who brought the cocaine into the White House according to the text?")?

The text does not definitively state who brought the cocaine into the White House. However, it provides some clues and details:

– The transcript mentions that the Secret Service is leading an investigation into "who brought cocaine into the white house."

– It states the "leading theory" is that "someone on a white house tour may have left a small bag of cocaine in the west wing."

– The cocaine was discovered near the visitors entrance and tour area of the West Wing.

– Visitors undergo background checks but are not screened for drugs.

So while the person responsible is not named, the main theory based on the location and details provided is that a visitor on a public White House tour likely left the bag of cocaine found in the West Wing. The investigation seems to be focused on identifying which visitor may have brought and left the drugs.

What about event/relation extraction ("Return a JSON object containing all of the events described in the text.")?

• {"events":[{"event":"Discovery of cocaine","description":"A bag of suspicious white powder, later confirmed to be cocaine, was discovered in the White House West Wing near the visitors entrance."},{"event":"White House lockdown","description":"The discovery of the powder led to a temporary lockdown of the White House complex until the substance was analyzed."},{"event":"Lab tests","description":"Laboratory tests were conducted which came back positive for cocaine."},{"event":"Secret Service investigation","description":"The Secret Service launched an investigation into who brought the illegal drugs into the White House."},{"event":"Senator Cotton's letter","description":"Senator Tom Cotton sent a letter criticizing White House security and requesting a plan to correct flaws."}]}
• {"events":[{"event":"Discovery of cocaine","details":"A bag of suspicious white powder was found near the visitors entrance in the West Wing of the White House"},{"event":"White House lockdown","details":"The White House was temporarily placed on lockdown after the discovery of the powder until it was analyzed"},{"event":"Lab tests","details":"The powder was lab tested and confirmed to be cocaine"},{"event":"Secret Service investigation","details":"The Secret Service launched an investigation into who brought the cocaine into the White House"},{"event":"Senator Cotton's letter","details":"Senator Cotton sent a letter criticizing White House security and requesting a plan to correct flaws"}]}
• {"events":[{"event":"Cocaine found in White House","details":"A bag of cocaine was discovered near the visitors entrance in the West Wing on Sunday July 16, 2023."},{"event":"White House temporarily locked down","details":"The White House was put on temporary lockdown after the discovery of the suspicious powder until it was analyzed."},{"event":"Substance tested and confirmed as cocaine","details":"Lab tests were conducted which confirmed the substance found was cocaine."},{"event":"Secret Service investigation launched","details":"The Secret Service started an investigation into who brought the illegal drugs into the White House."},{"event":"Senator Cotton criticizes White House security","details":"Senator Tom Cotton sent a letter saying security is not adequate and calling for a plan to correct flaws."}]}

Here we can see reasonable consistency across the results, with a similar event breakdown, though different wording.

What about changing the wording from "events" to "relationships" ("Return a JSON object containing all of the relationships described in the text."):

• {"relationships":[{"person1":"Secret Service","person2":"White House","relationship":"The Secret Service is responsible for security at the White House."},{"person1":"Secret Service","person2":"President Biden","relationship":"The Secret Service protects President Biden and his family."},{"person1":"White House","person2":"President Biden","relationship":"The White House is the official residence and workplace of President Biden."},{"person1":"Senator Cotton","person2":"Secret Service","relationship":"Senator Cotton sent a letter to the Secret Service director about the security lapse."},{"person1":"White House visitors","person2":"cocaine","relationship":"A visitor likely brought cocaine into the White House."}]}
• {"relationships":[{"person1":"President Biden","person2":"First Family","relationship":"President Biden is part of the First Family"},{"person1":"Weijia Jiang","person2":"CBS News","relationship":"Weijia Jiang is a reporter for CBS News"},{"person1":"Tom Cotton","person2":"U.S. Senate","relationship":"Tom Cotton is a Senator in the U.S. Senate"},{"person1":"Director of Secret Service","person2":"Secret Service","relationship":"The Director leads the Secret Service organization"}]}
• {"relationships":[{"person1":"Secret Service","person2":"White House","relationship":"provides security for"},{"person1":"Secret Service","person2":"President Biden","relationship":"protects"},{"person1":"President Biden","person2":"White House","relationship":"lives and works at"},{"person1":"Senator Cotton","person2":"Secret Service","relationship":"sent letter to"},{"person1":"Weijia Jiang","person2":"CBS News","relationship":"reporter for"}]}
• {"relationships":[{"person1":"Secret Service","person2":"White House","relationship":"is responsible for security of"},{"person1":"Secret Service","person2":"President Biden","relationship":"protects"},{"person1":"Visitors","person2":"White House","relationship":"take tours of"},{"person1":"Senator Cotton","person2":"Secret Service","relationship":"sent letter to"},{"person1":"Senator Cotton","person2":"White House security","relationship":"criticized"}]}

Like with our NER task, we can see the high variability with a lack of consistency in output.

What if we give it a bit more instruction ("Return a JSON object containing all of the relationships described in the text. Each relationship should have three fields: Actor 1, Relationship, Actor 2. Actor 1 is the first entity, Actor 2 is the second entity and Relationship is the relationship between them."):

• {"relationships":[{"Actor 1":"Secret Service","Relationship":"is investigating","Actor 2":"cocaine found in White House"},{"Actor 1":"White House","Relationship":"was temporarily locked down","Actor 2":"after cocaine discovery"},{"Actor 1":"Visitors","Relationship":"undergo","Actor 2":"background checks"},{"Actor 1":"Senator Cotton","Relationship":"criticized","Actor 2":"White House security"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service director"}]}
• {"relationships":[{"Actor 1":"Secret Service","Relationship":"is investigating","Actor 2":"cocaine found in White House"},{"Actor 1":"Visitors","Relationship":"take","Actor 2":"tours of the White House"},{"Actor 1":"Senator Cotton","Relationship":"criticized","Actor 2":"White House security"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service director"},{"Actor 1":"President Biden","Relationship":"was away at","Actor 2":"Camp David"}]}
• {"relationships":[{"Actor 1":"Secret Service","Relationship":"is investigating","Actor 2":"cocaine found in White House"},{"Actor 1":"Visitors","Relationship":"take","Actor 2":"tours of White House"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service"},{"Actor 1":"Senator Cotton","Relationship":"criticized","Actor 2":"White House security"},{"Actor 1":"White House","Relationship":"was on","Actor 2":"temporary lockdown"}]}

Again, quite a bit of variation between runs.

What if we change the formulation back to events ("Return a JSON object containing all of the events described in the text. Each event should have three fields: Actor 1, Relationship, Actor 2. Actor 1 is the first entity, Actor 2 is the second entity and Relationship is the relationship between them.")?

• {"events":[{"Actor 1":"Thieves","Relationship":"broke into","Actor 2":"the museum"},{"Actor 1":"Security guard","Relationship":"spotted","Actor 2":"the thieves"},{"Actor 1":"Police","Relationship":"arrived at","Actor 2":"the museum"},{"Actor 1":"Police","Relationship":"arrested","Actor 2":"the thieves"},{"Actor 1":"Museum curator","Relationship":"assessed","Actor 2":"the damage and stolen items"}]}
• Unfortunately there are no events described in the given text. The text does not contain any actions or occurrences. Since there are no events to extract, the JSON object will be empty.
• {"events":[{"Actor 1":"Secret Service","Relationship":"leading investigation into","Actor 2":"who brought cocaine into White House"},{"Actor 1":"Bag of cocaine","Relationship":"discovered","Actor 2":"near visitors entrance in West Wing"},{"Actor 1":"White House","Relationship":"temporarily closed","Actor 2":""},{"Actor 1":"Lab tests","Relationship":"came back positive for","Actor 2":"cocaine"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service Director"}]}
• {"events":[{"Actor 1":"Secret Service","Relationship":"leading investigation into","Actor 2":"who brought cocaine into White House"},{"Actor 1":"Bag of cocaine","Relationship":"discovered","Actor 2":"near visitors entrance in West Wing"},{"Actor 1":"White House","Relationship":"temporarily closed","Actor 2":null},{"Actor 1":"Lab tests","Relationship":"came back positive for","Actor 2":"cocaine"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service director"}]}
• Unfortunately there are no events described in the given text. The text does not contain any actions or occurrences. Since there are no events to extract, the JSON object will be empty.
• {"events":[{"Actor 1":"Secret Service","Relationship":"leading investigation into","Actor 2":"who brought cocaine into White House"},{"Actor 1":"Bag of cocaine","Relationship":"discovered","Actor 2":"near visitors entrance in West Wing"},{"Actor 1":"White House","Relationship":"temporarily closed","Actor 2":null},{"Actor 1":"Lab tests","Relationship":"came back positive for","Actor 2":"cocaine"},{"Actor 1":"Senator Cotton","Relationship":"sent letter to","Actor 2":"Secret Service director"}]}

Interestingly, this task posed the most challenges to the LLM, causing it to hallucinate a story out of whole cloth for one response and generating no results twice. However, in its three related responses, it was remarkably consistent.

Overall, Claude 2's responses are fairly on par with those we've seen from other major LLMs. It generates reasonable results across all tasks, but does hallucinate and generate inconsistent results across runs, as all contemporary LLMs are prone to do. It did generate better results than several other major LLMs in a few cases.