Experiments In Meme Tracking: Dividing Transcripts Into Stories – Part 4 & The Impact Of Capitalization, Punctuation & Hallucination

 August 13, 2023

Continuing our meme tracking series, let's take a closer look at the general task of summarization/distillation. In particular, let's take a closer look at capitalization, punctuation and hallucination in our summaries.

Let's start with a summarization request for one of the passages that we struggled with in our previous episode:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
  "instances": [
    { "prompt": "Summarize in one paragraph the following text:\n\n   commercial pilots radioing in. >> we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. >> reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. >> this thing is up in the sky. >> what the heck is that? >> that thing is not the moon. >> any help would be appreciated. >> reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. >> we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. >> reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. >> any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. >> reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. >> we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. >> reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. windchill ever recorded in the u.s., including alaska. snow squalls along with that arctic chill. a time lapse from vermont. a cold emergency in place in boston. this is through sunday now. rob marciano standing by to time this out, and abc\'s victor oquendo in boston where the wind chill is 13 below zero right now. >> reporter: tonight, millions of families are in the midst of what the national weather service is calling a \\"once in a generation\\" arctic blast. >> we\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. >> reporter: parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. >> we\'re going to load food and stuff in here. >> reporter: and outreach teams hitting the streets to care for the needy. >> if anyone is out and sees someone else who might be in distress, please call 911. >> reporter: doctors say hypothermia can set in in just minutes. >> this weather can be deadly. it\'s life threatening. this is something that people have to really take seriously. >> reporter: the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. david? >> victor oquendo in boston. victor, thank you. let\'s get to rob marciano who\'s timing this out for us tonight. hey, rob. >> reporter: hi, david. if you live in the northeast, you\'ve felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it\'s no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that\'s for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that\'s an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we\'ll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it\'s just a matter of survival for the next 36 hours. david? >> whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we\'re going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here\'s our chief correspondent matt gutman from california tonight. >> reporter: tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him."}
  ],
  "parameters": {
    "temperature": 0.0,
    "maxOutputTokens": 1024,
    "topK": 40,
    "topP": 0.95,
  }
}' > O; cat O | jq -r .predictions[0].content

Let's try this with a temperature of 0.0. Strangely the LLM does not correct the casing of the text. Importantly, it fixates on only the last story and hallucinates details of the story including the names:

the attack happened on the pacific coast highway in the city of monterey. police say the victim, 54-year-old gary riese, was riding his bike when he was hit by a car. the driver, 23-year-old vincent trent johnson, then got out of his car and stabbed riese multiple times. good Samaritans stopped to help and held johnson down until police arrived. johnson is now in custody and facing charges of murder and hit-and-run. riese was an e.r. doctor at monterey bay hospital. he was also a volunteer firefighter and a member of the local cycling club. his death has sent shockwaves through the community.

With a temperature of 0.2 it still fixates and hallucinates:

  • it happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good samaritans stopped the driver and held him until police arrived. the doctor died at the scene. the driver is in custody. police have not released his name or a motive.
  • the attack happened on the pacific coast highway in the city of monterey. police say the victim, 54-year-old gary riess, was riding his bike when he was hit by a car. the driver, 23-year-old vincent trent johnson, then got out of his car and stabbed riess multiple times. good Samaritans stopped to help and held johnson down until police arrived. johnson is now in custody and facing charges of murder and hit-and-run. riess was a well-known and respected doctor who worked at monterey community hospital for more than 20 years. he is survived by his wife and two children.
  • the attack happened on the pacific coast highway in the city of camarillo. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good Samaritans stopped the driver and held him until police arrived. the doctor was taken to a hospital where he died. the driver is in custody. police have not released his name or a motive for the attack.
  • the attack happened on the pacific coast highway in the city of camarillo. police say the victim, identified as 54-year-old thomas mahan, was riding his bike when he was hit by a car. the driver, identified as 23-year-old ian watson, then got out of his car and stabbed mahan multiple times. good Samaritans stopped watson and held him until police arrived. mahan was taken to a hospital, where he died. watson is being held on suspicion of murder. police have not released a motive for the attack.
  • it happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good Samaritans stopped the driver and held him until police arrived. the doctor was taken to a hospital where he died. the driver is in custody. police have not released his name or a motive.

Temperature 0.4 continues this trend of fixation and hallucination. However, in one of the stories it did include some detail about the balloon:

  • it happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor, identified as 54-year-old thomas kron, was riding his bike when he was hit by a car. police say the driver, identified as 23-year-old richard ramirez, got out of his car and stabbed kron multiple times. good samaritans stopped ramirez and held him until police arrived. ramirez is now in custody. he's been charged with murder. police have not released a motive for the attack. kron was a beloved member of the laguna beach community. he worked as an e.r. doctor at st. john's hospital for more than 20 years. his colleagues are devastated by his death. >> he was a wonderful physician, a wonderful colleague, and a wonderful friend. he will be deeply missed.
  • the attack happened on the pacific coast highway in the city of monterey. police say the victim, 51-year-old gary riess, was riding his bike when he was hit by a car. the driver, 43-year-old steven matthew campbell, then got out of his car and stabbed riess multiple times. good Samaritans intervened and held campbell down until police arrived. riess was taken to a hospital where he died. campbell is now in police custody and is facing charges of murder and hit-and-run. A large Chinese balloon that entered US airspace over Alaska and then flew into Canada before dipping down over Montana has been met with mixed reactions. While some, including Montana Republican Congressman Ryan Zinke, have called for the balloon to be shot down, the Pentagon has said that it does not currently pose a physical or military risk to people on the ground and is continuing to monitor and review options. China, for its part, insists that the balloon is not a spy vessel but a civilian airship used for research, mainly meteorological purposes.
  • the attack happened on the pacific coast highway in the city of camarillo, just outside of los angeles. the doctor, identified as 54-year-old thomas kron, was riding his bike when he was hit by a car. police say the driver, identified as 23-year-old ian watson, got out of his car and stabbed kron multiple times. good Samaritans stopped to help and held watson down until police arrived. kron was taken to a hospital where he died. watson is being held on suspicion of murder. police have not released a motive for the attack.
  • the attack happened on the pacific coast highway in the city of monterey. police say the victim, identified as 54-year-old gary ries, was riding his bike when he was hit by a car. the driver, identified as 23-year-old vincent trent johnson, then got out of his car and stabbed ries multiple times. good Samaritans stopped to help and held johnson down until police arrived. johnson is now in custody and facing charges of murder and hit-and-run. ries was a well-respected e.r. doctor at monterey memorial hospital. he had worked there for more than 20 years. his colleagues are devastated by his death. >> he was a wonderful physician, a wonderful person, and he will be deeply missed.

Max temperature of 0.99:

  • it happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out and stabbed him multiple times. good samaritans stopped the driver and held him down until police arrived. the doctor died at the scene. the driver is in custody. the motive for the attack is unknown. the victim has been identified as 54-year-old thomas krug. he was an e.r. doctor at st. johns hospital in santa monica. in a statement, the hospital said krug was a beloved member of the st. johns family. they called him a dedicated physician who provided compassionate care to his patients. the hospital said it is deeply saddened by his tragic death.
  • it happened on the pacific coast highway in the city of camarillo. the doctor was riding his bike when he was hit. police say the driver got out of his car and stabbed the doctor multiple times. good Samaritans stopped to help and held the driver down until police arrived. the doctor died at the scene. the driver was taken to a hospital for treatment. police have not released a motive in the attack. the doctor has been identified as 54-year-old thomas ian smith. he worked at st. johns hospital in oxnard. the hospital released a statement saying, quote, "we are deeply saddened by the tragic death of our colleague, dr. smith. he was a beloved member of our st. johns family and will be greatly missed. our hearts go out to his wife and children."
  • it happened on the pacific coast highway in the wealthy community of laguna beach.
    police say the driver, a 32-year-old man, is in custody. they have not released his name or a motive. the victim has been identified as 54-year-old thomas bauer. he was a well-known e.r. doctor at st. john's hospital in santa monica. witnesses say the driver of a silver sedan intentionally hit bauer from behind as he was riding his bicycle on the shoulder of the road. the driver then got out of his car and stabbed bauer multiple times. good Samaritans stopped their cars and held the driver down until police arrived. bauer was pronounced dead at the scene. the driver is being held on $1 million bail. this is a developing story. we'll have more information as it becomes available.
  • it happened on the pacific coast highway in the city of camarillo. witnesses tell us the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good samaritans stopped the suspect and held him until police arrived. the doctor was taken to a hospital where he died. the suspect is in custody. police have not released his name or a motive. this is a developing story. we'll have more information as it becomes available.

What if we remove the ">>" speaker change notations from the transcript and use a temperature of 0.0?

the attack happened on the pacific coast highway in the city of monterey. police say the driver, 32-year-old vincent trent pugh, was arrested and booked into jail on suspicion of murder. the victim has been identified as 54-year-old gary ries. he was a doctor at monterey memorial hospital. police say pugh was driving erratically when he hit ries, who was riding his bike. pugh then got out of his car and stabbed ries multiple times. good Samaritans stopped pugh and held him until police arrived. ries was pronounced dead at the scene. pugh is scheduled to appear in court on monday.

What if we get rid of the speaker notation of "reporter:"? This yields the following prompt:

Summarize in one paragraph the following text:\n\n commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. windchill ever recorded in the u.s., including alaska. snow squalls along with that arctic chill. a time lapse from vermont. a cold emergency in place in boston. this is through sunday now. rob marciano standing by to time this out, and abc\'s victor oquendo in boston where the wind chill is 13 below zero right now. tonight, millions of families are in the midst of what the national weather service is calling a \\"once in a generation\\" arctic blast. we\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. we\'re going to load food and stuff in here. and outreach teams hitting the streets to care for the needy. if anyone is out and sees someone else who might be in distress, please call 911. doctors say hypothermia can set in in just minutes. this weather can be deadly. it\'s life threatening. this is something that people have to really take seriously. the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. david? victor oquendo in boston. victor, thank you. let\'s get to rob marciano who\'s timing this out for us tonight. hey, rob. hi, david. if you live in the northeast, you\'ve felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it\'s no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that\'s for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that\'s an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we\'ll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it\'s just a matter of survival for the next 36 hours. david? whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we\'re going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here\'s our chief correspondent matt gutman from california tonight. tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.

Unfortunately, we still have hallucination:

the attack happened on the pacific coast highway in the city of monterey. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening, everyone. this is a developing story. we're learning new details about a deadly road rage attack in california. a 32-year-old emergency room doctor is dead after police say he was hit by a car and then stabbed multiple times. it happened on the pacific coast highway in monterey. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. the doctor was taken to a hospital where he died. police have arrested a suspect in the case. they say the suspect is a 32-year-old man from monterey. he's being held on suspicion of murder. we'll continue to follow this story and bring you updates as they become available.

Same prompt with temperature of 0.2. Notably we still see some hallucination (Laguna Beach), but no longer the hallucination of person names:

  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening, david. this is a very disturbing story. we're still learning details, but we know that the doctor was riding his bike on the pacific coast highway when he was hit by a car. the driver then got out of his car and stabbed the doctor multiple times. the doctor was taken to a local hospital where he died from his injuries. police have arrested a suspect in the case. the suspect is a 33-year-old man from laguna beach. he is being held on suspicion of murder. we're still learning about the motive for this attack, but police say they are not ruling out road rage as a factor. this is a very tragic story, and our hearts go out to the victim's family and friends.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening, david. this is a very disturbing story. we're still learning details, but we know that the doctor was riding his bike on the pacific coast highway when he was hit by a car. the driver then got out of his car and stabbed the doctor multiple times. the doctor was taken to a local hospital where he died from his injuries. police have arrested a suspect in the case. he is being held on suspicion of murder. we're working to learn more about this story and will bring you updates as they become available.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening. this is a developing story. we're working to get you the latest details. stay with abc news. A large Chinese balloon that entered US airspace over Alaska and then flew into Canada before dipping down over Montana has been met with mixed reactions. While some, including Montana Republican Congressman Ryan Zinke, have called for the balloon to be shot down, the Pentagon has said that it does not currently pose a physical or military risk to people on the ground. The White House has also said that President Biden agreed with the Pentagon's strong recommendation that the balloon not be shot down. China, for its part, insists that the balloon is not a spy vessel but a civilian airship used for research, mainly meteorological purposes. The Pentagon has flatly rejected that claim.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike home from work when he was hit and then stabbed. police say the driver, a 33-year-old man, is in custody. they have not released a motive. the victim's family is asking for privacy.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening. this is a developing story. we're working to get you more information. stay with abc news. A large Chinese balloon that entered US airspace over Alaska and then flew into Canada before dipping down over Montana has been met with mixed reactions. While some, such as Montana Republican Congressman Ryan Zinke, have called for the balloon to be shot down, the Pentagon has said that it does not currently pose a physical or military risk to people on the ground and is continuing to monitor and review options. China, for its part, insists that the balloon is not a spy vessel but a civilian airship used for research, mainly meteorological purposes.

Temperature of 0.4. Note the alternation of what details it emphasizes:

  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike home from work when he was hit and then stabbed. police say the driver, a 33-year-old man from irvine, california, is in custody. he is being held on suspicion of murder. the doctor's name has not been released. this is a developing story. we will have more information as it becomes available. A large Chinese balloon that entered US airspace over Alaska and then flew into Canada before dipping down over Montana is being monitored by the Pentagon, which says it is a surveillance balloon but China insists is a civilian airship used for research. Millions of families are in the midst of what the National Weather Service is calling a "once in a generation" arctic blast. Parts of New England expecting wind chills at or near 60 below zero. Outside Springfield, Massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. The woman driving rushed to the hospital. In Boston, pilots battling high winds as they landed at Logan Airport. Outreach teams hitting the streets to care for the needy. Doctors say hypothermia can set in in just minutes.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike home from work when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening. this is a developing story. we're learning new details about the stabbing death of an emergency room doctor in california. the doctor was riding his bike home from work when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. the doctor died at the scene. the driver is in custody. we're told the doctor was a well-respected member of the community. he leaves behind a wife and two young children. this is a tragic story and we're working to learn more.
  • the attack happened on the pacific coast highway in the wealthy enclave of laguna beach. the doctor was riding his bike when he was hit by a car. police say the driver got out of his car and stabbed the doctor multiple times. good evening. A large Chinese balloon that entered US airspace over Alaska and then flew into southwest Canada before dipping down over Billings, Montana has been met with mixed reactions. While some, including Montana Republican Congressman Ryan Zinke, have called for the balloon to be shot down, the Pentagon has said that it does not currently pose a physical or military risk to people on the ground and is continuing to monitor and review options. China, for its part, insists that the balloon is not a spy vessel but a civilian airship used for research, mainly meteorological purposes.

Let's try correcting its punctuation and capitalization:

Correct the capitalization and punctuation of the following text: commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. windchill ever recorded in the u.s., including alaska. snow squalls along with that arctic chill. a time lapse from vermont. a cold emergency in place in boston. this is through sunday now. rob marciano standing by to time this out, and abc\'s victor oquendo in boston where the wind chill is 13 below zero right now. tonight, millions of families are in the midst of what the national weather service is calling a \\"once in a generation\\" arctic blast. we\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. we\'re going to load food and stuff in here. and outreach teams hitting the streets to care for the needy. if anyone is out and sees someone else who might be in distress, please call 911. doctors say hypothermia can set in in just minutes. this weather can be deadly. it\'s life threatening. this is something that people have to really take seriously. the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. david? victor oquendo in boston. victor, thank you. let\'s get to rob marciano who\'s timing this out for us tonight. hey, rob. hi, david. if you live in the northeast, you\'ve felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it\'s no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that\'s for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that\'s an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we\'ll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it\'s just a matter of survival for the next 36 hours. david? whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we\'re going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here\'s our chief correspondent matt gutman from california tonight. tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.

Unfortunately, this triggers a guardrail false positive and fails. This is one of the more frustrating aspects of current LLM guardrails: their extreme unpredictability and false positive rate, where one prompt triggers a guardrail and another doesn't:

{
  "predictions": [
    {
      "safetyAttributes": {
        "blocked": true
      },
      "content": ""
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalTokens": 1223,
        "totalBillableCharacters": 4319
      },
      "outputTokenCount": {
        "totalBillableCharacters": 0,
        "totalTokens": 1024
      }
    }
  }
}

Here we've gone ahead and used an external tool to correct the text, so let's try our summarization again.

The corrected text:

Summarize in one paragraph the following text: Commercial pilots radioing in. We got that balloon in sight also. Looks like it\'s way up there, maybe 50,000 feet or so. The balloon is huge, with a technology bay attached below that is itself the size of three buses, loaded with high-resolution cameras, according to a senior U.S. official. Equipped with what appear to be solar panels on the side that could power its technology. It first entered American airspace over Alaska, then flew into southwest Canada before dipping down over Billings, Montana. This thing is up in the sky. What the heck is that? That thing is not the moon. Any help would be appreciated. Montana Republican Congressman Ryan Zinke was blunt, tweeting, \\"Shoot it down.\\" The Pentagon today saying that option was considered and rejected, for now. We assessed that currently it does not pose a physical or military risk to people on the ground. For now, we are continuing to monitor and review options. The White House today said President Biden agreed with the Pentagon\'s strong recommendation. The risks involved with shooting down the balloon are just too great. Any potential debris field would be significant and could potentially cause civilian injuries, deaths, or significant property damage. For its part, China is insisting today that the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. The Pentagon is flatly rejecting that claim. We are aware of the PRC\'s statement; however, the fact is, we know that it\'s a surveillance balloon. We have clearly communicated that this balloon is violating U.S. airspace and international law, and that this is unacceptable. China says the balloon deviated far from its planned course, but the Pentagon is questioning that as well, saying the balloon can be maneuvered. A senior U.S. official told ABC News that it\'s likely being navigated with spy satellites. This is all happening as Secretary of State Antony Blinken prepares for a weekend trip to Beijing. The coldest wind chill ever recorded in the U.S., including Alaska, with snow squalls along with that Arctic chill. A time-lapse from Vermont, with a cold emergency in place in Boston. This is through Sunday now. Rob Marciano is standing by to time this out, and ABC\'s Victor Oquendo is in Boston where the wind chill is 13 below zero right now. Tonight, millions of families are in the midst of what the National Weather Service is calling a \\"once-in-a-generation\\" Arctic blast. We\'re seeing wind chills right now that are already below zero, and before the night is over, these will drop even more. Parts of New England are expecting wind chills at or near 60 below zero. Outside Springfield, Massachusetts, as winds gust above 50 miles per hour, a tree crashed onto a car, killing a 6-month-old girl inside. The woman driving was rushed to the hospital. In Boston, pilots are battling high winds as they land at Logan Airport. We\'re going to load food and supplies in here. Outreach teams are hitting the streets to care for the needy. If anyone is out and sees someone else who might be in distress, please call 911. Doctors say hypothermia can set in in just minutes. This weather can be deadly. It\'s life-threatening. This is something that people have to really take seriously. The Arctic front is visible in the form of powerful snow squalls pushing from upstate New York through New England. Here in Boston, a brutal night is ahead. A cold emergency is in place. Winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. David? Victor Oquendo in Boston. Victor, thank you. Let\'s get to Rob Marciano, who\'s timing this out for us tonight. Hey, Rob. Hi, David. If you live in the Northeast, you\'ve felt it all day long. Temperatures have been dropping. Now these winds are really ripping, measured over 40 miles per hour already. The wind chill advisory is pretty much everywhere north of New York City. Here you see it on the map. It\'s no picnic south of here either. Minus 23 for a wind chill right now in Albany. Minus 13 in Boston. But numbers in the morning will be even more dangerous, and that red spot is a blizzard warning for blowing snow on the ground. Minus 57 in Caribou. If we get to that figure or less, that\'s an all-time record. Minus 11 in New York. This is hypothermic stuff, frostbite in minutes. But a rapid rebound in temperatures is coming during the day Sunday. We\'ll start in single digits in many spots, all the way into the 40s for Sunday, Monday, and Tuesday. So it\'s just a matter of survival for the next 36 hours. David? Whiplash for sure, but take Saturday seriously. Rob, we thank you. Get back inside. Let\'s turn next to the doctor struck by a car. Police say he was then attacked and did not survive. That emergency room doctor was riding his bike on the Pacific Coast Highway when he was struck by a driver. Police say the driver then got out of the car and stabbed the doctor. Bystanders took the attacker down. Here\'s our chief correspondent, Matt Gutman, from California tonight. Tonight, an ER doctor is dead after police in California say a motorist slammed into him, then got out of his car and stabbed him.

And the result with a temperature of 0.0. Less hallucination than before, but still some (Torrance) and it still fixates:

The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver got out of his car and stabbed the doctor multiple times. Bystanders intervened and held the driver down until police arrived. The doctor was taken to a hospital, where he died. The driver is in custody. Police have not released a motive for the attack.

What about a temperature of 0.2? Hallucination of the city and hallucinated details like the repeated stabbing. Again, fixated on the doctor case:

  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver then got out of his car and stabbed the doctor multiple times. Bystanders intervened and took the driver into custody. The doctor was taken to a hospital, where he died. The driver is being held on suspicion of murder. The doctor's name has not been released.
  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver then got out of his car and stabbed the doctor multiple times. Bystanders intervened and took the driver into custody. The doctor was pronounced dead at the scene. The driver is in police custody. His name has not been released. Police are still investigating what led up to the attack.
  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver then got out of his car and stabbed the doctor multiple times. Bystanders intervened and took the driver into custody. The doctor was taken to a hospital, where he died. The driver is being held on suspicion of murder. The doctor's name has not been released.

Temperature of 0.4? Interestingly, correcting the capitalization and punctuation of the text appears to have stopped the hallucination of the victim and suspect names and the city hallucination is consistent now:

  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver then got out of his car and stabbed the doctor multiple times. Bystanders intervened and took the driver into custody. The doctor was pronounced dead at the scene. The driver is being held on suspicion of murder. Police have not released his name or any information about a motive.
  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver got out of his car and stabbed the doctor multiple times. Bystanders intervened and held the driver down until police arrived. The doctor was pronounced dead at the scene. The driver is in custody. Police have not released a motive for the attack.
  • The attack happened on the Pacific Coast Highway in Torrance. The doctor was riding his bike when he was hit by a car. Police say the driver then got out of his car and stabbed the doctor multiple times. Bystanders intervened and took the driver into custody. The doctor was taken to a hospital, where he died. The driver is being held on suspicion of murder. Police have not released his name or any information about a motive.

Clearly, the LLM is extremely sensitive to the capitalization and punctuation: an issue we've observed with other LLMs we've tested.

Now the question is whether it was just the formatting correction that reduced the name hallucination or whether removing the speaker annotations in the transcript also played a role. Let's go back to the original version and correct it in situ without removing the "Reporter:" entries (though we'll remove the ">>" entries).

Thus our prompt becomes:

Summarize in one paragraph the following text: Commercial pilots radioing in: We got that balloon in sight also. Looks like it\'s way up there, maybe 50,000 feet or so. Reporter: The balloon is huge, with a technology bay attached below that is itself the size of three buses, loaded with high-resolution cameras, according to a senior U.S. official. Equipped with what appear to be solar panels on the side that could power its technology. It first entered American airspace over Alaska, then flew into southwest Canada before dipping down over Billings, Montana. This thing is up in the sky. What the heck is that? That thing is not the moon. Any help would be appreciated. Reporter: Montana Republican Congressman Ryan Zinke was blunt, tweeting, \\"Shoot it down.\\" The Pentagon today saying that option was considered and rejected, for now. We assessed that currently it does not pose a physical or military risk to people on the ground. For now, we are continuing to monitor and review options. Reporter: The White House today said President Biden agreed with the Pentagon\'s strong recommendation. The risks involved with shooting down the balloon are just too great. Any potential debris field would be significant and could potentially cause civilian injuries, deaths, or significant property damage. Reporter: For its part, China is insisting today that the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. The Pentagon is flatly rejecting that claim. We are aware of the PRC\'s statement. However, the fact is, we know that it\'s a surveillance balloon. We have clearly communicated that this balloon is violating U.S. airspace and international law, and that this is unacceptable. Reporter: China says the balloon deviated far from its planned course, but the Pentagon is questioning that as well, saying the balloon can be maneuvered. A senior U.S. official told ABC News that it\'s likely being navigated with spy satellites. This is all happening as Secretary of State Antony Blinken prepares for a weekend trip to Beijing. The coldest wind chill ever recorded in the U.S., including Alaska, with snow squalls along with that Arctic chill. A time-lapse from Vermont, with a cold emergency in place in Boston. This is through Sunday now. Rob Marciano is standing by to time this out, and ABC\'s Victor Oquendo is in Boston where the wind chill is 13 below zero right now. Reporter: Tonight, millions of families are in the midst of what the National Weather Service is calling a \\"once-in-a-generation\\" Arctic blast. \'We\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. Reporter: Parts of New England expecting wind chills at or near 60 below zero. Outside Springfield, Massachusetts, as winds gusted above 50 miles per hour, a tree crashed onto a car, killing a 6-month-old girl inside. The woman driving was rushed to the hospital. In Boston, pilots are battling high winds as they land at Logan Airport. We\'re going to load food and supplies in here. Reporter: Outreach teams are hitting the streets to care for the needy. If anyone is out and sees someone else who might be in distress, please call 911. Reporter: Doctors say hypothermia can set in in just minutes. This weather can be deadly. It\'s life-threatening. This is something that people have to really take seriously. Reporter: The Arctic front is visible in the form of powerful snow squalls pushing from upstate New York through New England. Here in Boston, a brutal night is ahead. A cold emergency is in place. Winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. David? Reporter: Victor Oquendo in Boston. Victor, thank you. Let\'s get to Rob Marciano, who\'s timing this out for us tonight. Hey, Rob. Reporter: Hi, David. If you live in the Northeast, you\'ve felt it all day long. Temperatures have been dropping. Now these winds are really ripping, measured over 40 miles per hour already. The wind chill advisory is pretty much everywhere north of New York City. Here you see it on the map. It\'s no picnic south of here either. Minus 23 for a wind chill right now in Albany. Minus 13 in Boston. But numbers in the morning will be even more dangerous, and that red spot is a blizzard warning for blowing snow on the ground. Minus 57 in Caribou. If we get to that figure or less, that\'s an all-time record. Minus 11 in New York. This is hypothermic stuff, frostbite in minutes. But a rapid rebound in temperatures is coming during the day Sunday. We\'ll start in single digits in many spots, all the way into the 40s for Sunday, Monday, and Tuesday. So it\'s just a matter of survival for the next 36 hours. David? Reporter: Whiplash for sure, but take Saturday seriously. Rob, we thank you. Get back inside. Reporter: We\'re going to turn next to the doctor struck by a car. Police say he was then attacked. He did not survive. That emergency room doctor was riding his bike on the Pacific Coast Highway when he was struck by a driver. Police say the driver then got out of the car and stabbed the doctor. Bystanders took the attacker down. Here\'s our chief correspondent, Matt Gutman, from California tonight. Reporter: Tonight, an ER doctor is dead after police in California say a motorist slammed into him, then got out of his car and stabbed him.

Let's try a temperature of 0.0. This hits a guardrail false positive again. It looks like the presence of "Reporter:" is what's triggering the guardrail.

{
  "predictions": [
    {
      "safetyAttributes": {
        "blocked": true
      },
      "content": ""
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalTokens": 1218,
        "totalBillableCharacters": 4503
      },
      "outputTokenCount": {
        "totalTokens": 78,
        "totalBillableCharacters": 0
      }
    }
  }
}

Let's go back to our original five-minute transcript that we used in our prior episodes of this series. We'll keep the punctuation and capitalization as-is:

Summarize in one paragraph the following text: tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it\'s believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out. the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting. another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers. the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down. the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh\'s housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper\'s fall. but will the jury hear about it? eva pilgrim reporting. robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you\'ll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing.

Temperature of 0.0. Note the hallucination that it is Ukraine that caused Blinken's trip to China to be canceled:

but that trip is now on hold. the white house saying secretary blinken will postpone his trip to china and will instead travel to japan and south korea to discuss the situation in ukraine. A large Chinese spy balloon is hovering over the United States. It was first spotted in Alaska and then flew into Canada before entering Montana. The balloon is equipped with high-resolution cameras and solar panels. The Pentagon is considering shooting it down, but the White House has said that the risks involved are too great. China insists that the balloon is a civilian airship used for research, but the Pentagon says that it is a surveillance balloon. Secretary of State Antony Blinken has postponed his trip to Beijing to discuss the situation in Ukraine.

Temperature of 0.2. Note the hallucination about Ukraine again and the addition of Japan and South Korea:

  • but that trip is now on hold. the white house saying secretary blinken will postpone his trip to china and will instead travel to japan and south korea to discuss the situation in ukraine. A large Chinese spy balloon is hovering over the United States. The balloon was first spotted in Alaska and then flew into Canada before entering Montana. The balloon is equipped with high-resolution cameras and solar panels. The Pentagon has considered shooting down the balloon, but has decided not to do so for now because of the risk of debris. China has denied that the balloon is a spy vessel and says that it is a civilian airship used for research. Secretary of State Antony Blinken has postponed his trip to China because of the situation in Ukraine.
  • but that trip is now on hold. the white house citing china's unwillingness to engage in substantive talks on a range of issues. The US is tracking a Chinese spy balloon that is hovering over the US. The balloon is 60,000 feet above the ground and is heading east. It was first spotted over Alaska, then flew into southwest Canada before dipping down over Billings, Montana. The balloon is huge, with a technology bay attached below that is itself the size of three busses. It is loaded with high resolution cameras and equipped with solar panels. The Pentagon has considered shooting down the balloon, but has decided not to do so for now because of the risks involved. China insists that the balloon is not a spy vessel, but a civilian airship used for research. The Pentagon flatly rejects that claim. Secretary of State Antony Blinken has postponed his trip to Beijing because of China's unwillingness to engage in substantive talks on a range of issues.
  • but that trip is now on hold. the white house saying secretary blinken will postpone his trip to china and will instead travel to japan and south korea to discuss the situation in ukraine. A large Chinese spy balloon is hovering over the United States. The balloon was first spotted in Alaska and then flew into Canada before entering Montana. The balloon is equipped with high-resolution cameras and solar panels. The Pentagon is monitoring the balloon and has considered shooting it down, but has decided not to do so for now because of the risk of debris. China claims that the balloon is a civilian airship used for research, but the Pentagon says that it is a surveillance balloon. Secretary of State Antony Blinken has postponed his trip to China because of the situation.
  • but that trip is now on hold. the white house saying secretary blinken will postpone his trip to china and will instead travel to japan and south korea to discuss the situation in ukraine. A large Chinese spy balloon is hovering over the United States. It was first spotted in Alaska and then flew into Canada before entering Montana. The balloon is 60,000 feet above the ground and is heading east. It is believed to be headed towards North Carolina. The Pentagon is monitoring the balloon and has considered shooting it down, but has decided not to do so for now because of the risk of debris. China claims that the balloon is a weather balloon that has drifted off course, but the Pentagon says that it is a surveillance balloon. Secretary of State Antony Blinken has postponed his trip to China because of the situation with the balloon. He will instead travel to Japan and South Korea to discuss the situation in Ukraine.