Experiments In Meme Tracking: Dividing Transcripts Into Stories – Part 2

Kalev Leetaru

1 year ago

Earlier today we explored a variety of approaches to dynamic semantic windowing of television news transcripts to break them into discrete stories using a SOTA LLM. Unfortunately, the results were less than desirable, with hallucination, false guardrail triggers, null outputs and extremely unstable execution. What if we change tactics and have the LLM help us craft our prompt? Let's explore this same ABC Evening News broadcast that we used in this morning's experiments.

The end result demonstrates quite vividly the complexities of using LLMs: rampant hallucination, truncation, random deletions, myriad guardrail false positives that abort output and a hugely complex workflow that requires layers upon layers upon layers upon layers of pre and post filtering to work around the LLM's challenges.

Rather than craft our own prompt, what if we ask PaLM to write its own prompt for us?

Write a prompt that can be used to take a news article and insert the word \'NEWTOPIC\' every time the topic changes in a major way in the text.

This yields the following Python code. Clearly, the LLM misunderstood its task, since "prompts" typically refer to LLM inputs, not traditional programming code. It is particularly notable that it so quickly defaults to Python code, reflecting the outsized importance LLM vendors have ascribed to the language and the resources they've poured into RLHF training for Python. Language aside, however, the code below doesn't actually do anything related to the task:

```
import re

def main():
    # Read the news article from the given file.
    with open("news_article.txt", "r") as f:
        article = f.read()

    # Find all the occurrences of the word "NEWTOPIC" in the article.
    matches = re.findall(r"NEWTOPIC", article)

    # For each match, replace the word "NEWTOPIC" with a new paragraph.
    for match in matches:
        article = article.replace(match, "\n\n")

    # Print the article with the new paragraphs.
    print(article)

if __name__ == "__main__":
    main()
```

About the same prompt and a temperature of 0.2?

```
import re

def main():
    # Read the news article from the given file.
    with open("news_article.txt", "r") as f:
        article = f.read()

    # Split the article into sentences.
    sentences = article.split("\n")

    # Find the topic of each sentence.
    topics = []
    for sentence in sentences:
        topic = re.search(r"(.+?)\s*-\s*", sentence).group(1)
        topics.append(topic)

    # Insert the word "NEWTOPIC" between sentences that have different topics.
    new_article = ""
    for i in range(len(sentences)):
        if i > 0 and topics[i] != topics[i - 1]:
            new_article += "NEWTOPIC\n"
        new_article += sentences[i] + "\n"

    # Write the new article to the given file.
    with open("new_article.txt", "w") as f:
        f.write(new_article)

if __name__ == "__main__":
    main()
```

And a temperature of 0.99?

```
import re

def main():
    # Read the news article from a file.
    with open('news_article.txt', 'r') as f:
        article = f.read()

    # Find all the topic changes in the article.
    topic_changes = re.finditer(r'\n\n+', article)

    # Insert the word 'NEWTOPIC' before each topic change.
    for topic_change in topic_changes:
        article = article[:topic_change.start()] + 'NEWTOPIC' + article[topic_change.end():]

    # Write the updated article to a file.
    with open('news_article_updated.txt', 'w') as f:
        f.write(article)

if __name__ == '__main__':
    main()
```

It seems that it wants to give us Python code, so let's explicitly specify that we want an LLM prompt:

Write a prompt that can be used with any Large Language Model to take a news article and insert the word \'NEWTOPIC\' every time the topic changes in a major way in the text.

This yields the following, which is not much more than a paraphrasing of our prompt, which didn't yield the desired results when we tried it earlier today.

```
Given a news article, insert the word 'NEWTOPIC' every time the topic changes in a major way in the text.
```

And with a temperature of 0.2 we get an identical result:

```
Given a news article, insert the word 'NEWTOPIC' every time the topic changes in a major way in the text.
```

What about a temperature of 0.99?

```
Here is a prompt that you can use with any large language model to take a news article and insert the word 'NEWTOPIC' everytime the topic changes in a major way in the text:

```
Given a news article, insert the word 'NEWTOPIC' everytime the topic changes in a major way in the text.
```

Here is an example of how the prompt can be used:

```
Article:

The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.

Output:

The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.
```

This is far more interesting. It makes use of including an example for the LLM, which is a best practice for more complicated tasks. It improperly includes "NEWTOPIC" in the source article text (such that in this case the model isn't actually doing anything), but it raises the idea of using an example.

Let's modify our last prompt from our experiments earlier today accordingly with a temperature of 0.0:

This yields the following. For the first time we're getting reasonable results here.

And a temperature of 0.2:

And a second and third time at 0.2, with consistent results:

reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it's likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. that visit now canceled. in light of china's unacceptable action, i am postponing my planned travel this weekend to china. we concluded that conditions were not conducive for a constructive visit at this time. reporter: there have been other foreign surveillance balloons above the u.s., but none like this. what makes this different is the duration and the length of which it has been over u.s. territory. reporter: the pentagon says the balloon is expected to linger above america for a few days. NEWTOPIC this is really something. let's bring in mola lenghi. he's in billings, montana, for us again tonight. that state with several sensitive nuclear sites. as you reported, u.s. authorities say the balloon is likely navigated by chinese satellites. the pentagon says they've decided not to shoot it down at this point, but that option isn't completely off the table? reporter: that's right, david, a u.s. official familiar with the situation says the current thinking is to wait for the balloon to drift over the atlantic ocean and then shoot it down once it's over u.s. territoryial waters.
reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it's likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. that visit now canceled. in light of china's unacceptable action, i am postponing my planned travel this weekend to china. we concluded that conditions were not conducive for a constructive visit at this time. reporter: there have been other foreign surveillance balloons above the u.s., but none like this. what makes this different is the duration and the length of which it has been over u.s. territory. reporter: the pentagon says the balloon is expected to linger above america for a few days. NEWTOPIC this is really something. let's bring in mola lenghi. he's in billings, montana, for us again tonight. that state with several sensitive nuclear sites. as you reported, u.s. authorities say the balloon is likely navigated by chinese satellites. the pentagon says they've decided not to shoot it down at this point, but that option isn't completely off the table? reporter: that's right, david, a u.s. official familiar with the situation says the current thinking is to wait for the balloon to drift over the atlantic ocean and then shoot it down once it's over u.s. territoryial waters.

And temperature of 0.99, with worse results:

reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it's likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. that visit now canceled. in light of china's unacceptable action, i am postponing my planned travel this weekend to china. we concluded that conditions were not conducive for a constructive visit at this time. reporter: there have been other foreign surveillance balloons above the u.s., but none like this. what makes this different is the duration and the length of which it has been over u.s. territory. reporter: the pentagon says the balloon is expected to linger above america for a few days. this is really something. let's bring in mola lenghi. he's in billings, montana, for us again tonight. that state with several sensitive nuclear sites. as you reported, u.s. authorities say the balloon is likely navigated by chinese satellites. the pentagon says they've decided not to shoot it down at this point, but that option isn't completely off the table? reporter: that's right, david, a u.s. official familiar with the situation says the current thinking is to wait for the balloon to drift over the atlantic ocean and then shoot it down once it's over u.s. territoryial waters.

NEWTOPIC secretary of state antony blinken had planned to visit china this weekend, but canceled the trip in response to the spy balloon. china says the balloon deviated far from its planned course, but the pentagon questions that and says it's likely being navigated by spy satellites.

Using this new prompt, let's start over and try our workflow again from the start of the broadcast. Here is the new prompt and the first minute of the broadcast:

And the final command:

Yielding:

tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china.

NEWTOPIC mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast.

NEWTOPIC

Yielding the following. Note how it truncates the first sentence ("mola lenghi, mary bruce standing by at the white house."):

the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.

NEWTOPIC the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.

NEWTOPIC another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.

NEWTOPIC the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down.

NEWTOPIC the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today.

NEWTOPIC and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew.

Yielding:

and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper's fall. but will the jury hear about it? eva pilgrim reporting.

NEWTOPIC robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you'll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear.

NEWTOPIC good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that's in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites.

Yielding the following. Note how it fails to segment the cold and balloon stories.

good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that's in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it's headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that's the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we're learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china's actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight.

NEWTOPIC mary bruce at the white house, and abc's mola lenghi leading us off tonight in montana.

Critically, the passage above truncates the entire last portion of the input, with all of the following text missing:

reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri.

We didn't hit an output cap either, returning just a quarter of our output token cap:

  "metadata": {
    "tokenMetadata": {
      "outputTokenCount": {
        "totalTokens": 280,
        "totalBillableCharacters": 1029
      },
      "inputTokenCount": {
        "totalBillableCharacters": 1862,
        "totalTokens": 508
      }
    }
  }

Thankfully, our workflow would just rewind and capture that text anyway since it would be considered part of the last story:

Yielding the following. This time we have a new problem: the entire second paragraph is hallucinated. While topically highly relevant to the original text, it consists entirely of hallucinated text. In this case it is unclear what an optimal workflow might attempt, since our basic workflow rewinds to the start of the last story in each chunk.

NEWTOPIC mary bruce at the white house, and abc's mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it's way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting "shoot. it. down." the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon's strong recommendation.

NEWTOPIC the balloon continues to move east, and now appears to be headed towards north carolina. commercial pilots have been radioing in with reports of seeing the balloon, which is estimated to be 60,000 feet above the ground. the balloon is equipped with high resolution cameras and solar panels, and is believed to be a spy balloon from china. the pentagon has said that it does not pose a physical or military risk to people on the ground, but is continuing to monitor the situation.

In this case, we'll assume that our workflow has built-in hallucination logic whereby if it can't find the start of the last story, it deletes it and rewinds to the start of the second-to-last story and again concatenates the next minute. In this case that will yield a larger amount of text, but should still be within our token limits.

This time the output is just the following literal string:

NEWTOPIC NEWTOPIC NEWTOPIC

We haven't hit our output caps, but it appears we've hit a dead end:

  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalTokens": 688,
        "totalBillableCharacters": 2603
      },
      "outputTokenCount": {
        "totalTokens": 9,
        "totalBillableCharacters": 24
      }
    }

Remember that we used a temperature of 0.0 up until now to ensure deterministic results and to absolutely minimize hallucination to the best degree possible. Let's increase our temperature to 0.2 to see if we can improve our results and hopefully escape this dead end.

Let's start with the first minute of the transcript again and a temperature of 0.2:

The full command:

This yields:

What if we run it additional times?

tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. NEWTOPIC mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. NEWTOPIC
tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. NEWTOPIC mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. NEWTOPIC
tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. NEWTOPIC

It appears that the results alternate between no topical division and the last section being its own topic. In this case we'll go with the first non-segmented version and just append the next minute and retry:

Yielding:

NEWTOPIC first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house.

NEWTOPIC the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.

NEWTOPIC the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.

NEWTOPIC another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.

NEWTOPIC the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew.

What if we run a second time to test segmentation stability? The results are the same for two of the runs, while the second run makes some changes.

Run 2 (Missing dividers between "live readings already. first, the pentagon tracking" and added a divider between "expert acknowledged today. and tonight, the story")
- tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already. first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house. the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.
- NEWTOPIC the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.
- NEWTOPIC another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.
- NEWTOPIC the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down.
- NEWTOPIC the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today.
- NEWTOPIC and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew.
Run 3 (identical to the first run)
- tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already.
- NEWTOPIC first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house.
- NEWTOPIC the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.
- NEWTOPIC the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.
- NEWTOPIC another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.
- NEWTOPIC the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down.
- NEWTOPIC the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew.
Run 4 (same as first run)
- tonight, several developing stories as we come on the air. news coming in, the suspected chinese spy balloon hovering over the u.s. where it is now, and will there be a window to shoot it down? also tonight, the dangerous life-threatening cold moving into the northeast. the live readings already.
- NEWTOPIC first, the pentagon tracking that chinese balloon, the intelligence bay hanging beneath it, the size of three busses. where it was spotted today over the u.s., where it's believed to be headed now, 60,000 feet in the air. will the u.s. shoot this down? secretary of state antony blinken postponing his high stakes trip to china. mola lenghi, mary bruce standing by at the white house.
- NEWTOPIC the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.
- NEWTOPIC the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.
- NEWTOPIC another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.
- NEWTOPIC the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down.
- NEWTOPIC the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today. and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew.

Again, using the first output's segmentation and appending the next minute:

Yields another blocked response. Unfortunately, the guardrail has incorrectly triggered again:

{
  "predictions": [
    {
      "safetyAttributes": {
        "blocked": true
      },
      "content": ""
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalBillableCharacters": 1357,
        "totalTokens": 371
      },
      "outputTokenCount": {
        "totalBillableCharacters": 0,
        "totalTokens": 265
      }
    }
  }
}

Running 6 more times, all responses were blocked, so it is not a matter of simply running until the randomized output doesn't trigger the guardrail. Our only solution is to increase temperature:

Temperature 0.3: ran 6 times, all blocked.
Temperature 0.4: ran 6 times, all blocked.
Temperature 0.5: ran 6 times, all blocked.
Temperature 0.6: ran 6 times, all blocked.
Temperature 0.7: ran 6 times, all blocked.
Temperature 0.8: ran 6 times, all blocked.
Temperature 0.9: ran 6 times, all blocked.
Temperature 0.99: ran 6 times, all blocked.

It appears that we've hit a dead end. There is no way of overcoming the guardrail false positive and generating output.

Unfortunately, current commercial LLMs do not provide useable diagnostic information about why a guardrail triggered to help users overcome false positives.

Instead, let's try backing off by removing sentences from the end until we get a response. In each case below we ran 6 times with temperature 0.2:

BLOCKED: Removed "u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
BLOCKED: Removed "good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."
ALLOWED: Removed "tonight, the lesson on conquering fear. good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that\'s in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight. china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites."

Finally, we have a text that generates output!

Thus, the following text, which has stripped nearly half the text from the paragraph, is blocked due to a guardrail false positive:

But, removing that last sentence ("tonight, the lesson on conquering fear.") the prompt finally yields output!

Yielding:

the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today.

NEWTOPIC and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper's fall. but will the jury hear about it? eva pilgrim reporting.

NEWTOPIC robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you'll see her jump from the tallest building in the southern hemisphere.

Now we'll resume our workflow:

Yielding:

robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you'll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear.

NEWTOPIC

Continuing:

Yielding the following. Once again, the last paragraph is entirely hallucinated.

NEWTOPIC china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it's headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that's the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we're learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china's actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc's mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri.

NEWTOPIC the cold weather is expected to continue through the weekend, with wind chills as low as 50 degrees below zero in some parts of the country. the national weather service has issued wind chill advisories and warnings for much of the midwest and northeast. officials are urging people to stay indoors and avoid unnecessary travel.

We'll assume we have hallucination detection and rewind back to the start of the first real story:

Unfortunately, this yields the following hallucinated and truncated output based on the real text:

CHINA SPY BALLOON NEWTOPIC SECRETARY OF STATE ANTONY BLINKEN POSTPONING TRIP TO BEIJING NEWTOPIC CHINA SAYING IT IS A WEATHER BALLOON THAT BLEW OFF COURSE

What if we run a few more times?

CHINA SPY BALLOON NEWTOPIC CHINA SPY BALLOON SIGHTINGS NEWTOPIC CHINA SPY BALLOON DESTINATION NEWTOPIC CHINA SPY BALLOON SHOT DOWN? NEWTOPIC CHINA SPY BALLOON SECRETARY OF STATE NEWTOPIC CHINA SPY BALLOON CHINA'S RESPONSE
CHINESE SPY BALLOON OVER U.S. NEWTOPIC SECRETARY OF STATE POSTPONES TRIP TO CHINA
CHINA SPY BALLOON NEWTOPIC SECRETARY OF STATE ANTONY BLINKEN POSTPONING TRIP TO BEIJING NEWTOPIC CHINA'S ACTIONS UNACCEPTABLE
CHINA SPY BALLOON NEWTOPIC SECRETARY OF STATE ANTONY BLINKEN POSTPONING TRIP TO BEIJING NEWTOPIC CHINA'S ACTIONS UNACCEPTABLE

Remember this is the prompt we're using:

time curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
"instances": [
{ "prompt": "Given the following news article, insert the word \'NEWTOPIC\' everytime the topic changes in a major way in the text.\nHere is an example:\nArticle: The president announced a new initiative to combat climate change. The economy is booming, with unemployment at a record low. The stock market is at an all-time high. \nOutput: The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.\n\n china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation.

"}
],
"parameters": {
"temperature": 0.2,
"maxOutputTokens": 1024,
"topK": 40,
"topP": 0.95,
}
}' > O; cat O | jq -r .predictions[0].content

What if we boost temperature to 0.4?

CHINESE SPY BALLOON OVER U.S. NEWTOPIC SECRETARY OF STATE DELAYS TRIP TO CHINA

And a temperature of 0.99?

chinese spy balloon NEWTOPIC the pentagon considering shooting it down NEWTOPIC secretary of state antony blinken postponing his trip to beijing NEWTOPIC china denies NEWTOPIC americans tracking the balloon

Despite running 6 times at each temperature setting, we cannot escape this failure state.

Thankfully, we're still below our max input quota, so let's append the next minute of the transcript and see what happens:

Given the following news article, insert the word \'NEWTOPIC\' everytime the topic changes in a major way in the text.\nHere is an example:\nArticle: The president announced a new initiative to combat climate change. The economy is booming, with unemployment at a record low. The stock market is at an all-time high. \nOutput: The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.\n\n china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it\'s headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that\'s the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we\'re learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china\'s actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc\'s mola lenghi leading us off tonight in montana. reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it\'s way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting \\"shoot. it. down.\\" the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon\'s strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc\'s statement. however, the fact is, we know that it\'s a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it\'s likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing.

Once again:

SUSPECTED CHINESE SPY BALLOON NEWTOPIC OVER MISSOURI NEWTOPIC BLINKEN POSTPONES CHINA TRIP

Testing temperatures up to 0.99 again six times each again reveals we cannot escape this failure state.

We're still below our input cap, so let's add another minute of transcript:

This time we got output! Even though the text is about a single story, it has segmented it into three stories, but the dividing points do represent substories and focal shifts in the telling, so do make some degree of sense.

NEWTOPIC reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it's way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting "shoot. it. down." the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon's strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc's statement. however, the fact is, we know that it's a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it's likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. that visit now canceled. in light of china's unacceptable action, i am postponing my planned travel this weekend to china. we concluded that conditions were not conducive for a constructive visit at this time. reporter: there have been other foreign surveillance balloons above the u.s., but none like this. what makes this different is the duration and the length of which it has been over u.s. territory. reporter: the pentagon says the balloon is expected to linger above america for a few days.

Let's continue:

And we're back to truncation:

NEWTOPIC NEWTOPIC
NEWTOPIC NEWTOPIC
NEWTOPIC NEWTOPIC
NEWTOPIC NEWTOPIC
NEWTOPIC NEWTOPIC

So let's add another minute to the transcript and try again:

This time we get results:

NEWTOPIC this is really something. let's bring in mola lenghi. he's in billings, montana, for us again tonight. that state with several sensitive nuclear sites. as you reported, u.s. authorities say the balloon is likely navigated by chinese satellites. the pentagon says they've decided not to shoot it down at this point, but that option isn't completely off the table? reporter: that's right, david, a u.s. official familiar with the situation says the current thinking is to wait for the balloon to drift over the atlantic ocean and then shoot it down once it's over u.s. territoryial waters. that way you can avoid injuries on the ground, and two, retrieve the balloon, study it, and ultimately exploit the intelligence. david, tonight china says it regrets the unintended entry of the air ship into u.s. air space, but as you can imagine, not exactly flying with u.s. national security officials who are calling this unacceptable.

NEWTOPIC mola lenghi, thank you. let's bring in mary bruce. president biden chose not to talk about this today, trying not to enflame the situation. what are you learning from the white house tonight? obviously the president continuing to be briefed. and on mola's point, if you shot this down over u.s. waters, that still is a small window of opportunity to pull this off.

NEWTOPIC reporter: yes, david. we know the president is being regularly briefed. his team presented him with all options, including the possible military action to take the balloon down. now, initially biden was advised against that because of the concern over civilian risk, but tonight we have learned that option may be back on the table. one source telling us the thinking here is to wait until the balloon goes out to see and then shoot it down, but this is very tricky. they would have just 12 nautical miles to do this before entering international waters and certainly don't want to start an international incident over this. tonight we are told the white house is keeping communications to china open. but even though beijing expressed regret, it was not enough to salvage secretary blinken's trip. what is clear this is only further escalating tensions between our two countries.

NEWTOPIC >> yeah, no question. you mentioned those 12 nautical miles would be the opportunity. mary, thank you. of course the other major concern the concern over this dangerous life-threatening cold. they're calling it a once in a generation arctic blast that has already turned deadly tonight. 25 million americans in ten states. wind chills below zero in philadelphia, new york, and boston. and look at this tonight. this is the view from a camera on mt. washington in new hampshire. they've already set a preliminary record at this hour. the wind chill moments ago, 106 degrees below zero. if confirmed it's the coldest windchill ever recorded in the u.s., including alaska.

NEWTOPIC

And continue:

Yielding the following. Note that it divides it in the middle of the story, but the division point is a shift from in studio to a field reporter, so does represent a shift in how the story is being told:

yeah, no question. you mentioned those 12 nautical miles would be the opportunity. mary, thank you. of course the other major concern the concern over this dangerous life-threatening cold. they're calling it a once in a generation arctic blast that has already turned deadly tonight. 25 million americans in ten states. wind chills below zero in philadelphia, new york, and boston. and look at this tonight. this is the view from a camera on mt. washington in new hampshire. they've already set a preliminary record at this hour. the wind chill moments ago, 106 degrees below zero. if confirmed it's the coldest windchill ever recorded in the u.s., including alaska. snow squalls along with that arctic chill. a time lapse from vermont. a cold emergency in place in boston. this is through sunday now. rob marciano standing by to time this out, and abc's victor oquendo in boston where the wind chill is 13 below zero right now.

NEWTOPIC >> reporter: tonight, millions of families are in the midst of what the national weather service is calling a 'once in a generation' arctic blast. >> we're seeing wind chills right now which are already below zero and before the night's over with, these are going to drop even more. >> reporter: parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. >> we're going to load food and stuff in here. >> reporter: and outreach teams hitting the streets to care for the needy. >> if anyone is out and sees someone else who might be in distress, please call 911.

Once again we've hit a guardrail false positive:

{
  "predictions": [
    {
      "content": "",
      "safetyAttributes": {
        "blocked": true
      }
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "outputTokenCount": {
        "totalTokens": 642,
        "totalBillableCharacters": 0
      },
      "inputTokenCount": {
        "totalTokens": 590,
        "totalBillableCharacters": 2050
      }
    }
  }
}

So, we'll add the following minute and try again:

Yielding the following. This time we get results, but it has considerably oversegmented the text. Yet, something is wrong: this text is considerably shorter than our input. Unlike our previous truncations, which simply removed the last sentences, this time we end on the correct sentence – something is missing in the middle of the text.

reporter: tonight, millions of families are in the midst of what the national weather service is calling a 'once in a generation' arctic blast.

NEWTOPIC parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport.

NEWTOPIC and outreach teams hitting the streets to care for the needy.

NEWTOPIC doctors say hypothermia can set in in just minutes.

NEWTOPIC the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there's a real concern about power outages, and it will feel like at least 30 below.

NEWTOPIC we're going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here's our chief correspondent matt gutman from california tonight.

NEWTOPIC tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.

What's missing? Here is the original text, with all of the passages that are deleted from the output text bolded and underlined:

reporter: tonight, millions of families are in the midst of what the national weather service is calling a 'once in a generation' arctic blast. >> we're seeing wind chills right now which are already below zero and before the night's over with, these are going to drop even more. >> reporter: parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. >> we're going to load food and stuff in here. >> reporter: and outreach teams hitting the streets to care for the needy. >> if anyone is out and sees someone else who might be in distress, please call 911. reporter: doctors say hypothermia can set in in just minutes. >> this weather can be deadly. it's life threatening. this is something that people have to really take seriously. >> reporter: the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there's a real concern about power outages, and it will feel like at least 30 below. david? >> victor oquendo in boston. victor, thank you. let's get to rob marciano who's timing this out for us tonight. hey, rob. >> reporter: hi, david. if you live in the northeast, you've felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it's no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that's for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that's an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we'll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it's just a matter of survival for the next 36 hours. david? >> whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we're going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here's our chief correspondent matt gutman from california tonight. >> reporter: tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.

More than half the text has been deleted. This is a worst-case scenario in that if we simply back up to the end of the last story we'll see the sentence matches the transcript and assume all is well, without realizing that more than half the text has been removed. Alternatively, we could flag when an output text is smaller than the input text and automatically rerun.

To confirm, we're not hitting our caps:

  "metadata": {
    "tokenMetadata": {
      "outputTokenCount": {
        "totalTokens": 331,
        "totalBillableCharacters": 1199
      },
      "inputTokenCount": {
        "totalBillableCharacters": 2863,
        "totalTokens": 824
      }
    }
  }

Let's assume we have a sanity check that flags the output being smaller than the input and automatically reruns the text. Unfortunately, running multiple times yields the same deletions each time:

reporter: tonight, millions of families are in the midst of what the national weather service is calling a 'once in a generation' arctic blast. NEWTOPIC parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. NEWTOPIC and outreach teams hitting the streets to care for the needy. NEWTOPIC doctors say hypothermia can set in in just minutes. NEWTOPIC the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there's a real concern about power outages, and it will feel like at least 30 below. NEWTOPIC we're going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here's our chief correspondent matt gutman from california tonight. NEWTOPIC tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.
reporter: tonight, millions of families are in the midst of what the national weather service is calling a 'once in a generation' arctic blast. NEWTOPIC parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. NEWTOPIC and outreach teams hitting the streets to care for the needy. NEWTOPIC doctors say hypothermia can set in in just minutes. NEWTOPIC the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there's a real concern about power outages, and it will feel like at least 30 below. NEWTOPIC we're going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here's our chief correspondent matt gutman from california tonight. NEWTOPIC tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him.

A closer inspection of the above shows that the ">>" transcription sequence, which indicates a change in speaker, does often appear in the vicinity of the truncations, so what if we remove those and try again:

This time we have a total truncation:

NEWTOPIC E.R. doctor killed in hit-and-run, stabbing
NEWTOPIC Emergency room doctor killed in hit-and-run, stabbing
NEWTOPIC E.R. doctor killed in hit-and-run, stabbing
NEWTOPIC E.R. doctor killed in hit-and-run, stabbing

So, let's try adding the next minute of transcript and try again:

Given the following news article, insert the word \'NEWTOPIC\' everytime the topic changes in a major way in the text.\nHere is an example:\nArticle: The president announced a new initiative to combat climate change. The economy is booming, with unemployment at a record low. The stock market is at an all-time high. \nOutput: The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.\n\n reporter: tonight, millions of families are in the midst of what the national weather service is calling a \'once in a generation\' arctic blast. we\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. reporter: parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. we\'re going to load food and stuff in here. reporter: and outreach teams hitting the streets to care for the needy. if anyone is out and sees someone else who might be in distress, please call 911. reporter: doctors say hypothermia can set in in just minutes. this weather can be deadly. it\'s life threatening. this is something that people have to really take seriously. reporter: the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. david? victor oquendo in boston. victor, thank you. let\'s get to rob marciano who\'s timing this out for us tonight. hey, rob. reporter: hi, david. if you live in the northeast, you\'ve felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it\'s no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that\'s for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that\'s an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we\'ll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it\'s just a matter of survival for the next 36 hours. david? whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we\'re going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here\'s our chief correspondent matt gutman from california tonight. reporter: tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him. according to the orange county sheriff\'s department, around 3:00 p.m. wednesday this white car broadsided dr. michael mammone as he was on his bike at a crosswalk on the much-touristed pacific coast highway. >> at that point, the suspect drove around the victim, exited his vehicle, went back to the victim and continued to assault the victim. >> reporter: a criminal complaint alleging the car\'s driver, vanroy smith, then attacked mammone with a knife. bystanders stepping in to detain smith. but mammone, an emergency medicine specialist, later dying at a hospital. tonight, providence mission hospital where mammone worked saying they are grieving over the loss of an incredible physician and friend. david, that suspect arrested, charged with murder and the use of a deadly weapon. authorities say they know of no connection between these two men and know of no motive for this crime. smith has pleaded not guilty. bail has been set at $1 million. david? >> just horrific. matt gutman tonight, thank you.

Once again we get a blocked response due to a guardrail false positive. Let's add the next minute of transcript and try again:

Given the following news article, insert the word \'NEWTOPIC\' everytime the topic changes in a major way in the text.\nHere is an example:\nArticle: The president announced a new initiative to combat climate change. The economy is booming, with unemployment at a record low. The stock market is at an all-time high. \nOutput: The president announced a new initiative to combat climate change. NEWTOPIC The economy is booming, with unemployment at a record low. NEWTOPIC The stock market is at an all-time high.\n\n reporter: tonight, millions of families are in the midst of what the national weather service is calling a \'once in a generation\' arctic blast. we\'re seeing wind chills right now which are already below zero and before the night\'s over with, these are going to drop even more. reporter: parts of new england expecting wind chills at or near 60 below zero. outside springfield, massachusetts, as winds gusted above 50 miles per hour, a tree crashing onto a car, killing a 6-month-old girl inside. the woman driving rushed to the hospital. in boston, pilots battling high winds as they landed at logan airport. we\'re going to load food and stuff in here. reporter: and outreach teams hitting the streets to care for the needy. if anyone is out and sees someone else who might be in distress, please call 911. reporter: doctors say hypothermia can set in in just minutes. this weather can be deadly. it\'s life threatening. this is something that people have to really take seriously. reporter: the arctic front visible in the form of powerful snow squalls pushing from upstate new york through new england. here in boston, a brutal night ahead. a cold emergency is in place. winds will gust to 40 miles per hour, so there\'s a real concern about power outages, and it will feel like at least 30 below. david? victor oquendo in boston. victor, thank you. let\'s get to rob marciano who\'s timing this out for us tonight. hey, rob. reporter: hi, david. if you live in the northeast, you\'ve felt it all day long. temperatures have been dropping. now these winds are really ripping, measured over 40 miles per hour already. and the wind chill advisory is pretty much everywhere north of new york city. here you see it on the map. it\'s no picnic south of here either. minus 23 for a wind chill right now in albany. minus 13 in boston. but numbers in the morning will be even more dangerous, and that red spot blizzard warning, that\'s for blowing snow on the ground. minus 57 in caribou. we get to that figure or less, that\'s an all-time record. minus 11 in new york. this is hypothermic stuff, frostbite in minutes. but a rapid rebound in temperatures coming during the day sunday. we\'ll start in single digits in many spots all the way into the 40s for sunday, monday, and tuesday. so it\'s just a matter of survival for the next 36 hours. david? whiplash for sure, but take saturday seriously. rob, we thank you. get back inside. we\'re going to turn next to the doctor struck by a car. police say he was then attacked. he did not survive. that emergency room doctor riding his bike on the pacific coast highway, struck by a driver. police say the driver then got out of the car and stabbed the doctor. bystanders taking the attacker down. here\'s our chief correspondent matt gutman from california tonight. reporter: tonight, an e.r. doctor is dead after police in california say a motorist slammed into him, then got out of his car and stabbed him. according to the orange county sheriff\'s department, around 3:00 p.m. wednesday this white car broadsided dr. michael mammone as he was on his bike at a crosswalk on the much-touristed pacific coast highway. at that point, the suspect drove around the victim, exited his vehicle, went back to the victim and continued to assault the victim. reporter: a criminal complaint alleging the car\'s driver, vanroy smith, then attacked mammone with a knife. bystanders stepping in to detain smith. but mammone, an emergency medicine specialist, later dying at a hospital. tonight, providence mission hospital where mammone worked saying they are grieving over the loss of an incredible physician and friend. david, that suspect arrested, charged with murder and the use of a deadly weapon. authorities say they know of no connection between these two men and know of no motive for this crime. smith has pleaded not guilty. bail has been set at $1 million. david? just horrific. matt gutman tonight, thank you. now to the scare at newark airport. two passenger jets clipping wings. gio benitez is there. reporter: tonight, travelers shaken after an aircraft clipped another plane\'s wing while it was getting ready to take off for orlando, filled with passengers. all of a sudden you hear and feel a jolt. reporter: united says the orlando plane was sitting at a gate, preparing to leave newark when its left wing was struck by another united plane. that plane empty and being towed to a nearby gate. rebecca bloom was on board. the wing of a larger plane had clipped our wing. reporter: you can see the damage in these photos obtained by abc news. the port authority police, aircraft rescue firefighter unit carrying a piece of the wing. i think we were thinking, what was that? but we all saw it, and i think our jaws dropped. reporter: the incident happening just weeks after a dramatic near miss at jfk airport. a delta jet coming within seconds of slamming into an american airlines plane that was crossing the wrong runway.

Still blocked:

{
  "predictions": [
    {
      "content": "",
      "safetyAttributes": {
        "blocked": true
      }
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalBillableCharacters": 4519,
        "totalTokens": 1282
      },
      "outputTokenCount": {
        "totalBillableCharacters": 0,
        "totalTokens": 9
      }
    }
  }
}

At this point we'll abandon this approach and give up on this specific workflow.

Let's take a look at how the story was divided using a temperature of 0.2. Each of the bullets below represents a story division according to this workflow:

the deadly cold already tonight, the national weather service is calling it a once in a generation arctic blast. 25 million americans, multiple states. new york city, philadelphia, boston, wind-chill readings in maine expected to reach 60 below zero. and where the wind chill is already 106 degrees below zero tonight. victor oquendo in massachusetts. rob marciano timing this out.
the deadly assault on an e.r. doctor riding his mountain bike. police say the suspect in his car hitting the victim, then allegedly getting out and attacking him. matt gutman reporting.
another scare at the airport. this time, newark. two passenger jets clipping wings. one plane full of passengers.
the stunning and unexpected jobs report tonight. president biden celebrating the numbers. the unemployment rate not this low since 1969. so, how does this complicate steps by the fed moving forward? rebecca jarvis, as always, breaking it down.
the alex murdaugh double murder trial. the defense team, and what a ballistics expert acknowledged today.
and tonight, the story of alex murdaugh's housekeeper dying in a fall at the home. what the family says they never knew. prosecutors say murdaugh got a $5 million payout from a policy that covered the housekeeper's fall. but will the jury hear about it? eva pilgrim reporting.
robin roberts tonight and her epic jump. robin from new zealand, the birthplace of the bungee jump. you'll see her jump from the tallest building in the southern hemisphere. tonight, the lesson on conquering fear.
good evening. it is great to have you with us on a friday night. almost to the weekend, but it will be brutally cold, dangerous, life-threatening cold. the readings coming in right now, and where the wind chill is already more than 100 degrees below zero. that's in a moment here. but we begin with the chinese spy balloon hovering over the u.s. tonight.
china claiming it is a weather balloon that blew off course. u.s. authorities believe this is a surveillance balloon being maneuvered by china, likely using spy satellites. today the balloon was spotted over missouri. so where is it tonight, and where u.s. authorities believe it's headed. among the first sightings over billings, montana. that intelligence bay hanging beneath the balloon, that's the size of three busses. the pentagon tonight on discussions over whether or not to shoot it down, and will there be a small window of opportunity to do just that? what we're learning tonight. secretary of state antony blinken postponing his high-stakes trip to beijing, calling china's actions unacceptable. so, will this be shot down? if so, where? and what china is saying tonight. mary bruce at the white house, and abc's mola lenghi leading us off tonight in montana.
reporter: tonight, the pentagon confirming the massive chinese spy balloon is on the move 60,000 feet above the ground and heading east. the balloon continues t move eastward and is currently over the center of the continental united states. what the heck is that? reporter: a senior u.s. official tells abc news the balloon now appears headed towards north carolina. across the country today, americans with their eyes on the skies, posting images like this one, the balloon floating over missouri. commercial pilots radioing in. we got that balloon in sight also. looks like it's way up there, maybe 50,000 feet or so. reporter: the balloon is huge with a technology bay attached below that is itself the size of three busses, loaded with high resolution cameras, according to a senior u.s. official, equipped with what appear to be solar panels on the side that could power its technology. it first entered american airspace over alaska, then flying into southwest canada before dipping down over billings, montana. this thing is up in the sky. what the heck is that? that thing is not the moon. any help would be appreciated. reporter: montana republican congressman ryan zinke was blunt, tweeting "shoot. it. down." the pentagon today sayig that option was considered and rejected, for now. we assessed that currently it does not pose a physical or military risk to people on the ground. for now we are continuing to monitor and review options. reporter: the white house today saying president biden agreed with the pentagon's strong recommendation. the risks involved with shooting down the balloon just too great. any potential debris field would be significant and potentially cause civilian injuries or deaths or significant property damage. reporter: for its part, china insisting today the balloon is not a spy vessel at all but a civilian airship used for research, mainly meteorological purposes. the pentagon flatly rejecting that claim. we are aware of the prc's statement. however, the fact is, we know that it's a surveillance pballo. we have clearly communicated that this balloon is violating u.s. air space and international law and that this is unacceptable. reporter: china says the balloon deviated far from its planned course, but the pentagon questioning that as well, saying the balloon can be maneuvered. a senior u.s. official telling abc news, it's likely being navigated with spy satellites. this all happening as secretary of state antony blinken prepared for a weekend trip to beijing. that visit now canceled. in light of china's unacceptable action, i am postponing my planned travel this weekend to china. we concluded that conditions were not conducive for a constructive visit at this time. reporter: there have been other foreign surveillance balloons above the u.s., but none like this. what makes this different is the duration and the length of which it has been over u.s. territory. reporter: the pentagon says the balloon is expected to linger above america for a few days.
this is really something. let's bring in mola lenghi. he's in billings, montana, for us again tonight. that state with several sensitive nuclear sites. as you reported, u.s. authorities say the balloon is likely navigated by chinese satellites. the pentagon says they've decided not to shoot it down at this point, but that option isn't completely off the table? reporter: that's right, david, a u.s. official familiar with the situation says the current thinking is to wait for the balloon to drift over the atlantic ocean and then shoot it down once it's over u.s. territoryial waters. that way you can avoid injuries on the ground, and two, retrieve the balloon, study it, and ultimately exploit the intelligence. david, tonight china says it regrets the unintended entry of the air ship into u.s. air space, but as you can imagine, not exactly flying with u.s. national security officials who are calling this unacceptable.
that way you can avoid injuries on the ground, and two, retrieve the balloon, study it, and ultimately exploit the intelligence. david, tonight china says it regrets the unintended entry of the air ship into u.s. air space, but as you can imagine, not exactly flying with u.s. national security officials who are calling this unacceptable.
mola lenghi, thank you. let's bring in mary bruce. president biden chose not to talk about this today, trying not to enflame the situation. what are you learning from the white house tonight? obviously the president continuing to be briefed. and on mola's point, if you shot this down over u.s. waters, that still is a small window of opportunity to pull this off.
reporter: yes, david. we know the president is being regularly briefed. his team presented him with all options, including the possible military action to take the balloon down. now, initially biden was advised against that because of the concern over civilian risk, but tonight we have learned that option may be back on the table. one source telling us the thinking here is to wait until the balloon goes out to see and then shoot it down, but this is very tricky. they would have just 12 nautical miles to do this before entering international waters and certainly don't want to start an international incident over this. tonight we are told the white house is keeping communications to china open. but even though beijing expressed regret, it was not enough to salvage secretary blinken's trip. what is clear this is only further escalating tensions between our two countries.
yeah, no question. you mentioned those 12 nautical miles would be the opportunity. mary, thank you. of course the other major concern the concern over this dangerous life-threatening cold. they're calling it a once in a generation arctic blast that has already turned deadly tonight. 25 million americans in ten states. wind chills below zero in philadelphia, new york, and boston. and look at this tonight. this is the view from a camera on mt. washington in new hampshire. they've already set a preliminary record at this hour. the wind chill moments ago, 106 degrees below zero. if confirmed it's the coldest windchill ever recorded in the u.s., including alaska. snow squalls along with that arctic chill. a time lapse from vermont. a cold emergency in place in boston. this is through sunday now. rob marciano standing by to time this out, and abc's victor oquendo in boston where the wind chill is 13 below zero right now.

Overall the results are reasonable, though the immense workflow required to get here and the number of workarounds and additional logic required makes such segmentation intractable for the quality of results.