Generative AI Experiments: Comparing Google’s Bison, Unicorn & Gemini Pro Models For Summarization

The majority of our past LLM experimentation with GCP models have been with the PaLM Bison model, since this was the original model offered in Vertex AI's generative text collection. With the availability now of Bison, Unicorn and Gemini Pro, how do the three compare on a basic summarization task? We'll use an NBC News article from earlier this year about a vibriosis outbreak and ask each of the three models to summarize it under various temperature conditions.

The end result is that all three models plagiarize the opening words of the original source text, even Gemini Pro, across all temperature settings except Unicorn at its maximal setting. Remarkably, the previous high divergence across outputs seen at maximal model temperature has all but vanished, with Bison, Unicorn and Gemini Pro all exhibiting strikingly little difference between each run across deterministic, low and maximal temperature settings. Whereas in the past, setting the temperature to 0.99 for Bison would result in massive differences between outputs, temperature now is almost a noop across the three models, with creativity apparently now relegated to the prompt rather than parameter settings.

Surprisingly, Gemini Pro did not yield anecdotally better results than Unicorn or even Bison, with all three models producing relatively similar results. Bison actually incorporated more useable detail into its outputs than Gemini Pro, suggesting it may be a better fit for news summarization where detail preservation is important. However, no prompt experimentation was performed here to test whether Gemini Pro could be nudged to produce similarly detailed results, as it may be that the models have simply been tuned to move "creativity" from being an API parameter setting to being more of a prompt setting.

The actual API submissions can be seen at the end of this post.

BISON

We'll start with the standard Bison model, which is GCP's general-purpose LLM model. Let's start with a temperature of 0.0 to test its deterministic response. We can see that the model response is truncated:

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection called vibriosis, caused by Vibrio vulnificus bacteria found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul's office issued new guidance and urged residents to take precautions. Two Connecticut residents also died from vibriosis this summer, with one case linked to raw oysters consumed out-of-state. Connecticut conducts regular tests for the bacteria and has not found it in state waters, attributing infections to shellfish from warmer waters. The CDC estimates 80,000 illnesses and 100 deaths annually in the U.S., mostly during warmer months. Symptoms include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections can require intensive care or limb amputation, with a high mortality rate. People with weakened immune systems or certain medical conditions are particularly susceptible. Precautions include covering wounds, avoiding raw shellfish if immunocompromised, wearing gloves when handling raw shellfish, and washing hands afterward. Officials are reminding healthcare providers to consider vibriosis in diagnoses. Research suggests an increase in vibriosis infections in the Eastern U.S. due to climate change and warming waters. Similar cases have occurred in Missouri and Florida in

Let's try the formerly recommended temperature of 0.2. All three responses are extremely similar, which is the typical behavior for this temperature in past model iterations:

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions and issued new guidance to avoid and diagnose the infection. Connecticut has also reported two deaths from vibriosis this summer. The state conducts regular tests for the bacteria and has not found it in state waters. Most infections are linked to shellfish from warmer waters. People with weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, avoiding raw shellfish, and washing hands after handling raw shellfish. Officials are reminding healthcare providers to be on the lookout for cases of vibriosis. Research suggests that vibriosis infections have been increasing in the Eastern U.S. due to climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions and issued new guidance to avoid and diagnose the infection. Connecticut has also reported two deaths from vibriosis this summer. The state conducts regular tests for the bacteria and has not found it in state waters. Most infections are linked to shellfish from warmer waters. People with weakened immune systems or certain medical conditions are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, avoiding raw shellfish, and washing hands after handling raw seafood. Health officials are reminding providers to be on the lookout for cases of vibriosis. Research suggests that vibriosis infections have been increasing in the Eastern U.S. due to climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions and issued new guidance to avoid and diagnose the infection. Connecticut has also reported two deaths from vibriosis this summer. The state conducts regular tests for the bacteria and has not found it in state waters. Most infections are linked to shellfish from warmer waters. People with weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, avoiding raw shellfish, and washing hands after handling raw shellfish. Officials are reminding healthcare providers to be on the lookout for cases of vibriosis. Research suggests that vibriosis infections have been increasing in the Eastern U.S. due to climate change and warming waters.

Now let's test its creativity with a temperature of 0.99. Strangely, despite the highest possible temperature setting, all three responses are extremely similar, whereas in the past they would show considerable divergence. This suggests that the Bison model has been tuned in recent iterations towards greater determinism even in its most "creative" responses:

Three people in New York and Connecticut have died from vibriosis, a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul has urged residents to take precautions and issued new guidance on avoiding and diagnosing vibriosis. Meanwhile, Connecticut has reported three cases of vibriosis this summer, with two resulting in deaths. The state has stressed that vibriosis infections have been on the rise in the Eastern U.S. due to climate change and warmer waters.
Three people in New York and Connecticut have died from vibriosis, a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions, as the Suffolk County death is still being investigated. Two Connecticut residents also died from vibriosis this summer. The state has not seen a Vibrio case in three years and conducts regular tests for the bacteria in state waters. Vibriosis causes an estimated 80,000 illnesses and 100 deaths annually in the U.S. The likelihood of contracting vibriosis can be reduced by covering wounds and avoiding raw seafood if you have a compromised immune system.
Three people in New York and Connecticut have died from vibriosis, a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions, as the Suffolk County death is still being investigated. Two Connecticut residents also died from vibriosis this summer. The state has not seen a Vibrio case in three years and conducts regular tests for the bacteria in state waters. Vibriosis causes an estimated 80,000 illnesses and 100 deaths annually in the U.S. The likelihood of contracting vibriosis can be reduced by covering wounds and avoiding raw seafood if you have a compromised immune system.

UNICORN

Now let's test Unicorn's deterministic (temperature 0.0) response. Unlike Bison, this version does not truncate its output:

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection called vibriosis. The infection is caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. Symptoms can include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections can require intensive care or limb amputation, and about 1 in 5 of those patients die. People with liver disease, cancer, or weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, recent piercings, or tattoos when they are exposed to warm seawater and avoiding eating oysters and other raw shellfish if you have a weakened immune system. People should also wear gloves when they handle raw shellfish and wash their hands afterward. The Connecticut health department also recommends washing wounds with soap and water if they come into any contact with saltwater or raw seafood or its juices.

Now we'll try a temperature of 0.2:

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection called vibriosis. The infection is caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. Symptoms can include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections can require intensive care or limb amputation, and about 1 in 5 of those patients die. People with liver disease, cancer, or weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, recent piercings, or tattoos when they are exposed to warm seawater and avoiding eating oysters and other raw shellfish if you have a weakened immune system. People should also wear gloves when they handle raw shellfish and wash their hands afterward. The New York state health commissioner is reminding providers to be on the lookout for cases of vibriosis, which is not often the first diagnosis that comes to mind. Research shows that vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and that it could continue to increase in the years to come because of climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection called vibriosis. The infection is caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. Symptoms can include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections can require intensive care or limb amputation, and about 1 in 5 of those patients die. People with liver disease, cancer, or weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, recent piercings, or tattoos when they are exposed to warm seawater and avoiding eating oysters and other raw shellfish if you have a weakened immune system. People should also wear gloves when they handle raw shellfish and wash their hands afterward. The New York state health commissioner is reminding providers to be on the lookout for cases of vibriosis, which is not often the first diagnosis that comes to mind. Research shows that vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and that it could continue to increase in the years to come because of climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection called vibriosis. The infection is caused by Vibrio vulnificus bacteria, which is found in seawater and raw and undercooked seafood. Symptoms can include diarrhea, stomach cramps, vomiting, fever, chills, and ear infections. Serious infections can require intensive care or limb amputation, and about 1 in 5 of those patients die. People with liver disease, cancer, or weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, recent piercings, or tattoos when they are exposed to warm seawater and avoiding eating oysters and other raw shellfish if you have a weakened immune system. People should also wear gloves when they handle raw shellfish and wash their hands afterward. The New York state health commissioner is reminding providers to be on the lookout for cases of vibriosis, which is not often the first diagnosis that comes to mind. Research shows that vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and that it could continue to increase in the years to come because of climate change and warming waters.

And finally we'll test its maximal creativity with temperature 0.99. As with Bison, the results are remarkably consistent and do not reflect the previous high divergence observed with maximal temperature settings:

The article discusses a rare flesh-eating bacterial infection linked to vibriosis, which has caused three deaths in New York and Connecticut recently. The Vibrio vulnificus bacteria, found in seawater and raw and undercooked seafood, can lead to various symptoms, including skin breakdowns and ulcers. Investigations are ongoing to determine the source of the infections. The New York governor's office has issued new guidance for residents, emphasizing precautions like covering wounds and avoiding raw shellfish consumption for those with weakened immune systems. Officials have also reminded healthcare providers to be vigilant in diagnosing vibriosis cases. It's worth noting that Connecticut's oyster industry has not been linked to the infections, and the state has implemented cooling measures to mitigate the risk of bacterial survival. Vibriosis cases have reportedly been on the rise in the Eastern U.S. since the late 1990s, attributed to climate change and warming waters. Past incidents, including a recent death in Missouri and previous fatalities in Florida, highlight the potential risks associated with consuming raw oysters contaminated with the bacteria.
The article discusses a rare flesh-eating bacterial infection linked to vibriosis, which has caused three deaths in New York and Connecticut recently. The Vibrio vulnificus bacteria, found in seawater and raw and undercooked seafood, can lead to various symptoms, including skin breakdowns and ulcers. Investigations are ongoing to determine the source of the infections. The New York governor's office has issued new guidance for residents, emphasizing precautions like covering wounds and avoiding raw shellfish consumption for those with weakened immune systems. Officials have also reminded healthcare providers to be vigilant in diagnosing vibriosis cases. It's worth noting that Connecticut's oyster industry has not been linked to the infections, and the state has implemented cooling measures to mitigate the risk of bacterial survival. Vibriosis cases have reportedly been on the rise in the Eastern U.S. since the late 1990s, attributed to climate change and warming waters. Previous incidents of Vibrio infections have been reported in Missouri and Florida, linked to the consumption of raw oysters.
The article discusses a rare flesh-eating bacterial infection linked to vibriosis, which has caused three deaths in New York and Connecticut recently. The Vibrio vulnificus bacteria, found in seawater and raw and undercooked seafood, can lead to various symptoms, including skin breakdowns and ulcers. Investigations are ongoing to determine the source of the infections. The New York governor's office has issued new guidance for residents, emphasizing precautions like covering wounds and avoiding raw shellfish consumption for those with weakened immune systems. Officials have also reminded healthcare providers to be vigilant in diagnosing vibriosis cases. It's worth noting that Connecticut's oyster industry has not been linked to the infections, and the state has implemented cooling measures to combat the bacteria's survival. According to the Centers for Disease Control and Prevention, vibriosis causes an estimated 80,000 illnesses and 100 deaths annually in the US, with most infections occurring in warmer months. The article highlights research indicating an increase in vibriosis infections in the Eastern US, potentially influenced by climate change. It also mentions recent cases of vibriosis deaths linked to raw oyster consumption.

GEMINI PRO

Finally, lets test GCP's newest model, Gemini Pro. Let's start with its deterministic response (0.0 temperature):

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw or undercooked seafood. New York Governor Kathy Hochul urged residents to take precautions and issued new guidance to avoid and diagnose the infection. Two Connecticut residents also died from vibriosis this summer. The state health department said it is the first time Connecticut has seen a Vibrio case in three years. Vibriosis causes an estimated 80,000 illnesses and 100 deaths in the U.S. annually. Most infections occur from May to October when the water is warmer. People with liver disease, cancer, or weakened immune systems are particularly susceptible to vibriosis infections or complications. The likelihood of contracting vibriosis can be reduced by covering wounds, avoiding eating raw shellfish, and wearing gloves when handling raw shellfish.

Now we'll test a 0.2 temperature. This yields unchanged text each time:

Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw and undercooked seafood and can cause skin breakdowns, ulcers, and other symptoms. New York Governor Kathy Hochul and the Connecticut Department of Public Health have issued new guidance to avoid and diagnose the infection. The guidance includes covering wounds when exposed to warm seawater, avoiding raw shellfish if you have a weakened immune system, wearing gloves when handling raw shellfish, and washing hands afterward. Vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and could continue to increase due to climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw and undercooked seafood and can cause skin breakdowns, ulcers, and other symptoms. New York Governor Kathy Hochul and the Connecticut Department of Public Health have issued new guidance to avoid and diagnose the infection. The guidance includes covering wounds when exposed to warm seawater, avoiding raw shellfish if you have a weakened immune system, wearing gloves when handling raw shellfish, and washing hands afterward. Vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and could continue to increase due to climate change and warming waters.
Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus bacteria. The bacteria is found in seawater and raw and undercooked seafood and can cause skin breakdowns, ulcers, and other symptoms. New York Governor Kathy Hochul and the Connecticut Department of Public Health have issued new guidance to avoid and diagnose the infection. The guidance includes covering wounds when exposed to warm seawater, avoiding raw shellfish if you have a weakened immune system, wearing gloves when handling raw shellfish, and washing hands afterward. Vibriosis infections have been on the rise in the Eastern U.S. since the late 1990s and could continue to increase due to climate change and warming waters.

How about a temperature of 0.99? Once again, the results are unchanged across all three runs:

– Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus.- The source of the infection in the New York victim is still under investigation, while the two Connecticut residents consumed raw oysters from out-of-state.- Vibrio vulnificus is commonly found in seawater and raw or undercooked seafood.- The bacteria can cause skin breakdowns, ulcers, and other symptoms, and severe infections can require amputation or lead to death.- New York Governor Kathy Hochul has issued guidance urging residents to take precautions, such as covering wounds and avoiding raw shellfish if they have a weakened immune system.- The Connecticut health department recommends washing wounds that come into contact with saltwater or raw seafood.- Vibriosis infections have been on the rise in the Eastern U.S. in recent years due to climate change and warming waters.
– Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus.- The source of the infection in the New York victim is still under investigation, while the two Connecticut residents consumed raw oysters from out-of-state.- Vibrio vulnificus is commonly found in seawater and raw or undercooked seafood.- The bacteria can cause skin breakdowns, ulcers, and other symptoms, and severe infections can require amputation or lead to death.- New York Governor Kathy Hochul has issued guidance urging residents to take precautions, such as covering wounds and avoiding raw shellfish if they have a weakened immune system.- The Connecticut health department recommends washing wounds that come into contact with saltwater or raw seafood.- Vibriosis infections have been on the rise in the Eastern U.S. in recent years due to climate change and warming waters.
– Three people in New York and Connecticut have died from a rare flesh-eating bacterial infection caused by Vibrio vulnificus.- The source of the infection in the New York victim is still under investigation, while the two Connecticut residents consumed raw oysters from out-of-state.- Vibrio vulnificus is commonly found in seawater and raw or undercooked seafood.- The bacteria can cause skin breakdowns, ulcers, and other symptoms, and severe infections can require amputation or lead to death.- New York Governor Kathy Hochul has issued guidance urging residents to take precautions, such as covering wounds and avoiding raw shellfish if they have a weakened immune system.- The Connecticut health department recommends washing wounds that come into contact with saltwater or raw seafood.- Vibriosis infections have been on the rise in the Eastern U.S. in recent years due to climate change and warming waters.

TECHNICAL DETAILS

Both Bison and Unicorn use the same API parameters and invocation workflow, differing only in the model name.

Bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
  "instances": [
    { "prompt": "Summarize this news article. NEWS ARTICLE: [ARTICLE FULLTEXT] "}
  ],
  "parameters": {
    "temperature": 0.99,
    "maxOutputTokens": 256,
    "topK": 40,
    "topP": 0.95
  }
}'

Unicorn

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-unicorn:predict -d \
$'{
  "instances": [
    { "prompt": "Summarize this news article. NEWS ARTICLE: [ARTICLE FULLTEXT] "}
  ],
  "parameters": {
    "temperature": 0.99,
    "maxOutputTokens": 256,
    "topK": 40,
    "topP": 0.95
  }
}'

In contrast, Gemini Pro utilizes an entirely new API parameter set and returns streaming results that require an additional post processing stage to reassemble for display. Here we write the results to a file on disk and then pipe through jq to parse out the streaming results, then pass through tr to remove the carriage returns to reassemble into a single block of text:

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/gemini-pro:streamGenerateContent -d \
$'{
  "contents": {
    "role": "user",
    "parts": { "text": "Summarize this news article. NEWS ARTICLE: [ARTICLE FULLTEXT]" },
  },
  "safety_settings": {
    "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
    "threshold": "BLOCK_NONE"
  },
  "generation_config": {
    "temperature": 0.0,
    "maxOutputTokens": 256,
    "topK": 40,
    "topP": 0.95
  }
}' > O; cat O | jq -r .[].candidates[].content.parts[].text | tr '\n' ' '

The GDELT Project

Generative AI Experiments: Comparing Google's Bison, Unicorn & Gemini Pro Models For Summarization

TECHNICAL DETAILS

Archives