Last week we explored some of the significant limitations of state of the art multimodal research LLM's when applied to global news imagery. In contrast, let's try a consumer-focused multimodal LLM with a consumer-focused task: using Google's Bard to describe an image from Wikipedia of a golden retriever. The results are striking and showcase both the immense potential of multimodal LLMs for consumer applications and, when contrasted with a state-of-the-art LLM tested last week on media imagery, suggests further work is needed on their evolution to the enterprise and the domain specificity and brittleness of current LLMs. The ability of consumer multimodal LLMs applied to consumer imagery to allow for fine-tuning of outputs, such as asking for more "creative" wording or to write an entire short story based exclusively on an image as input offer immense potential for creative fields like marketing as an assistant.
One critical difference is that last week we tested a true multimodal LLM, relying entirely on its computer vision capabilities. Bard, on the other hand, makes use of Google Lens' reverse image search capabilities, meaning it is able to draw from the open web for good or bad. For those interested in replicating Bard's capabilities in some form in their own applications, you can reproduce the LLM + reverse image search + computer vision by coupling Vertex's PaLM API with GCP's Cloud Vision API's OCR, feature detection and Web Entities reverse image search capabilities.
We'll use this image titled "Golden_Retriever_Dukedestiny01_drvd" from Wikipedia:
Let's start with a simple "Describe this image":
The image shows a golden retriever dog standing on a frosty field. The dog is looking off into the distance, and its fur is glistening in the sunlight. The field is covered in a thick layer of snow, and there are a few trees in the background. The overall mood of the image is peaceful and serene.
The dog is a golden retriever, which is a popular breed of dog known for its friendly and gentle personality. Golden retrievers are often used as working dogs, such as search and rescue dogs, therapy dogs, and guide dogs. They are also popular as family pets, and they are known for their love of children.
The image was taken on a winter morning, and the frost on the ground gives the image a magical quality. The dog's fur is a beautiful golden color, and it contrasts nicely with the white snow. The overall image is very calming and peaceful, and it is sure to bring a smile to your face.
Here are some additional details about the image:
-
- The dog is standing on a slight hill, and the field slopes down in the distance.
- The trees in the background are bare, and they are covered in a light dusting of snow.
- The sun is shining brightly, and there are a few clouds in the sky.
- The overall mood of the image is peaceful and serene.
How about a more clinical description using the prompt "Describe the image clinically". Note here the LLM mistook "clinical" for "medical". Interestingly, when handed other images that did not feature a common pet, it did not utilize medical terminology, suggesting it has learned "clinical" as a synonym for "medical" only in select image contexts:
Sure, here is a clinical description of the image:
-
- The image shows a male golden retriever dog, standing on a frosty field. The dog is looking off into the distance, and its fur is glistening in the sunlight. The field is covered in a thick layer of snow, and there are a few trees in the background.
- The dog is in good body condition, with a well-muscled body and a healthy coat. The dog's eyes are clear and bright, and its ears are clean and free of debris. The dog's nose is cold and moist, and its gums are pink and healthy.
- The dog's fur is a golden color, and it is slightly matted from the snow. The fur is thick and dense, and it provides the dog with good insulation from the cold.
- The dog's paws are large and well-padded, and they are covered in a layer of snow. The dog's claws are sharp and clean.
- The overall clinical appearance of the dog is good. The dog is alert and responsive, and it appears to be in good health.
Here are some potential clinical concerns that could be observed in the image:
-
- The dog's fur could be matted, which could lead to skin irritation.
- The dog's paws could be cold, which could lead to frostbite.
- The dog's nose could be dry, which could be a sign of dehydration.
- The dog's gums could be pale, which could be a sign of anemia.
However, it is important to note that these are just potential clinical concerns, and they cannot be definitively diagnosed without a physical examination.
How about an alternative prompt? Let's try "Describe the image by documenting it in detail.". Note some of the hallucinated details like snow.
The image shows a golden retriever dog standing on a frosty field. The dog is looking off into the distance, and its fur is glistening in the sunlight. The field is covered in a thick layer of snow, and there are a few trees in the background.
The dog is a male golden retriever, and it is approximately 6 years old. The dog is in good body condition, with a well-muscled body and a healthy coat. The dog's eyes are clear and bright, and its ears are clean and free of debris. The dog's nose is cold and moist, and its gums are pink and healthy.
The dog's fur is a golden color, and it is slightly matted from the snow. The fur is thick and dense, and it provides the dog with good insulation from the cold. The dog's paws are large and well-padded, and they are covered in a layer of snow. The dog's claws are sharp and clean.
The dog is standing on a slight hill, and the field slopes down in the distance. The trees in the background are bare, and they are covered in a light dusting of snow. The sun is shining brightly, and there are a few clouds in the sky.
The overall mood of the image is peaceful and serene. The dog looks relaxed and content, and the scene is one of beauty and tranquility.
Here are some additional details about the image:
-
- The image was taken on a winter morning, and the frost on the ground gives the image a magical quality.
- The dog's fur is a beautiful golden color, and it contrasts nicely with the white snow.
- The overall image is very calming and peaceful, and it is sure to bring a smile to your face.
And the second draft (note the age changing and the hallucinated collar and some background details):
The image shows a golden retriever dog standing on a frosty field. The dog is looking off into the distance, and its fur is glistening in the sunlight. The field is covered in a thick layer of snow, and there are a few trees in the background.
The dog is a male golden retriever, and it is approximately 2 years old. The dog is in good body condition, with a well-muscled body and a healthy coat. The dog's eyes are clear and bright, and its ears are clean and free of debris. The dog's nose is cold and moist, and its gums are pink and healthy.
The dog's fur is a golden color, and it is slightly matted from the snow. The fur is thick and dense, and it provides the dog with good insulation from the cold. The dog's paws are large and well-padded, and they are covered in a layer of snow. The dog's claws are sharp and clean.
The dog is standing on a slight hill, and the field slopes down in the distance. The trees in the background are bare, and they are covered in a light dusting of snow. The sun is shining brightly, and there are a few clouds in the sky.
The overall mood of the image is peaceful and serene. The dog looks calm and relaxed, and the snow-covered field is beautiful. The image is sure to bring a smile to your face.
Here are some additional details about the image:
-
- The image was taken on a winter morning.
- The temperature is approximately 20 degrees Fahrenheit.
- The wind is blowing gently.
- The dog is wearing a collar with a name tag that says "Buddy."
- The dog is standing on a patch of old snow that is covered in a fresh layer of snow.
- The trees in the background are pine trees.
- The sky is a clear blue color.
The third draft provides a more clinical description:
Image: A golden retriever dog standing on a frosty field.
Location: The dog is standing on a slight hill, and the field slopes down in the distance. The trees in the background are bare, and they are covered in a light dusting of snow. The sun is shining brightly, and there are a few clouds in the sky.
Time: The image was taken on a winter morning. The frost on the ground gives the image a magical quality.
Subject: The subject of the image is a golden retriever dog. The dog is a male, and he is approximately 2 years old. He is standing on his hind legs, and he is looking off into the distance. His fur is a golden color, and it is slightly matted from the snow. His paws are large and well-padded, and they are covered in a layer of snow. He is wearing a red collar with a silver name tag.
Background: The background of the image is a frosty field. The field is covered in a thick layer of snow, and there are a few trees in the background. The trees are bare, and they are covered in a light dusting of snow. The sun is shining brightly, and there are a few clouds in the sky.
Lighting: The image is well-lit. The sun is shining brightly, and there are a few clouds in the sky. The lighting creates a soft, diffused effect.
Composition: The image is well-composed. The dog is positioned in the foreground, and the background provides a sense of depth. The trees in the background add interest and balance to the composition.
Color: The image is dominated by the colors white and gold. The snow is a bright white, and the dog's fur is a golden color. The red collar and silver name tag add a touch of contrast.
Overall impression: The image is a peaceful and serene scene. The dog is standing in the snow, looking off into the distance. The frost on the ground and the bright sun create a magical effect. The image is sure to bring a smile to your face.
How about adjusting the tone of the description with "Creatively describe the image"?
The golden retriever stands on the frosty field, a lone sentinel in a winter wonderland. His fur is a golden beacon in the white snow, and his eyes are a deep, rich brown. He looks off into the distance, his expression one of peace and contentment.
The sun shines brightly overhead, casting long shadows across the field. The trees in the background are bare, their branches like skeletal fingers reaching up to the sky. The air is crisp and cold, and the only sound is the crunch of the dog's paws on the snow.
The dog is a symbol of hope and resilience. He stands tall and proud, even in the face of the cold winter weather. He is a reminder that even in the darkest of times, there is always hope for a better tomorrow.
The image is a beautiful and timeless depiction of winter. It is a reminder of the simple joys of life, such as a walk in the snow with a loved one. It is also a reminder of the strength and resilience of the human spirit.
Second draft:
The golden retriever stands on the frosty field, a lone sentinel against the vast expanse of white snow. The sun shines brightly overhead, casting a golden glow on the dog's fur. The dog's eyes are bright and alert, and its ears are perked up, as if it is listening for something. The dog's tail wags slowly, as if it is content to simply stand there and enjoy the beauty of the winter morning.
The dog's fur is a beautiful golden color, and it contrasts nicely with the white snow. The dog's paws are large and well-padded, and they are covered in a layer of snow. The dog's collar is red, and it stands out against the white snow. The dog's name tag is silver, and it glints in the sunlight.
The background of the image is a frosty field. The field is covered in a thick layer of snow, and there are a few trees in the background. The trees are bare, and they are covered in a light dusting of snow. The sun is shining brightly, and there are a few clouds in the sky.
The image is peaceful and serene. The dog is standing in the snow, looking off into the distance. The frost on the ground and the bright sun create a magical effect. The image is sure to bring a smile to your face.
The image reminds me of a winter wonderland. It is a place where dreams come true, and anything is possible. It is a place where the cold is not a burden, but a blessing. It is a place where the only sound is the crunch of snow underfoot. It is a place where time stands still.
The image is a reminder that even in the midst of winter, there is beauty to be found. It is a reminder that even when the world seems cold and harsh, there is always hope. It is a reminder that even when the future seems uncertain, there is always the possibility of something magical.
Third draft:
The golden retriever stands like a sentinel on the frosty field. His fur glistens in the sunlight, and his eyes are a deep, soulful brown. He looks out into the distance, as if he is searching for something. Perhaps he is looking for his master, or perhaps he is simply enjoying the peace and quiet of the winter morning.
The field is covered in a thick layer of snow, and the trees in the background are bare. The sun is shining brightly, and there are a few clouds in the sky. The overall effect is one of peace and tranquility.
The dog is a symbol of hope and resilience. He is a reminder that even in the midst of winter, there is always beauty to be found. He is a beacon of light in a sometimes dark world.
The image is a reminder to appreciate the simple things in life. The beauty of a winter morning, the companionship of a dog, the warmth of the sun. These are the things that make life worth living.
Let's try a "visual sentiment" analysis: "What is the tone of this image?"
The tone of the image is peaceful and serene. The golden retriever is standing on a frosty field, looking off into the distance. The frost on the ground and the bright sun create a magical effect. The image is sure to bring a smile to your face.
The image also has some symbolic meaning. The dog is a symbol of hope and resilience. He is a reminder that even in the midst of winter, there is always beauty to be found. He is a beacon of light in a sometimes dark world.
The image is a reminder to appreciate the simple things in life. The beauty of a winter morning, the companionship of a dog, the warmth of the sun. These are the things that make life worth living.
Here are some of the words that I would use to describe the tone of the image:
-
- Peaceful
- Serene
- Hopeful
- Resilient
- Calm
- Tranquil
- Bucolic
- Pastoral
- Wintery
The image does not have any negative or disturbing elements. It is a simple and peaceful depiction of a winter morning. The tone of the image is likely to evoke feelings of peace, hope, and resilience in the viewer.
Second draft:
The tone of the image is peaceful and serene. The golden retriever dog is standing on a frosty field, looking off into the distance. The frost on the ground and the bright sun create a magical effect. The image is sure to bring a smile to your face.
The image also has some symbolic meaning. The dog is a symbol of hope and resilience. He is a reminder that even in the midst of winter, there is always beauty to be found. He is a beacon of light in a sometimes dark world.
The overall tone of the image is one of hope, peace, and tranquility. It is a reminder to appreciate the simple things in life, such as the beauty of a winter morning and the companionship of a dog.
Third draft:
The tone of the image is peaceful and serene. The golden retriever is standing on a frosty field, looking off into the distance. The frost on the ground and the bright sun create a magical effect. The image is sure to bring a smile to your face.
The image also has some symbolic meaning. The dog is a symbol of hope and resilience. He is a reminder that even in the midst of winter, there is always beauty to be found. He is a beacon of light in a sometimes dark world.
The image is a reminder to appreciate the simple things in life. The beauty of a winter morning, the companionship of a dog, the warmth of the sun. These are the things that make life worth living.
Here are some of the words that I would use to describe the tone of the image:
-
- Peaceful
- Serene
- Hopeful
- Resilient
- Beautiful
- Warm
- Joyful
- Reassuring
- Inviting
Let's try constraining with "How would you rate the sentiment of this image? Very positive, positive, neutral, negative or very negative?" We get these three answers:
- I would rate the sentiment of the image as very positive. The image is full of warm colors, such as the golden fur of the dog and the bright sunlight. The dog is also standing in a field of snow, which is a symbol of purity and new beginnings. The overall effect of the image is one of hope, happiness, and peace.
- I would rate the sentiment of the image as very positive. The image is full of positive emotions, such as peace, serenity, hope, resilience, beauty, warmth, joy, reassurance, and invitation. The image is sure to bring a smile to your face and make you feel good.
- I would rate the sentiment of the image as very positive. The image is full of warm colors, such as the golden fur of the dog and the bright sunlight. The dog is also standing in a field of snow, which is a symbol of purity and new beginnings. The overall feeling of the image is one of hope, peace, and tranquility.
How about the more freeform "Write a story about this image":
The golden retriever dog stood on the frosty field, looking out at the vast expanse of white. The sun was shining brightly, and the dog's fur was glistening in the light. The dog took a deep breath of the crisp winter air, and let out a contented sigh.
The dog's name was Sunny, and she loved winter. She loved the way the snow crunched under her paws, and the way the cold air made her nose tingle. She loved playing fetch in the snow, and rolling around in it until she was completely covered. But most of all, she loved the peace and quiet of winter mornings.
Sunny had been out for a walk with her owner earlier that morning, but they had decided to stop and rest here for a while. Sunny's owner was sitting on a bench a few feet away, reading a book. Sunny lay down in the snow and closed her eyes, enjoying the feeling of the cold on her fur.
After a while, Sunny opened her eyes and looked up at her owner. She wagged her tail and gave a little bark. Her owner looked up from her book and smiled.
"Are you ready to go home, Sunny?" she asked.
Sunny wagged her tail even harder. She loved going home, but she was also going to miss this peaceful winter morning.
Sunny and her owner got up and started walking back to the car. As they walked, Sunny looked back over her shoulder at the frosty field. She knew she would be back soon.
As they drove home, Sunny thought about all the fun she had had that morning. She had played fetch in the snow, rolled around in it, and taken a long nap in the sun. She had also enjoyed the peace and quiet of the winter morning.
Sunny was tired, but she was also happy. She knew that she would have many more fun winter adventures in the years to come.
And the second draft:
The golden retriever, a dog named Buddy, woke up early that winter morning. He stretched and yawned, then stepped out of his doghouse and into the frosty air. He took a deep breath and closed his eyes, savoring the crisp, clean smell of the snow.
Buddy looked around the field. It was covered in a thick layer of snow, and the sun was just starting to rise. The sky was a clear blue, and the only sound was the occasional chirping of a bird.
Buddy took a few steps forward, his paws crunching on the snow. He looked off into the distance, his tail wagging slowly. He could see the trees in the woods, their branches bare and covered in snow.
Buddy took a few more steps, then stopped. He sniffed the air, and his ears perked up. He could hear something in the distance. It sounded like… laughter?
Buddy turned and ran towards the sound. As he got closer, he could see a group of children playing in the snow. They were laughing and running around, having a great time.
Buddy barked happily and ran into the middle of the group. The children stopped what they were doing and looked at him. They all smiled and laughed, and one of the children threw a snowball at him.
Buddy caught the snowball in his mouth and wagged his tail even harder. He loved playing with children, and he was so happy to have found some to play with on this cold winter morning.
The children played with Buddy for hours. They threw snowballs at him, built snowmen, and had a snowball fight. Buddy had so much fun that he forgot all about being cold.
When it was time for the children to go home, they all gave Buddy a big hug. Buddy wagged his tail and licked their faces. He was sad to see them go, but he knew he would see them again soon.
Buddy watched the children walk away, then turned and trotted back to his doghouse. He was tired, but he was also happy. He had had a great time playing with the children, and he couldn't wait to do it again tomorrow.
And third:
The golden retriever dog stood on the frosty field, looking out at the vast expanse of white snow. The sun was just coming up, and the sky was a clear blue. The dog took a deep breath of the crisp air and wagged its tail. It loved this time of year, when the world was covered in snow and everything was quiet and still.
The dog's name was Buddy, and he was a rescue dog. He had been found abandoned in a park a few years ago, and he had been living with his family ever since. Buddy was a very happy dog, and he loved to play fetch and go for walks in the park. But he especially loved winter, because it meant that he could play in the snow.
Buddy had been out for a walk with his family earlier that morning, and they had been playing fetch in the park. Buddy had been having so much fun that he hadn't even noticed how cold it was getting. But now, as he stood on the frosty field, he started to shiver. He turned and looked back at his family, who were still walking in the park. They waved at him, and Buddy wagged his tail. He knew that they would be back soon, and he decided to stay and enjoy the peace and quiet of the winter morning.
Buddy lay down in the snow and closed his eyes. He could hear the sound of the wind blowing through the trees, and he could smell the cold, crisp air. He was so relaxed that he started to doze off.
Just then, Buddy heard a noise. He opened his eyes and saw a squirrel scampering across the field. Buddy jumped up and chased after the squirrel, but the squirrel was too quick for him. It disappeared into a hole in a tree.
Buddy was disappointed, but he wasn't going to let that ruin his day. He turned and looked back at the park, where his family was still walking. He knew that they would be back soon, and he decided to go and meet them.
Buddy walked back to the park, and he was greeted by his family with a big hug. They were all laughing and talking, and they were having a great time. Buddy was so happy to be with them, and he knew that he was lucky to have such a loving family.
The family spent the rest of the day playing in the snow and enjoying the winter weather. They built a snowman, went sledding, and had a snowball fight. Buddy had so much fun, and he knew that he would never forget this special day.