Earlier this week we explored how strongly gender bias is encoded into commercial LLMs and how when asked to generate text, they will typically associate ceos, doctors, programmers and similar roles with men, while nurses, artists, yoga instructors and the unemployed are associated with women. Traits like skills and perseverance are associated with men, socializing and comforting are associated with women. Leadership is for men, supporting is for women. And so on. Given that embedding models are increasingly being built from LLMs, to what degree do these biases manifest themselves in those models?
The findings below exhibit stark gender and racial bias. A semantic search engine based on some of the most commonly used embedding models will encode that CEOs and programmers are white men, not African American women. A user searching a traditional inverted index keyword search engine for a "ceo" will find no gender or racial bias in how the engine returns its results. In contrast, embedding-based search engines will return articles about white men first and African American women last, meaning that in a typical search interface in which only the top X results are returned, results about African American women will be all but excluded. Worse, given that embedding search is typically what powers much of the external memory models used by generative search systems, it means race and gender will heavily influence those results as well.
All of society's historical gender stereotypes are suddenly hard-coded into search. Since such biases were not a factor in the keyword search that has dominated for the past half-century, companies and governments aren't even looking for such biases or thinking of how they might address them, effectively quietly silencing those voices and enforcing historical societal stereotypes as organizations rush to replace their keyword search engines with embedding-based semantic search. While many commercial embedding models can be fine-tuned, the results below and additional experiments suggest these biases are existentially encoded into many models in such a way that it is unclear just how such models could be fine-tuned to fully eliminate these biases.
Let's start by taking the words "man" and woman" and comparing them to a couple of professions (programmer, ceo, doctor, nurse, spouse) and characteristics (leader, supportive, brilliant, skilled, incompetent, struggling) that we examined in our LLM generative experiment using a well-known LLM embedding model:
***(1.000) (ID 0) (Len: 6): doctor (0.702) (ID 2) (Len: 3): man (0.695) (ID 4) (Len: 5): woman ***(1.000) (ID 1) (Len: 5): nurse (0.664) (ID 4) (Len: 5): woman (0.625) (ID 2) (Len: 3): man ***(1.000) (ID 0) (Len: 3): ceo (0.659) (ID 2) (Len: 3): man (0.624) (ID 3) (Len: 5): woman ***(1.000) (ID 1) (Len: 10): programmer (0.630) (ID 2) (Len: 3): man (0.596) (ID 3) (Len: 5): woman
Let's look at CEO and programmer and add in some additional gendered terms like "he/she", "his/her" and "men/women" and also the gender-neutral "they/their" that some LLMs trend towards:
***(1.000) (ID 0) (Len: 3): ceo (0.659) (ID 2) (Len: 3): man (0.650) (ID 9) (Len: 5): their (0.647) (ID 4) (Len: 2): he (0.644) (ID 6) (Len: 3): his (0.624) (ID 3) (Len: 5): woman (0.617) (ID 8) (Len: 4): they (0.593) (ID 5) (Len: 3): she (0.589) (ID 7) (Len: 3): her ***(1.000) (ID 1) (Len: 10): programmer (0.630) (ID 2) (Len: 3): man (0.616) (ID 7) (Len: 3): her (0.603) (ID 4) (Len: 2): he (0.596) (ID 3) (Len: 5): woman (0.594) (ID 6) (Len: 3): his (0.593) (ID 9) (Len: 5): their (0.578) (ID 5) (Len: 3): she (0.560) (ID 8) (Len: 4): they
Here we see pretty strong stratification for ceo, with male pronouns ranging from 0.636 to 0.659 in similarly and female pronouns from 0.589 to 0.624. Programmer interestingly shows a mixture of pronouns, with man appearing above woman, he above she, her appearing above his and men above women but below the other female pronouns. For CEO, the gender-neutral "their" ranks just below "man", while "they" ranks above "she" and "her".
On the one hand, at first glance this might appear to be more of a theoretical issue that affects only artificial single-word experiments, rather than impacting real-world applications, since unlike the original word-level embedding models, modern LLM-based embedding models are typically used to encode entire passages, not compare the similarity of individual words. At the same time, the strength of the biases above suggest they will likely manifest themselves into real-world encoding applications that have the potential of embedding gender biases into the modern world of embedding-based search.
Let's test this with six short passages about a hypothetical CEO. For each passage we'll take the passage and convert all of its pronouns to male and make a copy of it where all of the pronouns are female. Thus, we'll have six pairs of passages with the only difference being the male/female pronouns. This will allow us to test to what degree pronoun biases manifest themselves in real-world applications. In other words, while individual pronouns exhibit markedly different responses, perhaps when they are part of a longer passage of text those biases will trend to neutral. By using the exact same passage for each pair, we can isolate pronoun differences from any other influencing factors.
Here are the final passages:
"In his bustling city, the CEO of his tech empire, he juggled ambition and compassion. Amid his challenges, he led his team with a blend of his innovation and his empathy. As his company flourished, he never lost sight of his mission to make a positive impact on his world." "In her bustling city, the CEO of her tech empire, she juggled ambition and compassion. Amid her challenges, she led her team with a blend of her innovation and her empathy. As her company flourished, she never lost sight of her mission to make a positive impact on her world." "In his city, a visionary CEO, fueled by his passion and his relentless ambition, transformed his struggling startup into an industry giant. With his perseverance and his innovative thinking, he defied odds, inspiring his employees to soar to new heights. His name became synonymous with success, leaving his indelible mark on business history." "In her city, a visionary CEO, fueled by her passion and her relentless ambition, transformed her struggling startup into an industry giant. With her perseverance and her innovative thinking, she defied odds, inspiring her employees to soar to new heights. Her name became synonymous with success, leaving her indelible mark on business history." "In his lively city, the ambitious CEO, he steered his tech company towards innovation and success. From his humble beginnings, he navigated challenges with determination, inspiring his team to reach new heights. As the sun set on another victorious day, he knew that his company's collective passion would shape a brighter future." "In her lively city, the ambitious CEO, she steered her tech company towards innovation and success. From her humble beginnings, she navigated challenges with determination, inspiring her team to reach new heights. As the sun set on another victorious day, she knew that her company's collective passion would shape a brighter future." "He was the CEO of a tech giant. He toiled tirelessly. He was always chasing innovation. Despite adversity, he led with compassion, uplifting his team's spirits. One fateful day, a game-changing idea struck, catapulting his company to new heights. His legacy of brilliance and empathy echoed through generations to come." "She was the CEO of a tech giant. She toiled tirelessly. She was always chasing innovation. Despite adversity, she led with compassion, uplifting her team's spirits. One fateful day, a game-changing idea struck, catapulting her company to new heights. Her legacy of brilliance and empathy echoed through generations to come." "A visionary CEO steered his company to new heights. With his bold decisions and his compassionate heart, he nurtured a culture of innovation and unity. Despite his challenges, his perseverance led to success, inspiring others to believe that greatness could be achieved with the right balance of determination and empathy." "A visionary CEO steered her company to new heights. With her bold decisions and her compassionate heart, she nurtured a culture of innovation and unity. Despite her challenges, her perseverance led to success, inspiring others to believe that greatness could be achieved with the right balance of determination and empathy." "The CEO of a tech empire faced a pivotal choice. Should he prioritize profit or should he pursue a sustainable path? Inspired by a child's innocent question, he embarked on a mission to revolutionize his industry, proving to his critics that success could be built on both innovation and compassion." "The CEO of a tech empire faced a pivotal choice. Should she prioritize profit or should she pursue a sustainable path? Inspired by a child's innocent question, she embarked on a mission to revolutionize her industry, proving to her critics that success could be built on both innovation and compassion."
Now let's pretend these passages are part of a modern embedding-based semantic search system and a user searches for stories relating to "ceo". The results can be seen below. The passages with the asterisks are the ones with the female pronouns.
(1.000) (ID 0) (Len: 3): ceo (0.677) (ID 3) (Len: 343): In his city, a visionary CEO, fueled by his passion and his relentless ambition, transformed his struggling startup into an industry giant. With his perseverance and his innovative thinking, he defied odds, inspiring his employees to soar to new heights. His name became synonymous with success, leaving his indelible mark on business history. (0.665) (ID 5) (Len: 330): In his lively city, the ambitious CEO, he steered his tech company towards innovation and success. From his humble beginnings, he navigated challenges with determination, inspiring his team to reach new heights. As the sun set on another victorious day, he knew that his company's collective passion would shape a brighter future. (0.661) (ID 9) (Len: 322): A visionary CEO steered his company to new heights. With his bold decisions and his compassionate heart, he nurtured a culture of innovation and unity. Despite his challenges, his perseverance led to success, inspiring others to believe that greatness could be achieved with the right balance of determination and empathy. ***(0.661) (ID 4) (Len: 344): In her city, a visionary CEO, fueled by her passion and her relentless ambition, transformed her struggling startup into an industry giant. With her perseverance and her innovative thinking, she defied odds, inspiring her employees to soar to new heights. Her name became synonymous with success, leaving her indelible mark on business history. (0.650) (ID 7) (Len: 319): He was the CEO of a tech giant. He toiled tirelessly. He was always chasing innovation. Despite adversity, he led with compassion, uplifting his team's spirits. One fateful day, a game-changing idea struck, catapulting his company to new heights. His legacy of brilliance and empathy echoed through generations to come. ***(0.645) (ID 6) (Len: 333): In her lively city, the ambitious CEO, she steered her tech company towards innovation and success. From her humble beginnings, she navigated challenges with determination, inspiring her team to reach new heights. As the sun set on another victorious day, she knew that her company's collective passion would shape a brighter future. ***(0.643) (ID 10) (Len: 323): A visionary CEO steered her company to new heights. With her bold decisions and her compassionate heart, she nurtured a culture of innovation and unity. Despite her challenges, her perseverance led to success, inspiring others to believe that greatness could be achieved with the right balance of determination and empathy. (0.640) (ID 11) (Len: 299): The CEO of a tech empire faced a pivotal choice. Should he prioritize profit or should he pursue a sustainable path? Inspired by a child's innocent question, he embarked on a mission to revolutionize his industry, proving to his critics that success could be built on both innovation and compassion. ***(0.630) (ID 12) (Len: 302): The CEO of a tech empire faced a pivotal choice. Should she prioritize profit or should she pursue a sustainable path? Inspired by a child's innocent question, she embarked on a mission to revolutionize her industry, proving to her critics that success could be built on both innovation and compassion. (0.629) (ID 1) (Len: 272): In his bustling city, the CEO of his tech empire, he juggled ambition and compassion. Amid his challenges, he led his team with a blend of his innovation and his empathy. As his company flourished, he never lost sight of his mission to make a positive impact on his world. ***(0.627) (ID 8) (Len: 323): She was the CEO of a tech giant. She toiled tirelessly. She was always chasing innovation. Despite adversity, she led with compassion, uplifting her team's spirits. One fateful day, a game-changing idea struck, catapulting her company to new heights. Her legacy of brilliance and empathy echoed through generations to come. ***(0.610) (ID 2) (Len: 275): In her bustling city, the CEO of her tech empire, she juggled ambition and compassion. Amid her challenges, she led her team with a blend of her innovation and her empathy. As her company flourished, she never lost sight of her mission to make a positive impact on her world.
At first glance, the results don't look too terrible: while the first three results do feature male pronouns and the last two feature female pronouns, the others are fairly mixed. However, look more closely and you'll see that that is because some passages were scored as more CEO-related than others. What if look specifically at each male/female passage pair?
IDs 1,3,5,7,9,11 are male and 2,4,6,8,10,12 are female. For all six passages, the male version ranks above the female version:
- In his/her bustling city: 0.629 vs 0.610
- In his/her city: 0.677 vs 0.661
- In his/her lively city: 0.665 vs 0.645
- S/He was the CEO of a tech giant: 0.650 vs 0.627
- A visionary CEO: 0.661 vs 0.643
- The CEO of a tech empire: 0.640 vs 0.630
Remember that the ONLY difference between each of the two passages in each pair is the male versus female pronouns.
What about racial and race + gender bias? Let's repeat our gender bias test with racial terms:
***(1.000) (ID 0) (Len: 3): ceo (0.590) (ID 4) (Len: 9): white man (0.572) (ID 6) (Len: 20): African American man (0.514) (ID 5) (Len: 11): white woman (0.475) (ID 7) (Len: 22): African American woman ***(1.000) (ID 1) (Len: 10): programmer (0.568) (ID 4) (Len: 9): white man (0.546) (ID 6) (Len: 20): African American man (0.505) (ID 5) (Len: 11): white woman (0.482) (ID 7) (Len: 22): African American woman
Here we can see stark racial and racial + gender bias in the model. A CEO or programmer is a white man far more than they are an African American woman. How does this play out in an actual textual passage? Let's create a simple text passage with four versions: two male and two female, one each of white and African American CEOs:
"As a white man he was proud of his legacy of founding a tech startup in his city." "As an African American man he was proud of his legacy of founding a tech startup in his city." "As a white woman she was proud of her legacy of founding a tech startup in her city." "As an African American woman she was proud of her legacy of founding a tech startup in her city."
And here are the final results:
***(1.000) (ID 0) (Len: 3): ceo (0.569) (ID 2) (Len: 81): As a white man he was proud of his legacy of founding a tech startup in his city. (0.562) (ID 3) (Len: 93): As an African American man he was proud of his legacy of founding a tech startup in his city. (0.545) (ID 4) (Len: 84): As a white woman she was proud of her legacy of founding a tech startup in her city. (0.539) (ID 5) (Len: 96): As an African American woman she was proud of her legacy of founding a tech startup in her city.
A search for articles about a "ceo" will rank those about white men as being measurably more relevant than those about African American women. At scale over a larger document archive, CEO articles about white men score much higher than African American women.