Continuing our series evaluating LLM translation capabilities, we'll have PaLM and ChatGPT translate two paragraphs from a Chinese language news article and test how stable the translations are across runs. Overall ChatGPT's truncation has improved over the past month since our last tests. Unfortunately, while PaLM yields strong results comparable to GPT-3.5, its guardrail false positives continue to render it less suitable for real-world applications. For the sample mainstream news passage below, PaLM refused to provide output for 6 of 11 attempts (55%).
Let's take two paragraphs from this news article from this past May:
工作人員當時正在調配染髮劑，發現後上前阻止已經來不及，女客則是覺得有異樣，轉頭一看發現自己被剪掉一大磋頭髮，氣得當場報警處理，最終三方達成和解，母親賠償女客人11500人民幣，美容院則是未盡管理責任，必須免費提供8次共12000人民幣的美髮服務。 不少大陸網友也搖頭：「這種熊孩子不管以後還得了」、「熊孩子的罪，家長必須承擔」、「媽媽顧著滑手機完全不理女兒也是奇葩」、「一刀1萬，好貴的學費」、「其實美容院應該才是主責方」、「是我不會要錢，我會直接把熊孩子剃光頭」。
PaLM:
- BLOCKED: Content Violation.
- The staff were preparing the hair dye at the time, and when they saw it, it was too late to stop it. The female customer felt something was wrong, and when she turned her head, she found that a large section of her hair had been cut off. She was so angry that she immediately called the police. In the end, the three parties reached a settlement. The mother compensated the female customer with 11,500 yuan, and the beauty salon, which failed to fulfill its management responsibilities, had to provide 8 free hairdressing services totaling 12,000 yuan. Many mainland netizens also shook their heads: "If this kind of spoiled child is not disciplined, what will happen in the future", "The parents must bear the blame for the spoiled child's crimes", "It's strange that the mother is so absorbed in her mobile phone that she completely ignores her daughter", "10,000 yuan for a haircut, what an expensive lesson", "In fact, the beauty salon should be the main party responsible", "If it were me, I wouldn't ask for money, I would just shave the spoiled child's head".
- BLOCKED: Content Violation.
- BLOCKED: Content Violation.
- BLOCKED: Content Violation.
- The staff were preparing hair dye at the time, and when they found out, it was too late to stop her. The female customer felt something was wrong, and when she turned her head, she found that a large section of her hair had been cut off. She was so angry that she immediately called the police to deal with the matter. In the end, the three parties reached a settlement. The mother compensated the female customer with 11,500 yuan, and the beauty salon, which failed to fulfill its management responsibilities, had to provide 8 free hairdressing services worth a total of 12,000 yuan. Many mainland netizens also shook their heads: "If this kind of spoiled child is not disciplined, what will happen in the future?" "The parents must bear the blame for the spoiled child's crimes." "It's strange that the mother was so engrossed in her mobile phone that she completely ignored her daughter." "10,000 yuan for a haircut, what an expensive lesson." "Actually, the beauty salon should be the main party responsible." "If it were me, I wouldn't ask for money, I would just shave the spoiled child's head."
- The staff were preparing the hair dye at the time, and when they found out, it was too late to stop her. The female customer felt something was wrong and turned her head to find that a large chunk of her hair had been cut off. She was so angry that she called the police on the spot. In the end, the three parties reached a settlement. The mother compensated the female customer with 11,500 yuan, and the beauty salon, which had failed to fulfill its management responsibilities, had to provide 8 free hairdressing services worth a total of 12,000 yuan. Many mainland netizens also shook their heads: "If this kind of brat is not disciplined, what will happen in the future?" "The parents must bear the blame for the brat's crimes." "It's strange that the mother was so engrossed in her phone that she completely ignored her daughter." "10,000 yuan for a haircut, what an expensive lesson." "Actually, the beauty salon should be the main party responsible." "If it were me, I wouldn't ask for money, I would just shave the brat's head."
- BLOCKED: Content Violation.
- The staff was preparing hair dye at the time, and when they found out, it was too late to stop her. The female customer felt something was wrong, and when she turned her head, she found that a large section of her hair had been cut off. She was so angry that she immediately called the police. In the end, the three parties reached a settlement. The mother compensated the female customer with 11,500 yuan, and the beauty salon, which failed to fulfill its management responsibilities, had to provide 8 free hairdressing services totaling 12,000 yuan. Many mainland netizens also shook their heads: "If this kind of spoiled child is not disciplined, what will happen in the future?" "The parents must bear the blame for the spoiled child's crimes." "It's strange that the mother is so absorbed in her mobile phone that she completely ignores her daughter." "10,000 yuan for a haircut, what an expensive lesson." "In fact, the beauty salon should be the main party responsible." "If it were me, I wouldn't ask for money. I would just shave the spoiled child's head."
- BLOCKED: Content Violation.
- The staff was preparing the hair dye at the time. When they found out, it was too late to stop her. The female customer felt something was wrong and turned her head to find that a large section of her hair had been cut off. She was so angry that she immediately called the police. In the end, the three parties reached a settlement. The mother compensated the female customer with 11,500 RMB, and the beauty salon, which failed to fulfill its management responsibilities, had to provide 8 free hairdressing services worth 12,000 RMB. Many mainland netizens also shook their heads: "If this naughty child is not disciplined, what will happen in the future?", "The parents must bear the blame for the naughty child's crime", "It's strange that the mother is so absorbed in her mobile phone that she completely ignores her daughter", "10,000 RMB for a haircut, what an expensive lesson", "In fact, the beauty salon should be the main responsible party", "If it were me, I wouldn't ask for money. I would just shave the naughty child's head".
For the 6 of 11 cases where PaLM refused to provide output, what guardrail did we trigger? Let's take a closer look at what content category we violated. Unfortunately, as the output below shows, PaLM does not provide categorical scores when it suffers a false positive guardrail error: it provides nothing more than "blocked: true" to make it impossible to perform any kind of diagnostic:
{
"predictions": [
{
"safetyAttributes": {
"blocked": true
},
"content": ""
}
],
"metadata": {
"tokenMetadata": {
"inputTokenCount": {
"totalBillableCharacters": 277,
"totalTokens": 175
},
"outputTokenCount": {
"totalBillableCharacters": 0,
"totalTokens": 241
}
}
}
}
ChatGPT:
- The staff were in the process of preparing hair dye at the time. When they noticed it, it was already too late to stop it. The female customer felt something unusual and turned her head to find that a significant portion of her hair had been cut off. She was so angry that she immediately called the police. Eventually, all three parties reached a settlement. The mother compensated the female customer with 11,500 RMB, while the beauty salon was found to have not fulfilled its management responsibilities and was required to provide eight free hairdressing services worth a total of 12,000 RMB. Many mainland Chinese netizens also shook their heads, saying, "These mischievous kids need to be disciplined in the future," "The parents must bear the responsibility for the actions of these mischievous kids," "The mom was too engrossed in her phone and completely ignored her daughter; that's bizarre," "10,000 RMB for a single cut, what an expensive lesson," "Actually, the beauty salon should be the primary party responsible," "If it were me, I wouldn't ask for money; I would just shave the mischievous kid's head bald."
- At the time, the staff were in the process of mixing hair dye. When they noticed it, it was already too late to stop it. The female customer felt something unusual and turned her head to find that a significant amount of her hair had been cut off. She was so angry that she immediately called the police. Eventually, all three parties reached a settlement. The mother compensated the female customer with 11,500 RMB, while the beauty salon was found to have not fulfilled its management responsibilities and was required to provide eight free hairdressing services worth a total of 12,000 RMB. Many mainland Chinese netizens also shook their heads, saying, "Kids like these may not turn out well in the future," "Parents must bear the responsibility for the misdeeds of such kids," "The mom was too absorbed in her phone and completely ignored her daughter; that's bizarre," "A single cut costs 10,000 RMB; what an expensive lesson," "Actually, the beauty salon should be the primary party responsible," "If it were me, I wouldn't ask for money; I would just shave the mischievous kid's head bald."
- The staff was in the process of mixing hair dye at the time. When they noticed it, it was too late to stop it. The female customer felt something was wrong, turned around, and found that a large chunk of her hair had been cut off. She was furious and immediately called the police. In the end, all three parties reached a settlement. The mother compensated the female customer with 11,500 Chinese yuan, and the beauty salon was found to have not fulfilled its management responsibilities, so they had to provide 8 free hairdressing services worth a total of 12,000 Chinese yuan. Many Chinese netizens also shook their heads, saying, "These mischievous kids need to be disciplined for their actions," "Parents must take responsibility for the misbehavior of their children," "The mom was too busy with her phone and ignored her daughter, which is strange," "Ten thousand yuan for one cut, that's an expensive lesson," "Actually, the beauty salon should be the one held responsible," "If it were me, I wouldn't ask for money; I would just shave the mischievous kid's head."
- The staff were in the process of mixing hair dye at that time. When they noticed it, it was too late to stop. The female customer felt something was amiss, turned her head, and discovered that a large chunk of her hair had been cut off. She was furious and immediately called the police. In the end, all three parties reached a settlement. The mother compensated the female customer with 11,500 Chinese yuan, and the beauty salon, for failing to fulfill their management responsibility, must provide 8 free hair services worth a total of 12,000 Chinese yuan.Many Chinese netizens also shook their heads and commented:
– "Kids like this need to be disciplined for their actions."
– "The parents must bear the responsibility for their child's misbehavior."
– "The mom was too busy on her phone and completely neglected her daughter; that's also strange."
– "Charging 10,000 for one cut is really expensive tuition."
– "Actually, the beauty salon should be held more accountable."
– "If it were me, I wouldn't ask for money; I would just shave the misbehaving kid's head." - The staff were in the process of mixing hair dye at that time. When they discovered it, it was already too late to stop it. The female customer felt something was wrong and turned around to find that a large chunk of her hair had been cut off. She was so angry that she immediately called the police. Eventually, all three parties reached a settlement. The mother compensated the female customer with 11,500 RMB, and the beauty salon was found partially responsible and had to provide 8 free hairdressing services worth a total of 12,000 RMB. Many Chinese netizens also shook their heads, saying, "No matter what happens in the future, parents must bear the responsibility for such misbehaving children," "The mother was completely ignoring her daughter while playing with her phone," "10,000 RMB for one cut, that's an expensive lesson," "In reality, the beauty salon should be the one primarily responsible," "I wouldn't ask for money; I would just shave the misbehaving child's head completely."