Continuing our series of evaluating Meta's new SeamlessM4T multimodal translation model. Let's try translating some Weibo social media posts that we've looked at in the past. The end result is that for some posts it yields translations that compare favorably to both NMT and LLM translations, but with the added cost of having to use language-specific punctuation rules to split into sentences to translate a sentence at a time. For other posts, it yields subpar translations that can remove or truncate key details, suggesting promise but that it is not quite ready for production use.
Let's start with one of the Weibo posts we looked at last time:
【#他撑伞蹲了很久让女孩知道世界很美好#】近日，湖南长沙，一女子因感情问题一时想不开，独自坐在楼顶哭泣。消防员赶赴，温柔蹲在女孩身旁安抚情绪，夜色沉沉，天空下起小雨，救援人员为其撑伞暖心守护，经过近一小时的劝导，将女孩平安带下15楼天台。ps：答应我们，好好爱自己，世间还有很多美好~#女孩感情受挫消防员撑伞守护# 中国消防的微博视频
Let's try translating it in a single pass:
time python scripts/m4t/predict/predict.py "【#他撑伞蹲了很久让女孩知道世界很美好#】近日，湖南长沙，一女子因感情问题一时想不开，独自坐在楼顶哭泣。消防员赶赴，温柔蹲在女孩身旁安抚情绪，夜色沉沉，天空下起小雨，救援人员为其撑伞暖心守护，经过近一小时的劝导，将女孩平安带下15楼天台。ps：答应我们，好好爱自己，世间还有很多美好~#女孩感情受挫消防员撑伞守护# 中国消防的微博视频" t2tt eng –src_lang cmn
This captures the center sentence, but skips the text at the start and the end:
Recently, in Changsha, Hunan, a woman sat alone on the roof crying because of emotional problems, the firefighters rushed, gently crouched next to the girl to calm the emotions, the night was dark, the sky was raining, the rescuers guarded her umbrella with warmth, after almost an hour of persuasion, they brought the girl safely down to the 15th floor.
What if we translate sentence-by-sentence?
time python scripts/m4t/predict/predict.py "【#他撑伞蹲了很久让女孩知道世界很美好#】" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "近日，湖南长沙，一女子因感情问题一时想不开，独自坐在楼顶哭泣。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "消防员赶赴，温柔蹲在女孩身旁安抚情绪，夜色沉沉，天空下起小雨，救援人员为其撑伞暖心守护，经过近一小时的劝导，将女孩平安带下15楼天台。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "ps：答应我们，好好爱自己，世间还有很多美好~#女孩感情受挫消防员撑伞守护# 中国消防的微博视频" t2tt eng --src_lang cmn
Putting together this yields:
He stood there for a long time to let the girl know that the world was beautiful. Recently, in Changsha, Hunan, a woman sat alone on the roof crying because of emotional problems. The firefighters rushed in, gently crouched next to the girl to calm her emotions, the night was dark, the sky was raining, the rescuers were holding her umbrella and guarding her warmly, after almost an hour of persuasion, they brought the girl safely down the 15th floor attic. ps: Promise us, love yourself, there are still many good things in the world~ #girls feelings frustrated firefighters support umbrella #Chinese firefighters Weibo video
Which compares favorably with ChatGPT's translation, though SeamlessM4T mistranslated "rooftop" as "attic":
【#He Held an Umbrella and Squatted for a Long Time to Let the Girl Know that the World is Beautiful#】Recently, in Changsha, Hunan province, a woman, overwhelmed by relationship problems, sat alone on the rooftop crying. Firefighters rushed to the scene, gently squatting beside the girl to comfort her. The night was dark, and drizzling rain started to fall from the sky. The rescue team held an umbrella to protect and warm her heart. After nearly an hour of persuasion, they safely brought the girl down from the 15th-floor rooftop. PS: Promise us, love yourself well, there is still much beauty in the world ~ #Girl's Emotions Distressed, Firefighters Holding an Umbrella and Guarding# Video footage from the Chinese Fire Brigade's Weibo account.
What about the second Weibo post we tested last time?
一堂童画课，用色彩传递温暖！送困境儿童“美育盒子”，送自闭症儿童艺术疗愈课程。让我们与@延参法师 一堂童画课公益发起人，在这个充满爱与温暖的日子里，一起感受艺术的魅力，用童心绘制美好，为公益事业贡献一份力量！@一堂童画课 北京市行远公益基金会的微博视频
We'll chop it into sentences:
time python scripts/m4t/predict/predict.py "一堂童画课，用色彩传递温暖！送困境儿童“美育盒子”，送自闭症儿童艺术疗愈课程。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "让我们与@延参法师 一堂童画课公益发起人，在这个充满爱与温暖的日子里，一起感受艺术的魅力，用童心绘制美好，为公益事业贡献一份力量！" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "@一堂童画课 北京市行远公益基金会的微博视频" t2tt eng --src_lang cmn
Yielding the following, which compares far less favorably to the ChatGPT translation below, excluding Master Yan Can and the major details:
A children's drawing lesson, using color to convey warmth! Send a beauty box to children in need, send an art therapy course to children with autism. Let's celebrate the charm of art on this day of love and warmth, draw beauty with a child's heart, and contribute to the cause of public welfare! Weibo video of a children's drawing lesson from Beijing's Long Walk Public Welfare Foundation
Compared with ChatGPT's:
A children's painting class, conveying warmth through colors! Sending "Art Education Boxes" to children in difficult situations, and providing art therapy courses for children with autism. Let us, together with Master Yan Can, the initiator of the public welfare project "A Children's Painting Class," experience the charm of art on this loving and warm day. Let's use childlike hearts to create beauty and contribute to the cause of public welfare! @A Children's Painting Class, a video on the official Weibo account of Beijing Xingyuan Public Welfare Foundation.
What about a more complex example from this Quora post about Chinese text that is difficult to translate?
阿呆给领导送红包时，两人的对话颇有意思。 领导：你这是什么意思？ 阿呆：没什么意思，意思意思。 领导：你这就不够意思了。 阿呆：小意思，小意思。 领导：你这人真有意思。 阿呆：其实也没有别的意思。 领导：那我就不好意思了。 阿呆：是我不好意思。 领导：你肯定有什么意思。 阿呆：真的没有什么意思。 领导：既然没有什么意思，那你是什么意思？ 阿呆：其实，我的意思就是想意思意思。 领导：你既然是想意思意思，那就是有什么意思。 阿呆哭了：我就是想意思意思。但是，真的没有什么别的意思。这么个小红包能有什么意思？也就是意思意思而已。 领导笑了：呵呵。我对你有点意思了。 阿呆心想：嘻嘻。我就是这个意思。
We'll translate sentence-by-sentence:
time python scripts/m4t/predict/predict.py "阿呆给领导送红包时，两人的对话颇有意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：你这是什么意思？" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：没什么意思，意思意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：你这就不够意思了。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：小意思，小意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：你这人真有意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：其实也没有别的意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：那我就不好意思了。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：是我不好意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：你肯定有什么意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：真的没有什么意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：既然没有什么意思，那你是什么意思？" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆：其实，我的意思就是想意思意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导：你既然是想意思意思，那就是有什么意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆哭了：我就是想意思意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "但是，真的没有什么别的意思。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "这么个小红包能有什么意思？" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "也就是意思意思而已。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "领导笑了：呵呵。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "我对你有点意思了。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "阿呆心想：嘻嘻。" t2tt eng --src_lang cmn time python scripts/m4t/predict/predict.py "我就是这个意思。" t2tt eng --src_lang cmn
Recall that the original author translated it as:
One day, Adele wanted to bribed her boss with money. Their conversation was pretty interesting. Boss: What do you mean? Adele: No bad purposes. It's just my tiny present to you. Boss: Then you are so mean. It could do harm to my career. Adele: Just a tiny present, just a tiny present. Boss: Oh, you are very funny. Adele: Actually, I have no bad purposes. Boss: Well, I am a little embarrassed….I shouldn’t get you wrong… Adele: Oh,no, I am the one who should feel shamed. Boss: But to be honest,what do you want from me? Adele: I want nothing from you, my boss. Boss: Since you want nothing, then why do you bribe me? Adele: Actually I just want to show you my heart. Boss: Since you just want to show me your heart, then you must want something! Adele: Please don't get me wrong! I just want to show my heart, no other bad purposes! Money can represent nothing at all! Boss( smiled): Then I have feelings for you. Adele(thought): Haha, that’s what I want from you.
For comparison, Bing Translator offered:
When Dun gave red envelopes to the leader, the conversation between the two was quite interesting. Leader: What do you mean by that? Dumb: It's not interesting, it means. Leader: You're not interesting enough. Dumb: Small meaning, small meaning. Leader: You're an interesting person. Dumb: Actually, it doesn't mean anything else. Leader: Then I'm embarrassed. Dumb: I'm sorry. Leader: You must have something to say. Dumb: It's really not interesting. Leader: Since it doesn't mean anything, what do you mean? Dumb: Actually, I mean what I mean. Leader: Since you mean something, what does it mean. Dumb cried: I just meant it. But, really, there is nothing else. What can such a small red envelope mean? That's what it means. The leader laughed: hehe. I'm kind of interesting to you. Dumb thought to himself: hee-hee. That's what I meant.
Seamless provides the following translation, which captures far less of the meaning and shifts the tone:
The conversation between the two was interesting when Amad gave the red envelope to the leader. Leader: What do you mean? It doesn't mean anything, it means something. Leader: You're not annoying enough. It's not that big of a deal. Leader: You are so funny. It doesn't really mean anything else. Then I'm ashamed of myself. I'm not ashamed of it. Leader: You must mean something. It doesn't really mean anything. Leader: If it doesn't mean anything, what do you mean? AB: Actually, what I mean is to think about meaning. Leader: If you're thinking about meaning, what does it mean? I'm just trying to make sense of it. But it doesn't really mean anything else. What does this little red envelope mean? That's what it means. The leader laughed. I feel sorry for you. I thought, "Wow, wow, wow". That's what I mean.