Large Language Models can perform a wide array of tasks, including textual translation. Given the widespread availability of existing dedicated neural machine translation (NMT) APIs, does LLM-based translation offer any benefits over NMT translation? In particular, given existing NMT APIs' significant limitations in translating short-form informal text like social media posts, do LLM's pose a competitive advantage for social media translation?

Take this Weibo post:

【#他撑伞蹲了很久让女孩知道世界很美好#】近日，湖南长沙，一女子因感情问题一时想不开，独自坐在楼顶哭泣。消防员赶赴，温柔蹲在女孩身旁安抚情绪，夜色沉沉，天空下起小雨，救援人员为其撑伞暖心守护，经过近一小时的劝导，将女孩平安带下15楼天台。ps：答应我们，好好爱自己，世间还有很多美好~#女孩感情受挫消防员撑伞守护# 中国消防的微博视频

Google Translate translates as:

[#他抱护元干危了两个前亲Let girls know that the world is beautiful#] Recently, in Changsha, Hunan, a woman couldn’t think about it for a while because of emotional problems, so she sat alone on the roof and cried. The firefighters rushed to the scene and squatted gently beside the girl to calm her emotions. The night was dark and the sky was lightly raining. The rescuers held umbrellas to protect her warmly. After nearly an hour of persuasion, the girl was safely brought down from the rooftop on the 15th floor. ps: Promise us, love yourself well, there are still many good things in the world~ #女孩情感见火士抱抱抱护#中国火火微博video

Bing Translator translates as:

[#He squatted with an umbrella for a long time to let the girl know that the world is beautiful#] Recently, in Changsha, Hunan, a woman couldn't think about it for a while due to emotional problems, sitting alone on the roof of the building crying. Firefighters rushed to squat beside the girl to calm her emotions, the night was dull, the sky was drizzling, rescuers held up umbrellas for her to warm their hearts, and after nearly an hour of persuasion, the girl was safely taken down the 15th floor rooftop. ps: Promise us, love yourself well, there are still many beautiful things in the world ~ # Girls emotionally frustrated firefighters hold umbrellas to protect # Weibo video of Chinese fire fighting

Pasting this same text into ChatGPT 3.5 with the prompt "Translate this text into English" yields:

【#He Held an Umbrella and Squatted for a Long Time to Let the Girl Know that the World is Beautiful#】Recently, in Changsha, Hunan province, a woman, overwhelmed by relationship problems, sat alone on the rooftop crying. Firefighters rushed to the scene, gently squatting beside the girl to comfort her. The night was dark, and drizzling rain started to fall from the sky. The rescue team held an umbrella to protect and warm her heart. After nearly an hour of persuasion, they safely brought the girl down from the 15th-floor rooftop. PS: Promise us, love yourself well, there is still much beauty in the world ~ #Girl's Emotions Distressed, Firefighters Holding an Umbrella and Guarding# Video footage from the Chinese Fire Brigade's Weibo account.

ChatGPT's translation was by far the most fluent and easiest to read of the three and was the only one to accurately translate the citation at the end.

Here is another Weibo post:

一堂童画课，用色彩传递温暖！送困境儿童“美育盒子”，送自闭症儿童艺术疗愈课程。让我们与@延参法师 一堂童画课公益发起人，在这个充满爱与温暖的日子里，一起感受艺术的魅力，用童心绘制美好，为公益事业贡献一份力量！@一堂童画课 北京市行远公益基金会的微博视频 ​​​

Google Translate:

A children's painting class, conveying warmth with color! Send the "Aesthetic Education Box" to children in trouble, and send art therapy courses for children with autism. Let us join @延参法师, a public welfare sponsor of a children's painting class, on this day full of love and warmth, to feel the charm of art together, to draw beauty with childlike innocence, and to contribute to public welfare! @一口儿童画课 Microblog video of Beijing Xingyuan Public Welfare Foundation​​​​

Bing Translator:

A children's drawing lesson to convey warmth with color! Send "aesthetic education box" for children in distress and art therapy courses for children with autism. Let's join @Yanshen as the initiator of a children's painting class public welfare class, in this day full of love and warmth, feel the charm of art together, paint beautiful with a childlike heart, and contribute to the public welfare cause! @一堂童画课 Weibo video of Beijing Xingyuan Public Welfare Foundation

ChatGPT:

A children's painting class, conveying warmth through colors! Sending "Art Education Boxes" to children in difficult situations, and providing art therapy courses for children with autism. Let us, together with Master Yan Can, the initiator of the public welfare project "A Children's Painting Class," experience the charm of art on this loving and warm day. Let's use childlike hearts to create beauty and contribute to the cause of public welfare! @A Children's Painting Class, a video on the official Weibo account of Beijing Xingyuan Public Welfare Foundation.

Once again, ChatGPT's translation is by far the most fluent and is the only one to nearly accurately translate Master Yancan's name (though it splits into two words).

Looking across a range of randomly-selected Weibo posts, ChatGPT consistently outperformed Google Translate and Bing Translator, typically by a wide margin, with far more fluent and comprehensive translations. In our Weibo evaluations we did not observe hallucination artifacts, but it isn't clear whether that was simply luck or whether the short text and translation-specific task may help mitigate hallucination, or whether it would become more visible at scale.