Continuing our series of evaluating Meta's new SeamlessM4T multimodal translation model, there are actually two versions of the model: large and medium. We've been testing the large model up until now under the assumption that it would provide the highest accuracy, while the medium model would provide lower accuracy but faster performance. Here we will test that theory. Surprisingly, the medium model actually correctly handles sentences that the large model struggles with: where the large model enters a failure state or truncates, the medium model provides a more comprehensive translation. However, the medium model also truncates text and mistranslates portions, as well as providing more stilted translations. Equally surprising is that both models take around the same time to load and translate on a V100 GPU, despite their different sizes. Overall, the medium model provides substantially superior results to the large model, offering an unexpected reminder that for AI models, larger is not always better and organizations should not automatically assume that the largest available model will yield the best possible results.
Returning to the full-length news article we examined, let's take one of the longer sentences and test it under both large and medium models:
time m4t_predict "李志辉告诉新黄河记者，他目前在沧州，女朋友是刁窝镇东辛庄村人，8月1日几位家人有的已经搬到附近的白塔村住，有的还在东辛庄村，跟张俊一样，李志辉女朋友一家也没想到，当地的水涨得如此快。" t2tt eng --src_lang cmn --model_name seamlessM4T_large time m4t_predict "李志辉告诉新黄河记者，他目前在沧州，女朋友是刁窝镇东辛庄村人，8月1日几位家人有的已经搬到附近的白塔村住，有的还在东辛庄村，跟张俊一样，李志辉女朋友一家也没想到，当地的水涨得如此快。" t2tt eng --src_lang cmn --model_name seamlessM4T_medium
Below are the results. Note how the large model truncates the last part of the sentence about his girlfriend's family not expecting the water to rise so quickly. In contrast, the medium model yields more stilted results and transliterates the names differently, but translates the complete passage, matching Google Translate's results more closely:
- seamlessM4T_large: Li Zhihui told the New Yellow River reporter that he is currently in Yangzhou, and his girlfriend is from the village of Dongxingzhou in the town of Zhejiang. On August 1, several families had moved to the nearby village of White Tower, and some were still living in Dongxingzhou.
- seamlessM4T_medium: Li Zhiyuan told reporters that he is currently in Zhejiang, his girlfriend is from the village of Dongsinzhou in Zhejiang Town. On August 1st, some of his family had moved to the nearby village of White Tower, some were still in the village of Dongsinzhou, just like Zhang Jun, Li Zhiyuan's girlfriend's family didn't expect the local water to flow so quickly.
- Google Translate: Li Zhihui told the New Yellow River reporter that he is currently in Cangzhou, and his girlfriend is from Dongxinzhuang Village, Diaowo Town. Like Zhang Jun, Li Zhihui's girlfriend's family did not expect the local water to rise so quickly.
What about the sentence that caused the large model to fail? Recall that one of the sentences yielded the phrase "the Little River" repeated over and over again instead of the correct translation. Could the medium model yield correct results?
time m4t_predict "目前，涿州境内北拒马河、小清河、白沟河等多条河流流量较 大，小清河分洪区、兰沟洼蓄滞洪区已相继启动。" t2tt eng --src_lang cmn --model_name seamlessM4T_large time m4t_predict "目前，涿州境内北拒马河、小清河、白沟河等多条河流流量较 大，小清河分洪区、兰沟洼蓄滞洪区已相继启动。" t2tt eng --src_lang cmn --model_name seamlessM4T_medium
Below are the results. Note how the large model enters a failure state and cannot translate the text. The medium model, however, does partially translate the sentence, but it truncates a substantial portion and mistranslates several of the names, suggesting this sentence is problematic across both models:
- seamlessM4T_large: At present, the flow of several rivers in the state, such as the North River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River, the Little River.
- seamlessM4T_medium: Currently, there is a large flow of rivers in the northern part of Zhejiang Province, such as the Ma River, the Little Qing River, the White Gorge River, and many other rivers in the Zhejiang Province.
- Google Translate: At present, the North Juma River, Xiaoqing River, Baigou River and other rivers in Zhuozhou have relatively large flows, and the Xiaoqing River flood diversion area and the Langou depression flood storage and detention area have been activated one after another.