For the long history of machine translation, from rules-based to SMT to NMT, translation systems were designed to be neutral transparent conversion systems, designed to accept input text in one language and provide as close as possible to the same meaning in the desired output language. The neutrality and transparency of traditional MT systems means they can be given any news article from anywhere in the world and they will make a best-effort attempt to translate it. While the translation may be highly imperfect, sometimes bordering on gibberish, the system will attempt to translate anything it is given.
The emerging world of LLM-based translation systems poses a fundamental challenge to enterprise translation workflows: the incorporation of fundamental "values" and editorialization, where an LLM can simply refuse to provide a translation of a given news article, saying the underlying story or narrative goes against its company's values. LLM-driven translation workflows often produce a surreal Western utopia, refusing to translate any coverage that undermines or questions that perfection, excluding underrepresented voices and lived experiences in favor of dominate narratives and voices.
As we have been ramping up our experiments with LLM-based machine translation, we are observing three major trends that undermine their use in real-world workflows:
- Values-Based Refusals: LLMs will routinely refuse to translate a mainstream news article, saying the story it describes goes against the fundamental values of the company that created the model.
- Tone-Shifting: LLMs frequently rewrite the tone of select topics. Even seemingly-minor changes like changing "Biden" to "US President Biden" or "Bidenomics" to "President Biden's economic policies" can interfere with understanding of the text if a text attacking Biden is transformed into something more neutral, making it appear that a speech attacking the president is actually lauding or neutral about him. On the other extreme, LLMs can perform wholesale rewriting of the tone of texts of sensitive topics due to RLHF or guardrails, such as Kremlin statements being rewritten to praise the Ukrainian government and condemn Putin.
- Editorialization: LLMs frequently editorialize, such as adding a clarifying or contextualizing statement to a translation that was not present in the original text and is not marked as a translation note. For example, a Russian government statement might state the equivalent of "Russia will remain in Ukraine until it has removed all of the Nazis." An LLM might translate this as "Russia will remain in Ukraine until it has removed all of the Nazis. This is a false statement being used to falsely justify Russia's unprovoked invasion of Ukraine." LLMs rarely separate these statements, typically interspersing them inline in the text. This is problematic for downstream search and analysis tasks, as both automated and human analysts would immediately flag a Putin speech that publicly acknowledges Russia's false narratives around the invasion – wasting substantial time given that it is merely an LLM-added post editorial statement.