Comparing Human Vs Machine Transcription And Translation Of A Russian Television News Broadcast

 October 30, 2022

Yesterday the Russian Media Monitor tweeted out a clip from a Russian television news broadcast with human-translated English language subtitles. Given the enormous advances in speech recognition and machine translation, could a fully automated transcription and translation come close to those human subtitles? To test this, we applied OpenAI's Whisper ASR to translate the original broadcast into English and compared it with the Russian Media Monitor's translations. The end result is that Whisper's fully automated English translation is extremely similar to Russian Media Monitor's human translation to the point of being nearly interchangeable. Only five significant differences are observed, two of which change the underlying meaning.

It turns out that despite Russian Media Monitor's clip appearing to be a single extracted excerpt from the broadcast, it is actually composed of 9 independent clips from the broadcast woven together out of order. While we are working on fully autonomous clip searching and deconstruction analysis, for this comparison the TV News Archive's founder Roger Macdonald manually deconstructed the Russian Media Monitor clip into its component excerpts from the original broadcast to make it possible to more directly compare their transcripts.

In addition to running Whisper over the original NTV broadcast, we also applied it to RMM's excerpted video, just in case the edits made in the video clipped any of the words from the original broadcast in a way that would change how Whisper translated it.

Original Broadcast:

View The Entire Broadcast In Visual Explorer.
Whisper-Generated Transcription (Broadcast).
Whisper-Generated Translation Into English (Broadcast).
Google STT-Generated Transcript (Broadcast).

Twitter Video:

View The Russian Media Monitor Twitter Video.
Whisper-Generated Translation Into English (Twitter Video). [SRT] [VTT]

Comparison

  • Clip 1: Start and End 
    • RMM's translation: "There is one month left until winter, until the chill. This is how we're conducting our military operation including strikes against energy infrastructure"
    • Whisper's translation (Broadcast): "We explain to everyone who needs it, that we have a month left before the beginning of winter, before the cold. We are now conducting our military operation like this, like this, like this. Including hitting the energy infrastructure."
    • Whisper's translation (Twitter): "There is a month left before the beginning of winter, before the cold. We are now conducting our military operation like this, like this, like this. Including hitting the energy infrastructure."
  • Clip 2: Start and End
    • RMM's translation: "Some in the West say that this is effective, the magazine 'Economist' wrote that our strikes against Ukraine's energy infrastructure are effective and are working. It works in terms of the absence of electricity, but not in the sense of changing their minds, of forcing decision-makers in Kyiv into negotiations."
    • Whisper's translation: "Someone in the West says that this is effective. Well, some newspaper wrote it. This is, in fact, the IKONOMIST, the IKONOMIST magazine. It says that our strikes on the energy infrastructure of Ukraine are effective and it works. But it works from the point of view of the lack of electricity somewhere. But so far it does not work from the point of view of changing consciousness. With the compulsion to negotiations. Yes. People who make some decisions in Kiev."
    • Whisper's translation (Twitter): "Someone in the West says that this is effective. Well, some newspaper wrote it. This is, in fact, the IKONOMIST, the IKONOMIST magazine. The fact is that our strikes on the energy infrastructure of Ukraine are effective and it works. Well, it works from the point of view of the lack of electricity somewhere, but so far it does not work from the point of view of the change of consciousness. The compulsion to negotiations. Yes. People who make some decisions in Kiev."
  • Clip 3: Start and End
    • RMM's translation: "The problem is that international law was violated. The territorial integrity of a sovereign nation was violated on February 24th. These borders of a sovereign nation that have been violated, those weren't Russian borders. And that's about it."
    • Whisper's translation: "But the problem is that the international law has been violated. And the territorial integrity of a sovereign state has been violated. It doesn't matter the degree of sovereignty. It depends on the United States. It depends on the United States. It was on February 24th. And these borders, and this violation of the borders of a sovereign state, were not Russian borders. That's all."
    • Whisper's translation (Twitter): "The problem is that the international law has been violated and the territorial integrity of the sovereign state has been violated. It was on February 24. And these borders and this violation of the borders of the sovereign state were not Russian borders. That's all."
  • Clip 4: Start and End
    • RMM's translation: "So far, there aren't any signs that anyone is ready to make a deal with Russia about anything whatsoever. What about the piles of would-be intermediaries? A pile of intermediaries is so far just a pile and no more. For Ukraine, territorial concessions are seen not only as impossible, but as unconscionable. If we accept the thesis of Vladimir Vladimirovich that we are one people, perhaps not in the sense of nationalities, since we are clearly different people, but only in the sense that these are former Soviet people. In that instance, it is pointless to anticipate that the people of Ukraine would surrender or demand surrender from their government because they're forced to cook bacon over a candle. These are former Soviet people. Geography aside, these former Soviet people have survived the siege of Leningrad. They are filled with hatred. For them, it is their Patriotic War. So, neither us nor them could be broken this way. No, they can't be broken."
    • Whisper's translation: "So far, there are no signs that someone is ready to make a deal with Russia today. No matter what it is. What about a bunch of intermediaries? A bunch of intermediaries. It remains a bunch of intermediaries, nothing more. The concessions for Ukraine are territorial. Ukraine is now recognized as Ukraine. Ukraine is recognized as impossible and unthinkable. If we take the thesis of Vladimir Vladimirovich that it is one nation, and one nation may not be in the national sense, because it is clear that the nations are different, but only in one sense, that it is a former Soviet nation, then in such a case, it makes no sense to expect from the population of Ukraine to surrender their positions to their government due to the fact that they are frying lard on a candle there. This is a former Soviet nation. This former Soviet nation, well, let's leave the geographical things aside, has survived the Leningrad blockade. They are filled with hatred. For them, this is a domestic war. So, you are not going to break us with such methods, nor are you going to break them. Am I right? No, we are not going to break them. I see."
    • Whisper's translation (Twitter): "So far, there are no signs that someone is ready to negotiate with Russia today. Whatever it is. And a bunch of intermediaries who will show something now? A bunch of intermediaries. It remains a bunch of intermediaries, nothing more. So. The concessions for Ukraine are territorial. Ukraine is recognized as Ukraine today. I emphasize. They are recognized as impossible and unthinkable. If we accept the thesis of Vladimir Vladimirovich that this is one nation, and one nation, it may not be in the national sense, because it is clear that the peoples are different. But only in one sense, that this is a former Soviet nation. In this case, to expect from the population of Ukraine the surrender and the demand for its government, the surrender of their positions due to the fact that they are frying lard on a candle, is pointless. This is a former Soviet nation. This former Soviet nation, well, let's leave the geographical things aside, has survived the Leningrad blockade. They are filled with hatred. For them, this is a domestic war. That is, you will not break us with such methods, nor will you break them. Am I right? No, they will not be broken. It is clear."
    • Key difference: RMM's "patriotic war" becomes Whisper's "domestic war"
  • Clip 5: Start and End
    • RMM's translation: "I don't understand what kind of a just world order we could be talking about, when we are destroying power plants and the infrastructure in Ukraine. You haven't been here for a while, he saved up some stuff in his soul. He decided to throw a turd onto the field. This is cruelty and sadism, a war against civilians. This is a war against the people. You're forcing me to explain everything that as said in our studio in your absence for the last 3, 4, 5 years. As a citizen, I'm responsible, but I did not approve the bombing of the Ukrainian power plants. Well, if YOU didn't approve, we can all go home. Let's pretend we listened to the rest of your manifesto, do you have anything of substance to say? When this is the kind of thing that we do, the way they feel towards us is clearly understandable."
    • Whisper's translation: "In fact, I don't understand at all what kind of fair world order we can talk about when we are engaged in the fact that we are actually destroying the power plants and the Ukrainian infrastructure. This man hasn't been here for a long time. He has accumulated in his soul. Now I will come and throw this shit on the collective farm. This is cruelty and sadism. This is a war against the civilian population. Alexey Valerievich, don't make me explain to you everything that has been said during your absence in our history. That is, the last three or four years. Yes, the last five years. Well, five years. Anyway, yes. As a citizen, you should understand that I was involved in this. I didn't allow to bomb the power plants. Well, that's it, let's disperse. Thank you. I didn't allow it anymore. That's it. Let's disperse. This is a thing that… I don't even know what to say. Alexey Valerievich, let's pretend that we have listened to your manifesto to the end. Is there anything to say in fact? In fact, there is. Well, tell me. That's why."
    • Whisper's translation (Twitter): "In fact, it is not at all clear to me what kind of fair world order we can speak of when we are engaged in the fact that we are actually destroying these very power plants and the Ukrainian infrastructure. This man has not been here for a long time. He has accumulated, you know, in his soul. Now I will come, as we say, and throw this shit on the kolkhoz field. This is cruelty and sadism. This is a war against the civilian population. Alexei Valentinovich, do not make me explain to you everything that has been said during your absence in our studio. That is, the last three or four years. Yes, the last five years. Well, five years. As a citizen, you should understand that I was involved in this. I did not allow to bomb power plants in Ukraine. Well, that's it, let's part ways. Thank you. I did not allow it anymore. That's it. Let's part ways.This is a thing that… I do not even know if we should oppose this or not. Alexei Valentinovich, let's pretend that we have listened to your manifesto to the end. And in fact, is there anything to say? In fact, there is. Well, say it. That is why, that is, when we are engaged in this, the attitude towards us is absolutely clear. Absolutely clear. Thank you."
  • Clip 6: Start and End
    • RMM's translation: "Pardon, I'm amazed at all these Smerdyakov's. You're not an expert, you're Smerdyakov reincarnate. He crawled out and is starting this. Doestoyevsky is a genius. We all understand you wanted to return to television, but I'm afraid you went up the wrong porch. No one is interested in what you're saying. All of that was discussed and forgotten long ago."
    • Whisper's translation: "I'm sorry. I'm amazed how many Smerdyakovs are on the other side. You are not an expert. You are Smerdyakov. You are an embodied Smerdyakov. Here he got out. That's how he starts. Yes, of course. Dostoevsky is a genius. And he will be successful. So, too. Mr. Alexey, we understood that you really want to return to the TV screen. But I'm afraid that you decided not to go through that wing. So. And? Now what you are saying is no longer interesting to anyone. A long time ago, everything was said and forgotten."
    • Whisper's translation (Twitter): "Excuse me, I am amazed how many Smerdyakovs are on that side. You are not an expert, you are Smerdyakov, embodied Smerdyakov. That's how he is described. That's how he starts. Yes, of course. Dostoevsky is a genius. Alexei Valentinovich, we understood that you really want to return to the TV, but I am afraid that you have decided not to go beyond 100 wings. So, and? Now, what you are saying is no longer interesting to anyone. A long time ago, everything was discussed and forgotten."
  • Clip 7: Start and End
    • RMM's translation: "For 20 minutes we're talking about negotiations. What negotiations? The main message of Putin's speech yesterday is that the West is finished. For an hour he went on about the death of the Western hegemony. Was that an offer to negotiate? He brought up the word 'negotiations' 8 times. What's the difference? He could have said it 20 times. The balance of interests can't be achieved without negotiations, even if the West is finished. There are also other parts of the world. How was it all described yesterday? The world after the hegemony. Yesterday, Putin painted the outline of the next, new world order, that is truly more just, there's no West in it. What negotiations? I come to the negotiations and say, 'This used to be your apartment and not it's not'."
    • Whisper's translation: "We've been talking about negotiations for 20 minutes. What kind of negotiations? The main idea of yesterday's speech of Putin, the main, general idea. The West is fine. And what? Putin has been talking about the death of the Western hegemony for an hour. And this is an offer to negotiate? He said it eight times. He said it 20 times. First, the West is not the whole West. Secondly, besides not the whole West, there are other parts of the world. It's not very strange. What was it called yesterday? The West is everything.The world after hegemony. So what? Yesterday, Putin drew the outlines of another, the next world order, really more fair. In this world order… Why am I confused? He says that the countries of the West should get sovereignty. He says, then I can negotiate with him. I come to the negotiations and say, you know what? You had this apartment, you won't have it anymore."
    • Whisper's translation (Twitter): "We are talking about negotiations for 20 minutes. What negotiations? The main idea of yesterday's speech of Putin, the main, general idea. The West, everything is fine. And what? He is talking about the death of the Western hegemony for an hour, and this is an offer to negotiations. Well, he said it eight times. He said it 20 times. First of all, the West is everything. The West is not the whole West. Secondly, in addition to not the whole West, there are still different parts of the world. It's not very strange. In general, what was it all called yesterday? The West is everything, and the domination of the West is a different thing. The world after the hegemony. And what? Yesterday, Putin drew the outlines of another, the next world order, really more fair. In this world order, he says that the countries of the West should acquire sovereignty. He says, then I will be able to talk to him. I come to the negotiations and say, so, you had this apartment, you will not have it anymore."
    • Key difference: RMM's "the West is finished" becomes Whisper's "The West is fine."
  • Clip 8: Start and End
    • RMM's translation: "About the changes, there really are changes. I agree, we traveled the long way. In 2 years, we traveled the long winding road, from deriding the BLM, to the President of Guinea-Bissau acting as a mediator between Russia and Ukraine."
    • Whisper's translation: "Now, as for the changes. Indeed, there are changes, I agree. We have come a long way. For two years, we have come a long way from the criticism of the BLM to the fact that the president of Guinea-Bissau acts as a mediator between Russia and Ukraine."
    • Whisper's translation (Twitter): "As for the change, there really is a change, I agree. We have come a long way. You know, in two years we have gone a long way from the criticism of the BLM, to the fact that President Gveney Besayev acts as a mediator between Russia and Ukraine."
  • Clip 9: Start and End
    • RMM's translation: "Before asserting that the world order has changed, that we're on the fairway of the new world, that everything has changed in our favor. The Russian president set very ambitious goals before the Russian Armed Forces. But to celebrate our victory, or say everything changed. Nobody is celebrating! You agreed with Iosif Evgenyevich that changes did take place. Then, for some reason, you threw in the President of Guinea-Bissau. As of October 28, 2022, these changes are not in Russia's favor. God willing, they will be in Russia's favor and then we'll drink to that."
    • Whisper's translation: "before they say that the world order has been changed, that we are somewhere in the process of creating a new world, that everything has already changed in our side. Yes, indeed, the Russian president set very ambitious tasks before the Russian armed forces, before the Russian society. But to celebrate the victory or to say that everything has changed? Nobody celebrates the victory. You agreed with Yevgeny that there were really changes. Changes? Then, for some reason, you threw the president of Guinea-Bissau and Belarus here. Because on October 28, 2022, these changes are not in the interest of Russia. And God forbid that they are in the interest of Russia. Then we will go for it."
    • Whisper's translation (Twitter): "Of course, before you say that the world order has changed, that we are somewhere in the forerunner of a new world, that everything has already changed in our side. Yes, indeed, Russia has set a very ambitious task for the Russian Armed Forces, for the Russian society. But to celebrate the victory or to say that everything has changed? No one celebrates. You agreed with Yevgeny that changes have really taken place. Changes, if so far. Then, for some reason, you threw President Gveney Besayev and the White here. Because on October 28, 2022, these changes are not in the interest of Russia. And God forbid that they are in the interest of Russia. And then we will stand up for it."
    • Key difference: RMM's "President of Guinea-Bissau" becomes Whisper's "president of Guinea-Bissau and Belarus".
    • Key difference: RMM's "God willing" becomes Whisper's "God forbid".
    • Key difference: RMM's we'll drink to that" becomes Whisper's "we will go for it".