Last month OpenAI released an open source ASR system called Whisper, trained on 680,000 hours of multilingual data. How does Whisper perform on Russian television news broadcasts? To test this, we ran a 30 second clip from Russia 1's 60 Minutes show through Whisper using a publicly-available demo space on Hugging Face.
You can download the clip above as an MP4 file or view it on the Visual Explorer:
You can see the results below, along with their complete submission-to-results execution times. We submitted each multiple times to ensure timings were consistent.
As expected, the Medium and Large models produce the best results, with a few small improvements when moving from the Medium to Large models, but with the Tiny, Small and Base models producing results that deviated too far or which substantively changed the meaning. The results of the Medium and Large models are extremely fluent and showcase the tremendous power of this open model, though the need to use the largest two of the available model sizes means the computational cost of transcribing large video archives will be substantial and limits the ability to perform near-realtime transcription at this time.
MODEL=TINY (4s):
The pre-crepling of the former bomber. It is a fact that the nuclear power supply is not a single-way operation. It is a work that is not a means of the fight. Let's go. You are not a scientist, you are a bluff. Putin is sure that he is not a bluff. But we are not a bluff and we are the ones who are trying to make a decision. We are the two-way nuclear power supply countries. We are the one who is not a nuclear weapon.
MODEL=SMALL (25s):
to the bomber fighter. It is remarkable that he is working on the German Luftwaffe, including the German one. Crazy. Putin is saying that he is not bluffing. Putin is saying that he is not bluffing. But we are not bluffing, the European Union and the United States are not bluffing. We will answer any nuclear strike in Ukraine. We will answer any nuclear weapon.
MODEL=BASE (8s):
to the fighters of Bombarderov. It is wonderful that the nuclear weapons in Russia are being developed, including the German Luftwaffe. Without it. Putin is not a shent at all. Putin believes that he is not a blip, but not a blip and we, and the European Union, the United States. We are responsible for any nuclear weapons in Ukraine. We are responsible for the nuclear weapons.
MODEL=MEDIUM (73s):
to attach it to the bomber. It is remarkable that the nuclear strikes on Russia are being worked out, including the German Luftwaffe. It's crazy. Putin assures that he is not bluffing. But we are not bluffing either. The European Union, the USA and NATO. We will respond to any nuclear strike on Ukraine. We will respond with non-nuclear weapons.
MODEL=LARGE (153s):
to the bombers. It is remarkable that the nuclear strikes on Russia are being worked out by the German Luftwaffe. It's crazy. Putin is saying that he is not bluffing. But we are not bluffing either. The European Union, the United States and NATO. We will respond to any nuclear strike on Ukraine. We will respond with non-nuclear weapons.