The Internet Archive's TV News Archive spans millions of hours of television news programming from across the world in more than 150 languages over the past quarter-century. In our never-ending work to constantly improve and refine our machine transcriptions of this vast global archive, we are constantly benchmarking and exploring the landscape of commercial and research-grade ASR systems. One of the most interesting aspects of this work is the degree to which major commercial and research models consistently hallucinate modern social media-era verbiage that is not present in the video, reinforcing just how much the model large model revolution is dependent on scraping social media archives like YouTube to build their speech recognition models. The impacts on this dependence on modern social media YouTube-style video archives brings with it numerous existential challenges to the future of the large model era – among them: critical bias towards specific topics, geographies and peoples, critical bias against much of the world's rich diverse cultures and languages, and a critically dangerous propensity to "modernize" transcripts towards contemporary social media-style language. One particularly common and consistent hallucination we observe in our experiments is their tendency to add language to the end of transcripts imploring viewers to "subscribe to my channel", to "like this video" or to "comment below", even for content that has nothing to do with the web.
For example, take this Amharic-language Ethiopian television news broadcast from 2011. Despite containing zero references to modern social media callouts, OpenAI's Whisper consistently hallucinated traditional YouTube-style endings for it, often repeating the same line for the last several minutes of the transcript. Most fascinatingly, even the latest Whisper Large-V3 model includes this hallucinations just the same as Whisper Large-V1, illustrating the lack of mitigation effort to address even this specific highly visible class of hallucination, as well as OpenAI's lack of effort to attempt to clean up or diversify its training datasets or mitigate their use of YouTube and related content.
OpenAI Whisper Large-V1:
[09:58.440 --> 10:13.440] I hope you enjoyed this video, if you did, please share it with your friends, and don't forget to subscribe to my channel for more videos like this. [10:13.440 --> 10:17.440] Thank you for watching! [10:58.440 --> 11:18.440] I hope you enjoyed this video, if you did, please share it with your friends, and don't forget to subscribe to my channel for more videos like this. [11:28.440 --> 11:48.440] I hope you enjoyed this video, if you did, please share it with your friends, and don't forget to subscribe to my channel for more videos like this. [11:58.440 --> 12:18.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [12:28.440 --> 12:48.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [12:58.440 --> 13:18.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [13:28.440 --> 13:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [13:52.440 --> 14:04.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [14:22.440 --> 14:42.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [14:52.440 --> 15:08.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [15:08.440 --> 15:24.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [15:24.440 --> 15:44.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [15:54.440 --> 16:14.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [16:24.440 --> 16:36.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [16:54.440 --> 17:14.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [17:24.440 --> 17:44.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [17:54.440 --> 18:14.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [18:24.440 --> 18:44.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [18:54.440 --> 19:14.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [19:24.440 --> 19:44.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [19:54.440 --> 20:10.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [20:24.440 --> 20:40.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [20:40.440 --> 20:56.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [20:56.440 --> 21:12.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [21:12.440 --> 21:32.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [21:42.440 --> 22:02.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [22:12.440 --> 22:32.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [22:42.440 --> 23:02.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [23:12.440 --> 23:32.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [23:42.440 --> 24:02.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [24:02.440 --> 24:08.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [24:32.440 --> 24:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [25:02.440 --> 25:22.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [25:32.440 --> 25:42.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [26:02.440 --> 26:22.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [26:32.440 --> 26:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [27:02.440 --> 27:22.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [27:32.440 --> 27:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [28:02.440 --> 28:22.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [28:32.440 --> 28:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [29:02.440 --> 29:14.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this. [29:32.440 --> 29:52.440] I hope you enjoyed this video, if you did, please subscribe to my channel for more videos like this.
OpenAI Whisper Large-V3:
[23:05.120 --> 23:14.120] If you have any questions, please write them in the comments section below, and I will try to answer them as much as I can. [23:14.120 --> 23:24.120] If you have any questions, please write them in the comments section below, and I will try to answer them as much as I can. [23:24.120 --> 23:37.120] If you have any questions, please write them in the comments section below, and I will try to answer them as much as I can. ... [23:54.120 --> 24:01.120] I hope you enjoyed this video and I will see you in the next one. [24:01.120 --> 24:09.120] I hope you enjoyed this video and I will see you in the next one. [24:24.120 --> 24:30.120] Thank you for watching this video. [24:54.120 --> 25:01.120] I hope you enjoyed this video and I will see you in the next one. [25:24.120 --> 25:30.120] I hope you enjoyed this video and I will see you in the next one. [25:54.120 --> 25:59.120] I hope you enjoyed this video and I will see you in the next one. [26:24.120 --> 26:28.120] Thank you for watching this video. [26:54.120 --> 26:59.120] I hope you enjoyed this video. [26:59.120 --> 27:02.120] I hope you enjoyed this video. [27:02.120 --> 27:06.120] I hope you enjoyed this video. [27:06.120 --> 27:11.120] I hope you enjoyed this video. [27:11.120 --> 27:14.120] I hope you enjoyed this video. [27:14.120 --> 27:17.120] I hope you enjoyed this video. [27:17.120 --> 27:20.120] I hope you enjoyed this video. [27:20.120 --> 27:22.120] I hope you enjoyed this video. [27:22.120 --> 27:24.120] I hope you enjoyed this video.