Announcing The New LowerThird Television News Chyron Search API!

 January 6, 2020

We're tremendously excited to announce today the new Television News Chyron Search API! Similar to the Television News API, the new Television News Chyron Search API allows you to perform keyword and phrase searches across two and a half years of television news chyrons from BBC News, CNN, MSNBC and Fox News, generating timelines, station breakdowns, wordclouds and clip summaries, complete with links out to the Archive's website to view the matching clips themselves. Chyrons offer a powerful correlate to the spoken word transcripts of closed captioning, offering additional contextual detail, editorialization and even offering a proxy for the airtime of various guests.

Since August 2017, the Internet Archive's Television News Archive has extracted the chyrons of CNN, MSNBC, Fox News and BBC News by OCR'ing a small bounding box at the bottom of the screen every 1 second. Last month we began reprocessing the raw OCR data from the Archive into a new research chyron dataset.  Taking the raw per-second as-is OCR output from the Archive, we clean and reshape it into a new research-grade dataset using language modeling and edit distance clustering to yield a maximally comprehensive chyron dataset that surfaces the "best" available onscreen chyron text for each minute.

To date these chyrons have primarily been manually examined by journalists looking to compare the coverage of major events across the four stations. Typically the side-by-side comparison viewer is used to view the chyrons minute by minute and flag periods of particular difference.

The new LowerThird Television News Chyron Search API allows you to fulltext search these chyrons using an API interface almost identical to that of the Television News API.

Note that there is a very high level of OCR error in the current chyrons and the chyron feed for each station can experience outages, meaning results are incomplete, but offer a useful holistic summary of chyron appearances.

QUICK START EXAMPLES

Here are some simple examples to get you started.

  • Daily timeline of "brexit" mentions. Displays the number of seconds of airtime on each station mentioning "brexit" by day. [View]
  • Station breakdown of "brexit" mentions. Displays the number of seconds of airtime for each station mentioning "brexit". [View]
  • Wordcloud of words co-occurring with "Pelosi". Displays a list of top words that appeared in chyrons mentioning "Pelosi." [View]
  • Clips mentioning "trump" and "war" together in the same chyron text ordered by relevancy. Displays a list of chyrons that mentioned the words "trump" and "war" together in the same chyron, ordered by most relevant. [View]
  • Clips mentioning "trump" and "war" together in the same chyron text ordered by date. Displays a list of chyrons that mentioned the words "trump" and "war" together in the same chyron, ordered by date. [View]

FULL DOCUMENTATION

The GDELT LOWERTHIRD 2.0 API is accessed via a simple URL with the following parameters. Under each parameter is the list of operators that can be used as the value of that parameter.

  • QUERY. This contains your search query and supports keyword and keyphrase searches, OR statements and a variety of advanced operators. NOTE – all of the operators below must be used as part of the value of the QUERY field, separated by spaces, and cannot be used as URL parameters on their own.
    • "". Anything found inside of quote marks is treated as an exact phrase search. Thus, you can search for "Donald Trump" to find all matches of his name.
      • "donald trump"
    • (a OR b). You can specify a list of keywords to be boolean OR'd together by enclosing them in parentheses and placing the capitalized word "OR" between each keyword or phrase. Boolean OR blocks cannot be nested at this time. For example, to search for mentions of Clinton, Sanders or Trump, you would use "(clinton OR sanders OR trump)".
      • (clinton OR sanders OR trump)
    • -. You can place a minus sign in front of any operator, word or phrase to exclude it. For example "-sanders" would exclude results that contained "sanders" from your results.
      • -sanders
    • Show. This narrows your search to a particular television show. This must be the complete show name as returned by the TV API. To find a particular show, search the API and use the "clipgallery" mode to display matching clips and their source show. For example, to limit your search to the show Hardball With Chris Matthews, you'd search for "show:"Hardball With Chris Matthews"". Note that you must surround the show name with quote marks. Remember that the TV API only searches shows monitored by the Internet Archive's Television News Archive, which may not include all shows.
      • show:"Hardball With Chris Matthews"
    • Station. This narrows your search to a particular television station. Remember that the TV API only searches stations monitored by the Internet Archive's Television News Archive and not all of those stations have been monitored for the entire 2009-present time period. Do not use quote marks around the name of the station. At this time the available stations are "CNN", "MSNBC", "FOXNEWS" and "BBCNEWS".
      • station:CNN
      • (station:CNN OR station:MSNBC OR station:FOXNEWS OR station:BBCNEWS)
  • MODE. This specifies the specific output you would like from the API, ranging from timelines to word clouds to clip galleries.
    • ClipGallery. This displays up to the top 50 most relevant clips matching your search. Each returned clip includes the name of the source show and station, the time the clip aired, a thumbnail, the actual text of the snippet and a link to view the full one minute clip on the Internet Archive's website. This allows you to see what kinds of clips are matching and view the full clip to gain context on your search results. In HTML output, this mode displays a "high design" visual layout suitable for creating magazine-style collages of matching coverage. When embedded as an iframe, the API uses the same postMessage resize model as the DOC 2.0 API.
    • StationChart. This compares how many results your search generates from each of the selected stations over the selected time period, allowing you to assess the relative attention each is paying to the topic. Using the DATANORM parameter you can control whether this reports results as raw clip counts or as normalized percentages of all coverage (the most robust way of comparing stations). Note that in HTML mode, you can use the button at the top right of the display to save it as a static image or save its underlying data.
    • TimelineVol. This tracks how many results your search generates by day/hour over the selected time period, allowing you to assess the relative attention each is paying to the topic and how that attention has varied over time. Using the DATANORM parameter you can control whether this reports results as raw clip counts or as normalized percentages of all coverage (the most robust way of comparing stations). By default, the timeline will not display the most recent 24 hours, since those results are still being generated (it can take up to 2-12 hours for a show to be processed by the Internet Archive and ready for analysis), but you can include those if needed via the LAST24 option. You can also smooth the timeline using the TIMELINESMOOTH option and combine all selected stations into a single time series using the DATACOMB option. Note that in HTML mode, you can toggle the station legend using the button that appears at the top right of the display or export the timeline as a static image or save its underlying data.
    • WordCloud. This mode returns the top words that appear most frequently in clips matching your search. It takes the 200 most relevant clips that match your search and displays a word cloud of up to the top 200 most frequent words that appeared in those clips (common stop words are automatically removed). This is a powerful way of understanding the topics and words dominating the relevant coverage and suggesting additional contextual search terms to narrow or evolve your search. Note that if there are too few matching clips for your query, the word cloud may be blank. Note that in HTML mode, you can use the options at the bottom right of the display to save it as a static image or save its underlying data.
  • FORMAT. This controls what file format the results are displayed in. Not all formats are available for all modes.  To assist with website embedding, the CORS ACAO header for all output of the API is set to the wildcard "*", permitting universal embedding.
    • HTML. This is the default mode and returns a browser-based visualization or display. Some displays, such as word clouds, are static images, some, like the timeline modes, result in interactive clickable visualizations, and some result in simple HTML lists of images or articles. The specific output varies by mode, but all are intended to be displayed directly in the browser in a user-friendly intuitive display and are designed to be easily embedded in any page via an iframe.
    • CSV. This returns the requested data in comma-delimited (CSV) format. The specific set of columns varies based on the requested output mode. Note that since some modes return multilingual content, the CSV is encoded as UTF8 and includes the UTF8 BOM to work around Microsoft Excel limitations handling UTF8 CSV files.
    • JSON. This returns the requested data in UTF8 encoded JSON. The specific fields varies by output mode.
    • JSONP. This mode is identical to "JSON" mode, but accepts an additional parameter in the API URL "callback=XYZ" (if not present defaults to "callback") and wraps the JSON in that callback to return JSONP compliant JavaScript code.
    • RSS. This is only available in ClipGallery mode and returns the results in RSS 2.0 format.
    • JSONFeed. This is only available in ClipGallery mode and returns the results in JSONFeed 1.0 format.
  • DATACOMB. By default, both timeline and station chart modes separate out each matching station's data to make it possible to compare the relative attention paid to a given topic by each station. Sometimes, however, the interest is in overall media attention, rather than specific per-station differences in that coverage. Setting this parameter to "combined" will collapse all matching data into a single "Combined" synthetic station and make it much easier to understand macro-level patterns.
  • DATERES. By default results are returned in minute resolution (for <1 day timespans), hourly resolution (for <7 day timespans), daily resolution for <3 year timespans and monthly resolution otherwise. You can override these settings and manually set the time resolution to any of the following values. NOTE that this will automatically adjust the STARTDATETIME and ENDDATETIME parameters to their start/stop dates/times at the given date resolution.
    • Min. This is only available for timespans of 1 day or less. Rounds start/stop dates to their nearest full minute.
    • Hour. This is only available for timespans of 7 or fewer days. Rounds start/stop dates to their nearest full hours.
    • Day. Rounds start/stop dates to their nearest full day.
    • Week. Rounds start/stop dates to their nearest full week.
    • Month. Rounds start/stop dates to their nearest full month.
    • Year. Rounds start/stop dates to their nearest full year.
  • MAXRECORDS. This option only applies to ClipGallery mode. By default 50 clips are displayed in HTML mode. In JSON, JSONP or CSV formats, up to 3,000 clips can be returned.
  • SORT. By default results are sorted by relevance to your query. Sometimes you may wish to sort by date or tone instead.
    • DateDesc. Sorts matching clips by broadcast date, displaying the most recent clips first.
    • DateAsc. Sorts matching clips by broadcast date, displaying the oldest clips first.
  • STARTDATETIME/ENDDATETIME. These parameters allow you to specify the precise start and end date/times to search, instead of using an offset like with TIMESPAN.
    • STARTDATETIME. Specify the precise date/time in YYYYMMDDHHMMSS format to begin the search – only articles published after this date/time stamp will be considered. The earliest available date is July 2, 2009. If you do not specify an ENDDATETIME, the API will search from STARTDATETIME through the present date/time.
    • ENDDATETIME. Specify the precise date/time in YYYYMMDDHHMMSS format to end the search – only articles published before this date/time stamp will be considered. The earliest available date is July 2, 2009. If you do not specify a STARTDATETIME, the API will search from July 2, 2009 through the specified ENDDATETIME.
  • TIMELINESMOOTH. This option is only available in timeline mode and performs moving window smoothing over the specified number of time steps, up to a maximum of 30. Timeline displays can sometimes capture too much of the chaotic noisy information environment that is the television landscape, resulting in jagged displays. Use this option to enable moving average smoothing up to 30 time steps to smooth the results to see the macro-level patterns. Note that since this is a moving window average, peaks will be shifted to the right, up to several days or weeks at the heaviest smoothing levels.
  • TIMESPAN. By default the LOWERTHIRD API searches the entirety of the Internet Archive's Television News Archive's chyron holdings, which extend back to August 25, 2017. You can narrow this range by using this option to specify the number of months, weeks, days or hours (minimum of 1 hour). The API then only searches airtime within the specified timespan backwards from the present time. If you would instead like to specify the precise start/end time of the search instead of an offset from the present time, you should use the STARTDATETIME/ENDDATETIME parameters.
    • Minutes. Specify a number followed by "min" to provide the timespan in minutes.
    • Hours. Specify a number followed by "h" or "hours" to provide the timespan in hours.
    • Days. Specify a number followed by "d" or "days" to provide the timespan in days.
    • Weeks. Specify a number followed by "w" or "weeks" to provide the timespan in weeks.
    • Months. Specify a number followed by "m" or "months" to provide the timespan in months.
    • Years. Specify a number followed by "y" or "years" to provide the timespan in years.
  • TIMEZONEADJ. By default STARTDATETIME, ENDDATETIME and all returned results are interpreted and reported in the UTC timezone. Use this parameter to adjust to any of the following major timezones: -12:00, -11:00, -10:00, -09:30, -09:00, -08:00, -07:00, -06:00, -05:00, -04:00, -03:30, -03:00, -02:30, -02:00, -01:00, +00:00, +01:00, +02:00, +03:00, +03:30, +04:00, +04:30, +05:00, +05:30, +05:45, +06:00, +06:30, +07:00, +08:00, +08:45, +09:00, +09:30, +10:00, +10:30, +11:00, +12:00, +12:45, +13:00, +13:45 or +14:00. Note that UTC offsets are treated as-is over the entire timespan, meaning that the offset is not adjusted to account for periods in which daylight savings is honored.
  • TIMEZOOM. This option is only available for timeline modes in HTML format output and enables interactive zooming of the timeline using the browser-based visualization. Set to "yes" to enable and set to "no" or do not include the parameter, to disable. By default, the browser-based timeline display allows interactive examination and export of the timeline data, but does not allow the user to rezoom the display to a more narrow time span. If enabled, the user can click-drag horizontally in the graph to select a specific time period. If the visualization is being displayed directly by itself (it is the "parent" page), it will automatically refresh the page to display the revised time span. If the visualization is being embedded in another page via iframe, it will use postMessage to send the new timespan to the parent page with parameters "startdate" and "enddate" in the format needed by the STARTDATETIME and ENDDATETIME API parameters. The parent page can then use these parameters to rewrite the URLs of any API visualizations embedded in the page and reload each of them. This allows the creation of dashboard-like displays that contain multiple LOWERTHIRD API visualizations where the user can zoom the timeline graph at the top and have all of the other displays automatically refresh to narrow their coverage to that revised time frame.