For those interested in applying today's 'Trump Is…" analysis (in which every mention of "trump is" on CNN, MSNBC and Fox News June 2015 to present was scanned for every mention of "trump is" and the following two words aggregated by month to compile a monthly list of the top descriptions associated with him) to other public figures and topics, here is the process we used.
Note that the scripts below assume that your query returns fewer than 3,000 results per month – you will have to customize the "makefetchcmds.pl" script to use a shorter time horizon until each query returns fewer results.
First, download "makefetchcmds.pl" to your computer. Decide the start and stop month for your query and convert your query into URL-escaped form (quotes become "%22", spaces become "%20" and so on). Then run once for each station of interest. For example, to compile monthly files from May 2015 to November 2019 for the query "trump is" on CNN, MSNBC and Fox News you would use the following commands:
time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" CNN time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" MSNBC time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" FOXNEWS
After running each of the commands above, the script will output a "run.sh" shell file with the necessary list of CURL queries to compile the cache of "trump is" responses for each station. The script will automatically save the results of each query to "./CACHE/" subdirectory. Run the script as-is, do not change the "sleep 5" delay at the end of each fetch – this ensures you do not overburden the TV API.
To get an idea of the output these scripts generate, here is the CURL query they create for CNN for June 2015:
time curl "https://api.gdeltproject.org/api/v2/tv/tv?startdatetime=20150601000000&enddatetime=20150630235959&datanorm=raw&query=%22trump%20is%22%20station:CNN%20&mode=clipgallery&format=csv&maxrecords=5000" -o ./CNN.201506.csv
The result is CSV file containing the URL of the clip in the Internet Archive's Television News Archive (so you can watch the clip), the date/time the clip aired (in UTC), the station, show name, the Archive's unique ShowID for the broadcast, the preview thumbnail for the clip and the brief matching snippet of the 15 second clip containing your query. Note that in a small number of cases your phrase may appear at the end of the snippet or may appear cut off at the end of the snippet – this indicates that your phrase straddled the 15 second boundary between clips. The API looks ahead of each clip to ensure that phrases that span clips are attributed to the clip the phrase begins in, but this means that the results snippets may occasionally be truncated – these comprise a very small number of cases and are dropped by the script below.
Next, convert these CSV files into monthly word histograms. Download "makehist.pl" and run it over your CSV cache downloaded above. A simple way of doing this is to download GNU parallel and use it:
time find ./CACHE/*.csv | parallel --eta './makehist.pl {}'
This will parse each CSV file, compile the two words following "trump is" and output a histogram of the matches for that month.
Note – to adapt for other queries, just replace the word "trump" in the regular expression with your query and adjust the "(is|was|are|were)" exclusions appropriately.
At this point you have a list of the top two-word phrases by month for your query. Of course, the most common phrases will likely be consistent across months, typically grammatical phrases like "going to" or "trying to." Use a TFIDF filter to eliminate these common phrases that appear every month and surface the most important ones.
Finally, download "maketimeline.pl" – this script compiles all of the histograms above into a single output timeline, including performing modified TFIDF scoring to surface the most significant phrases by month. Run as:
time ./maketimeline.pl > MYTIMELINE.TXT
By default the script outputs the timeline according to date, with all stations listed under each date, but you can change the sort order by simply uncommenting the two lines in the timeline construction portion of the script and commenting the two above them to sort by station -> date.
Note that this script uses a modified TFIDF score in which the IDF component treats each month as a "document" (rather than each clip) and thus adjusts by how many months a phrase appeared in, while the TF component is the standard raw frequency count. This normalizes against the number of months a phrase appears in and results in more meaningful results in this case. You might experiment with other TFIDF definitions for your particular use case.
Remember that these phrases are a mixture of the wording of a station's own personalities and the words of others that it chooses to broadcast. Regardless of source, it captures the descriptive words a viewer of that station see associated with a given person/topic.
The final result for the "Trump is" analysis can be seen below, showcasing the power of this approach for tracing how media portrayals of public figures and topics evolve over time.
| Jun-15 | CNN | almost double, like almost, out at, catching on, threatening to |
| FOXNEWS | running for, your loser, hitting some, sharp guy, being talked | |
| MSNBC | making another, eliminating, paying some, apparently launching, in second | |
| Jul-15 | CNN | wrong on, also creating, not backing, political car, using your |
| FOXNEWS | political fraud, here tonight, leading, firing up, employing is | |
| MSNBC | modern day, bringing a, surviving this, spiking to, surging | |
| Aug-15 | CNN | interesting man, leading, dominating, leading for, on top |
| FOXNEWS | good for, hitting a, so popular, boss i, resinating well | |
| MSNBC | raising, sucking all, tapping into, featured on, touching a | |
| Sep-15 | CNN | entertainer, maybe, leading, criticizing jeb, relatively new |
| FOXNEWS | entertainer, shallow, narcissist, wonderful entertainer, now becoming | |
| MSNBC | entertainer, for donald, surging with, insecure, seeing a | |
| Oct-15 | CNN | probably going, campaigning now, now even, entertaining, weak on |
| FOXNEWS | entertaining, now number, still on, here with, on top | |
| MSNBC | committed christian, entertaining, pre-bragging about, dictating, showing people | |
| Nov-15 | CNN | sharpening his, hosting 'saturday, insult, tweeting out, still on |
| FOXNEWS | hosting 'saturday, still number, funny, not slamming, ratings machine | |
| MSNBC | now convinced, make up, not worth, here spewing, obviously winning | |
| Dec-15 | CNN | on video, jerk, sending around, center stage, a jerk |
| FOXNEWS | playing cruz, big christmas, calling for, posing a, genuine conservative | |
| MSNBC | playing right, on video, jerk, sending around, calling for | |
| Jan-16 | CNN | welcoming, bringing up, suggesting, not used, maintaining his |
| FOXNEWS | out, top seed, not there, brilliant, lasagna | |
| MSNBC | running, fool, your guy, debuting his, welcoming | |
| Feb-16 | CNN | very rattled, this campaign, not christian, our campaign, intensifying this |
| FOXNEWS | behind me, very rattled, right candidate, this campaign, big winner | |
| MSNBC | nominee, way ahead, this campaign, that man, leading | |
| Mar-16 | CNN | nominee, pathological liar, ahead, winner, the nominee |
| FOXNEWS | nominee, phony, 20 points, clown, front-runner | |
| MSNBC | nominee, con man, gift wrapped, ted cruz, phony | |
| Apr-16 | CNN | nominee, over, poised to, whining, behind hillary |
| FOXNEWS | nominee, expect to, predicting a, close to, unstoppable | |
| MSNBC | nominee, winning by, denied, not elected, heavy favorite | |
| May-16 | CNN | presumptive nominee, qualified to, presumptive republican, reliable conservative, nominee |
| FOXNEWS | last man, presumptive nominee, presumptive republican, qualified to, nominee | |
| MSNBC | presumptive nominee, nominee, terrible person, so stupid, no nothing | |
| Jun-16 | CNN | loud, picking on, temperamentally unfit, criticizing judge, learning how |
| FOXNEWS | temperamentally unfit, on with, unfit to, featured for, drawing an | |
| MSNBC | loud, certainly a, genuine threat, defeated, temperamentally unfit | |
| Jul-16 | CNN | so dangerous, now fully, good man, also ignoring, in colorado |
| FOXNEWS | good man, campaigning hard, best choice, far better, compelling | |
| MSNBC | his hypocrisy, you're fired, not raising, defeated, bad person | |
| Aug-16 | CNN | their candidate, not qualified, up 10, comfortable about, very plugged |
| FOXNEWS | because a, everywhere, pounding, airing tomorrow, pretty strong | |
| MSNBC | losing by, down ten, up 10, unfit to, reinforcing harmful | |
| Sep-16 | CNN | phony, closer, disgusting fraud, amateur, not qualified |
| FOXNEWS | doing better, potentially, now leading, changing, phony | |
| MSNBC | phony, best debater, no republican, biased against, unfit for | |
| Oct-16 | CNN | definitely back, already making, seizing on, nazi, trailing |
| FOXNEWS | person best, pushing back, hitting, in michigan, vehemently denied | |
| MSNBC | sexual predator, frankly pathetic, behind, headline, clearly going | |
| Nov-16 | CNN | phony, president-elect, ahead, challenge unlike, uniquely unqualified |
| FOXNEWS | very man, president-elect, 70000 votes, not their, good guy | |
| MSNBC | phony, president-elect, sworn, serious about, almost surely | |
| Dec-16 | CNN | inaugurated, sworn, president-elect, fully briefed, uniquely unqualified |
| FOXNEWS | sworn, promising to, already acting, president-elect, assembling a | |
| MSNBC | sworn, very committed, echoing, officially off, surrounding himself | |
| Jan-17 | CNN | sworn, legitimate president, inaugurated, still unsure, so unaware |
| FOXNEWS | sworn, watching right, mentally unstable, pledging to, he t | |
| MSNBC | sworn, 44% which, legitimate president, very concerned, illegitimate | |
| Feb-17 | CNN | horrible, embodiment of, often, facing his, still moving |
| FOXNEWS | top global, billy jack, not boring, kicking off, horrible | |
| MSNBC | attacking judges, learning, skipping cpac, horrible, no exception | |
| Mar-17 | CNN | extremely confident, requesting, destroying, requesting as, to my |
| FOXNEWS | exercising his, in cahoots, negotiator, somewhere between, clouding your | |
| MSNBC | requesting, exercising his, 70-year-old rookie, in campaign, president right | |
| Apr-17 | CNN | making fools, considering military, on pace, facing several, now considering |
| FOXNEWS | white isis, wall, most dangerous, him, dumbest president | |
| MSNBC | learning, caving on, more mysterious, not relishing, likable | |
| May-17 | CNN | in brussels, weighing whether, back, in saudi, heading out |
| FOXNEWS | terrorizing, still here, on with, seeking is, up next | |
| MSNBC | dangerous, acting, abusing his, considering firing, recording conversations | |
| Jun-17 | CNN | traitor, considering firing, actively engaging, not under, not well |
| FOXNEWS | not under, traitor, sexist, radicalized whack, slamming its | |
| MSNBC | under investigation, when it's, hitting on, not under, narcissist on | |
| Jul-17 | CNN | seen having, already chafing, discussing possibly, guest of, very different |
| FOXNEWS | militarizing, half-term president, wall, phenomenon, media's x | |
| MSNBC | overseas, already looking, looking into, already basically, in warsaw | |
| Aug-17 | CNN | driving, morally bankrupt, in arizona, turning into, throwing his |
| FOXNEWS | assassinated, promising 100, so evil, tearing, anti-christ | |
| MSNBC | officially on, tearing, fuming to, dividing our, and | |
| Sep-17 | CNN | white supremacist, biggest tv, moving to, reconsidering plans, trolling north |
| FOXNEWS | white supremacist, weighing his, donating $1, unicorn, vindicated | |
| MSNBC | white supremacist, staying quiet, original, about getting, in alabama | |
| Oct-17 | CNN | not finished, constantly undermining, telling you, quote seething, mentally deranged |
| FOXNEWS | mentally deranged, right people, route, moved into, calling | |
| MSNBC | moron, stopping, mentally ill, because they, treating | |
| Nov-17 | CNN | in china, same bezos, everything, rejecting steve, leader of |
| FOXNEWS | in beijing, back from, in south, in china, arriving | |
| MSNBC | same bezos, in china, accused 15, committing obstruction, president which | |
| Dec-17 | CNN | loathsome human, attacking fbi, warning iran, inaugurated, making me |
| FOXNEWS | head for, our president, billionaire we, an [bleep, packing | |
| MSNBC | being recruited, russia, owed a, madman, one half | |
| Jan-18 | CNN | like negotiating, completely capable, fighting for, racially ignorant, racist |
| FOXNEWS | like negotiating, mentally unfit, turn, taking too, making america | |
| MSNBC | like negotiating, pretty significant, interesting, known to, above water | |
| Feb-18 | CNN | model of, either totally, okay with, incredibly supportive, living embodiment |
| FOXNEWS | not my, already given, expressing an, every bit, getting land | |
| MSNBC | different at, phony, frustrated with, running this, always a | |
| Mar-18 | CNN | watching this, adding a, manchurian candidate, not supported, upset with |
| FOXNEWS | in your, picking a, proposing to, planner, very predictable | |
| MSNBC | serious threat, not consulting, watching this, not here, joking | |
| Apr-18 | CNN | considering firing, killing us, now considering, considering sitting, his only |
| FOXNEWS | challenging, unethical, more entertaining, pretty funny, taking real | |
| MSNBC | subject, killing us, offered up, not worried, worried | |
| May-18 | CNN | compiling a, embarrassed by, unclear, viewed more, entering his |
| FOXNEWS | lying on, making history, to work, not probable, in very | |
| MSNBC | what he, compiling a, on one, at walter, exculpatory of | |
| Jun-18 | CNN | unfit for, at camp, working behind, using you, america first |
| FOXNEWS | not ever, crashing economy, hopeful, not necessary, never going | |
| MSNBC | being taken, meaningless to, turning america's, spy's dream, lobbying to | |
| Jul-18 | CNN | considering stripping, shutting down, using sports, completely innocent, inviting him |
| FOXNEWS | not ever, disaster, not tough, historic president, live | |
| MSNBC | not prepared, dealing away, being manipulated, on it, hours away | |
| Aug-18 | CNN | breaking records, so panicked, as long, turning to, being urged |
| FOXNEWS | con, guilty of, not well, framing you, about 15 | |
| MSNBC | getting more, con, doing any, drunk on, disingenuous about | |
| Sep-18 | CNN | now demanding, to get, with you, now calling, drag on |
| FOXNEWS | complicit, harming, making america's, in full, sociopath | |
| MSNBC | symptom of, one driver, escalating america's, largely through, very angry | |
| Oct-18 | CNN | anti-christ, speaking out, talking up, on his, weak |
| FOXNEWS | not responsible, doing great, i feel, no prize, autocrat | |
| MSNBC | good role, getting away, not campaigning, entitled to, already talking | |
| Nov-18 | CNN | in paris, pissed at, going too, campaigning like, certainly campaigning |
| FOXNEWS | rallying, undoing, very happy, job loser, taking no | |
| MSNBC | fully stable, beginning to, rather insecure, allegedly back, taking mueller's | |
| Dec-18 | CNN | not removed, resorting to, most polarizing, privately telling, telling |
| FOXNEWS | plunging, throwing a, for it, fixated on, versus | |
| MSNBC | gleeful, target, telling, genuinely melting, traitor to | |
| Jan-19 | CNN | great deal, agent of, that it, calling on, clinging to |
| FOXNEWS | manufacturing a, rude, like spinach, delivering on, having an | |
| MSNBC | calling on, wholly owned, considering declaring, so nervous, fraud | |
| Feb-19 | CNN | russian asset, hours away, denying, overseas next, increasingly fixated |
| FOXNEWS | acting outside, different he, first us, in very, in vietnam | |
| MSNBC | his children, not 6'3, national emergency, around, making decisions | |
| Mar-19 | CNN | not worth, effectively cleared, mostly right, morally unfit, golfing |
| FOXNEWS | not worth, bad for, so powerful, confident, different from | |
| MSNBC | cheat, unindicted co-conspirator, losing ground, behaving like, not exonerated | |
| Apr-19 | CNN | putting his, pressing pause, re-elected, still under, lashing out |
| FOXNEWS | tearing, presumed innocent, printed, declaring as, inspecting a | |
| MSNBC | only president, more dangerous, 72, not above, having done | |
| May-19 | CNN | goading us, bleep, in japan, warning iran, disobeying |
| FOXNEWS | stunning, not scared, taking credit, literal treasonous, considered collaborating | |
| MSNBC | goading us, primed to, lying when, individual 1, patriotic man | |
| Jun-19 | CNN | in south, not enough, an illegitimate, never be, wrapping up |
| FOXNEWS | showing restraint, in japan, even harder, done is, horrible | |
| MSNBC | in london, floor, telling friends, not enough, in ireland | |
| Jul-19 | CNN | not above, playing on, because of, charging ahead, fortifying his |
| FOXNEWS | because of, working for, racist, going out, doing through | |
| MSNBC | not above, racist, because of, doing at, spending all | |
| Aug-19 | CNN | white supremacist, white nationalist, greatest president, sounding upbeat, asian man |
| FOXNEWS | racist, as destructive, asian man, responsible for, so fast | |
| MSNBC | white supremacist, underwater, tweeting insults, roiling financial, getting worse | |
| Sep-19 | CNN | searching for, gun shy, hugely worried, done testifying, confirming |
| FOXNEWS | mental illness, reelected, doing this, jerk card, too old | |
| MSNBC | demanding to, holding up, re-elected, doing this, taking away | |
| Oct-19 | CNN | legal, impeached, hedging his, showing small, effectively rejecting |
| FOXNEWS | most corrupt, upset, here exclusively, being accused, so entertaining | |
| MSNBC | upset, pressing for, practically daring, that even, being impeached | |
| Nov-19 | CNN | welcome to, war room, gigantic impediment, ditching new, now formal |
| FOXNEWS | a better, also better, still entertaining, defeated, welcome to | |
| MSNBC | welcome to, being impeached, based on, clearly rattled, fourth president |