The Making Of The 'Trump Is' Analysis

For those interested in applying today's 'Trump Is…" analysis (in which every mention of "trump is" on CNN, MSNBC and Fox News June 2015 to present was scanned for every mention of "trump is" and the following two words aggregated by month to compile a monthly list of the top descriptions associated with him) to other public figures and topics, here is the process we used.

Note that the scripts below assume that your query returns fewer than 3,000 results per month – you will have to customize the "makefetchcmds.pl" script to use a shorter time horizon until each query returns fewer results.

First, download "makefetchcmds.pl" to your computer. Decide the start and stop month for your query and convert your query into URL-escaped form (quotes become "%22", spaces become "%20" and so on). Then run once for each station of interest. For example, to compile monthly files from May 2015 to November 2019 for the query "trump is" on CNN, MSNBC and Fox News you would use the following commands:

time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" CNN
time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" MSNBC
time ./makefetchcmds.pl 201505 201911 "%22trump%20is%22" FOXNEWS

After running each of the commands above, the script will output a "CMDS.TXT" file with the necessary list of CURL queries to compile the cache of "trump is" responses for each station. Run each of the "makefetchcmds.pl" commands above, stopping to execute all of the CURL queries in its CMDS.TXT file before moving to the next one. Do not change the "sleep 5" delay at the end of each fetch – this ensures you do not overburden the TV API.

To get an idea of the output these scripts generate, here is the CURL query they create for CNN for June 2015:

time curl "https://api.gdeltproject.org/api/v2/tv/tv?startdatetime=20150601000000&enddatetime=20150630235959&datanorm=raw&query=%22trump%20is%22%20station:CNN%20&mode=clipgallery&format=csv&maxrecords=5000" -o ./CNN.201506.csv

The result is  CSV file containing the URL of the clip in the Internet Archive's Television News Archive (so you can watch the clip), the date/time the clip aired (in UTC), the station, show name, the Archive's unique ShowID for the broadcast, the preview thumbnail for the clip and the brief matching snippet of the 15 second clip containing your query. Note that in a small number of cases your phrase may appear at the end of the snippet or may appear cut off at the end of the snippet – this indicates that your phrase straddled the 15 second boundary between clips. The API looks ahead of each clip to ensure that phrases that span clips are attributed to the clip the phrase begins in, but this means that the results snippets may occasionally be truncated – these comprise a very small number of cases and are dropped by the script below.

Next, convert these CSV files into monthly word histograms. Download "makehist.pl" and run it over your CSV cache downloaded above. A simple way of doing this is to download GNU parallel and use it:

time find ./CACHE/*.csv | parallel --eta './makehist.pl {}'

This will parse each CSV file, compile the two words following "trump is" and output a histogram of the matches for that month.

Note – to adapt for other queries, just replace the word "trump" in the regular expression with your query and adjust the "(is|was|are|were)" exclusions appropriately.

At this point you have a list of the top two-word phrases by month for your query. Of course, the most common phrases will likely be consistent across months, typically grammatical phrases like "going to" or "trying to." Use a TFIDF filter to eliminate these common phrases that appear every month and surface the most important ones.

Finally, download "maketimeline.pl" – this script compiles all of the histograms above into a single output timeline, including performing modified TFIDF scoring to surface the most significant phrases by month. Run as:

time ./maketimeline.pl > MYTIMELINE.TXT

By default the script outputs the timeline according to date, with all stations listed under each date, but you can change the sort order by simply uncommenting the two lines in the timeline construction portion of the script and commenting the two above them to sort by station -> date.

Note that this script uses a modified TFIDF score in which the IDF component treats each month as a "document" (rather than each clip) and thus adjusts by how many months a phrase appeared in, while the TF component is the standard raw frequency count. This normalizes against the number of months a phrase appears in and results in more meaningful results in this case. You might experiment with other TFIDF definitions for your particular use case.

Remember that these phrases are a mixture of the wording of a station's own personalities and the words of others that it chooses to broadcast. Regardless of source, it captures the descriptive words a viewer of that station see associated with a given person/topic.

The final result for the "Trump is" analysis can be seen below, showcasing the power of this approach for tracing how media portrayals of public figures and topics evolve over time.

Jun-15 CNN almost double, like almost, out at, catching on, threatening to
FOXNEWS running for, your loser, hitting some, sharp guy, being talked
MSNBC making another, eliminating, paying some, apparently launching, in second
Jul-15 CNN wrong on, also creating, not backing, political car, using your
FOXNEWS political fraud, here tonight, leading, firing up, employing is
MSNBC modern day, bringing a, surviving this, spiking to, surging
Aug-15 CNN interesting man, leading, dominating, leading for, on top
FOXNEWS good for, hitting a, so popular, boss i, resinating well
MSNBC raising, sucking all, tapping into, featured on, touching a
Sep-15 CNN entertainer, maybe, leading, criticizing jeb, relatively new
FOXNEWS entertainer, shallow, narcissist, wonderful entertainer, now becoming
MSNBC entertainer, for donald, surging with, insecure, seeing a
Oct-15 CNN probably going, campaigning now, now even, entertaining, weak on
FOXNEWS entertaining, now number, still on, here with, on top
MSNBC committed christian, entertaining, pre-bragging about, dictating, showing people
Nov-15 CNN sharpening his, hosting 'saturday, insult, tweeting out, still on
FOXNEWS hosting 'saturday, still number, funny, not slamming, ratings machine
MSNBC now convinced, make up, not worth, here spewing, obviously winning
Dec-15 CNN on video, jerk, sending around, center stage, a jerk
FOXNEWS playing cruz, big christmas, calling for, posing a, genuine conservative
MSNBC playing right, on video, jerk, sending around, calling for
Jan-16 CNN welcoming, bringing up, suggesting, not used, maintaining his
FOXNEWS out, top seed, not there, brilliant, lasagna
MSNBC running, fool, your guy, debuting his, welcoming
Feb-16 CNN very rattled, this campaign, not christian, our campaign, intensifying this
FOXNEWS behind me, very rattled, right candidate, this campaign, big winner
MSNBC nominee, way ahead, this campaign, that man, leading
Mar-16 CNN nominee, pathological liar, ahead, winner, the nominee
FOXNEWS nominee, phony, 20 points, clown, front-runner
MSNBC nominee, con man, gift wrapped, ted cruz, phony
Apr-16 CNN nominee, over, poised to, whining, behind hillary
FOXNEWS nominee, expect to, predicting a, close to, unstoppable
MSNBC nominee, winning by, denied, not elected, heavy favorite
May-16 CNN presumptive nominee, qualified to, presumptive republican, reliable conservative, nominee
FOXNEWS last man, presumptive nominee, presumptive republican, qualified to, nominee
MSNBC presumptive nominee, nominee, terrible person, so stupid, no nothing
Jun-16 CNN loud, picking on, temperamentally unfit, criticizing judge, learning how
FOXNEWS temperamentally unfit, on with, unfit to, featured for, drawing an
MSNBC loud, certainly a, genuine threat, defeated, temperamentally unfit
Jul-16 CNN so dangerous, now fully, good man, also ignoring, in colorado
FOXNEWS good man, campaigning hard, best choice, far better, compelling
MSNBC his hypocrisy, you're fired, not raising, defeated, bad person
Aug-16 CNN their candidate, not qualified, up 10, comfortable about, very plugged
FOXNEWS because a, everywhere, pounding, airing tomorrow, pretty strong
MSNBC losing by, down ten, up 10, unfit to, reinforcing harmful
Sep-16 CNN phony, closer, disgusting fraud, amateur, not qualified
FOXNEWS doing better, potentially, now leading, changing, phony
MSNBC phony, best debater, no republican, biased against, unfit for
Oct-16 CNN definitely back, already making, seizing on, nazi, trailing
FOXNEWS person best, pushing back, hitting, in michigan, vehemently denied
MSNBC sexual predator, frankly pathetic, behind, headline, clearly going
Nov-16 CNN phony, president-elect, ahead, challenge unlike, uniquely unqualified
FOXNEWS very man, president-elect, 70000 votes, not their, good guy
MSNBC phony, president-elect, sworn, serious about, almost surely
Dec-16 CNN inaugurated, sworn, president-elect, fully briefed, uniquely unqualified
FOXNEWS sworn, promising to, already acting, president-elect, assembling a
MSNBC sworn, very committed, echoing, officially off, surrounding himself
Jan-17 CNN sworn, legitimate president, inaugurated, still unsure, so unaware
FOXNEWS sworn, watching right, mentally unstable, pledging to, he t
MSNBC sworn, 44% which, legitimate president, very concerned, illegitimate
Feb-17 CNN horrible, embodiment of, often, facing his, still moving
FOXNEWS top global, billy jack, not boring, kicking off, horrible
MSNBC attacking judges, learning, skipping cpac, horrible, no exception
Mar-17 CNN extremely confident, requesting, destroying, requesting as, to my
FOXNEWS exercising his, in cahoots, negotiator, somewhere between, clouding your
MSNBC requesting, exercising his, 70-year-old rookie, in campaign, president right
Apr-17 CNN making fools, considering military, on pace, facing several, now considering
FOXNEWS white isis, wall, most dangerous, him, dumbest president
MSNBC learning, caving on, more mysterious, not relishing, likable
May-17 CNN in brussels, weighing whether, back, in saudi, heading out
FOXNEWS terrorizing, still here, on with, seeking is, up next
MSNBC dangerous, acting, abusing his, considering firing, recording conversations
Jun-17 CNN traitor, considering firing, actively engaging, not under, not well
FOXNEWS not under, traitor, sexist, radicalized whack, slamming its
MSNBC under investigation, when it's, hitting on, not under, narcissist on
Jul-17 CNN seen having, already chafing, discussing possibly, guest of, very different
FOXNEWS militarizing, half-term president, wall, phenomenon, media's x
MSNBC overseas, already looking, looking into, already basically, in warsaw
Aug-17 CNN driving, morally bankrupt, in arizona, turning into, throwing his
FOXNEWS assassinated, promising 100, so evil, tearing, anti-christ
MSNBC officially on, tearing, fuming to, dividing our, and
Sep-17 CNN white supremacist, biggest tv, moving to, reconsidering plans, trolling north
FOXNEWS white supremacist, weighing his, donating $1, unicorn, vindicated
MSNBC white supremacist, staying quiet, original, about getting, in alabama
Oct-17 CNN not finished, constantly undermining, telling you, quote seething, mentally deranged
FOXNEWS mentally deranged, right people, route, moved into, calling
MSNBC moron, stopping, mentally ill, because they, treating
Nov-17 CNN in china, same bezos, everything, rejecting steve, leader of
FOXNEWS in beijing, back from, in south, in china, arriving
MSNBC same bezos, in china, accused 15, committing obstruction, president which
Dec-17 CNN loathsome human, attacking fbi, warning iran, inaugurated, making me
FOXNEWS head for, our president, billionaire we, an [bleep, packing
MSNBC being recruited, russia, owed a, madman, one half
Jan-18 CNN like negotiating, completely capable, fighting for, racially ignorant, racist
FOXNEWS like negotiating, mentally unfit, turn, taking too, making america
MSNBC like negotiating, pretty significant, interesting, known to, above water
Feb-18 CNN model of, either totally, okay with, incredibly supportive, living embodiment
FOXNEWS not my, already given, expressing an, every bit, getting land
MSNBC different at, phony, frustrated with, running this, always a
Mar-18 CNN watching this, adding a, manchurian candidate, not supported, upset with
FOXNEWS in your, picking a, proposing to, planner, very predictable
MSNBC serious threat, not consulting, watching this, not here, joking
Apr-18 CNN considering firing, killing us, now considering, considering sitting, his only
FOXNEWS challenging, unethical, more entertaining, pretty funny, taking real
MSNBC subject, killing us, offered up, not worried, worried
May-18 CNN compiling a, embarrassed by, unclear, viewed more, entering his
FOXNEWS lying on, making history, to work, not probable, in very
MSNBC what he, compiling a, on one, at walter, exculpatory of
Jun-18 CNN unfit for, at camp, working behind, using you, america first
FOXNEWS not ever, crashing economy, hopeful, not necessary, never going
MSNBC being taken, meaningless to, turning america's, spy's dream, lobbying to
Jul-18 CNN considering stripping, shutting down, using sports, completely innocent, inviting him
FOXNEWS not ever, disaster, not tough, historic president, live
MSNBC not prepared, dealing away, being manipulated, on it, hours away
Aug-18 CNN breaking records, so panicked, as long, turning to, being urged
FOXNEWS con, guilty of, not well, framing you, about 15
MSNBC getting more, con, doing any, drunk on, disingenuous about
Sep-18 CNN now demanding, to get, with you, now calling, drag on
FOXNEWS complicit, harming, making america's, in full, sociopath
MSNBC symptom of, one driver, escalating america's, largely through, very angry
Oct-18 CNN anti-christ, speaking out, talking up, on his, weak
FOXNEWS not responsible, doing great, i feel, no prize, autocrat
MSNBC good role, getting away, not campaigning, entitled to, already talking
Nov-18 CNN in paris, pissed at, going too, campaigning like, certainly campaigning
FOXNEWS rallying, undoing, very happy, job loser, taking no
MSNBC fully stable, beginning to, rather insecure, allegedly back, taking mueller's
Dec-18 CNN not removed, resorting to, most polarizing, privately telling, telling
FOXNEWS plunging, throwing a, for it, fixated on, versus
MSNBC gleeful, target, telling, genuinely melting, traitor to
Jan-19 CNN great deal, agent of, that it, calling on, clinging to
FOXNEWS manufacturing a, rude, like spinach, delivering on, having an
MSNBC calling on, wholly owned, considering declaring, so nervous, fraud
Feb-19 CNN russian asset, hours away, denying, overseas next, increasingly fixated
FOXNEWS acting outside, different he, first us, in very, in vietnam
MSNBC his children, not 6'3, national emergency, around, making decisions
Mar-19 CNN not worth, effectively cleared, mostly right, morally unfit, golfing
FOXNEWS not worth, bad for, so powerful, confident, different from
MSNBC cheat, unindicted co-conspirator, losing ground, behaving like, not exonerated
Apr-19 CNN putting his, pressing pause, re-elected, still under, lashing out
FOXNEWS tearing, presumed innocent, printed, declaring as, inspecting a
MSNBC only president, more dangerous, 72, not above, having done
May-19 CNN goading us, bleep, in japan, warning iran, disobeying
FOXNEWS stunning, not scared, taking credit, literal treasonous, considered collaborating
MSNBC goading us, primed to, lying when, individual 1, patriotic man
Jun-19 CNN in south, not enough, an illegitimate, never be, wrapping up
FOXNEWS showing restraint, in japan, even harder, done is, horrible
MSNBC in london, floor, telling friends, not enough, in ireland
Jul-19 CNN not above, playing on, because of, charging ahead, fortifying his
FOXNEWS because of, working for, racist, going out, doing through
MSNBC not above, racist, because of, doing at, spending all
Aug-19 CNN white supremacist, white nationalist, greatest president, sounding upbeat, asian man
FOXNEWS racist, as destructive, asian man, responsible for, so fast
MSNBC white supremacist, underwater, tweeting insults, roiling financial, getting worse
Sep-19 CNN searching for, gun shy, hugely worried, done testifying, confirming
FOXNEWS mental illness, reelected, doing this, jerk card, too old
MSNBC demanding to, holding up, re-elected, doing this, taking away
Oct-19 CNN legal, impeached, hedging his, showing small, effectively rejecting
FOXNEWS most corrupt, upset, here exclusively, being accused, so entertaining
MSNBC upset, pressing for, practically daring, that even, being impeached
Nov-19 CNN welcome to, war room, gigantic impediment, ditching new, now formal
FOXNEWS a better, also better, still entertaining, defeated, welcome to
MSNBC welcome to, being impeached, based on, clearly rattled, fourth president