Using GSG TV News Embeddings For Multimodal Search: Search A Decade Of TV News Using An Online News Article

Earlier today we demonstrated how to use our new GSG TV News Embeddings to semantically search television news for references to a known fact check claim. By converting a three-sentence fact check claim into an embedding using the same model we used for television news, we can identify references to it across television news simply by computing its cosine similarity against the other 189 million sentences we've processed and identifying those with the highest similarity scores. Since the DAN-based Universal Sentence Encoder model we use for the GSG TV News Embeddings doesn't have a fixed input token size and has an essentially flat inference time with respect to input length, we can convert entire documents to embeddings. This means that instead of just searching television news for a three-sentence fact check claim, we can search it for statements highly similar to any arbitrary online news article!

Let's start with this CNN article from earlier today titled "California hydropower plant forced to shut down as water levels fall at Lake Oroville" that connects climatic-induced drought to critically low water levels at Lake Oroville that forced its hydropower plant to be taken offline.

First, we need to convert the article into a USE embedding. To do so, create a free new Colab notebook and run the following code, replacing the "FULL TEXT OF ONLINE ARTICLE" below with the copy-pasted text of the CNN article:

#load libraries...
import tensorflow_hub as hub
import tensorflow as tf
!pip install tensorflow_text
import tensorflow_text as text # Needed for loading universal-sentence-encoder-cmlm/multilingual-preprocess
import numpy as np

#normalize...
def normalization(embeds):
norms = np.linalg.norm(embeds, 2, axis=1, keepdims=True)
return embeds/norms

sent = tf.constant(["FULL TEXT OF ONLINE ARTICLE"])
embed_use = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")
sente = embed_use(sent)
sente = normalization(sente)
print(repr(sente))

It loads the necessary libraries and converts the verbatim three-sentence fact check claim summary as-is into the USE vector representation, printing a 512-dimension array:

<tf.Tensor: shape=(1, 512), dtype=float32, numpy= array([[ 0.04630536, -0.04385197, -0.05280872, -0.04773887, -0.00309931, 0.05450295, -0.02673355, -0.05545911, 0.05327506, 0.04394556, -0.03168765, 0.05503967, 0.00442976, -0.05656279, 0.05117863, 0.0158463 , 0.05592725, 0.03598066, -0.03839573, -0.00109333, -0.05644736, -0.05434908, -0.02864576, -0.04568801, 0.05557097, -0.02160742, -0.03676632, -0.00822768, 0.00673328, 0.05361567, 0.02790285, 0.01862041, -0.04964391, -0.00603407, 0.02612479, -0.05509547, 0.01189164, 0.02481853, 0.05586883, 0.04517639, 0.0466411 , 0.050674 , 0.04127763, -0.05451256, -0.05663854, 0.05264238, -0.01035258, 0.02292279, 0.01242081, -0.04307327, -0.05607928, 0.01975088, -0.0522441 , -0.03337459, 0.00374916, 0.03055446, -0.05663315, -0.01754221, 0.05500044, 0.04655377, 0.01067017, -0.04976329, 0.05180267, -0.05604182, -0.05621086, -0.05376201, 0.05308988, 0.05225808, 0.02736193, 0.05039906, 0.04467546, 0.04993341, -0.01599472, 0.05476574, -0.03768978, 0.03524011, 0.038475 , -0.04673229, 0.05104368, 0.03713217, 0.05663855, -0.03777216, 0.04949455, 0.05450713, 0.0121822 , 0.04588546, 0.03606191, -0.05663596, -0.00443359, -0.00486312, 0.04559771, -0.03937602, 0.05089479, 0.05131284, -0.05663429, 0.011048 , -0.03472014, 0.05651702, 0.05602308, -0.03779978, -0.04273533, 0.02687778, -0.00161195, 0.0483719 , 0.05653317, -0.02144088, -0.02948533, -0.03920264, 0.05462149, -0.04935208, -0.05296955, -0.04880546, -0.00295182, 0.04659003, -0.05461466, -0.01888734, 0.05398951, 0.02089173, 0.05544703, 0.05631928, 0.05661533, -0.01530546, 0.02082386, 0.0565156 , 0.0480477 , 0.05597575, 0.05648562, -0.03431455, -0.02098618, 0.05572988, 0.04933405, 0.05657511, 0.05287464, -0.04752011, -0.0127286 , -0.05533136, -0.05246781, -0.05286954, -0.01377515, -0.05659974, -0.05340074, 0.05663871, -0.05304778, 0.0356891 , 0.05625559, -0.03166396, -0.02535708, 0.05566574, -0.05334753, 0.03626539, -0.05652655, -0.02445913, -0.04480174, -0.0536372 , -0.05637573, 0.05520798, 0.00620194, 0.00769248, -0.00251708, 0.02002978, 0.01431433, -0.05497971, 0.05050481, 0.00775918, -0.05434256, 0.05292607, 0.05663849, 0.03883202, -0.05636555, 0.02466718, 0.05662578, -0.0565693 , 0.05112001, -0.03461725, -0.05163195, 0.05614306, 0.01156624, -0.02751734, -0.05663678, -0.04754735, -0.03358746, 0.04703908, -0.05295394, -0.0298285 , 0.0561368 , -0.00339463, -0.04886793, -0.05578013, -0.00150067, -0.04190002, -0.03766162, -0.05042543, -0.0564624 , 0.05586027, -0.05423312, -0.05428811, 0.01233629, -0.03560453, 0.0438605 , -0.01390999, 0.05649086, 0.04363002, -0.01915026, -0.04412376, -0.04625799, 0.00250358, -0.01445448, -0.0549707 , -0.03246034, 0.05128665, 0.02786998, 0.05596329, -0.00467303, 0.05363059, -0.05618523, -0.05351293, 0.05294608, 0.05418331, 0.05172252, -0.05613088, 0.04158759, -0.0531192 , -0.00016934, 0.04018888, 0.04438653, -0.05590613, -0.05545995, -0.05568601, 0.04614003, 0.0515261 , -0.05565028, 0.05064454, 0.04195472, -0.05663871, 0.02045964, 0.05661396, 0.01868009, 0.04616543, -0.05425839, 0.02895739, -0.00870576, -0.05060951, 0.05448073, -0.03053513, 0.05588634, 0.05284701, 0.0387537 , 0.05663617, 0.01202832, 0.05647126, 0.05554701, -0.04252514, -0.01016596, -0.05591315, 0.04815601, 0.02560365, 0.03293224, 0.05663871, -0.03906305, -0.05614728, -0.00922975, 0.03632371, -0.04679849, 0.00541904, -0.0483457 , 0.04948195, 0.05629317, -0.05365576, -0.0540406 , 0.04029153, -0.04976862, 0.05627936, 0.00623676, 0.04831374, 0.04152348, -0.05591218, 0.03268768, -0.05616672, 0.00461178, 0.05557774, -0.02102494, 0.04548164, -0.01191826, -0.03333873, -0.03785552, -0.05644065, 0.05057264, 0.0535511 , -0.03758802, 0.05241463, -0.03128109, 0.04006368, 0.05663619, 0.02352799, 0.03468762, -0.04845323, 0.05241161, -0.05661327, -0.03323733, 0.0562074 , -0.0532062 , 0.05125945, 0.05163905, -0.03261847, -0.03992615, 0.05551983, 0.05439377, 0.05566144, 0.01520137, -0.04974468, -0.05259595, -0.05453914, -0.05373696, 0.00099202, 0.03560246, -0.04520666, 0.05654072, -0.04186178, 0.05656121, -0.05306372, -0.03237957, 0.04866744, 0.05546519, -0.05446809, 0.02790952, -0.02287924, 0.01771197, -0.02117336, -0.05658579, 0.02544056, -0.05613875, -0.00273917, -0.04361826, 0.05231266, -0.04763988, 0.05401538, 0.05242627, 0.00802161, -0.03727852, -0.05454857, 0.05630051, -0.05627939, -0.03195082, 0.05663869, 0.03092248, -0.0566386 , -0.05579765, 0.04760486, -0.04988462, -0.04649271, 0.05289843, 0.04974864, -0.02068705, -0.03956971, -0.04818833, 0.01898351, -0.05663871, -0.05455592, 0.04776748, 0.05336981, -0.01439432, -0.03554876, 0.05509063, 0.0529646 , -0.01606726, 0.04869639, 0.0565936 , 0.05117821, 0.05582917, 0.04640149, -0.04383011, 0.05116829, -0.04927861, -0.05622511, 0.03731203, 0.03827041, 0.05329274, 0.03752604, 0.04686715, -0.05653588, 0.0525238 , -0.00743251, -0.04215745, -0.05066058, 0.03327134, -0.05321286, 0.0284178 , 0.05511777, -0.04277489, -0.04043424, 0.02011093, 0.05063638, 0.05647369, -0.05660866, -0.05375618, -0.02472765, -0.05039553, 0.0210466 , -0.02123819, -0.05449731, -0.0495085 , -0.05599762, 0.01672727, -0.03641875, -0.0109718 , 0.0197558 , -0.05473058, -0.03824357, -0.04720744, -0.02279065, 0.05560668, -0.04888083, -0.0543465 , 0.05614474, 0.05656378, -0.05269594, -0.04969141, 0.03275722, -0.04788459, -0.00249315, -0.02697386, -0.05160728, 0.04478324, 0.04320117, -0.05504712, -0.05186576, -0.03642318, 0.05662105, 0.0219523 , -0.02522418, -0.02286198, -0.05325816, -0.01484585, 0.05663819, -0.03145841, -0.04733086, 0.02927428, -0.02253605, 0.04772612, 0.04731531, -0.05653518, 0.01324066, -0.03877051, -0.02176844, -0.05550387, -0.05629394, 0.03758418, -0.05630903, -0.02910103, -0.04853559, -0.04942301, -0.02937661, 0.04367953, -0.04979736, -0.0544776 , -0.05663146, 0.00049248, 0.05663855, -0.05391856, -0.03783586, 0.03743649, 0.04993442, 0.04803006, 0.0566255 , -0.05663816, 0.03799942, 0.00489313, 0.0294361 , -0.03560543, 0.01455918, -0.0566387 , -0.0045967 , -0.05587058, -0.04253622, 0.02515171, -0.04162445, 0.05623724, -0.02933892, 0.05354614, -0.05659237, 0.01207218, 0.05556235, -0.00630465, -0.05662564, -0.01850525, 0.05583531, 0.04880809, 0.03923202, 0.05379379, -0.05649344, 0.01326263, -0.02756828, -0.03858707, 0.02329104, -0.05156119, 0.03547219, 0.05663704, 0.02843156, -0.00453846, -0.03434811, -0.03759045, -0.05455792, -0.01064915, -0.05663871, 0.05413051, 0.05407911, -0.05658937, 0.01777141, -0.04965514, 0.0557703 , -0.05490747, -0.05608686]], dtype=float32)>

Now we need to compute the cosine similarity of this vector against all of the sentence-level vectors in the Global Similarity Graph Television News Sentence Embeddings dataset to find potential references to it.

We just copy-paste the vector above into a simple SQL + UDF query in BigQuery which yields:

CREATE TEMPORARY FUNCTION cossim(a ARRAY<FLOAT64>, b ARRAY<FLOAT64>)
RETURNS FLOAT64 LANGUAGE js AS '''
var sumt=0, suma=0, sumb=0;
for(i=0;i<a.length;i++) {
sumt += (a[i]*b[i]);
suma += (a[i]*a[i]);
sumb += (b[i]*b[i]);
}
suma = Math.sqrt(suma);
sumb = Math.sqrt(sumb);
return sumt/(suma*sumb);
''';

WITH query AS (
select [0.04630536, -0.04385197, -0.05280872, -0.04773887, -0.00309931,
0.05450295, -0.02673355, -0.05545911, 0.05327506, 0.04394556,
-0.03168765, 0.05503967, 0.00442976, -0.05656279, 0.05117863,
0.0158463 , 0.05592725, 0.03598066, -0.03839573, -0.00109333,
-0.05644736, -0.05434908, -0.02864576, -0.04568801, 0.05557097,
-0.02160742, -0.03676632, -0.00822768, 0.00673328, 0.05361567,
0.02790285, 0.01862041, -0.04964391, -0.00603407, 0.02612479,
-0.05509547, 0.01189164, 0.02481853, 0.05586883, 0.04517639,
0.0466411 , 0.050674 , 0.04127763, -0.05451256, -0.05663854,
0.05264238, -0.01035258, 0.02292279, 0.01242081, -0.04307327,
-0.05607928, 0.01975088, -0.0522441 , -0.03337459, 0.00374916,
0.03055446, -0.05663315, -0.01754221, 0.05500044, 0.04655377,
0.01067017, -0.04976329, 0.05180267, -0.05604182, -0.05621086,
-0.05376201, 0.05308988, 0.05225808, 0.02736193, 0.05039906,
0.04467546, 0.04993341, -0.01599472, 0.05476574, -0.03768978,
0.03524011, 0.038475 , -0.04673229, 0.05104368, 0.03713217,
0.05663855, -0.03777216, 0.04949455, 0.05450713, 0.0121822 ,
0.04588546, 0.03606191, -0.05663596, -0.00443359, -0.00486312,
0.04559771, -0.03937602, 0.05089479, 0.05131284, -0.05663429,
0.011048 , -0.03472014, 0.05651702, 0.05602308, -0.03779978,
-0.04273533, 0.02687778, -0.00161195, 0.0483719 , 0.05653317,
-0.02144088, -0.02948533, -0.03920264, 0.05462149, -0.04935208,
-0.05296955, -0.04880546, -0.00295182, 0.04659003, -0.05461466,
-0.01888734, 0.05398951, 0.02089173, 0.05544703, 0.05631928,
0.05661533, -0.01530546, 0.02082386, 0.0565156 , 0.0480477 ,
0.05597575, 0.05648562, -0.03431455, -0.02098618, 0.05572988,
0.04933405, 0.05657511, 0.05287464, -0.04752011, -0.0127286 ,
-0.05533136, -0.05246781, -0.05286954, -0.01377515, -0.05659974,
-0.05340074, 0.05663871, -0.05304778, 0.0356891 , 0.05625559,
-0.03166396, -0.02535708, 0.05566574, -0.05334753, 0.03626539,
-0.05652655, -0.02445913, -0.04480174, -0.0536372 , -0.05637573,
0.05520798, 0.00620194, 0.00769248, -0.00251708, 0.02002978,
0.01431433, -0.05497971, 0.05050481, 0.00775918, -0.05434256,
0.05292607, 0.05663849, 0.03883202, -0.05636555, 0.02466718,
0.05662578, -0.0565693 , 0.05112001, -0.03461725, -0.05163195,
0.05614306, 0.01156624, -0.02751734, -0.05663678, -0.04754735,
-0.03358746, 0.04703908, -0.05295394, -0.0298285 , 0.0561368 ,
-0.00339463, -0.04886793, -0.05578013, -0.00150067, -0.04190002,
-0.03766162, -0.05042543, -0.0564624 , 0.05586027, -0.05423312,
-0.05428811, 0.01233629, -0.03560453, 0.0438605 , -0.01390999,
0.05649086, 0.04363002, -0.01915026, -0.04412376, -0.04625799,
0.00250358, -0.01445448, -0.0549707 , -0.03246034, 0.05128665,
0.02786998, 0.05596329, -0.00467303, 0.05363059, -0.05618523,
-0.05351293, 0.05294608, 0.05418331, 0.05172252, -0.05613088,
0.04158759, -0.0531192 , -0.00016934, 0.04018888, 0.04438653,
-0.05590613, -0.05545995, -0.05568601, 0.04614003, 0.0515261 ,
-0.05565028, 0.05064454, 0.04195472, -0.05663871, 0.02045964,
0.05661396, 0.01868009, 0.04616543, -0.05425839, 0.02895739,
-0.00870576, -0.05060951, 0.05448073, -0.03053513, 0.05588634,
0.05284701, 0.0387537 , 0.05663617, 0.01202832, 0.05647126,
0.05554701, -0.04252514, -0.01016596, -0.05591315, 0.04815601,
0.02560365, 0.03293224, 0.05663871, -0.03906305, -0.05614728,
-0.00922975, 0.03632371, -0.04679849, 0.00541904, -0.0483457 ,
0.04948195, 0.05629317, -0.05365576, -0.0540406 , 0.04029153,
-0.04976862, 0.05627936, 0.00623676, 0.04831374, 0.04152348,
-0.05591218, 0.03268768, -0.05616672, 0.00461178, 0.05557774,
-0.02102494, 0.04548164, -0.01191826, -0.03333873, -0.03785552,
-0.05644065, 0.05057264, 0.0535511 , -0.03758802, 0.05241463,
-0.03128109, 0.04006368, 0.05663619, 0.02352799, 0.03468762,
-0.04845323, 0.05241161, -0.05661327, -0.03323733, 0.0562074 ,
-0.0532062 , 0.05125945, 0.05163905, -0.03261847, -0.03992615,
0.05551983, 0.05439377, 0.05566144, 0.01520137, -0.04974468,
-0.05259595, -0.05453914, -0.05373696, 0.00099202, 0.03560246,
-0.04520666, 0.05654072, -0.04186178, 0.05656121, -0.05306372,
-0.03237957, 0.04866744, 0.05546519, -0.05446809, 0.02790952,
-0.02287924, 0.01771197, -0.02117336, -0.05658579, 0.02544056,
-0.05613875, -0.00273917, -0.04361826, 0.05231266, -0.04763988,
0.05401538, 0.05242627, 0.00802161, -0.03727852, -0.05454857,
0.05630051, -0.05627939, -0.03195082, 0.05663869, 0.03092248,
-0.0566386 , -0.05579765, 0.04760486, -0.04988462, -0.04649271,
0.05289843, 0.04974864, -0.02068705, -0.03956971, -0.04818833,
0.01898351, -0.05663871, -0.05455592, 0.04776748, 0.05336981,
-0.01439432, -0.03554876, 0.05509063, 0.0529646 , -0.01606726,
0.04869639, 0.0565936 , 0.05117821, 0.05582917, 0.04640149,
-0.04383011, 0.05116829, -0.04927861, -0.05622511, 0.03731203,
0.03827041, 0.05329274, 0.03752604, 0.04686715, -0.05653588,
0.0525238 , -0.00743251, -0.04215745, -0.05066058, 0.03327134,
-0.05321286, 0.0284178 , 0.05511777, -0.04277489, -0.04043424,
0.02011093, 0.05063638, 0.05647369, -0.05660866, -0.05375618,
-0.02472765, -0.05039553, 0.0210466 , -0.02123819, -0.05449731,
-0.0495085 , -0.05599762, 0.01672727, -0.03641875, -0.0109718 ,
0.0197558 , -0.05473058, -0.03824357, -0.04720744, -0.02279065,
0.05560668, -0.04888083, -0.0543465 , 0.05614474, 0.05656378,
-0.05269594, -0.04969141, 0.03275722, -0.04788459, -0.00249315,
-0.02697386, -0.05160728, 0.04478324, 0.04320117, -0.05504712,
-0.05186576, -0.03642318, 0.05662105, 0.0219523 , -0.02522418,
-0.02286198, -0.05325816, -0.01484585, 0.05663819, -0.03145841,
-0.04733086, 0.02927428, -0.02253605, 0.04772612, 0.04731531,
-0.05653518, 0.01324066, -0.03877051, -0.02176844, -0.05550387,
-0.05629394, 0.03758418, -0.05630903, -0.02910103, -0.04853559,
-0.04942301, -0.02937661, 0.04367953, -0.04979736, -0.0544776 ,
-0.05663146, 0.00049248, 0.05663855, -0.05391856, -0.03783586,
0.03743649, 0.04993442, 0.04803006, 0.0566255 , -0.05663816,
0.03799942, 0.00489313, 0.0294361 , -0.03560543, 0.01455918,
-0.0566387 , -0.0045967 , -0.05587058, -0.04253622, 0.02515171,
-0.04162445, 0.05623724, -0.02933892, 0.05354614, -0.05659237,
0.01207218, 0.05556235, -0.00630465, -0.05662564, -0.01850525,
0.05583531, 0.04880809, 0.03923202, 0.05379379, -0.05649344,
0.01326263, -0.02756828, -0.03858707, 0.02329104, -0.05156119,
0.03547219, 0.05663704, 0.02843156, -0.00453846, -0.03434811,
-0.03759045, -0.05455792, -0.01064915, -0.05663871, 0.05413051,
0.05407911, -0.05658937, 0.01777141, -0.04965514, 0.0557703 ,
-0.05490747, -0.05608686] as sentEmbed
)
SELECT cossim(doc.sentEmbed, query.sentEmbed) sim, date, lead, previewUrl FROM `gdelt-bq.gdeltv2.gsg_iatvsentembed` doc, query order by sim desc limit 10000

This computes the cosine similarity of the online CNN article against all 189 million television news sentences we have indexed, sorts them and returns the top 10,000 results in just 2 minutes 19 seconds, processing 740GB of data. This includes results like:

• "and the evidence of drought is most obvious at lake mead, the largest reservoir in the u.s., now below 40% capacity and still falling, already at an all-time low with the federal government poised to declare a formal wattle to water shortage in which likely triggering major water restrictions for million people in nevada, arizona, and california." (Fox News)
• "the 0roville dam, which is about 150 miles north—east of san francisco, is the tallest in the us." (BBC News)
• "the grizzly creek fire near denver, colorado has burned 25000 acres and forced to shut down of i 70, the main east to west artery, the shutdown is training local in supply chains, the brittle heat is putting a strain on california power grid, forcing the states power grid operator to declare a stage three emergency to conserve energy, the blackout lasted for 20 minutes last night, they were rotating power outages throughout the state on friday to, a stage iii emergency like this has not been declared in california in nearly 20 years, also a red flag warning is going to remain in effect through tomorrow morning for the bay area because more thunderstorms and lightning is expected overnight which could lead to more wildfires in the state." (Fox News)
• "100,000 cubic feet per second of water, rushing out of lake 0roville in california." (BBC News)
• "california's lake orville, the states second largest reservoir is still seeing its water levels falling so low it's expected the hydroelectric power plant will be forced to shutdown for the first time in its history." (CNN)
• "in spring of 2001 and 2005 nearly 1 in 10 acres in montana was facing a severe drought, and about 9 billion of ground water are drawn in idaho everyday for agriculture, and 2 billion in montana and 90% of it for irrigation, but the experts warn that we should not take it for granted and water scarcity is going to be a bigger issue when times come by, and they say that it is going to make our current fight over the keystone pipeline look silly by comparison." (MSNBC)

Looking at the examples above, you can see they range from references to the Oroville dam (note in particular the second example which includes a typo of the number zero instead of the letter "O" as the first character – despite this, it was successfully matched by the embedding due either to the surrounding context or that particular error being learned by the model. Other matches include similar drought scenarios in other states and dams. Thus, we can see that our similarity matches span a range of related topics, ranging from the Oroville dam to other dams and drought conditions.

We hope this inspires you with your own ideas for multimodal search of television using online news coverage!