Using Global Similarity Graph Document Embeddings & BigQuery For "More Like This" Search

Earlier today we announced the new Global Similarity Graph Document Embeddings dataset that uses the Universal Sentence Encoder V4 to compute document-level embeddings for each news article we monitor in realtime across 65 languages using machine translation. Using BigQuery, we can select an article from the dataset and perform a brute-force cosine similarity comparison of that article against every other article in the dataset and return the top most similar articles. In a production application this approach would be far too slow and resource-intensive and you would use something like Vertex Matching Engine, but for simple explorative analysis it demonstrates the tremendous power of embeddings for encoding the semantic focus of articles.

It turns out that Google has already published a sample embedding similarity search query for BigQuery that required only minimal customization to work on the GSG Document Embeddings dataset:

SELECT
  c.k1 as query_title,
  c.k2 as match_title,
  SUM(vv1*vv2) / (SQRT(SUM(POW(vv1,2))) * SQRT(SUM(POW(vv2,2)))) AS similarity,
  ANY_VALUE(c.u2) as match_url
FROM
(
  SELECT
    a.key k1, a.val v1, b.key k2, b.val v2, a.url u1, b.url u2
  FROM
  (
    SELECT title key, url url, docembed val FROM `gdelt-bq.gdeltv2.gsg_docembed` WHERE DATE(date) = "2021-07-30" and url='https://www.marketwatch.com/story/cdc-expected-to-release-data-that-informed-change-on-face-mask-policy-says-delta-variant-of-covid-means-war-has-changed-11627655532' LIMIT 1
   ) a
  CROSS JOIN
  (
    SELECT title key, url url, docembed val FROM `gdelt-bq.gdeltv2.gsg_docembed` WHERE DATE(date) = "2021-07-30" 
  ) b
) c
, UNNEST(c.v1) vv1 with offset ind1 JOIN UNNEST(c.v2) vv2 with offset ind2 ON (ind1=ind2)
GROUP BY c.k1, c.k2
ORDER BY similarity DESC
LIMIT 100

Here we select a MarketWatch article titled "CDC releases data that informed reconsideration of face masks, says delta variant of COVID means the ‘war has changed’" and compute its similarity to every other article in the dataset on the same day (we could easily compare against the entire dataset, but often you might only be interested in other coverage from the same day). You can see the results in the table below showing how it is able to find related coverage even if it doesn't use the same exact words. While most of the results below are in English, you can also see several articles in other languages, including Bulgarian, French, Greek, Hungarian, Romanian and Turkish.

Thus, starting with an article of interest it is trivial to identify others like it using this dataset!