Google's Cloud Vision API offers reverse image search, allowing you to specify a still image and it will locate other instances of that image across the open web, returning the URLs of pages they appeared on, along with the most common topics found in the descriptive captions of those other appearances. This means you can take an arbitrary image and determine whether it is a novel image, if not, how often it has appeared across the web and the kinds of objects and activities associated with it in the textual descriptions of it across the web.
How might we use this capability to learn more about an image seen on television news?
Using Cloud Video we searched the OCR'd onscreen text of television news for coverage of "defund the police" protests. One of the more common images shown across television news features a sea of people with a large stenciled sign being held aloft. Yet few details were provided for most appearances of this image – when and where was it taken, what is its context? One could imagine that when analyzing foreign state-owned media, there might also be questions about verification and whether an image actually depicts what it is claimed to show – classic fact checking questions.
Take the frame below from MSNBC's Live With Stephanie Rule at 6AM PDT on June 8, 2020.
Perhaps you came across a clip of the broadcast somewhere on the web and wanted to verify its contents and the details of what it depicts.
Using image segmentation you would crop just the photograph portion of the frame, dropping the bottom portion of the image with the chyron and other overlay text. In this case this was done manually, but in a production workflow this would be automated. This results in:
Then pass the image to Cloud Vision API and request its Web Entities feature:
curl -s -H "Content-Type: application/json" -H "Authorization: Bearer $(gcloud auth print-access-token)" https://vision.googleapis.com/v1/images:annotate -d '
{
"requests": [
{
"image": {
"source": {
"gcsImageUri": "[CROPPED IMAGE URL]"
}
},
"features": [
{
"maxResults": 200,
"type": "WEB_DETECTION"
},
]
}
]
}' > OUT2
Download The Full JSON Results File.
This returns a large JSON file that includes a number of results, including a list of the top entities most commonly associated with the image across the web, a list of partial cropped versions of this image seen across the web and a list of complete instances of the image, along with their URLs and page titles.
Here are the entities the Vision API shows are most commonly associated with this image across the web (note that in this case since we are checking this image more than a month after publication the details below are particularly rich as the image has propagated across the web and been heavily captioned, whereas if this image was being scanned in realtime there might have been fewer or less rich labels below):
{
"entityId": "/m/0gnwz4",
"score": 0.6056,
"description": "Demonstration"
},
{
"entityId": "/t/26tx7xy9_tgvr",
"score": 0.5578
},
{
"entityId": "/g/11jmpggx90",
"score": 0.5322,
"description": "Killing of George Floyd"
},
{
"entityId": "/m/01lvn4",
"score": 0.4995,
"description": "New York City Police Department"
},
{
"entityId": "/g/11fwjyk5ws",
"score": 0.478,
"description": "Defund the police"
},
{
"entityId": "/g/1td33zb0",
"score": 0.46137303,
"description": "Cable News Network"
},
{
"entityId": "/m/016497",
"score": 0.4386,
"description": "Police brutality"
},
{
"entityId": "/m/01wsk3",
"score": 0.4141,
"description": "Civil disorder"
},
{
"entityId": "/m/03qtwd",
"score": 0.31333035,
"description": "Crowd"
},
{
"entityId": "/t/2d94b73_3fn02",
"score": 0.2948
},
{
"entityId": "/m/02_286",
"score": 0.25900054,
"description": "New York"
},
{
"entityId": "/m/09c7w0",
"score": 0.006492,
"description": "United States"
}
In its list of other instances of this image across the web instances of it on CNN's website include (many of these are inset images pointing to related pages):
{
"url": "https://www.cnn.com/2020/06/28/us/nypd-car-hit-with-debris/index.html",
"pageTitle": "NYPD cars hit with glass bottles and other debris in Harlem after ...",
"partialMatchingImages": [
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-medium-plus-169.jpg"
},
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-large-169.jpg"
}
]
},
{
"url": "https://www.cnn.com/2020/07/14/us/police-violence-defund-debate-trnd/index.html",
"pageTitle": "Crime is surging in US cities. Some say defunding the police will ...",
"partialMatchingImages": [
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-medium-plus-169.jpg"
},
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-large-169.jpg"
}
]
},
{
"url": "https://www.cnn.com/2020/06/24/business/gun-sales-spike/index.html",
"pageTitle": "Gun and ammunition sales soar as defund-the-police movement ...",
"partialMatchingImages": [
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-medium-plus-169.jpg"
},
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-large-169.jpg"
}
]
},
{
"url": "https://www.cnn.com/2020/06/27/us/big-city-police-department-budgets/index.html",
"pageTitle": "NYPD, the nation's largest police force, faces financial reckoning ...",
"partialMatchingImages": [
{
"url": "https://cdn.cnn.com/cnnnext/dam/assets/200623134215-defund-the-police-protester-0602-restricted-super-tease.jpg"
},
{
"url": "https://dynaimage.cdn.cnn.com/cnn/c_fill,g_auto,w_1200,h_675,ar_16:9/https%3A%2F%2Fcdn.cnn.com%2Fcnnnext%2Fdam%2Fassets%2F200623134215-defund-the-police-protester-0602-restricted.jpg"
}
]
},
Examples of the image on other sites include:
{
"url": "https://www.msn.com/en-us/news/politics/corey-lewandowski-the-way-roger-stone-was-treated-by-this-government-should-scare-every-american/vi-BB16Cj7x",
"pageTitle": "Corey Lewandowski: The way Roger Stone was treated by this ...",
"partialMatchingImages": [
{
"url": "https://img-s-msn-com.akamaized.net/tenant/amp/entityid/BB16DRPc.img?h=90&w=120&m=6&q=60&u=t&o=t&l=f&f=jpg"
}
]
},
{
"url": "https://finance.yahoo.com/video/defund-police-cities-rising-crime-124056474.html",
"pageTitle": "In 'defund police' cities with rising crime, getting a gun is no easy task",
"partialMatchingImages": [
{
"url": "https://s.yimg.com/hd/cp-video-transcode/prod/2020-08/11/5f32949e13c1af3ca852945b/5f32949e13c1af3ca852945c_o_U_v2.jpg"
}
]
},
{
"url": "https://www.ctvnews.ca/world/as-u-s-cities-cut-police-budgets-the-nation-s-largest-force-faces-financial-reckoning-1.5002855",
"pageTitle": "As U.S. cities cut police budgets, the nation's largest force faces ...",
"partialMatchingImages": [
{
"url": "https://beta.ctvnews.ca/content/dam/ctvnews/images/2020/7/10/1_5002862.jpg?cache_timestamp=1594382574242"
},
{
"url": "https://www.ctvnews.ca/polopoly_fs/1.5002862.1593289839!/httpImage/image.jpg_gen/derivatives/landscape_620/image.jpg"
},
{
"url": "https://www.ctvnews.ca/polopoly_fs/1.5002862.1593289839!/httpImage/image.jpg_gen/derivatives/landscape_300/image.jpg"
},
{
"url": "https://www.ctvnews.ca/polopoly_fs/1.5002862.1593289839!/httpImage/image.jpg_gen/derivatives/landscape_960/image.jpg"
}
]
},
US Rep. Bass included the image on her official website:
{
"url": "https://bass.house.gov/media-center/in-the-news/rep-bass-participates-virtual-town-hall-meeting-members-congressional-black",
"pageTitle": "Rep. Bass Participates in Virtual Town Hall Meeting with Members of ...",
"partialMatchingImages": [
{
"url": "https://i.ytimg.com/vi/nAFqIqUAO6Y/hqdefault.jpg"
}
]
},
These are just a fraction of the entries in the complete API results.
What about the provenance of the image itself? Who took it and when, where and what does it depict?
Using the above list of copies of the image across the web, we can download each of these copies from the web and scan it for embedded EXIF metadata. Here is a copy of the photograph that appeared on DNyuz:
https://dnyuz.com/wp-content/uploads/2020/06/As-US-cities-cut-police-budgets-the-nation%E2%80%99s-largest-force-1024x576.jpg
Scanning this for EXIF metadata returns a wide range of embedded data including:
ApertureValue: 18135/5017 Artist: Pacific Press Copyright: 2020 Pacific Press DateTime: 2020-06-23T17:38:42-04:00 DateTimeDigitized: 2020:06:02 00:00:00 DateTimeOriginal: 2020:06:02 00:00:00 ExifImageLength: 1067 ExifImageWidth: 1600 ExifOffset: 656 ExifVersion: 48, 50, 50, 48 ExposureBiasValue: 0/1 ExposureMode: 1 ExposureProgram: 1 ExposureTime: 1/200 Flash: 16 FNumber: 7/2 FocalLength: 35/1 FocalPlaneResolutionUnit: 3 FocalPlaneXResolution: 1600/1 FocalPlaneYResolution: 1600/1 ImageDescription: NEW YORK CITY, NEW YORK, UNITED STATES - 2020/06/02: Thousands protesters continued to defy an 8 p.m. city-wide curfew across New York City, protesting the death of George Floyd, police brutality, racism and marched through the city remaining predominantly peaceful. (Photo by Michael Nigro/Pacific Press/LightRocket via Getty Images) ISOSpeedRatings: 320 Make: Canon MaxApertureValue: 1/1 MeteringMode: 5 Model: Canon EOS 5D Mark III ResolutionUnit: 2 SceneCaptureType: 0 ShutterSpeedValue: 25691/3361 Software: Adobe Photoshop Lightroom Classic 9.2.1 (Macintosh) SubSecTimeDigitized: 00 WhiteBalance: 0 XResolution: 72/1 YResolution: 72/1 colorspace: 2
The "DateTime" field reporting June 23, 2020 appears to be the date the image was prepared for web use on this particular site, while "DateTimeOriginal" tells us that the image was taken on June 2, 2020 and was syndicated by Pacific Press. We know it was taken with a Canon EOS 5D Mark III professional DSLR camera common with news photographers and have the shutter speed, metering and other details on how it was taken and that it was edited using Adobe Photoshop's Lightroom Classic prior to publication, though this could have been something as simple as conversion from RAW to a web publication file format.
Most importantly, we have a complete description from the photographer as to what the image depicts, telling us its location, date, photographer, syndication chain and description:
NEW YORK CITY, NEW YORK, UNITED STATES – 2020/06/02: Thousands protesters continued to defy an 8 p.m. city-wide curfew across New York City, protesting the death of George Floyd, police brutality, racism and marched through the city remaining predominantly peaceful. (Photo by Michael Nigro/Pacific Press/LightRocket via Getty Images)
Thus, we started with an image that appeared on television news and by using Cloud Vision API's Web Entities feature, we identified a selection of other sites it appears on across the web, along with its most common captioned topics and by scanning one of those other images we identified a range of EXIF metadata that tells us the full story of the image.