Yesterday we announced the new Visual Explorer Channel Inventory JSON file for programmatic discovery. As a simple example of how this new inventory can be used in scripts when working with the Visual Explorer, take our experiment from last week where we scanned Russian television news for Tucker Carlson appearances. To download all of the inventory files for a given channel, we have to know the start date of the channel. For example, to scan Russia 24 from start to present, we historically had to know that the Internet Archive's archive of Russia 24 begins April 26, 2022 and hardcode that into our scripts:
apt-get -y install jq start=20220426; end=20230521; while [[ ! $start > $end ]]; do echo $start; start=$(date -d "$start + 1 day" "+%Y%m%d"); done > DATES
This required manually checking the start date of each channel, which was cumbersome when working with multiple channels. Using the new channel inventory file, checking the start date of Russia 24 is as simple as:
curl -s https://api.gdeltproject.org/api/v2/tvv/tvv?mode=chaninv | jq -r '.channels[] | select(.id=="RUSSIA24") | .startDate' >20220426
We can simply wrap this inside an interpolation in our standard date script to yield a one-liner that generates the complete date set for any channel from its start:
start=$(curl -s https://api.gdeltproject.org/api/v2/tvv/tvv?mode=chaninv | jq -r '.channels[] | select(.id=="RUSSIA24") | .startDate'); end=20230521; while [[ ! $start > $end ]]; do echo $start; start=$(date -d "$start + 1 day" "+%Y%m%d"); done > DATES
Using this approach, far more complex operations are now possible using simple shell one-liners!