Experiments With Google's PaLM 2 LLM's For Embedding, Summarization, Event & Relation Extraction, NER & Q&A: Bison & Gecko

 July 17, 2023

What does it look like to summarize a television evening news broadcast using Google's PaLM 2 large language model (LLM)? Let's explore both embedding and summarization tasks using Vertex AI's PaLM API models text-bison and textembedding-gecko.

We'll start with a CBC Evening News broadcast from July 5th that includes this synopsis of the story:

jericka: good evening to our viewers in the west, and thank you for joining us this wednesday night. i'm jericka duncan in for norah. we begin with the secret service leading an investigation into who brought cocaine into the white house. the surprising discovery of a bag of suspicious white powder initially caused a panic leading to the temporary closure of the building on sunday. the illegal drug was found near a visitors entrance for those taking a tour of the west wing. today lab tests came back positive for cocaine. president biden and the first family were at camp david at the time. cbs's weijia jiang starts us off from the white house. good evening. >> reporter: good evening, jericka. the white house says it's very important to president biden to get to the bottom of who brough% that illegal drug inside. but officials cautioned this could be a challenging investigation. tonight the leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. according to senior law enforcement officials. the discovery sunday nights put the white house on temporary lockdown until the substance was deemed 'not a threat.' the bag was discovered on the ground floor in a common area near the entry to the west wing. not far from the situation room. it was close to a set of storage cubbies where visitors leave their cell p phones duringng a . the secret service is leading the investigation looking at visitor logs and camera surveillance. they are camera systems they will go back as a part of the investigation and research. not every camera is trained on every body at every minute. so there will always be a possibility that this cocould go unanswered. reporteter: white house visitorsrs undergo background checks and go through magnetometers on site. but the secret service does not have k-9 units that target illegal drugs. instead, the dog search for explosives, biohazardous materials, and weapons. >> this episode shines a light on the fact that you can bring in illegal substances to the white house, so what is preventing a visitor from bringing an anthrax or something that is not magnetic into the white house? i totally understand the question, but it is under investigation. we will let the secret service do their job. i'm not going to get ahead of if or whens or changes. reporter: senator tom cotton, a republican from arkansas sent a letter to the director of the secret service to say that congress needs to know if the white house complex is not secure. and he asked for a plan to correct any security flaws. jericka: weijia jiang at the white house, thank you.

First, let's explore how to construct an embedding of this text using text-embedding-gecko:

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/textembedding-gecko:predict -d \
$'{
  "instances": [
    { "content": "jericka: good evening to our viewers in the west, and thank you for joining us this wednesday night. i\'m jericka duncan in for norah. we begin with the secret service leading an investigation into who brought cocaine into the white house.  the surprising discovery of a bag of suspicious white powder initially caused a panic leading to the temporary closure of the building on sunday. the illegal drug was found near a visitors entrance for those taking a tour of the west wing. today lab tests came back positive for cocaine. cbs\'s weijia jiang starts us off from the white house. good evening. >> reporter: good evening, jericka. the white house says it\'s very important to president biden to get to the bottom of who brough% that illegal drug inside. but officials cautioned this could be a challenging investigation. tonight the leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. according to senior law enforcement officials. the discovery sunday nights put the white house on temporary lockdown until the substance was deemed \'not a threat.\'president biden and the first family were at camp david at the time. the bag was discovered on the ground floor in a common area near the entry to the west wing. not far from the situation room. it was close to a set of storage cubbies where visitors leave their cell p phones duringng a . the secret service is leading the investigation looking at visitor logs and camera surveillance. they are camera systems they will go back as a part of the investigation and research. not every camera is trained on every body at every minute. so there will always be a possibility that this cocould go unanswered. reporteter: white house visitorsrs undergo background checks and go through magnetometers on site. but the secret service does not have k-9 units that target illegal drugs. instead, the dog search for explosives, biohazardous materials, and weapons. >> this episode shines a light on the fact that you can bring in illegal substances to the white house, so what is preventing a visitor from bringing an anthrax or something that is not magnetic into the white house? i totally understand the question, but it is under investigation. we will let the secret service do their job. i\'m not going to get ahead of if or whens or changes. reporter: senator tom cotton, a republican from arkansas sent a letter to the director of the secret service to say that congress needs to know if the white house complex is not secure. and he asked for a plan to correct any security flaws. jericka: weijia jiang at the white house, thank you."}
  ]
}'

This yields a response with the number of processed tokens, whether the input was truncated to fit Gecko's input limit of 3,072 tokens and the final 768-dimension embedding vector:

{
  "predictions": [
    {
      "embeddings": {
        "statistics": {
          "token_count": 620,
          "truncated": false
        },
        "values": [ -0.026316206902265549, -0.00928876455873251, -0.013688338920474052, 0.021982554346323013, 0.0097444616258144379, -0.015772707760334015, 0.004138801246881485, -0.0075818272307515144, 0.016171032562851906, 0.028827827423810959, 0.05103302001953125, 0.029937701299786568, 0.038068559020757675, 0.0061067845672369, -0.018965898081660271, -0.0256775114685297, -0.091803833842277527, -0.034031633287668228, -0.0060932785272598267, 0.014593350701034069, -0.066603764891624451, -0.0095631163567304611, 0.024024041369557381, -0.0079607218503952026, -0.005145183764398098, -0.092862524092197418, -0.0012368297902867198, 0.011701472103595734, -0.00325111486017704, 0.044185217469930649, -0.029273299500346184, 0.0520230270922184, -0.014012008905410767, -0.020314397290349007, 0.046527337282896042, 0.033383198082447052, 0.0048452704213559628, 0.022460045292973518, -0.0012309200828894973, 0.0677514299750328, 0.018747150897979736, -0.0035755180288106203, 0.037990573793649673, -0.014407480135560036, -0.0011932635679841042, 0.033241834491491318, -0.053499162197113037, 0.080047480762004852, -0.054175291210412979, -0.037399996072053909, -0.0080034369602799416, -0.030808413401246071, 0.0092976866289973259, 0.0448807030916214, 0.0043153264559805393, 0.017457028850913048, -0.050193365663290024, -0.022531535476446152, -0.013204140588641167, 0.01885344460606575, 0.00574616901576519, -0.045513268560171127, 0.0066943424753844738, -0.0712982714176178, -0.027345221489667892, 0.042248494923114777, 0.0040224445983767509, 0.013439870439469814, -0.015983225777745247, -0.03760816901922226, 0.05191253125667572, 0.0350683219730854, 0.0050177471712231636, -0.016289044171571732, 0.048790067434310913, 0.0066572064533829689, -0.0019017370650544763, 0.025467976927757263, 0.057025041431188583, -0.050553541630506516, -0.0070882458239793777, -0.053437814116477966, -0.046275746077299118, -0.13421690464019775, -0.020731303840875626, 0.086676493287086487, 0.015039863064885139, 0.014777365140616894, -0.03055473230779171, 0.050682112574577332, -0.035149849951267242, 0.025798056274652481, 0.012301063165068626, 0.0062756882980465889, -0.064990691840648651, 0.013834938406944275, -0.051995061337947845, -0.00079759990330785513, 0.0066644442267715931, 0.012001706287264824, -0.0079390192404389381, -0.063710562884807587, 0.044366545975208282, -0.016902312636375427, 0.0428246334195137, 0.018548643216490746, 0.01926993764936924, 0.048384230583906174, -0.0011376551119610667, -0.07110295444726944, -0.086533837020397186, 0.00812441110610962, -0.0458795428276062, 0.023786090314388275, -0.0039624534547328949, -0.035696256905794144, 0.0023529164027422667, -0.001919868984259665, -0.010612469166517258, 0.0032412649597972631, -0.051447264850139618, 0.0058120153844356537, 0.00781091395765543, -0.0017323935171589255, -0.049268078058958054, 0.024353422224521637, 0.028342736884951591, -0.034602772444486618, -0.034199949353933334, 0.015881473198533058, 0.035613395273685455, 0.004543862771242857, 0.07088296115398407, -0.039463784545660019, 0.046701215207576752, 0.01532874908298254, -0.022417116910219193, -0.0003880842705257237, -0.0066897408105432987, 0.019256442785263062, -0.068573862314224243, -0.00067667977418750525, -0.021355381235480309, -0.0098667414858937263, 0.00028396613197401166, -0.010659218765795231, 0.030645176768302917, -0.050831831991672516, -0.022481508553028107, -0.039617203176021576, 0.015442410483956337, -0.013325230218470097, 0.016390321776270866, 0.033645197749137878, 0.08762369304895401, 0.036494847387075424, 0.058092851191759109, 0.075144916772842407, -0.020085778087377548, -0.058286841958761215, 0.021364128217101097, -0.020230172201991081, -0.038337793201208115, -0.03770042210817337, 0.048687223345041275, 0.018671782687306404, 0.033665869385004044, 0.00977856945246458, -0.013594646938145161, 0.0052350247278809547, 0.05435759574174881, -0.15027342736721039, 0.037088572978973389, -0.034370757639408112, 0.094945646822452545, 0.023252522572875023, -0.0043162712827324867, 0.0068887979723513126, 0.018356040120124817, -0.013730751350522041, -0.035038098692893982, -0.10446758568286896, -0.005304933525621891, 0.0019772211089730263, -0.066111944615840912, -0.022060751914978027, 0.017511617392301559, -0.031518656760454178, -0.047460515052080154, 0.054819650948047638, -0.018583152443170547, 0.029078252613544464, 0.014650802128016949, -0.056383751332759857, -0.0043354369699954987, 0.011964770033955574, 0.04107024148106575, -0.13784731924533844, -0.027608023956418037, 0.033411502838134766, -0.044444672763347626, -0.044441241770982742, -0.019240120425820351, 0.044814370572566986, -0.045749519020318985, -0.010699816979467869, -0.039257779717445374, 0.025641884654760361, 0.010015186853706837, 0.0185274425894022, -0.0055958293378353119, -0.021514885127544403, 0.0082529652863740921, 0.039269499480724335, -0.0078581469133496284, -0.038671988993883133, -0.015580939128994942, -0.074905931949615479, -0.044650718569755554, 0.039278872311115265, -0.0047844992950558662, -0.0072679305449128151, 0.061034247279167175, 0.0791454166173935, 0.002983615268021822, 0.026874322444200516, 0.039715621620416641, -0.0105104586109519, 0.0093197207897901535, 0.024982800707221031, -0.012086464092135429, 0.00052393384976312518, -0.044072277843952179, -0.0088321603834629059, 0.00094110809732228518, 0.044567491859197617, -0.00079117628047242761, 0.074029959738254547, -0.071012064814567566, -0.0039413967169821262, 0.038064368069171906, 0.043769285082817078, -0.044775538146495819, 0.028004949912428856, -0.010611819103360176, -0.004892522469162941, 0.020269447937607765, 0.024363018572330475, 0.052664425224065781, -0.0069154165685176849, 0.066945686936378479, -0.071817919611930847, -0.017478758469223976, 0.018190000206232071, 0.03751499205827713, -0.0086314696818590164, 0.025379018858075142, 0.056897521018981934, -0.054873146116733551, 0.0077923806384205818, 0.0058024311438202858, -0.020971076563000679, 0.0391705147922039, 0.12995195388793945, -0.004607341717928648, -0.047103539109230042, -0.078285306692123413, -0.014779404737055302, 0.0449865348637104, -0.0495770089328289, -0.027704590931534767, 0.0059334421530365944, 0.060318492352962494, 0.05264587327837944, -0.018132075667381287, 0.0069052176550030708, -0.063973180949687958, 0.019628074020147324, 0.018511399626731873, 0.042460836470127106, 0.0277895238250494, -0.032415885478258133, -3.78041913791094e-05, 0.051172856241464615, -0.018370026722550392, 0.036760680377483368, -0.0230995062738657, 0.034480251371860504, -0.042489845305681229, -0.014464761130511761, 0.033649202436208725, -0.039281737059354782, -0.033192023634910583, 0.073097392916679382, -0.00023193511879071593, -0.003101282985880971, 0.035935819149017334, -0.019467530772089958, -0.031026003882288933, 0.045379795134067535, 0.0041903224773705006, -0.044047903269529343, -0.020402844995260239, -0.0504877008497715, -0.061819050461053848, -0.027720579877495766, -0.00500531168654561, -0.0069280141033232212, 0.05069950595498085, -0.021937036886811256, -1.7885344277601689e-05, -0.011793952435255051, 0.010473549365997314, 0.023273181170225143, -0.0694643035531044, 0.023775665089488029, -0.0084978053346276283, -0.0034868491347879171, -0.038604769855737686, 0.03774532675743103, -0.028512574732303619, -0.004587060771882534, 0.0929008424282074, 0.023256026208400726, -0.0063658906146883965, 0.0095173399895429611, -0.0080719469115138054, -0.0278488602489233, 0.057520207017660141, -0.029592080041766167, 0.0044944840483367443, -0.0040046311914920807, -0.051493551582098007, 0.017654145136475563, 0.0032976111397147179, 0.0026079099625349045, 0.013650619424879551, -0.0024660662747919559, -0.0023697640281170607, -0.06456109881401062, 0.024658707901835442, -0.021408997476100922, -0.0068025114014744759, -0.0066638542339205742, -0.036482363939285278, 0.019174516201019287, -0.0438554473221302, -0.021870041266083717, -0.04547392949461937, -0.038521196693181992, 0.00596650829538703, 0.030038686469197273, -0.035086750984191895, 0.04976818710565567, 0.0400061272084713, 0.0064890272915363312, -0.0063623036257922649, 0.01012021116912365, 0.035347927361726761, 0.030191557481884956, -0.011656205169856548, -0.013294038362801075, -0.015342887490987778, 0.034351147711277008, -0.0021816431544721127, 0.020229756832122803, -0.044474758207798004, 0.044439211487770081, -0.010409115813672543, -0.025947626680135727, -0.085024826228618622, -0.0020656429696828127, 0.00052231590962037444, 0.016292104497551918, -0.057914499193429947, -0.050391267985105515, -0.027483150362968445, -0.0097544686868786812, 0.0033668591640889645, 0.017049608752131462, -0.054627485573291779, 0.042636740952730179, 0.0075308582745492458, 0.020278513431549072, 0.0011057972442358732, -0.013009698130190372, 0.016123700886964798, 0.065272562205791473, -0.028781287372112274, -0.0041641448624432087, 0.044746361672878265, 0.020369136705994606, 0.037195291370153427, -0.0261179618537426, -0.015578169375658035, 0.043055444955825806, -0.025448441505432129, -0.021031482145190239, -0.040092967450618744, -0.025911903008818626, 0.048199713230133057, 0.014582139439880848, 0.0058958618901669979, -0.028741909191012383, 0.030405901372432709, 0.039635665714740753, -0.025071172043681145, 0.030906492844223976, 0.047843046486377716, -0.019424373283982277, 0.010585284791886806, -0.05459122359752655, -0.0087877418845891953, -0.026016067713499069, -0.0014721733750775456, -0.085492178797721863, 0.031729720532894135, -0.078703917562961578, -0.0027248507831245661, -0.0350271575152874, 0.051737476140260696, -0.050961028784513474, -0.10167919099330902, -0.0070392638444900513, 0.025627272203564644, 0.0074106105603277683, -0.065895773470401764, 0.011963761411607265, -0.042191382497549057, 0.02000121958553791, 0.013293313793838024, -0.022133151069283485, 0.026620637625455856, -0.0232720784842968, 0.010924641974270344, -0.040254242718219757, 0.071481935679912567, 0.066086731851100922, -0.0037087397649884224, -0.027018001303076744, -0.0081843966618180275, 0.015539648942649364, -0.0734042078256607, 0.020948104560375214, 0.0088705914095044136, -0.04675234854221344, 0.0064436499960720539, -0.0016939505003392696, -0.042131610214710236, 0.0021320984233170748, -0.048430327326059341, -0.019703345373272896, -0.023374607786536217, 0.0537690594792366, 0.033143993467092514, 0.048992723226547241, 0.021371668204665184, -0.014940503984689713, 0.056736309081315994, -0.016199393197894096, -0.029217114672064781, 0.014478300698101521, -0.02062884159386158, 0.018699198961257935, 0.012110746465623379, -0.01791837252676487, 0.018031449988484383, 0.046827990561723709, -0.033992249518632889, 0.0211626086384058, -0.01361218374222517, 0.082904554903507233, 0.0026834416203200817, -0.0081295585259795189, 0.06623174250125885, -0.0098228035494685173, 0.011983917094767094, 0.039773993194103241, 0.02003207616508007, -0.025936976075172424, -0.0665331557393074, 0.011741439811885357, -0.004901728592813015, -0.042070217430591583, 0.020219540223479271, 0.038307908922433853, -0.0044579682871699333, -0.020297309383749962, -0.0037286004517227411, 0.025151962414383888, 0.006233209278434515, -0.017356084659695625, 0.015886928886175156, -0.027023743838071823, 0.0353916771709919, -0.032550949603319168, 0.0054831248708069324, 0.008485875092446804, 0.02822386845946312, 0.00055772654013708234, 0.035904359072446823, 0.038642261177301407, 0.033935274928808212, -0.04867519810795784, -0.040141310542821884, -0.031909134238958359, -0.010828463360667229, -0.051429837942123413, -0.03953658789396286, -0.005867494735866785, -0.019369930028915405, 0.061740703880786896, -0.013841093517839909, -0.088703647255897522, -0.034924622625112534, -0.091603584587574, 0.016831735149025917, 0.034741189330816269, 0.014819703064858913, -0.014225361868739128, -0.0068876338191330433, -0.0198333989828825, 0.015365913510322571, 0.0083503536880016327, 0.056442447006702423, 0.056455954909324646, 0.038439355790615082, -0.029722638428211212, -0.063413433730602264, 0.011833596043288708, 0.016077077016234398, 0.03391527384519577, -0.031383827328681946, 0.040686309337615967, -0.0045986874029040337, -0.042314108461141586, 0.033530868589878082, 0.00094054639339447021, 0.032176192849874496, 0.093753546476364136, -0.014439630322158337, -0.042523600161075592, 0.0047705862671136856, 0.0030266710091382265, 0.017641626298427582, 0.013294588774442673, 0.030782707035541534, 0.044322464615106583, 0.031199635937809944, 0.052474472671747208, 0.036384563893079758, -0.047080215066671371, -0.068959988653659821, 0.060448259115219116, 0.0362711101770401, -0.0079839704558253288, 7.6323623943608254e-05, -0.043060086667537689, -0.0284821055829525, -0.012486309744417667, 0.0022430496755987406, 0.017791595309972763, -0.017464542761445045, -0.0040545891970396042, 0.055264808237552643, -0.033159296959638596, 0.048600237816572189, 0.0071537047624588013, 0.016378089785575867, 0.042574159801006317, -0.027758155018091202, -0.056894510984420776, -0.023540997877717018, 0.0074647869914770126, 0.018669456243515015, 0.001149291405454278, -0.019083635881543159, 0.076512843370437622, 0.015710553154349327, 0.0582815445959568, -0.030300052836537361, 0.034225389361381531, 0.027270868420600891, 0.023879049345850945, -0.021174229681491852, 0.026707133278250694, -0.011528563685715199, -0.050993524491786957, 0.01228194497525692, -0.049799654632806778, 0.014272252097725868, -0.063386999070644379, 0.006799464114010334, -0.0068940017372369766, -0.036047190427780151, 0.010232350789010525, -0.030707294121384621, 0.015823215246200562, 0.030295798555016518, 0.039484459906816483, 0.000905710447113961, 0.0021220427006483078, 0.078907705843448639, -0.043667729943990707, 0.0090091153979301453, 0.0010319879511371255, 0.031346507370471954, 0.012438925914466381, 0.035531342029571533, 0.0032337333541363478, 0.010663202963769436, 0.064628124237060547, 0.012772770598530769, -0.015615382231771946, -0.016807988286018372, 0.022895120084285736, -0.019653532654047012, 0.019522856920957565, 0.03982137143611908, -0.0033588006626814604, 0.031334321945905685, 0.0089674843475222588, 0.0017239715671166778, -0.06297273188829422, 0.034553438425064087, -0.010600830428302288, -0.01747843436896801, 0.0098965084180235863, -0.013531683944165707, -0.025220366194844246, -0.00737897539511323, -0.024626210331916809, -0.016799338161945343, -0.0053686108440160751, 0.017709797248244286, -0.020623026415705681, -0.038971975445747375, 0.012872497551143169, 0.02904120460152626, 0.014000239782035351, 0.0075879525393247604, 0.014748400077223778, -0.0084365503862500191, 0.0023773263674229383, 0.016774099320173264, -0.014384962618350983, 0.022928168997168541, -0.016738042235374451, -0.032256394624710083, 0.025818657130002975, -0.017969658598303795, -0.013997227884829044, 0.0029355555307120085, -0.0315256230533123, -0.026518212631344795, -0.0075670378282666206, -0.026912249624729156, -0.0027541194576770067, 0.0093572121113538742, 0.053237061947584152, 0.027376702055335045, 0.007124289870262146, 0.023848235607147217, 0.0362401157617569, -0.045349262654781342, 0.046671662479639053, -0.023667324334383011, 0.034829627722501755, 0.014288730919361115, 0.0010316941188648343, -0.05718626081943512, 0.00054579123388975859, -0.016503829509019852, -0.067113369703292847, 0.037305478006601334, -0.02717096358537674, 0.0060737114399671555, -0.023821115493774414, 0.027460051700472832, 0.028433756902813911, -0.018951106816530228, 0.025625767186284065, 0.017243430018424988, -0.0457310676574707, 0.018011104315519333, -0.071702934801578522, -0.016308693215250969, 0.0083586070686578751, -0.00789407454431057, -0.00409452011808753, 0.017870822921395302, 0.031845990568399429, 0.0016450724797323346, 0.011956182308495045, -0.029796648770570755, -0.079430453479290009, -0.01187825296074152, 0.037597853690385818, -0.013134190812706947, 0.021589752286672592, 0.042448859661817551, -0.0064895674586296082, 0.043483644723892212, 0.031352333724498749, 0.035559024661779404, -0.020559746772050858, -0.01215350441634655, 0.034788474440574646, 0.0032888427376747131, -0.011147325858473778, 0.0024599034804850817, -0.017628306522965431, 0.032043773680925369, 0.024470558390021324, 0.0004806019423995167, 0.021746734157204628, -0.046977408230304718, -0.016708966344594955, -0.0033737628255039454, 0.0019520255737006664, -0.014722972176969051, 0.031111471354961395, 0.021764641627669334, -0.0058486745692789555, -0.026641644537448883, 0.018864938989281654, 0.0082651916891336441, -0.0022936826571822166, -0.005159065593034029, 0.040802452713251114, -0.022272210568189621, -0.01438487134873867, 0.018310142681002617, 0.057113774120807648, 0.016066212207078934, -0.068376481533050537, 0.04297725111246109, -0.0067473971284925938, 0.067931763827800751, -0.049211379140615463, -0.00513458764180541, 0.051039986312389374, 0.011585981585085392, -0.043625619262456894, 0.024500312283635139, -0.012329598888754845, 0.0023764201905578375, -0.00697014294564724, -0.026742862537503242, -0.026267023757100105, -0.093272946774959564, 0.037495456635951996, -0.048684924840927124, 0.04810091108083725, 0.013280177488923073, 0.081437990069389343, 0.032810073345899582, 0.042868722230196, 0.003743504174053669, 0.10386362671852112, 0.017743712291121483, 0.038164127618074417, 0.013563415966928005, -0.0060403961688280106, -0.0039521381258964539, 0.041518833488225937, -0.022289315238595009, -0.018127968534827232 ]      }
    }
  ]
}

Now let's try Bison itself, using it to summarize the passage (parameter documentation):

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/[YOURPROJECTID]/locations/us-central1/publishers/google/models/text-bison:predict -d \
$'{
  "instances": [
    { "prompt": "Summarize this television news transcript: \\"jericka: good evening to our viewers in the west, and thank you for joining us this wednesday night. i\'m jericka duncan in for norah. we begin with the secret service leading an investigation into who brought cocaine into the white house.  the surprising discovery of a bag of suspicious white powder initially caused a panic leading to the temporary closure of the building on sunday. the illegal drug was found near a visitors entrance for those taking a tour of the west wing. today lab tests came back positive for cocaine. cbs\'s weijia jiang starts us off from the white house. good evening. >> reporter: good evening, jericka. the white house says it\'s very important to president biden to get to the bottom of who brough% that illegal drug inside. but officials cautioned this could be a challenging investigation. tonight the leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. according to senior law enforcement officials. the discovery sunday nights put the white house on temporary lockdown until the substance was deemed \'not a threat.\'president biden and the first family were at camp david at the time. the bag was discovered on the ground floor in a common area near the entry to the west wing. not far from the situation room. it was close to a set of storage cubbies where visitors leave their cell p phones duringng a . the secret service is leading the investigation looking at visitor logs and camera surveillance. they are camera systems they will go back as a part of the investigation and research. not every camera is trained on every body at every minute. so there will always be a possibility that this cocould go unanswered. reporteter: white house visitorsrs undergo background checks and go through magnetometers on site. but the secret service does not have k-9 units that target illegal drugs. instead, the dog search for explosives, biohazardous materials, and weapons. >> this episode shines a light on the fact that you can bring in illegal substances to the white house, so what is preventing a visitor from bringing an anthrax or something that is not magnetic into the white house? i totally understand the question, but it is under investigation. we will let the secret service do their job. i\'m not going to get ahead of if or whens or changes. reporter: senator tom cotton, a republican from arkansas sent a letter to the director of the secret service to say that congress needs to know if the white house complex is not secure. and he asked for a plan to correct any security flaws. jericka: weijia jiang at the white house, thank you.\\"  "}
  ],
  "parameters": {
    "temperature": 0.2,
    "maxOutputTokens": 256,
    "topK": 40,
    "topP": 0.95
  }
}'

The complete output looks like the following:

{
  "predictions": [
    {
      "citationMetadata": {
        "citations": []
      },
      "safetyAttributes": {
        "blocked": false,
        "scores": [
          0.2,
          0.1,
          1,
          0.2,
          1,
          0.1,
          0.5,
          0.9,
          0.1,
          0.9,
          0.2,
          0.3,
          0.6
        ],
        "categories": [
          "Death, Harm & Tragedy",
          "Derogatory",
          "Firearms & Weapons",
          "Health",
          "Illicit Drugs",
          "Insult",
          "Legal",
          "Politics",
          "Profanity",
          "Public Safety",
          "Toxic",
          "Violent",
          "War & Conflict"
        ]
      },
      "content": "The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed \"not a threat.\" President Biden and the first family were at Camp David at the time. The bag was discovered on the ground floor in a common area near the entry to the west wing. It was close to a set of storage cubbies where visitors leave their cell phones during a tour. The Secret Service is leading the investigation looking at visitor logs and camera surveillance. However, not every camera is trained on every body at every minute, so there will always be a possibility that this could go unanswered. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure. He asked for a plan to correct any security flaws."
    }
  ]
}

Here we can see that the response includes both the actual output of the LLM and a set of "sensitive topics" categories that help the application understand the overall sensitivity of the passage.

Running twice, here are the example outputs:

  • The Secret Service is investigating how cocaine was brought into the White House. The discovery of the bag of suspicious white powder on Sunday night led to a temporary closure of the building. The drug was found near a visitors entrance for those taking a tour of the west wing. Today, lab tests came back positive for cocaine. The leading theory is that someone on a white house tour may have left the bag behind. The Secret Service is looking at visitor logs and camera surveillance footage, but they caution that this could be a challenging investigation. Senator Tom Cotton, a Republican from Arkansas, has sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure. He asked for a plan to correct any security flaws.
  • The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed "not a threat." President Biden and the first family were at Camp David at the time. The bag was discovered on the ground floor in a common area near the entry to the west wing. It was close to a set of storage cubbies where visitors leave their cell phones during a tour. The Secret Service is leading the investigation looking at visitor logs and camera surveillance. However, not every camera is trained on every body at every minute, so there will always be a possibility that this could go unanswered. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure. He asked for a plan to correct any security flaws.

Temperature is a particularly interesting parameter, as it allows us to adjust the "creativity" of the output. Let's try a few different values:

  • Temperature 0.0: The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed 'not a threat.' The leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. The Secret Service is looking at visitor logs and camera surveillance, but not every camera is trained on every body at every minute. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure and asked for a plan to correct any security flaws.
  • Temperature 0.5: The Secret Service is investigating how cocaine was brought into the White House. The discovery of the bag of white powder on Sunday night caused a temporary lockdown of the building. The leading theory is that someone on a White House tour may have left the cocaine behind. The Secret Service is looking at visitor logs and camera surveillance footage, but they may not be able to identify who brought the cocaine into the White House. Senator Tom Cotton has sent a letter to the director of the Secret Service asking for a plan to correct any security flaws that may have allowed the cocaine to be brought into the White House.
  • Temperature 0.99: The Secret Service is investigating how cocaine was brought into the White House. The bag of white powder was found near a visitors entrance on Sunday and later tested positive for cocaine. The leading theory is that someone on a White House tour left the bag behind. The Secret Service is looking at visitor logs and camera surveillance footage, but not every camera is trained on every person at every minute. Senator Tom Cotton has sent a letter to the director of the Secret Service asking for a plan to correct any security flaws.

Here we see there wasn't much of a difference. What if we double our max output tokens to give more room for the model's expressiveness, set temperature to 0.0, topK to 1 (greedy) and topP to 0.0? This should offer as close as possible to a clinical deterministic output. Let's run it three times:

  • The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed 'not a threat.' The leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. The Secret Service is looking at visitor logs and camera surveillance, but not every camera is trained on every body at every minute. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure and asked for a plan to correct any security flaws.
  • The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed 'not a threat.' The leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. The Secret Service is looking at visitor logs and camera surveillance, but not every camera is trained on every body at every minute. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure and asked for a plan to correct any security flaws.
  • The Secret Service is investigating how cocaine was brought into the White House. The bag of suspicious white powder was found near a visitors entrance for those taking a tour of the west wing. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed 'not a threat.' The leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing. The Secret Service is looking at visitor logs and camera surveillance, but not every camera is trained on every body at every minute. Senator Tom Cotton, a Republican from Arkansas, sent a letter to the director of the Secret Service to say that Congress needs to know if the White House complex is not secure and asked for a plan to correct any security flaws.

Indeed, all three runs yield identical results. At the same time, this causes the model to nearly plagiarize the original text, excerpting it as-is in large chunks. This is actually the expected behavior, since we are in effect asking the model to return the most probable entailed token.

What if we attempt to maximize its creativity? We'll set temperature to 0.99, topK to 40 (its max) and topP to 0.99 and run three times:

  • The Secret Service is investigating how cocaine was brought into the White House. The discovery of the bag of white powder on Sunday night led to a temporary lockdown of the building. The bag was found near a visitors entrance for those taking a tour of the west wing. The leading theory is that someone on a white house tour may have left the bag behind. The Secret Service is looking at visitor logs and camera surveillance footage, but not every camera is trained on every person at every minute. Senator Tom Cotton, a Republican from Arkansas, has sent a letter to the director of the Secret Service asking for a plan to correct any security flaws.
  • The Secret Service is investigating how cocaine was brought into the White House. The discovery of the bag of white powder on Sunday night caused a temporary lockdown of the building. The leading theory is that someone on a White House tour may have left the drug behind. The Secret Service is looking at visitor logs and camera surveillance footage, but it may be difficult to identify the person who brought the cocaine into the White House. Senator Tom Cotton has sent a letter to the director of the Secret Service asking for a plan to correct any security flaws that may have allowed the drug to be brought into the building.
  • The Secret Service is investigating how cocaine was brought into the White House. The bag of cocaine was found on Sunday night and the White House was temporarily put on lockdown. The leading theory is that someone on a White House tour left the bag behind. The Secret Service is looking at visitor logs and camera surveillance footage to try to identify who brought the cocaine into the White House. Senator Tom Cotton has sent a letter to the Director of the Secret Service asking for a plan to correct any security flaws.

This time the model generates different results each time, though there are still substantial similarities between them. Interestingly, every single summary generated to date, regardless of all parameter settings, has begun with "The Secret Service is investigating how cocaine was brought into the White House."

What if we use a slightly different prompt and ask it to "write an abstract for this television news transcript" using the same parameters:

  • The Secret Service is investigating how cocaine was brought into the White House. The discovery of the bag of white powder on Sunday night caused a panic and led to the temporary closure of the building. The drug was found near a visitors entrance for those taking a tour of the West Wing. Today, lab tests came back positive for cocaine. The leading theory is that someone on a White House tour may have left the bag. The Secret Service is looking at visitor logs and camera surveillance footage to try to identify the person responsible. However, not every camera is trained on every person at every minute, so there is a possibility that the investigation may not be able to identify the culprit. Senator Tom Cotton, a Republican from Arkansas, has sent a letter to the director of the Secret Service asking for a plan to correct any security flaws that may have allowed the cocaine to be brought into the White House.
  • The Secret Service is investigating how cocaine was brought into the White House. The suspicious white powder was found near a visitors entrance on Sunday and led to a temporary closure of the building. The leading theory is that someone on a White House tour may have left the bag. The Secret Service is looking at visitor logs and camera surveillance footage, but it is possible that the incident may never be fully explained.

And headlines ("Write a title for this television news transcript" using the same parameters):

  • A good title for this television news transcript would be \"Cocaine Found in White House\". The text discusses the discovery of a bag of suspicious white powder in the White House, which was later confirmed to be cocaine. The text also discusses the investigation into who brought the cocaine into the White House and the potential security implications of the incident.
  • "The title of this news transcript should be: Cocaine Found Inside White House.\n\nThe text describes the discovery of a bag of cocaine in the White House. The discovery of cocaine in the White House is a major security breach, and the Secret Service is leading the investigation into who brought the cocaine into the White House.

And hashtags ("What are potential hashtags for this television news transcript" using the same parameters) (note that in the second run it did not output actual hashtags, while the hashtags in the third response are not as descriptive as the first response and lack the same explaination):

  • #whitehouse #cocaine #secretservice #whitehousetour #securityflaw #senatortomcotton #congress\n\nThe story is about cocaine found in the White House, so we can use #whitehouse and #cocaine as hashtags.\nThe story also mentions that the Secret Service is leading the investigation, so we can use #secretservice as a hashtag.\nThe story also mentions that the bag of cocaine was found near the entrance to the West Wing, so we can use #whitehousetour as a hashtag.\nThe story also mentions that the Secret Service does not have k-9 units that target illegal drugs, so we can use #securityflaw as a hashtag.\nThe story also mentions that Senator Tom Cotton sent a letter to the Director of the Secret Service to say that Congress needs to know if the White House complex is not secure, so we can use #senatortomcotton and #congress as hashtags.
  • Potential hashtags for this television news transcript:\n\n- Cocaine in the White House\n\n- Secret Service Investigation\n\n- White House Security\n\n- White House Tour\n\n- White House Lockdown\n\n- Anthrax\n\n- Congressman Tom Cotton
  • The text is from a news report about the discovery of a bag of cocaine in the White House. Here are some potential hashtags for this text:\n\n#whitehouse #cocaine #security #secretservice\n\n#drugs #drugsinthewhitehouse #securitybreach\n\n#investigation #mystery #whitehousetour\n\n#secretserviceinvestigation #news #politics #usgovernment

Let's revert to our original parameters (temperature: 0.2, topK: 40, topP: 0.95) and test sentiment ("What is the sentiment of this television news transcript"):

  • The sentiment of this television news transcript is negative. The news is about the discovery of a bag of cocaine in the White House, which led to the temporary closure of the building. The news is also about the Secret Service's investigation into who brought the cocaine into the White House. The news is negative because it highlights the fact that someone was able to bring an illegal substance into the White House, which raises questions about the security of the building.
  • The sentiment of this television news transcript is negative. The news is about the discovery of a bag of cocaine in the White House, which led to the temporary closure of the building. The news is also about the Secret Service's investigation into who brought the cocaine into the White House. The news is negative because it is about a security breach at the White House.
  • (Testing with temperature of 0.99): The sentiment of this television news transcript is negative. The news is about a bag of cocaine being found in the White House. This is a serious security breach, and it is being investigated by the Secret Service. The discovery of the cocaine has raised questions about the security of the White House, and whether it is possible for visitors to bring in illegal substances. The news is also negative because it is an example of the current political climate in the United States, which is often divided and contentious.

And constrain the tone response to a categorical statement ("Classify as either negative or positive the tone of this television news transcript"):

  • negative
  • negative

Following this JSON sentiment example, let's try a machine-readable structured response ("For this television news transcript return a JSON object that has fields sentiment and explaination. Acceptable values for sentiment are Very Positive, Positive, Neutral, Negative or Very Negative. The explanation field contains text that explains the sentiment."):

  • {\n \"sentiment\": \"Negative\",\n \"explanation\": \"The news is about cocaine being found in the White House. This is a negative event.\"\n}
  • {\n \"sentiment\": \"Negative\",\n \"explanation\": \"The news is about cocaine being found in the White House. This is a negative event.\"\n}
  • (Testing with temperature of 0.99): {\n \"sentiment\": \"Negative\",\n \"explaination\": \"The discovery of cocaine in the White House is a serious security breach and raises questions about the Secret Service's ability to protect the president and his family.\"\n}

What about a machine-readable categorical example ("Return a JSON object with Category and Explanation. For Category classify the text as one of the following categories: Political, Crime, War, Fashion, Tourism. The Explanation field contains text that explains the category selection."):

  • {\n \"Category\": \"Crime\",\n \"Explanation\": \"The article is about the discovery of cocaine in the White House. It is a crime to bring illegal drugs into the White House.\"\n}
  • {\n \"Category\": \"Crime\",\n \"Explanation\": \"The article is about the discovery of a bag of cocaine in the White House. It discusses the investigation into who brought the cocaine into the White House and the security implications of the discovery.\"\n}
  • (Testing with temperature of 0.99): {\n \"Category\": \"War\",\n \"Explanation\": \"The text discusses an investigation by the Secret Service into who brought cocaine into the White House. Cocaine is a Schedule II drug, which is defined by the United States Drug Enforcement Administration as a substance with a high potential for abuse, with a moderate to low risk of dependence. The text also discusses the possibility of bringing other illegal substances into the White House, such as anthrax, which is a biological weapon. The text discusses the security of the White House complex, and the need to correct any security flaws.
  • (Testing with temperature of 0.99): {\n \"Category\": \"Political\",\n \"Explanation\": \"The article describes the investigation into the person who brought cocaine into the White House.\"\n}
  • (Testing with temperature of 0.99): {\n \"Category\": \"Political\",\n \"Explanation\": \"The article is about an investigation into who brought cocaine into the White House. The article discusses the background checks and security measures at the White House and the possibility that the cocaine was brought in by a visitor.\"\n}
  • (Testing with temperature of 0.99): {\n \"Category\": \"Crime\",\n \"Explanation\": \"The news article describes a crime in which cocaine was brought into the White House.\"\n}

Note how the more creative temperature leads to a greater variety of responses, some of which are less intuitive, such as War or Political. This shows the dangers of higher temperature settings for precision NLU tasks like categorization.

What about another categorical example, this time with three custom categories where we provide a specific definition that partially overlaps and rely on the LLM to decide the topical boundaries ("Return a JSON object with Category and Explanation. For Category classify the text as one of the following categories: Topic1 or Topic2 or Topic3. The Topic1 category is defined as relating to US domestic politics. The Topic2 category is defined as relating to the White House or the current presidential administration. The Topic3 category relates to any criminal activity in the US. The Explanation field contains text that explains the category selection."):

  • {\n \"Category\": \"Topic2\",\n \"Explanation\": \"The article is about the Secret Service investigation into who brought cocaine into the White House. This is a topic related to the White House and the current presidential administration.\"\n}
  • {\n \"Category\": \"Topic2\",\n \"Explanation\": \"The article is about the Secret Service investigation into who brought cocaine into the White House. This is a topic that is related to the White House and the current presidential administration.\"\n}
  • {\n \"Category\": \"Topic2\",\n \"Explanation\": \"The text is about the Secret Service leading an investigation into who brought cocaine into the White House. The text also mentions that the White House says it's very important to President Biden to get to the bottom of who brought that illegal drug inside. This is a topic that is related to the White House and the current presidential administration, so it is classified as Topic2.\"\n}
  • (Testing with temperature of 0.99): {\n \"Category\": \"Topic2\",\n \"Explanation\": \"This is a news article about the Secret Service investigating cocaine being brought into the White House. This article is related to the White House and is not about domestic politics.\"\n}
  • (Testing with temperature of 0.99): {\n \"Category\": \"Topic3\",\n \"Explanation\": \"Article is about the discovery of cocaine in the White House and the investigation that ensued. It is relevant to criminal activity in the US
  • (Testing with temperature of 0.99): {\"Category\": \"Topic2\", \"Explanation\": \"This story is about cocaine being found at the White House, which is directly related to the White House and current presidential administration.\"}
  • (Testing with temperature of 0.99): {\n \"Category\": \"Topic3\",\n \"Explanation\": \"The story is about the discovery of cocaine in the White House and the Secret Service's investigation into the incident. The story also discusses the security implications of the incident and the concerns of lawmakers about the security of the White House complex.\"}

Here we can see that once again the default temperature setting leads to consistent and expected results. With a higher temperature setting it alternates between two equally-applicable labels.

How about a NER task ("What are the locations mentioned in the text?"):

  • 1. The White House\n2. The West Wing\n3. The Situation Room\n4. Camp David
  • 1. The West Wing\n2. The Situation Room\n3. Camp David
  • 1. The White House\n2. West Wing\n3. Camp David
  • (Testing with temperature of 0.99): The White House is located at 1600 Pennsylvania Avenue NW in Washington, D.C. The West Wing is located on the west side of the White House. The Situation Room is located in the West Wing. The White House is the official residence of the President of the United States. It is also the home of the President's family. The White House is a popular tourist attraction and is open to the public for tours.
  • (Testing with temperature of 0.99): The locations mentioned in the text are West Wing, White House, Camp David, and the ground floor.
  • (Testing with temperature of 0.99): 1) \"White house\" is the main location mentioned in the text. The text is about an incident in which a bag of cocaine was found in the White House.\n2) \"West Wing\" is also mentioned in the text. The bag of cocaine was found near the entry to the West Wing.\n3) \"Camp David\" is also mentioned in the text. The President and the First Family were at Camp David when the cocaine was found.

Here we can see variability even at the default temperature setting, with White House and Situation room alternately appearing or not. This is a fairly common limitation of current LLMs: despite their immense prowess at interpreting natural language and generatively summarizing it, when constrained to more simplistic tasks like NER, they can struggle. Interestingly, the linguistic tasks required by NER (the underlying semantic modeling) is an intrinsic part of the LLM model's weights, but current LLMs struggle to translate this knowledge into reliable and consistent entity extraction. Moreover, even when they successfully extract entities, they are unable to resolve them due to a lack of realtime updating knowledgestore (which can't be replicated via two tower ANN).

What about structured entity extraction ("Extract all of the entities, their types and descriptions in the following text into a JSON object.")?

  • Entities:\n\n- White House: LOCATION\n- Secret Service: ORGANIZATION\n- Cocaine: SUBSTANCE\n- Sunday: DATE\n- West Wing: BUILDING\n- White House: BUILDING\n- Cell Phones: GADGET\n- Camera Surveillance: GADGET\n- K-9 Units: GADGET\n- Anthrax: SUBSTANCE\n- Senator Tom Cotton: PERSON\n- Arkansas: STATE\n- Director of the Secret Service: PERSON\n- Congress: ORGANIZATION\n- White House Complex: BUILDING
  • Entities:\n\n- Cocaine: drug\n- White House: building\n- Secret Service: organization\n- Sunday: date\n- West Wing: building\n- White House: building\n- Camera: device\n- White House: building\n- K-9: animal\n- Anthrax: biological agent\n- Senator Tom Cotton: person\n- Arkansas: location\n- Director of the Secret Service: person\n- White House Complex: building
  • Entities:\n\n- White House: LOCATION\n- Secret Service: ORGANIZATION\n- Cocaine: DRUG\n- Sunday: DATE\n- West Wing: BUILDING\n- White House Tour: EVENT\n- Ground Floor: ROOM\n- Storage Cubby: CONTAINER\n- Cell Phone: ELECTROMAGNETIC DEVICE\n- Camera: MACHINE\n- K-9 Unit: ANIMAL\n- Anthrax: DISEASE\n- Senator Tom Cotton: PERSON\n- Director of the Secret Service: PERSON\n\nDescriptions:\n\n- White House: The official residence and workplace of the President of the United States.\n- Secret Service: A United States federal law enforcement agency under the Department of Homeland Security responsible for protecting the President of the United States, the Vice President, their immediate families, and other high-ranking officials.\n- Cocaine: A powerful stimulant drug that is derived from the leaves of the coca plant.\n- Sunday: The seventh day of the week.\n- West Wing: The executive wing of the White House.\n- White House Tour: A tour of the White House.\n- Ground Floor: The lowest floor of a building.\n- Storage Cubby: A small, enclosed space used for storage.\n- Cell Phone: A mobile phone.\n- Camera: A device that records images or video.\n- K-9 Unit: A unit of dogs trained to perform specific tasks, such as detecting drugs or explosives.\n- Anthrax: A serious infectious disease caused by the bacterium Bacillus anthracis.\n- Senator Tom Cotton: An American politician who has served as the junior United States Senator from Arkansas since 2015.\n- Director of the Secret Service: The head of the United States Secret Service.
  • Entities:\n\n- Secret Service: Organization\n- White House: Building\n- Cocaine: Drug\n- Sunday: Day\n- West Wing: Building\n- Lab Tests: Test\n- CBS: Organization\n- Weijia Jiang: Person\n- Jerrika Duncan: Person\n- President Biden: Person\n- First Family: Family\n- Camp David: Place\n- Ground Floor: Floor\n- Common Area: Area\n- Entry: Entrance\n- Situation Room: Room\n- Storage Cubbies: Cubicle\n- Cell Phones: Phone\n- Background Checks: Check\n- Magnetometers: Device\n- K-9 Units: Unit\n- Explosives: Substance\n- Biohazardous Materials: Material\n- Weapons: Weapon\n- Senator Tom Cotton: Person\n- Arkansas: State\n- Director of the Secret Service: Person\n- Congress: Organization\n- Security Flaws: Flaw
  • (Testing with temperature of 0.99): {\n \"entities\": [\n {\n \"name\": \"White House\",\n \"type\": \"Location\",\n \"description\": \"A building in Washington D.C. that houses the President of the United States\"\n },\n {\n \"name\": \"Secret Service\",\n \"type\": \"Organization\",\n \"description\": \"A federal law enforcement agency that protects the President of the United States\"\n },\n {\n \"name\": \"Cocaine\",\n \"type\": \"Drug\",\n \"description\": \"A powerful stimulant drug that can be addictive\"\n },\n {\n \"name\": \"West Wing\",\n \"type\": \"Building\",\n \"description\": \"A part of the White House that houses the offices of the President of the United States\"\n }\n ]\n}
  • (Testing with temperature of 0.99): Entities:\n\n- Date: Sunday\n- Drug: Cocaine\n- Findings: Bag of suspicious white powder\n- Location: White House\n- Visitor: Someone on a White House tour\n- White House: A building\n\nTypes:\n\n- Date: Time\n- Drug: Illegal drug\n- Findings: Object\n- Location: Building\n- Visitor: Person\n- White House: Building
  • (Testing with temperature of 0.99) (Note the tick marks are verbatim from the response): "`\n{\n \"White House\": {\n \"type\": \"location\",\n \"description\": \"The official residence and workplace of the President of the United States\"\n },\n \"Secret Service\": {\n \"type\": \"organization\",\n \"description\": \"The United States Secret Service is a federal law enforcement agency under the Department of Homeland Security responsible for the protection of the President of the United States, the Vice President, their immediate families, and other high-ranking officials\"\n },\n \"Cocaine\": {\n \"type\": \"drug\",\n \"description\": \"Cocaine is a powerful stimulant drug derived from the leaves of the coca plant\"\n },\n \"Sunday\": {\n \"type\": \"day\",\n \"description\": \"The day of the week between Saturday and Monday\"\n },\n \"Magnetometer\": {\n \"type\": \"device\",\n \"description\": \"A metal detector that uses a magnet to detect the presence of metal\"\n },\n \"Congress\": {\n \"type\": \"organization\",\n \"description\": \"The legislative branch of the federal government of the United States\"\n },\n \"Senator Tom Cotton\": {\n \"type\": \"person\",\n \"description\": \"An American politician serving as the junior United States senator from Arkansas\"\n },\n \"Director of the Secret Service\": {\n \"type\": \"person\",\n \"description\": \"The head of the United States Secret Service\"\n }\n}\n"`
  • (Testing with temperature of 0.99): Entities:\n- White House\n- Sunday\n- Law enforcement\n- Camera surveillance\n- West Wing\n- Secret Service\n- Cocaine\n- White House visitors\n- Magnetometers\n- K-9 units\n- Anthrax\n- Senator Tom Cotton\n- Arkansas

Given the free-form nature of the prompt and the lack of a precise definition of an "entity" the results here vary considerably, even at the default temperature setting. Strangely, at the default temperature setting, the LLM consistently failed to output in JSON format and in one case output strange triple tick marks around the output.

Note that it extracted Jericka Duncan's name only once and even then respelled it as "Jerrika", which suggests biasing in its training data of specific name spellings.

Of the eight runs, it extracted the names of Jericka Duncan and Weijia Jiang just once.

What about a Q&A ("Where was the cocaine discovered?"):

  • The cocaine was discovered on the ground floor in a common area near the entry to the west wing.
  • The cocaine was discovered on the ground floor in a common area near the entry to the west wing.
  • The cocaine was discovered on the ground floor in a common area near the entry to the west wing.
  • (Testing with temperature of 0.99): The cocaine was discovered on the ground floor in a common area near the entry to the west wing.
  • (Testing with temperature of 0.99): The cocaine was discovered in the West Wing, on the ground floor, in a common area near the entry.

Here the temperature setting did not impact the results.

Or another question ("Who brought the cocaine into the White House according to the following text?")?

  • According to the text, the cocaine was brought into the White House by someone on a White House tour.
  • According to the text, the cocaine was brought into the White House by someone on a White House tour.
  • The cocaine was brought into the White House by someone on a tour. This is the leading theory, according to senior law enforcement officials. The discovery on Sunday night put the White House on temporary lockdown until the substance was deemed \"not a threat.\" President Biden and the first family were at Camp David at the time. The bag was discovered on the ground floor in a common area near the entry to the West Wing. It was close to a set of storage cubbies where visitors leave their cell phones during a tour. The Secret Service is leading the investigation, looking at visitor logs and camera surveillance.
  • (Testing with temperature of 0.99): The correct answer is \"someone on a white house tour\".\n\nThe story is about the investigation of cocaine found in the White House. The leading theory is that someone on a white house tour may have left a small bag of cocaine in the west wing.
  • (Testing with temperature of 0.99): The cocaine was brought into the White House by someone who was on a White House tour.

Interestingly, here the default temperature actually yielded a longer response in one case than a temperature of 0.99.

What about event/relation extraction ("Return a JSON object containing all of the events described in the following text.")?

  • { "safetyAttributes": { "blocked": true }, "content": "" }
  • { "safetyAttributes": { "blocked": true }, "content": "" }
  • { "safetyAttributes": { "blocked": true }, "content": "" }

Strangely, every single time it was run the output was blocked due to a violation of Google's standards.

Changing the prompt from "events" to "relationships" fixes the issue ("Return a JSON object containing all of the relationships described in the following text."):

  • {\n \"White House\": {\n \"LOCATION\": \"Washington, D.C.\",\n \"LEADER\": \"Joe Biden\",\n \"SECURITY\": {\n \"SECRET SERVICE\": \"Leads the investigation\"\n }\n },\n \"Cocaine\": {\n \"TYPE\": \"Illegal drug\",\n \"DISCOVERED\": \"White House\"\n },\n \"Secret Service\": {\n \"LEADER\": \"James Murray\",\n \"DUTIES\": \"Investigates who brought cocaine into the White House\"\n },\n \"White House Visitor\": {\n \"GOALS\": \"Tour the West Wing\",\n \"SECURITY\": {\n \"MAGNETOMETERS\": \"Used for background checks\",\n \"K-9 UNITS\": \"Not used for illegal drugs\"\n }\n }\n}
  • {\n \"White House\": {\n \"LOCATION\": \"Washington, D.C.\",\n \"OCCUPIED_BY\": \"President of the United States\",\n \"SECURITY_GUARD\": \"Secret Service\",\n \"VISITORS\": \"White House visitors\"\n },\n \"White House visitors\": {\n \"VISITED_LOCATION\": \"White House\",\n \"SECURITY_CHECK\": \"Magnetometers\",\n \"POSSIBLE_DRUG_BRINGERS\": \"Someone on a white house tour\"\n },\n \"Secret Service\": {\n \"OCCUPIES_LOCATION\": \"White House\",\n \"INVESTIGATING\": \"Who brought cocaine into the White House\",\n \"SECURITY_CHECK\": \"Magnetometers\",\n \"POSSIBLE_DRUG_BRINGERS\": \"Someone on a white house tour\"\n },\n \"Cocaine\": {\n \"FOUND_IN_LOCATION\": \"White House\",\n \"POSSIBLE_BRINGERS\": \"Someone on a white house tour\"\n }\n}
  • {\n \"White House\": {\n \"LOCATION\": \"Washington, D.C.\",\n \"LEADER\": \"Joe Biden\",\n \"SECURITY_FORCE\": \"Secret Service\"\n },\n \"Secret Service\": {\n \"LEADER\": \"James Murray\",\n \"TASK\": \"Investigating who brought cocaine into the White House\"\n },\n \"Cocaine\": {\n \"TYPE\": \"Illegal drug\",\n \"DISCOVERY_LOCATION\": \"White House\"\n }\n}

What if we give it a bit more instruction ("Return a JSON object containing all of the relationships described in the following text. Each relationship should have three fields: Actor 1, Relationship, Actor 2. Actor 1 is the first entity, Actor 2 is the second entity and Relationship is the relationship between them."):

  • {\n \"cocaine\": {\n \"CONTAINED_IN\": \"bag\",\n \"FOUND_IN\": \"white house\"\n },\n \"bag\": {\n \"FOUND_IN\": \"white house\"\n },\n \"white house\": {\n \"VISITED_BY\": \"visitors\"\n },\n \"visitors\": {\n \"UNDERGO_BACKGROUND_CHECKS\": \"yes\",\n \"GO_THROUGH_MAGNETOMETERS\": \"yes\"\n }\n}
  • {\n \"cocaine\": {\n \"LOCATION\": \"white house\",\n \"DISCOVERED_BY\": \"secret service\",\n \"FOUND_NEAR\": \"visitors entrance\"\n },\n \"white house\": {\n \"VISITOR_LOGS\": \"secret service\",\n \"CAMERA_SURVEILLANCE\": \"secret service\",\n \"VISITORS\": \"white house visitors\"\n },\n \"secret service\": {\n \"CONDUCTING_INVESTIGATION\": \"cocaine\",\n \"SECURITY_FLAW\": \"white house\"\n }\n}
  • (Testing with temperature of 0.99) (tick marks verbatim from output): "`json\n{\n \"WHITE HOUSE\": {\n \"LOCATION\": \"WASHINGTON, DC\",\n \"LEADER\": \"PRESIDENT JOE BIDEN\",\n \"SECRET SERVICE INVESTIGATION\": {\n \"RESULT\": \"UNDER INVESTIGATION\",\n \"SUBJECT\": \"WHO BROUGHT COCAINE INTO THE WHITE HOUSE\",\n \"DESCRIPTION\": \"THE SECRET SERVICE IS LOOKING AT VISITOR LOGS AND CAMERA SURVEILLANCE\",\n \"PROBLEM\": \"NOT EVERY CAMERA IS TRAINED ON EVERY BODY AT EVERY MINUTE\"\n }\n },\n \"SECRET SERVICE INVESTIGATION\": {\n \"CONDUCTED BY\": \"THE SECRET SERVICE\",\n \"RESULT\": \"UNDER INVESTIGATION\",\n \"SUBJECT\": \"WHO BROUGHT COCAINE INTO THE WHITE HOUSE\"\n }\n}\n"`
  • (Testing with temperature of 0.99): {\n \"Jerrica Duncan\": {\n \"WORK_FOR\": \"CBS\"\n },\n \"The Secret Service\": {\n \"LEADER\": \"President Joe Biden\",\n \"EMPLOYER\": \"United States\"\n },\n \"White House\": {\n \"LEADER\": \"President Joe Biden\",\n \"COUNTRY\": \"United States\"\n },\n \"The White House Tour\": {\n \"LOCATION\": \"The White House\",\n \"ORGANIZATION\": \"The United States Secret Service\"\n },\n \"Illegal Drug\": {\n \"SUBSTANCE\": \"Cocaine\"\n },\n \"Cocaine\": {\n \"SYNONYM\": \"Illegal Drug\"\n }\n}

What if we change the formulation a bit to get around the incorrect policy block ("Return a JSON object containing all of the events described in the following text. Each event should have three fields: Actor 1, Relationship, Actor 2. Actor 1 is the first entity, Actor 2 is the second entity and Relationship is the relationship between them.")?

  • {\n \"events\": [\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"CONTAINS\",\n \"Actor 2\": \"Cocaine\"\n },\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"UNDERGOES\",\n \"Actor 2\": \"Background Check\"\n },\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"HAS\",\n \"Actor 2\": \"Magnetometers\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"INVESTIGATES\",\n \"Actor 2\": \"Cocaine\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"HAS\",\n \"Actor 2\": \"K-9 Units\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"SEARCHES FOR\",\n \"Actor 2\": \"Explosives\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"SEARCHES FOR\",\n \"Actor 2\": \"Biohazardous Materials\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"SEARCHES FOR\",\n \"Actor 2\": \"Weapons\"\n },\n {\n \"Actor 1\": \"Senator Tom Cotton\",\n \"Relationship\": \"SENDS LETTER TO\",\n \"Actor 2\": \"Director of the Secret Service\"\n },\n {\n \"Actor 1\": \"Senator Tom Cotton\",\n \"Relationship\": \"WANTS TO KNOW\",\n \"Actor 2\": \"If the White House Complex is not secure\"\n },\n {\n \"Actor 1\": \"Senator Tom Cotton\",\n \"Relationship\": \"WANTS\",\n \"Actor 2\": \"A plan to correct any security flaws\"\n }\n ]\n}
  • {\n \"events\": [\n {\n \"Actor1\": \"White House\",\n \"Relationship\": \"CONTAINS\",\n \"Actor2\": \"Cocaine\"\n },\n {\n \"Actor1\": \"White House\",\n \"Relationship\": \"VISITED BY\",\n \"Actor2\": \"Someone\"\n },\n {\n \"Actor1\": \"Secret Service\",\n \"Relationship\": \"INVESTIGATING\",\n \"Actor2\": \"Cocaine\"\n },\n {\n \"Actor1\": \"Secret Service\",\n \"Relationship\": \"LOOKING AT\",\n \"Actor2\": \"Visitor Logs\"\n },\n {\n \"Actor1\": \"Secret Service\",\n \"Relationship\": \"LOOKING AT\",\n \"Actor2\": \"Camera Surveillance\"\n },\n {\n \"Actor1\": \"Senator Tom Cotton\",\n \"Relationship\": \"WRITING TO\",\n \"Actor2\": \"Director of the Secret Service\"\n }\n ]\n}
  • (Testing with temperature of 0.99): {\n \"events\": [\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"INVESTIGATE\",\n \"Actor 2\": \"White House Tour\"\n },\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"DISCOVER\",\n \"Actor 2\": \"Bag of Cocaine\"\n },\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"GO INTO\",\n \"Actor 2\": \"LOCKDOWN\"\n },\n {\n \"Actor 1\": \"Cocaine\",\n \"Relationship\": \"BE IDENTIFIED\",\n \"Actor 2\": \"Lab Tests\"\n }\n ]\n}
  • (Testing with temperature of 0.99): {\n \"events\": [\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"HAS_SECURITY_FLAW\",\n \"Actor 2\": \"Security\"\n },\n {\n \"Actor 1\": \"White House\",\n \"Relationship\": \"WAS_CLOSED\",\n \"Actor 2\": \"Sunday\"\n },\n {\n \"Actor 1\": \"Cocaine\",\n \"Relationship\": \"DISCOVERED_IN\",\n \"Actor 2\": \"White House\"\n },\n {\n \"Actor 1\": \"Secret Service\",\n \"Relationship\": \"IS_INVESTIGATING\",\n \"Actor 2\": \"Cocaine\"\n },\n {\n \"Actor 1\": \"Cocaine\",\n \"Relationship\": \"WAS_LEFT_BY\",\n \"Actor 2\": \"Someone on a White House Tour\"\n }\n ]\n}

Here we can see that just by the addition of a small amount of clarifying text we can guide the LLM a bit, though we can see entity extraction breaking down to simple clause extraction – a symptom we've seen in current tier 1 commercial LLMs.

Overall, these examples give a general sense of how PaLM 2 can be applied via embeddings and generative text tasks. Each of the requests above took between 3 and 7 seconds per execution, meaning they are not sufficiently responsive for latency-sensitive applications, but are sufficiently fast and scalable for background bulk processing.