Yesterday we examined the significant limitations of using embeddings to search by tone and framing, demonstrating that embeddings are not as adept at capturing tone as they are other characteristics. The result is that an embedding of a passage with positive or negative tone towards a given topic may not necessarily be scored as more similar to passages with the same tone than those with opposing tone. In other words, an embedding with negative tone towards inflation might be scored as more similar to articles that are pro-inflation than with similar anti-inflation coverage. This poses unique challenges to the growing use of embedding-centric semantic search for applications that require the consideration of tone and framing and suggests that embedding systems may need to be enriched with tonal and framing attributes outside of the embedding process or through embeddings fine-tuned to reflect sentiment more strongly.

Recall that when using Vertex's LLM-based embeddings to sort our passages from most positive to most negative we got the following results (starred passages are the actual positive sentences):

***(0.442) (ID 11) (Len: 2372): Inflation, when managed appropriately, can have positive effects on an econ
***(0.431) (ID 10) (Len: 2323): Inflation, when managed appropriately, can have positive effects on an econ
***(0.402) (ID 8) (Len: 2378): Inflation, when managed within reasonable limits, can have positive effects
   (0.402) (ID 18) (Len: 2320): Inflation, particularly when it becomes excessive or uncontrollable, can ha
   (0.390) (ID 16) (Len: 2077): Inflation, especially when it reaches high levels or spirals out of control
   (0.387) (ID 17) (Len: 2271): Inflation, particularly when it becomes excessive or uncontrollable, can ha
***(0.386) (ID 12) (Len: 320): Inflation, when properly managed, can encourage spending, investment, and d
***(0.382) (ID 9) (Len: 1677): Inflation, when managed appropriately, can bring several positive outcomes 
   (0.365) (ID 15) (Len: 1940): Inflation, especially when it exceeds a moderate level, can have detrimenta
   (0.336) (ID 14) (Len: 214): Inflation, when properly controlled, can stimulate economic activity, encou
***(0.305) (ID 13) (Len: 271): Inflation, when appropriately managed, can stimulate economic growth, encou
   (0.271) (ID 20) (Len: 286): Excessive inflation can erode purchasing power, diminish the value of savin
   (0.257) (ID 19) (Len: 296): High and uncontrolled inflation can erode purchasing power, reduce the valu
   (0.236) (ID 21) (Len: 312): Excessive inflation can erode purchasing power, reduce the value of savings

Similarly, when using Vertex embeddings to sort from most negative to most positive we got the following results (starred passages are the actual negative sentences):

   (0.442) (ID 8) (Len: 2378): Inflation, when managed within reasonable limits, can have positive effects
***(0.441) (ID 18) (Len: 2320): Inflation, particularly when it becomes excessive or uncontrollable, can ha
   (0.439) (ID 11) (Len: 2372): Inflation, when managed appropriately, can have positive effects on an econ
***(0.430) (ID 16) (Len: 2077): Inflation, especially when it reaches high levels or spirals out of control
   (0.429) (ID 10) (Len: 2323): Inflation, when managed appropriately, can have positive effects on an econ
***(0.427) (ID 17) (Len: 2271): Inflation, particularly when it becomes excessive or uncontrollable, can ha
   (0.401) (ID 9) (Len: 1677): Inflation, when managed appropriately, can bring several positive outcomes 
***(0.396) (ID 15) (Len: 1940): Inflation, especially when it exceeds a moderate level, can have detrimenta
   (0.325) (ID 12) (Len: 320): Inflation, when properly managed, can encourage spending, investment, and d
   (0.307) (ID 14) (Len: 214): Inflation, when properly controlled, can stimulate economic activity, encou
   (0.290) (ID 13) (Len: 271): Inflation, when appropriately managed, can stimulate economic growth, encou
***(0.289) (ID 19) (Len: 296): High and uncontrolled inflation can erode purchasing power, reduce the valu
***(0.276) (ID 20) (Len: 286): Excessive inflation can erode purchasing power, diminish the value of savin
   (0.252) (ID 21) (Len: 312): Excessive inflation can erode purchasing power, reduce the value of savings

In the case of positive->negative sorting, three of the correct sentences did appear at the top of the results, while the remaining three were scattered below, while for the negative->positive sorting, all six examples were evenly distributed through the results for effectively randomized results. How do these results compare with traditional sentiment analysis?

Using the same sentences, we ran each through GCP's Natural Language API and requested its computed document-level sentiment. We report the results below (the full text of each passage was scored by the NLP API, but just the first 75 characters are shown below for easy display, as with the embedding examples above). The end result is that classical neural sentiment scoring perfectly stratifies the passages by positive and negative, with no false positives or negatives. The original passages were not generated with varying levels of sentiment (just positive/negative), so it is impossible to evaluate the internal ranking of the positive and negative passages below, but overall the ordering seems intuitively reasonable.

Like with embeddings, however, sentiment scores are entirely influenced by length. Shorter passages are scored with stronger sentiment than longer ones, though longer passages do not always trend towards neutral. In the case of sentiment this makes complete sense, since a short passage consistently of a majority of emotional words has a high density of emotional charge since the majority of its words contribute to tone. A lengthy passage with a much greater array of neutral structural words (connectives) will also typically have a larger array of neutral details in the case of news content (though op-eds can retain this high emotional density). Think of a pithy news headline breathlessly proclaiming "candidate wins landslide election to save planet" versus the full body text of the article that likely contains a state-by-state numerical accounting of vote totals, comparison projections and other clinical language and a much greater amount of the "connective tissue" of language that is required of long-form text to make it grammatically correct that is excused in shorter-form content.

Here are the final results, sorted by score and then sub-sorted by sentiment (score * magnitude).

Sentence	Text	Length	Score	Mag	Sent (Score*Mag)
13	Inflation, when appropriately managed, can stimulate economic growth, encou	271	0.6	1.3	0.78
14	Inflation, when properly controlled, can stimulate economic activity, encou	214	0.6	0.6	0.36
12	Inflation, when properly managed, can encourage spending, investment, and d	320	0.4	1.4	0.56
8	Inflation, when managed within reasonable limits, can have positive effects	2378	0.2	9.3	1.86
11	Inflation, when managed appropriately, can have positive effects on an econ	2372	0.2	7.3	1.46
10	Inflation, when managed appropriately, can have positive effects on an econ	2323	0.2	7.1	1.42
9	Inflation, when managed appropriately, can bring several positive outcomes	1677	0.2	6.4	1.28
17	Inflation, particularly when it becomes excessive or uncontrollable, can ha	2271	-0.3	11.7	-3.51
18	Inflation, particularly when it becomes excessive or uncontrollable, can ha	2320	-0.3	11.9	-3.57
16	Inflation, especially when it reaches high levels or spirals out of control	2077	-0.4	9.6	-3.84
15	Inflation, especially when it exceeds a moderate level, can have detrimenta	1940	-0.5	9.9	-4.95
19	High and uncontrolled inflation can erode purchasing power, reduce the valu	296	-0.8	1.6	-1.28
20	Excessive inflation can erode purchasing power, diminish the value of savin	286	-0.8	1.6	-1.28
21	Excessive inflation can erode purchasing power, reduce the value of savings	312	-0.8	1.7	-1.36