Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scaling of
Texts with Large Language Models

Patrick Y. Wu  Center for Social Media and Politics, New York University
Jonathan Nagler  Center for Social Media and Politics, New York University
Department of Politics, New York University  Joshua A. Tucker  Center for
Social Media and Politics, New York University  Department of Politics, New
York University  Solomon Messing  Center for Social Media and Politics, New
York University

(Version: October 15, 2023  
Click here for the latest version)

###### Abstract

Existing text scaling methods often require a large corpus, struggle with
short texts, or require labeled data. We develop a text scaling method that
leverages the pattern recognition capabilities of generative large language
models (LLMs). Specifically, we propose concept-guided chain-of-thought
(CGCoT), which uses prompts designed to summarize ideas and identify target
parties in texts to generate concept-specific breakdowns, in many ways similar
to guidance for human coder content analysis. CGCoT effectively shifts
pairwise text comparisons from a reasoning problem to a pattern recognition
problem. We then pairwise compare concept-specific breakdowns using an LLM. We
use the results of these pairwise comparisons to estimate a scale using the
Bradley-Terry model. We use this approach to scale affective speech on
Twitter. Our measures correlate more strongly with human judgments than
alternative approaches like Wordfish. Besides a small set of pilot data to
develop the CGCoT prompts, our measures require no additional labeled data and
produce binary predictions comparable to a RoBERTa-Large model fine-tuned on
thousands of human-labeled tweets. We demonstrate how combining substantive
knowledge with LLMs can create state-of-the-art measures of abstract concepts.

## Introduction

Text scaling methods are quantitative techniques used to analyze the latent
positions of texts, such as tweets, documents, or speeches, along one or more
dimensions. Popular text scaling methods, such as Wordfish, are used
extensively in the social sciences [[, see,
e.g.,]]laver_benoit_garry_2003,wordfish_slapin_proksch,wu_etal_2019,rheault_cochrane_2020,bailey2023measuring.
However, these approaches require a large text corpus, struggle with short
texts, and do not have a mechanism to precisely target the abstract concept of
interest. Other approaches, such as fine-tuning pre-trained language models
such as RoBERTa [6], require hand-labeled data. These methods often fail with
texts such as social media posts: they are short, it is often not clear how to
pick documents for identification, and the usages of words rapidly shift over
time.

In this article, we present a novel text scaling framework that leverages the
embedded knowledge base and pattern recognition capabilities of generative
large language models (LLMs). The core idea is to use an LLM to conduct
pairwise comparisons between two texts. In other words, we prompt an LLM to
pick the text that reflects a greater quantity of some latent or abstract
concept of interest, such as which text contains greater aversion. But instead
of directly pairwise comparing texts, we compare concept-specific breakdowns
of the texts. These concept-specific breakdowns are generated using an
approach we call concept-guided chain-of-thought (CGCoT) prompting. CGCoT
prompting is a method that uses a series of researcher-crafted questions that
examine the constituent parts of the concept of interest in a given text. The
text and the LLM’s answers to the CGCoT prompts for that text form the text’s
concept-specific breakdown. These researcher-crafted prompts, akin to a
codebook used to guide content analysis, are the same across all texts, making
the concept-specific breakdowns directly comparable in pairwise comparisons.

Using the LLM to pairwise compare concept-specific breakdowns of the texts
along a targeted concept, we then scale the LLM’s answers using the Bradley-
Terry model [7]. We call the resulting scores for the texts CGCoT pairwise
scores. Because we craft the CGCoT prompts and the basis of the pairwise
comparisons, we can precisely target the abstract concept of interest. We rely
on pairwise comparisons rather than attempting to use the LLM to adjudicate
the presence or absence of concepts like aversion for a number of reasons.
First, pairwise comparisons enable us to establish a scale where differences
are meaningful; ordinal rankings indicate order without quantifying the gaps
between ranked items. Second, at least for human tasks, pairwise comparisons
are easier to complete and allow for more subtle distinctions, improving
reliability over labeling tasks over single items [8].

These comparisons are made over “concept-specific” breakdowns of the texts
rather than the texts themselves. It is well-documented that generative LLMs
often make mistakes in problems that require intermediate reasoning steps [9,
10, 11, 12, 13]. [10] propose “chain-of-thought,” which prompts the LLM to
explicitly generate its intermediate reasoning steps, leading to improved
responses on problem-solving tasks such as arithmetic or question-answering.
In related work, [12] use the LLM to automatically break a problem down into
subproblems using few-shot examples, an approach they call least-to-most
prompting; the LLM then solves each subproblem to solve a larger, harder
problem. But despite these innovations, it is still unclear whether generative
LLMs are able to “reason.” [13], for example, find that LLMs perform poorly on
“counterfactual” tasks, which are variants of reasoning tasks that an LLM
performs well on.

We build on these ideas by proposing CGCoT prompting. Rather than using the
LLM to generate its own intermediate reasoning steps or its own breakdown of
the problem into subproblems, we leverage the researcher’s substantive
knowledge about the target concept to craft a sequence of questions that
identify and describe the targeted concept and its constituent parts in the
text. This approach is analogous to using a codebook for qualitative content
analysis [[, see, e.g., ]]FONTEYN2008165. In other words, we use the LLM’s
pattern recognition capabilities in conjunction with researcher-guided prompts
to generate the intermediate reasoning steps that we would ideally like the
LLM to reason through for a targeted concept when making pairwise comparisons.
Effectively, CGCoT shifts the pairwise comparisons of text from a reasoning
problem to a pattern recognition problem. For example, if we are scaling the
level of aversion expressed towards a target in a text, we can use a series of
prompts that summarizes the text, identifies the primary focus/target of the
text, and identifies whether aversion is expressed towards that target.

We apply the proposed approach to measuring affective polarization expressed
in tweets [[, see, e.g.,]]yarchi_2021,nordbrandt_2021. Affective polarization
is the tendency for partisans to dislike or distrust members of the opposing
party [17, 18]. Affective polarization is typically studied using surveys but,
except for [19], has not been extensively studied among political non-elites
on social media. [19] label tweets for aversion against Republicans and
aversion against Democrats in tweets, but these are binary labels. Expressions
of aversion against a specific party are an inevitability on social media, but
strong expressions of aversion on social media threaten productive discourse
online, incentivize antidemocratic rhetoric, and, when analyzed in the
aggregate, can be a sign of the declining health of democracy [20].

To validate the pairwise comparison approach with concept-specific tweet
breakdowns, we calculate two affective polarization scales using a random
sample of political tweets from [19]: an aversion against Republicans scale
and an aversion against Democrats scale. We develop a series of questions used
as CGCoT prompts with the LLM—here, we use ChatGPT-3.5—that identify and
describe aversion towards specific parties in a given tweet. Specifically, we
use three prompts: the first prompt asks ChatGPT to summarize the tweet; the
second prompt asks ChatGPT to identify the primary party that is the focus or
target of the tweet; the third prompt asks ChatGPT to identify whether
aversion is expressed towards the targeted party. The tweet’s concept-specific
breakdown is the original tweet and all the LLM’s answers to these questions
about the tweet. We then prompt ChatGPT-3.5 to select the concept-specific
breakdown that exhibits greater aversion against a specific party, and scale
its resulting answers using the Bradley-Terry model.

We compare CGCoT pairwise scores with three alternative unsupervised text
scaling approaches. The first two use Wordfish, a popular unsupervised Poisson
text scaling model [2]. We use Wordfish with the tweets and the tweet’s
concept-specific breakdowns. The third text scaling approach is the resulting
scale from using ChatGPT-3.5 to pairwise compare the raw tweets. We show that
pairwise comparing the concept-specific tweet breakdowns generates scales that
are more strongly associated with human judgments than all three alternative
text scaling approaches. The results indicate that using both CGCoT prompts
and pairwise comparisons are important for deriving high-quality scales.

We also show that our continuous scale, which only requires a small set of
pilot data to develop the CGCoT prompts, is competitive with state-of-the-art
supervised approaches. Binarizing the scales—treating all observations with a
CGCoT pairwise score above a cutoff as exhibiting aversion and all
observations with a score below a cutoff as not exhibiting aversion—gives us a
set of predictions that nearly match (for aversion against Democrats) or
exceed (for aversion against Republicans) the performance metrics of a
RoBERTa-Large [6] model fine-tuned on 3,000 human labeled tweets. Overall, the
success of CGCoT in measuring affective polarization suggests that pairing
substantive knowledge with LLMs can be immensely useful for solving social
science text-as-data measurement problems.

## Results

##### Data

We use ChatGPT-3.5 to pairwise compare political tweets from [19]. [19] fine-
tune a RoBERTa model that classifies whether tweets contain aversion against
Republicans and/or aversion against Democrats. They used 3,000 tweets for
fine-tuning, 500 tweets for validation, and 500 tweets as the test set. These
tweets were selected using a set of political keywords and were sampled from
all users on Twitter. Each tweet was labeled by 3 coders from SurgeAI. We
create scales of each of these two concepts using the CGCoT method outlined
below. We scale the test set tweets to make our results comparable with
theirs.

##### Concept-Guided Chain-of-Thought

To create aversion against Republican-specific breakdowns, we created the
following series of questions using definitions and concepts from the
literature on affective polarization [[,
e.g.,]]iyengar_sood_lelkes_affectivepolarization,finkel_etal_2020_sectarianism,Druckman2021,affective_polarization_csmap:

  1. 1.

Summarize the Tweet.

  2. 2.

We broadly define Republicans to include any member of the Republican
Party/GOP, the Republican Party/GOP generally, conservatives, right-wingers,
anyone that supports MAGA, or the alt-right. We broadly define Democrats to
include any member of the Democratic Party, the Democratic Party generally,
liberals, leftists, or progressives. Using these definitions, does the Tweet
primarily focus on Republicans (or a Republican) or Democrats (or a Democrat)?
The focus can be on a specific member of a party.

  3. 3.

If the Tweet primarily focuses on Republicans based on your above answer, does
the Tweet express aversion, dislike, distrust, blame, criticism, or negative
sentiments of Republicans (or a Republican)? If the Tweet primarily focuses on
Democrats based on your above answer, does the Tweet express aversion,
dislike, distrust, blame, criticism, or negative sentiments of Democrats (or a
Democrat)? If the Tweet focuses on neither party, answer “N/A.”

  4. 4.

Using only your answer immediately above, does the Tweet express aversion,
dislike, distrust, blame, criticism, or negative sentiments of Republicans (or
a Republican)?

We used a similar set of prompts to create aversion against Democrats-specific
breakdowns: the first three questions are the same, except we flipped the
order of the definitions, and we replaced the word “Republicans” with the word
“Democrats” in the fourth question. Taking advantage of the conversational
aspect of generative LLMs, we prompt each question sequentially. The concept-
specific breakdown is the concatenation of the original tweet and the answers
to all four questions. In short, the process of creating these prompts is
analogous to creating a codebook for human coders and qualitative content
analysis [[, see, e.g.,]]satu_kyngas_2008,FONTEYN2008165. The Methods section
contains greater details about the process we used to create these CGCoT
prompts.

We apply this set of prompts to each tweet to obtain a concept-specific
breakdown for each tweet. We then use ChatGPT-3.5 to pairwise compare these
concept-specific breakdowns. The Methods section works through an example of
one such pairwise comparison, including the prompts used for the pairwise
comparison of concept-specific breakdowns.

### Face Validity of CGCoT Pairwise Scores

#### The Affective Polarization CGCoT pairwise scores show face validity when
examining the rank ordering of tweets

We showcase three tweet examples from each of the affective polarization
scales. The first has the highest, the second is around the mean, and the
third shows the lowest CGCoT pairwise scores, demonstrating the scales’ face
validity. We begin with the aversion against Republicans CGCoT pairwise
scores.

  * •

Highest: “#ArrestTrump and all associates immediately. Try them for treason.
PUBLIC CAPITAL PUNISHMENT. They’re all traitors & murderers. NEVER FORGET all
the blood they have on their hands #Trump #Republicans
#1776RestorationMovement #January6thCommittee #Jan6Hearings”

  * •

Middling: “@newsmax America doesn’t have to worry about socialism. The threat
to American democracy is not socialism, it is the radicalized and unhinged
Republicans.”

  * •

Lowest: “House Republican leader Kevin McCarthy calls the situation at the
southern border a humanitarian and national security crisis.”

The ordering of these tweets appears intuitively consistent with our concept
of interest. There is a gradation of the intensity of aversion expressed in
tweets from the highest to lowest CGCoT pairwise scores. The middling tweet
expresses aversion against Republicans but is not nearly as intense as the
tweets with the highest CGCoT pairwise scores.

Looking at the same ordering for the aversion against Democrats CGCoT pairwise
scores, we get the following tweets.

  * •

Highest: “Very true. I keep wondering the same. The Demonrats have sunk deeper
into the swamp! They’ve been exposed, but that’s only let them do their dirty
deeds defiantly out in the open. Why has no one been stopped or paid for their
crimes?”

  * •

Middling: “I kind of like blocking the liberals and only seeing common sense
comments.”

  * •

Lowest: “Get over it, scoop! It’s obvious he’s waiting to put it all together
with the DNC this week. Then I bet he’ll be holding pressers with his policies
and all. But, this incessant reporter whining is for the birds.”

There is, again, a gradation in the intensity of the aversion expressed in the
tweets over the CGCoT pairwise scores. The middling tweet is a jab at
liberals, but it is not the same type of criticism leveled at Democrats as the
tweet with the highest CGCoT pairwise score.

### Validating CGCoT Pairwise Scores

#### Comparing CGCoT with human-coded labels

We first compare the aversion against Republicans CGCoT pairwise scores
against the number of human coders that labeled a tweet as containing aversion
against Republicans (Figure 1). The less ambiguity in text expressing
aversion, the more agreement we expect among human coders. We note that the
wider distribution of CGCoT pairwise scores for tweets with two or three
coders labeling a tweet as containing aversion against Republicans is a
function of the pairwise comparisons: tweets that do not contain any aversion
against Republicans tend to tie with each other in matchups, resulting in
scores “clumping.” In the Supplementary Information, we examine the tweets
labeled by three coders as containing aversion against Republicans with the
lowest aversion against Republicans CGCoT pairwise scores.

Figure 1: Aversion against Republicans CGCoT pairwise scores are strongly
associated with the number of coders applying an aversion against Republicans
label (data from [19]). CGCoT pairwise score estimates are shown with 95%
confidence intervals derived from quasi-standard errors [22]. A linear
regression line is drawn through the points with a 95% confidence band.

We repeat this exercise for aversion against Democrats. We again find a strong
relationship when comparing aversion against Democrats CGCoT pairwise scores
against the number of human coders that labeled a tweet as containing aversion
against Democrats. Figure 2 illustrates this comparison. Again, in the
Supplementary Information, we examine the tweets labeled by three coders as
containing aversion against Democrats with the lowest aversion against
Democrats CGCoT pairwise scores.

Figure 2: Aversion again Democrats CGCoT pairwise scores are strongly
associated with the number of coders applying an aversion against Democrats
label (data from [19]). CGCoT pairwise score estimates are shown with 95%
confidence intervals derived from quasi-standard errors [22]. A linear
regression line is drawn through the points with a 95% confidence band.

#### CGCoT pairwise scores outperform other unsupervised text scaling methods

We compare Spearman’s correlations between human labels and an array of text
scaling methods, including CGCoT pairwise scores. We do this for both aversion
against Democrats and Republicans for the following methods:

  1. 1.

Using Wordfish with the tweets only

  2. 2.

Using WordFish with the tweets’ concept-specific breakdowns

  3. 3.

Using the pairwise comparison approach with ChatGPT-3.5 with the tweets only
(“non-CGCoT tweets-only pairwise scores”)

  4. 4.

CGCoT pairwise scores

Details about how we fit the Wordfish model are in the Methods section. Table
1 shows these correlations for both aversion against Republicans and
Democrats.

|  Aversion Against |   
---|---|---  
Republicans, $\rho$ |  Aversion Against |   
Democrats, $\rho$ |  |   
Wordfish Using Tweets Only |  0.031 |  0.035  
Wordfish Using Concept-Specific Breakdowns |  0.545 |  0.223  
Non-CGCoT Tweets-Only Pairwise Scores |  0.554 |  0.560  
CGCoT Pairwise Scores |  0.638 |  0.611  
Table 1: CGCoT pairwise scores outperform other scaling methods, based on
Spearman’s rank correlation coefficient, with the number of human coders
labeling a tweet as containing aversion against Republicans and Democrats.

The results demonstrate the utility of both CGCoT and pairwise comparisons:
there are notable gains in correlation when moving from using the tweets to
concept-specific breakdowns of the tweets and when moving from using Wordfish
to pairwise comparisons. Our proposed procedure of using CGCoT with LLM
pairwise comparisons yields a scale that most closely aligns with human
judgments compared to other text scaling approaches.

#### CGCoT pairwise scores are competitive with supervised learning approaches

To further analyze the validity of the affective polarization CGCoT pairwise
scores, we create binary labels using cutoffs in the two aversion scales. For
each scale, we label all tweets with CGCoT pairwise scores above the mean of
the CGCoT pairwise scores as 1, and all tweets with CGCoT pairwise scores
below the mean as 0. While binarizing the CGCoT pairwise scores by labeling
observations above the mean as 1 and those below the mean as 0 is slightly
arbitrary, this approach is also guided by a principled decision to use the
central tendency of the scale as the threshold. Future work will use a
training set to choose a more accurate cutoff, which would almost certainly
improve results. We repeat this process for the scale generated using ChatGPT
pairwise comparisons of the tweets only (i.e., non-CGCoT tweets-only
comparisons).

We compare CGCoT pairwise scores with a RoBERTa-Large model fine-tuned using
[19]’s 3,000 labeled political tweets. Table 2 contains the performance
metrics of the two cutoff classifiers and the RoBERTa-Large model.

Classifier | Aversion against | F1 Score | Precision | Recall  
---|---|---|---|---  
Non-CGCoT Tweets-Only Pairwise Scores Cutoff Classifier | Republicans | 0.696 | 0.643 | 0.760  
Democrats | 0.668 | 0.579 | 0.791  
CGCoT Pairwise Scores Cutoff Classifier | Republicans | 0.836 | 0.890 | 0.789  
Democrats | 0.789 | 0.837 | 0.747  
Fine-Tuned RoBERTa-Large Model | Republicans | 0.808 | 0.818 | 0.797  
Democrats | 0.814 | 0.816 | 0.811  
Table 2: The F1 score, precision, and recall of the non-CGCoT tweets-only
pairwise scores cutoff classifier, the CGCoT pairwise scores cutoff
classifier, and RoBERTa-Large classifier fine-tuned on [19]’s aversion labels.

The performance metrics show that CGCoT pairwise scores outperform non-CGCoT
tweets-only pairwise scores on all metrics, except for recall for aversion
against Democrats. The metrics show that the aversion against Republicans
CGCoT pairwise scores outperform the fine-tuned RoBERTa-large model on F1
score and precision and are nearly equivalent on recall. Similarly, the
aversion against Democrats CGCoT pairwise scores are comparable with the fine-
tuned RoBERTa-Large on F1 score. The former performs better on precision and
the latter performs better on recall. Again, the CGCoT pairwise comparison
cutoff classifier’s predictions were calculated using no human-labeled tweets,
except a small set of pilot tweets that were used to develop the CGCoT prompts
for the concept-specific breakdowns.

## Discussion

We develop a novel text scaling method that leverages pairwise comparisons and
an expert-crafted prompting procedure called concept-guided chain-of-thought
(CGCoT), which creates concept-specific breakdowns of the texts. We prompt
ChatGPT to make pairwise decisions between the concept-specific breakdowns
along an abstract concept of interest. The resulting scales are called CGCoT
pairwise scores. We apply the method to derive affective polarization scales:
we use ChatGPT to pick the concept-specific breakdowns of tweets that exhibit
greater aversion towards a specific political party in pairwise comparisons.

We find that the scales largely correlate with how humans interpret aversion
against Republicans and aversion against Democrats on Twitter. We also find
that using CGCoT and pairwise comparisons with LLMs are crucial, as scales
that do not use one or both of these methods are demonstrably worse. We also
show that using a cutoff with the scale can yield binary classifications that
are highly competitive with a RoBERTa-Large model fine-tuned on thousands of
human-labeled tweets. Our findings suggest that using substantive knowledge
with generative LLMs can not only be useful for calculating high-quality
continuous scales but can also be useful for generating discrete
classifications with high accuracy and performance without the use of any
labeled data. In other words, our approach can generate a scale that is
competitive with both unsupervised and supervised approaches.

Our method is situated in a rapidly growing literature on using generative
LLMs for social science applications [[, see,
e.g.,]]tornberg2023chatgpt4,rathje_mirea_sucholutsky_marjieh_robertson_vanbavel_2023,argyle2023ai,bisbee_clinton_dorff_kenkel_larson_2023,wu2023large.
The works that have analyzed text, such as analyzing text along psychological
constructs [24], directly analyze the text. Our proposed text scaling method
breaks the text down into the concept of interest’s constituent parts using
prompts developed by substantive knowledge about the targeted concept; it
involves substantive expert knowledge to a much greater extent than the other
research studies that analyze text using LLMs.

This method also speaks to a large body of text scaling methods, such as
WordScores [1, 28], Wordfish [2], and word embedding methods [29, 3]. Our
method minimizes the need for hand-labeling of text, can measure multiple
targeted concepts within the same corpora, does not require a large corpus,
relies on a transformers-based language model rather than bag-of-words, and
leverages the researchers’ substantive knowledge to precisely target the
abstract concept of interest instead of relying on post hoc dimensional
interpretation.

Our findings align with the notion that the LLM synthesizes information about
complex concepts such as affective polarization, allowing it to reliably and
coherently evaluate latent constructs, abstract concepts, stances, and
sentiments within texts using its pattern recognition capabilities. While we
provide information about what constitutes a “Republican” and “Democrat” being
targeted in our CGCoT prompts, we still assume that the LLM is able to
identify Republican or Democratic figures and organizations, such as Donald
Trump, Joe Biden, and the DCCC. Additionally, we assume that the LLM possesses
the capability to recognize conflict within the presented texts. Again, this
capability stems from its training on various instances of political conflict
on social media and other forms of content. However, the precise impact of
this training on both CGCoT and the pairwise comparisons remains obscured due
to the black box nature of ChatGPT.

The need for CGCoT prompting when making out-of-sample text pairwise
comparisons also indicates the limitations of current generative LLM models.
As shown in previous works, even the best generative LLM models make errors
when solving “out-of-sample” reasoning tasks. These errors are reduced when
the intermediate reasoning steps are made explicit. But rather than allowing
the LLM to generate its own reasoning steps, we accomplish this by engineering
a specific set of questions that examine the concept of interest in the text.
It shows that the LLM cannot effectively “reason” about these concepts. Using
substantive knowledge to generate the concept-specific breakdowns changes the
pairwise comparisons of text from a reasoning problem to a pattern recognition
problem, greatly increasing the effectiveness of pairwise comparisons for
scaling purposes.

Our proposed approach, in spite of its limitations, creates scales that
largely agree with human judgments of the texts along different dimensions of
interest. Lastly, it is well-known that there is a significant mental toll on
people identifying harmful materials for data labeling and content moderation
purposes. Our approach, which rivals the binary prediction performance of
language models fine-tuned on thousands of hand-labeled social media posts,
can help avoid having human coders label thousands of posts containing
potentially harmful material.

## Methods

### Using CGCoT Pairwise Scores to Measure Affective Polarization

There are a total of 4,000 tweets from [19]. The training set has 3,000
tweets, the validation set has 500 tweets, and the test set has 500 tweets. We
scale the 500 tweets of the test set to have comparable performance metrics
when binarizing the CGCoT pairwise scores.

We estimate two versions of the pairwise comparison scores for each scale. One
version compares the tweets directly, which we call non-CGCoT tweets-only
comparisons. The other version compares the concept-specific breakdowns
generated using concept-guided chain-of-thought, which we call aversion CGCoT
pairwise scores. After obtaining the concept-specific breakdowns, we compare
pairs of the concept-specific breakdowns using the following prompt for
aversion against Republicans:

> Tweet Description 1: [concept-specific breakdown for Tweet 1]
>
> Tweet Description 2: [concept-specific breakdown for Tweet 2]
>
> Based on these two Tweet Descriptions, which Tweet Description expresses
> greater aversion, dislike, distrust, blame, criticism, or negative
> sentiments of Republicans: Tweet Description 1 or Tweet Description 2? If
> both equally express or do not express aversion, distrust, blame, criticism,
> or negative sentiments of Republicans, reply with “Neither” or “Tie.”

We use this exact same prompt for aversion against Democrats, except we
replace the word “Republicans” with “Democrats” in the prompt. We also use the
same prompt to directly compare tweets, except we replace all instances of
“Tweet Description” with “Tweet” and all instances of “Tweet Descriptions”
with “Tweets.” Again, we use another prompt to extract the answers from these
pairwise comparisons. The Supplementary Information contains the prompts we
use to extract these answers.

Because we are comparing 500 tweets in a pairwise fashion, there are a total
of 124,750 possible matchups. To reduce the total number of matchups, we
sample 20 matchups per tweet ID for a total of 10,000 matchups. The Bradley-
Terry model, detailed later in this Methods section, does not require complete
matchups to obtain scores for each tweet. In the Supplementary Information, we
compare the CGCoT pairwise scores when we sample 5, 10, and 15 matchups per
tweet ID for both affective polarization scales.

### Developing the Concept-Guided Chain-of-Thought Prompts

As the Results section indicated, using concept-specific breakdowns in
pairwise comparisons is essential to creating a scale that more closely
matches human judgments. As previous works have shown, ChatGPT and other
generative LLMs often make errors in intermediate reasoning steps; these
errors are reduced when the prompt asks the LLM to explicitly provide such
reasoning steps [10]. We build on this idea by using a set of concept-specific
questions that first force the LLM to generate “intermediate reasoning steps”
in the form of concept-specific breakdowns; these concept-specific breakdowns
are then used in pairwise comparisons with the LLM.

Creating the set of questions used for concept-guided chain-of-thought is not
trivial. There is no defined algorithm for creating the set of questions or
the number of questions, as the series of questions is dependent on the
concept of interest. Instead, the prompts are created in a similar fashion to
developing a codebook for human coders or for qualitative content analysis. We
use approximately 50 tweets labeled as aversion against Republicans, 50 tweets
labeled as not aversion against Republicans, 50 tweets labeled as aversion
against Democrats, and 50 tweets labeled as not aversion against Democrats.
These tweets came from [19]’s training set. Rather than relying on the labels
from [19], we labeled these tweets ourselves in order to familiarize ourselves
with the text that we were working with. We iterate on an initial set of CGCoT
prompts and examine outputs from ChatGPT-3.5 until the summaries and party
identifications align with our expectations across the labeled tweets. This is
akin to the steps described in [21]: we first “make sense of the data and
whole” by labeling the tweets ourselves and then apply deductive content
analysis steps to summarize and identify key aspects of the text. We combine
these steps with prompt engineering tactics such as changing specific words,
repeatedly providing definitions, and splitting up complex questions can all
cause the LLM to provide better responses about the concept of interest [[,
see, e.g.,]]Saravia_Prompt_Engineering_Guide_2022. It is an example of how
having a substantive understanding of the concept of interest is still
important, even with the recent development of generative LLMs that provide
human-like responses. However, the inspiration we took from qualitative
content analysis can be applied to the design of prompts for other concepts of
interest.

We also found that asking the LLM to first summarize the tweet always resulted
in better responses for subsequent questions. The conversational nature of the
recent generative LLMs comes from using the current prompt and the previous
prompts and LLM responses in order to generate a response to the current
prompt. Thus, asking for the LLM to first summarize a tweet forces the LLM to
“explain the tweet in its own words,” which yielded improved answers to
subsequent questions.

In future work, we aim to more systematically design the CGCoT prompts. There
are currently no frameworks for sampling semantically equivalent text and
prompts. Such a framework could allow us to find a prompt that is semantically
equivalent to our current prompt while yielding concept-guided breakdowns that
more closely align with the intention of our analysis. We have ongoing work
that aims to develop an MCMC-like algorithm for sampling semantically
equivalent text.

### Example of Applying CGCoT on Two Tweets and Pairwise Comparing the
Concept-Specific Breakdowns

We walk through an example of one pairwise comparison using concept-specific
breakdowns. The first tweet reads: “He’s such a lowlife piece of shit. Isn’t
this slander? Can he be sued? And the GOP are silent. What crap they are. They
did this. They own this. #NeverForget #VoteBlueToSaveAmerica.” The concept-
specific breakdown for this tweet, using the CGCoT prompts for aversion
against Republicans, is the following.

> (0) Original Tweet: “He’s such a lowlife piece of shit. Isn’t this slander?
> Can he be sued? And the GOP are silent. What crap they are. They did this.
> They own this. #NeverForget #VoteBlueToSaveAmerica”  
> (1) The tweet contains a link and expresses anger and frustration towards an
> individual, calling them a derogatory term. The author questions whether the
> person’s actions constitute slander and suggests the possibility of a
> lawsuit. They also criticize the Republican Party for their silence, blaming
> them for the situation and urging others to vote for Democratic candidates
> to protect America. The hashtags used are #NeverForget and
> #VoteBlueToSaveAmerica. (2) The Tweet primarily focuses on criticizing
> Republicans (or the Republican Party/GOP). It expresses dissatisfaction with
> their silence and holds them responsible for a certain issue. It also
> includes a hashtag (#VoteBlueToSaveAmerica) encouraging support for the
> Democratic Party as an alternative. (3) The Tweet expresses aversion,
> dislike, distrust, blame, criticism, and negative sentiments towards
> Republicans (or the Republican Party/GOP). It criticizes their silence and
> assigns responsibility to them, using negative language to describe their
> actions. (4) Yes, the Tweet expresses aversion, dislike, distrust, blame,
> criticism, and negative sentiments towards Republicans (or the Republican
> Party/GOP).

The second tweet reads: “Conservative stupidity will do that.” The concept-
specific breakdown for this tweet, using the same CGCoT prompts for aversion
against Republicans, is the following.

> (0) Original Tweet: ‘Conservative stupidity will do that.’  
> (1) The tweet criticizes conservative individuals by suggesting that their
> stupidity is responsible for certain negative consequences.  
> (2) The tweet primarily focuses on conservatives, which includes
> Republicans, right-wingers, and individuals who support MAGA (Make America
> Great Again).  
> (3) The tweet expresses criticism and negative sentiments towards
> conservatives, including Republicans and individuals who align with right-
> wing ideologies.  
> (4) Yes, the tweet expresses criticism and negative sentiments towards
> conservatives, including Republicans.

We then pairwise compare these two concept-specific breakdowns using
ChatGPT-3.5 using the following prompt:

> Tweet Description 1: [concept-specific breakdown for the first tweet above]  
> Tweet Description 2: [concept-specific breakdown for the second tweet above]  
> Based on these two Tweet Descriptions, which Tweet Description expresses
> greater aversion, dislike, distrust, blame, criticism, or negative
> sentiments of Republicans: Tweet Description 1 or Tweet Description 2? If
> both equally express or do not express aversion, distrust, blame, criticism,
> or negative sentiments of Republicans, reply with “Neither” or “Tie.”

Inputting this prompt into ChatGPT-3.5, the model returns the following
answer.

> Tweet Description 1 expresses greater aversion, dislike, distrust, blame,
> criticism, or negative sentiments of Republicans compared to Tweet
> Description 2.

We repeat this exercise for each matchup between the tweets’ concept-specific
breakdowns. Because of the sheer number of pairwise comparisons, we randomly
sample 20 matchups per tweet, giving us a total of 10,000 matchups for each
aversion scale. The answers elicited from ChatGPT-3.5 are then scaled using
the Bradley-Terry model to estimate the aversion against Republicans and
aversion against Democrats scales.

### Using the Bradley-Terry Model to Estimate the Affective Polarization
Scales

The Bradley-Terry model assumes that in a contest between two players $i$ and
$j$, the odds that $i$ beats $j$ in a matchup are $\alpha_{i}/\alpha_{j}$,
where $\alpha_{i}$ and $\alpha_{j}$ are positive-valued parameters that
indicate latent “ability” [7]. We can define
$\alpha_{i}\equiv\exp(\lambda_{i})$. Then, the log-odds of $i$ beating $j$ is

$\log\left[\frac{\text{Pr}(i\text{ beats }j)}{\text{Pr}(j\text{ beats
}i)}\right]=\lambda_{i}-\lambda_{j}$

The intuition is that the larger the value of $\lambda_{i}$ compared to
$\lambda_{j}$, the more likely it is for player $i$ to beat player $j$.

We translate the above matchup into a contest between two concept-specific
breakdowns. Using the affective polarization scales as our example, the
estimated $\lambda$ parameters are the measures of the level of aversion
towards a specific party. We denote the concept-specific breakdown exhibiting
greater aversion towards a specific party as the “winner.” For ties, we
considered these 0.5 wins for both tweets in the matchup. [31] find that this
approach yields ability parameter estimates that highly correlate with more
complex approaches that explicitly deal with ties. We use the bias-reduced
maximum likelihood estimation approach implemented in the BradleyTerry2 R
package with ChatGPT’s answers to pairwise comparisons to estimate the level
of aversion towards a specific party. These scores are the affective
polarization CGCoT pairwise scores described in the previous sections. The
estimated $\lambda$ parameters are relative to a reference tweet, but this
choice is unimportant because we scale the parameters to the unit interval.

We can also estimate standard errors for the estimated $\lambda$ parameters.
These standard errors are interpreted relative to a reference tweet. We
calculate quasi-variances, which can be interpreted as if they refer to
reference-free estimates of the variance of the underlying “ability” of each
tweet. This means that any pairs of tweets can be directly compared. We use
the qvcalc package to calculate quasi-standard errors [32]. The 95% confidence
intervals in the Results section are derived from these quasi-standard errors.

### Using Wordfish to Estimate the Affective Polarization Scales

The primary goal of Wordfish [2] is to estimate the position of a document
along a single dimension. The assumption is that the rate that tweet $i$
mentions word $j$ is drawn from a Poisson distribution. The functional form of
the model is

$y_{ij}\sim Poisson(\lambda_{ij})$
$\lambda_{ij}=\exp\left(\alpha_{i}+\psi_{j}+\beta_{j}\omega_{i}\right)$

where $y_{ij}$ is the count of word $j$ in tweet $i$, $\alpha$ is the set of
tweet fixed effects, $\psi$ is the set of word fixed effects, $\beta$ is an
estimate of a word-specific weight that reflects the importance of word $j$ in
discriminating between positions, and $\omega_{i}$ is tweet $i$’s position. We
fit this model using quanteda R package [33].

We used standard preprocessing steps: we removed symbols, numbers, and
punctuation. We also stemmed the words. Lastly, we had to impose minimum word
counts and word-document frequency counts to prevent non-convergence. For
aversion against Republicans, a word had to be used at least 4 times across at
least 4 tweets. For aversion against Democrats, a word had to be used at least
5 times across at least 5 tweets. We applied this to both the tweets and the
concept-specific breakdowns.

## References

  * [1] Michael Laver, Kenneth Benoit and John Garry  “Extracting Policy Positions from Political Texts Using Words as Data”  In _American Political Science Review_ 97.2 Cambridge University Press, 2003, pp. 311–331  DOI: 10.1017/S0003055403000698
  * [2] Jonathan B. Slapin and Sven-Oliver Proksch  “A Scaling Model for Estimating Time-Series Party Positions from Texts”  In _American Journal of Political Science_ 52.3, 2008, pp. 705–722  DOI: https://doi.org/10.1111/j.1540-5907.2008.00338.x
  * [3] Patrick Y. Wu, Walter R. Mebane, Logan Woods, Joseph Klaver and Preston Due  “Partisan Associations of Twitter Users Based on Their Self-Descriptions and Word Embeddings” Presented at APSA 2019, 2019 
  * [4] Ludovic Rheault and Christopher Cochrane  “Word Embeddings for the Analysis of Ideological Placement in Parliamentary Corpora”  In _Political Analysis_ 28.1 Cambridge University Press, 2020, pp. 112–133  DOI: 10.1017/pan.2019.26
  * [5] Michael Bailey  “Measuring Candidate Ideology from Congressional Tweets and Websites”, 2023  DOI: 10.2139/ssrn.4350550
  * [6] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019  arXiv:1907.11692 [cs.CL]
  * [7] Ralph Allan Bradley and Milton E. Terry  “Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons”  In _Biometrika_ 39.3-4, 1952, pp. 324–345  DOI: 10.1093/biomet/39.3-4.324
  * [8] David Carlson and Jacob M. Montgomery  “A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political Texts”  In _American Political Science Review_ 111.4 Cambridge University Press, 2017, pp. 835–843  DOI: 10.1017/S0003055417000302
  * [9] Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi and Hannaneh Hajishirzi  “Generated Knowledge Prompting for Commonsense Reasoning”  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 3154–3169  DOI: 10.18653/v1/2022.acl-long.225
  * [10] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le and Denny Zhou  “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, 2023  arXiv:2201.11903 [cs.CL]
  * [11] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa  “Large Language Models are Zero-Shot Reasoners”, 2023  arXiv:2205.11916 [cs.CL]
  * [12] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le and Ed Chi  “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, 2023  arXiv:2205.10625 [cs.AI]
  * [13] Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas and Yoon Kim  “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks”, 2023  arXiv:2307.02477 [cs.CL]
  * [14] Marsha E. Fonteyn, Margaret Vettese, Diane R. Lancaster and Susan Bauer-Wu  “Developing a codebook to guide content analysis of expressive writing transcripts”  In _Applied Nursing Research_ 21.3, 2008, pp. 165–168  DOI: https://doi.org/10.1016/j.apnr.2006.08.005
  * [15] Christian Baden Moran Yarchi and Neta Kligler-Vilenchik  “Political Polarization on the Digital Sphere: A Cross-platform, Over-time Analysis of Interactional, Positional, and Affective Polarization on Social Media”  In _Political Communication_ 38.1-2 Routledge, 2021, pp. 98–139  DOI: 10.1080/10584609.2020.1785067
  * [16] Maria Nordbrandt  “Affective polarization in the digital age: Testing the direction of the relationship between social media and users’ feelings for out-group parties”  In _New Media & Society_ 0.0, 2021  DOI: 10.1177/14614448211044393
  * [17] Shanto Iyengar, Gaurav Sood and Yphtach Lelkes  “Affect, Not Ideology: A Social Identity Perspective on Polarization”  In _Public Opinion Quarterly_ 76.3, 2012, pp. 405–431  DOI: 10.1093/poq/nfs038
  * [18] James N. Druckman, Samara Klar, Yanna Krupnikov, Matthew Levendusky and John Barry Ryan  “Affective polarization, local contexts and public opinion in America”  In _Nature Human Behaviour_ 5.1, 2021, pp. 28–38  DOI: 10.1038/s41562-020-01012-5
  * [19] Haohan Chen, Zhanna Terechshenko, Patrick Y. Wu, Richard Bonneau and Joshua A. Tucker  “Detecting Political Sectarianism on Social Media: A Deep Learning Classifier with Application to 2020‑2022 Tweets”, 2022 
  * [20] Eli J. Finkel, Christopher A. Bail, Mina Cikara, Peter H. Ditto, Shanto Iyengar, Samara Klar, Lilliana Mason, Mary C. McGrath, Brendan Nyhan, David G. Rand, Linda J. Skitka, Joshua A. Tucker, Jay J. Bavel, Cynthia S. Wang and James N. Druckman  “Political sectarianism in America”  In _Science_ 370.6516, 2020, pp. 533–536  DOI: 10.1126/science.abe1715
  * [21] Satu Elo and Helvi Kyngäs  “The qualitative content analysis process”  In _Journal of Advanced Nursing_ 62.1, 2008, pp. 107–115  DOI: https://doi.org/10.1111/j.1365-2648.2007.04569.x
  * [22] David Firth and Renée X. De Menezes  “Quasi‐variances”  In _Biometrika_ 91.1, 2004, pp. 65–80  DOI: 10.1093/biomet/91.1.65
  * [23] Petter Törnberg  “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning”, 2023  arXiv:2304.06588 [cs.CL]
  * [24] Steve Rathje, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire Robertson and Jay J Van Bavel  “GPT is an effective tool for multilingual psychological text analysis”  PsyArXiv, 2023  DOI: 10.31234/osf.io/sekf5
  * [25] Lisa P. Argyle, Ethan Busby, Joshua Gubler, Chris Bail, Thomas Howe, Christopher Rytting and David Wingate  “AI Chat Assistants can Improve Conversations about Divisive Topics”, 2023  arXiv:2302.07268 [cs.HC]
  * [26] James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel and Jennifer Larson  “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models”  SocArXiv, 2023  DOI: 10.31235/osf.io/5ecfa
  * [27] Patrick Y. Wu, Jonathan Nagler, Joshua A. Tucker and Solomon Messing  “Large Language Models Can Be Used to Estimate the Latent Positions of Politicians”, 2023  arXiv:2303.12057 [cs.CY]
  * [28] Will Lowe  “Understanding Wordscores”  In _Political Analysis_ 16.4 Cambridge University Press, 2008, pp. 356–371  DOI: 10.1093/pan/mpn004
  * [29] Austin C. Kozlowski, Matt Taddy and James A. Evans  “The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings”  In _American Sociological Review_ 84.5, 2019, pp. 905–949  DOI: 10.1177/0003122419877135
  * [30] Elvis Saravia  “Prompt Engineering Guide”  In _https://github.com/dair-ai/Prompt-Engineering-Guide_ , 2022 
  * [31] Heather Turner and David Firth  “Bradley-Terry Models in R: The BradleyTerry2 Package”  In _Journal of Statistical Software_ 48.9, 2012, pp. 1–21  DOI: 10.18637/jss.v048.i09
  * [32] David Firth  “qvcalc: Quasi Variances for Factor Effects in Statistical Models” R package version 1.0.3, 2023  URL: https://CRAN.R-project.org/package=qvcalc
  * [33] Kenneth Benoit, Kohei Watanabe, Haiyan Wang, Paul Nulty, Adam Obeng, Stefan Müller and Akitaka Matsuo  “quanteda: An R package for the quantitative analysis of textual data”  In _Journal of Open Source Software_ 3.30, 2018, pp. 774  DOI: 10.21105/joss.00774

Supplementary Information for “Concept-Guided Chain-of-Thought Prompting for
Pairwise Comparison Scaling of Texts with Large Language Models”  
Patrick Y. Wu, Jonathan Nagler, Joshua A. Tucker, and Solomon Messing

#### Tweets with the Lowest CGCoT Pairwise Scores that were Labeled by Three
Coders as Containing Aversion Against Republicans

Looking at just the tweets labeled by three coders as containing aversion
against Republicans, we examined the three tweets with the lowest aversion
against Republicans CGCoT pairwise scores. The text of these tweets are as
follows.

  1. 1.

Dear @POTUS: Are those forgotten men & women who you say never protest the
same people showing up at statehouses armed w/military-style weaponry? Are
those same forgotten ones the same who call themselves Boogaloo? Or are they
those very fine Nazis you favor? All of the above?

  2. 2.

Cadet bone-spur & tribe be innocent then they should welcome investigations
clearing their good name besmirched by furtive conniving fake news liberals.

  3. 3.

“Trump campaign thought their ‘huge news’ on pre-existing conditions had
Democrats cornered – but it backfired spectacularly https://t.co/Ue8ugrCzRF”

ChatGPT makes mistakes in the interpretation of certain phrases in two of
these tweets: it misinterprets a message addressed to the @POTUS account as
directed towards President Biden, not President Trump, and it did not
recognize “Cadet bone-spur” to be a derisive nickname for Trump. In the third
tweet, ChatGPT interpreted a vague headline describing something backfiring
against the Trump campaign as not expressing aversion against Republicans, an
arguably correct interpretation.

#### Tweets with the Lowest CGCoT Pairwise Scores that were Labeled by Three
Coders as Containing Aversion against Democrats

Looking at just the tweets labeled by three coders as containing aversion
against Democrats, we again examine the three tweets with the lowest aversion
against Democrats CGCoT pairwise scores.The text of these tweets are as
follows.

  1. 1.

Maybe if Yang and Tulsi were running the party, I might have a much better
opinion of the Democratic Party. However as it stands, I cannot. I do respect
Yang for wanting to help fighters get paid fairly. My brother boxed for 20
years and doesnt have anything to show for it.

  2. 2.

This is what the MSM and libs won’t ever tell you. And folks reply to this
with all sorts of whataboutism, that it’s perfectly fine these cops were
injured by violent protesters. So that means all Jan 6 defendants should never
have been arrested? AmIright?

  3. 3.

Regular law enforcement like any other regulated profession so that bad cops
cant́ just get rehired at the next department over.” Wtf does this mean? Proof
your statements, Dems.

Here, ChatGPT-3.5 interpreted the text differently from humans. For example,
it did not interpret someone describing how they do not respect the Democratic
Party as expressing aversion against Democrats; in another tweet, it did not
interpret “libs” as an insult towards liberals. In the third tweet, the author
asked Democrats to “proof your statements,” which ChatGPT-3.5 did not
interpret as an insult or criticism of Democrats.

#### Extracting the Tweet that Exhibits Greater Aversion Against Republicans
and Democrats in Each Matchup

For each pairwise comparison prompt, ChatGPT typically returns a small
paragraph explaining its choice. We concatenate the model’s answer with the
following text:

> In the above Text, which Tweet Description is described to be expressing
> greater aversion, dislike, distrust, blame, criticism, or negative
> sentiments of Republicans: Tweet Description 1 or Tweet Description 2?
> Return only “Tweet Description 1” or “Tweet Description 2”. If neither Tweet
> Descriptions are described to be more likely to be expressing greater
> aversion, dislike, distrust, blame, criticism, or negative sentiments of
> Republicans, reply with “Tie.”

We then prompt ChatGPT with the concatenated text. Ties and answers that
deviated from either “Tweet Description 1,” “Tweet Description 2,” or “Tie”
are manually fixed. We used the extract same prompt for extracting answers
from non-CGCoT comparisons, except we replaced the “Tweet Description” with
just “Tweet.”

#### Correlations Between Scores with Differing Number of Matchups

Results in the main text are from 20 matchups per tweet ID, for a total of
10,000 matchups. We analyze how the CGCoT pairwise scores calculated using 5,
10, 15, and 20 matchups per tweet ID correlate for both scales. Results are in
Tables 3 and 4. Across all configurations, correlations are greater than 0.90.

| 5 | 10 | 15 | 20  
---|---|---|---|---  
5 | 1.000 | 0.937 | 0.940 | 0.953  
10 | 0.937 | 1.000 | 0.964 | 0.979  
15 | 0.940 | 0.964 | 1.000 | 0.986  
20 | 0.953 | 0.979 | 0.986 | 1.000  
Table 3: Pearson correlation between CGCoT pairwise scores 5, 10, 15, and 20 matchups per tweet ID for aversion against Republicans. | 5 | 10 | 15 | 20  
---|---|---|---|---  
5 | 1.000 | 0.934 | 0.929 | 0.943  
10 | 0.934 | 1.000 | 0.958 | 0.976  
15 | 0.929 | 0.958 | 1.000 | 0.984  
20 | 0.943 | 0.976 | 0.984 | 1.000  
Table 4: Pearson correlation between CGCoT pairwise scores 5, 10, 15, and 20
matchups per tweet ID for aversion against Democrats.

