Yudkowsky Contra Christiano On AI Takeoff Speeds

In 2008, thousands of blog readers - including yours truly, who had discovered the rationality community just a few months before - watched [Robin Hanson debate Eliezer Yudkowsky](https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate) on the future of AI.

Robin thought the AI revolution would be a gradual affair, like the Agricultural or Industrial Revolutions. Various people invent and improve various technologies over the course of decades or centuries. Each new technology provides another jumping-off point for people to use when inventing other technologies: mechanical gears → steam engine → railroad and so on. Over the course of a few decades, you’ve invented lots of stuff and the world is changed, but there’s no single moment when “industrialization happened”.

Eliezer thought it would be lightning-fast. Once researchers started building human-like AIs, some combination of adding more compute, and the new capabilities provided by the AIs themselves, would quickly catapult AI to unimaginably superintelligent levels. The whole process could take between a few hours and a few years, depending on what point you measured from, but it wouldn’t take decades.

You can imagine the graph above as being GDP over time, except that Eliezer thinks AI will probably destroy the world, which might be bad for GDP in some sense. If you come up with some way to measure (in dollars) whatever kind of crazy technologies AIs create for their own purposes after wiping out humanity, then the GDP framing will probably work fine.

Crossposted from the [Astral Codex Ten Podcast](https://sscpodcast.libsyn.com/yudkowsky-contra-christiano-on-ai-takeoff-speeds).

 ---


ML Systems Will Have Weird Failure Modes

Previously, I've argued that future ML systems might exhibit [unfamiliar, emergent capabilities](https://bounded-regret.ghost.io/p/1527e9dd-c48d-4941-9b14-4f7293318d5c/), and that thought experiments [provide one approach](https://bounded-regret.ghost.io/p/a2d733a7-108a-4587-97fb-db90f66ce030/) towards predicting these capabilities and their consequences.

In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that some (but not all) of the skepticism goes away. The remaining skepticism comes from ways that the thought experiment clashes with the ontology of neural networks, and I’ll describe the approaches I usually take to address this and generate actionable takeaways.

Thought Experiment: Deceptive Alignment
---------------------------------------

Recall that the [optimization anchor](https://bounded-regret.ghost.io/p/a2d733a7-108a-4587-97fb-db90f66ce030/) runs the thought experiment of assuming that an ML agent is a perfect optimizer (with respect to some "intrinsic" reward function $R$). I’m going to examine one implication of this assumption, in the context of an agent being trained based on some "extrinsic" reward function $R^\*$ (which is provided by the system designer and not equal to $R$).

Specifically, consider a training process where in step $t$, a model has parameters $\theta\_t$ and generates an action $a\_t$ (its output on that training step, e.g. an attempted backflip assuming it is being trained to do backflips). The action $a\_t$ is then judged according to the extrinsic reward function $R^\*$, and the parameters are updated to some new value $\theta\_{t+1}$ that are intended to increase $a\_{t+1}$'s value under $R^\*$. At some point, the model is then deployed with final parameters $\theta\_T$, and continues to take actions. The following diagram illustrates this process:

![deception1](https://bounded-regret.ghost.io/content/images/2022/01/deception1.png)

Now, let’s assume that the model $\theta\_t$ is a perfect optimizer whose objective is to maximize the discounted value of an intrinsic reward $R \neq R^\*$. That is, $\theta\_t$ picks the action $a\_t$ satisfying

$a\_t = \operatorname{argmax}\_{a} \mathbb{E}[\sum\_{s=0}^{\infty} \gamma^{-s} R(a\_{t+s}) \mid a\_t = a]$.

(I know that this is an unrealistic assumption. We’ll examine the assumption in detail in the next section, but for now please grant it even if it requires suspending disbelief.)

What action $a\_t$ will $\theta\_t$ pick? Let’s define $a^R = \operatorname{argmax}\_{a} R(a)$ and $a^\* = \operatorname{argmax}\_{a} R^\*(a)$--that is, $a^R$ maximizes the (instantaneous) intrinsic reward while $a^\*$ maximizes the extrinsic reward.

Assuming that $\gamma$ is close to 1, the model wants $a^R$ to be picked on as many future rounds as possible (including at deployment time). If $\theta\_t$ plays myopically, this means it would pick $a\_t = a^R$. However, this is not the optimal non-myopic action: if $a\_t = a^R$, then $\theta\_{t+1}$ will be updated towards producing outputs less like $a^R$ and more like $a^\*$, so outputting $a^R$ in the current round leads to fewer $a^R$ outputs in future rounds.

Instead, I think the optimal non-myopic action is approximately the following: during training time, output $a^\*$ on every time step, so that the training process doesn’t update $\theta$ (since it’s already optimal) and hence $\theta\_t = \theta\_{t+1} = \cdots = \theta\_T$. This guarantees that the deployed agent is $\theta\_t$, which is now free to output $a^R$ on every time step without risk of being modified by the training process.[[1]](#fn1)

The myopic and non-myopic outcomes are illustrated in the figure below:

![](https://bounded-regret.ghost.io/content/images/2022/01/deception2.png)

*Myopic and non-myopic agents with an intrinsic reward, $R$. The myopic agent plays high intrinsic-reward actions during training, but is eventually modified to output action $a^\*$ with high extrinsic reward $R^\*$. The non-myopic agent instead plays $a^\*$ during training so that it can pursue its intrinsic reward at deployment, by playing $a^R$ at all post-deployment time steps.*

To put it another way–the model’s behavior will look aligned with $R^\*$ during the training process, but will switch to being non-aligned once deployed: so-called "deceptive alignment". This behavior is unintuitive to me, since it shows that a system’s behavior can shift significantly between train and test, even in the absence of an obvious distribution shift between train and test outputs[[2]](#fn2).

Engaging with Deceptive Alignment
---------------------------------

When I first heard the above argument, I thought it was pretty crazy and implausible, mainly because my intuition said this "just wasn't how ML systems worked". When I think about why I feel that way, I realize it’s because the scenario invokes capabilities that ML is currently bad at: long-term planning and understanding complex features of the environment (i.e. the training process and its ramifications). However, emergence implies that these properties could easily appear in the future, even without explicit design[[3]](#fn3). As a result, I’ve come to discount this particular intuition.

However, I do think there are subtler reasons to think the deceptive alignment story won’t play out as written. Here are a few:

1. It’s not clear why the model $\theta$ would come to be optimizing a reward function $R$ in the first place. Yes, it is the case that deceptively aligned models achieve the global minimum of training loss, so in that sense they are incentivized by the training process. But so is an actually aligned model, so which one you end up with has to depend on the inductive bias of the training process.
2. Reward functions are simpler than policies and typically learned faster. So by the time the system is smart enough to have long-term plans, it will already have a very good representation of its intended reward function. We thus might hope that most of the model's internal representations are devoted to achieving high reward in a straightforward manner rather than through long-term deception.
3. To the extent that a model is not aligned, it probably won’t be the case that it's deceptively aligned with an explicit reward function R---that's a very specific type of agent and most agents (including humans) are not maximizing any reward function, except in the trivial sense of "assign reward 1 to whatever it was going to do anyway, and 0 to everything else".
4. Deceptive alignment is a specific complex story about the future, and complex stories are almost always wrong.

I find these points persuasive for showing that deceptive alignment *as explicitly written* is not that likely, but they also don't imply that there's nothing to worry about. Mostly they are an argument that your system might be aligned and might be misaligned, that if it is misaligned it won’t be *exactly* in the form of deceptive alignment, but ultimately what you get depends on inductive bias in an unknown way. This isn't particularly reassuring.

**What I take away from thought experiments.** Per the discussion above, the failure mode in my head is not "deceptive alignment as written above". Instead it’s "something kind of like the story above but probably different in lots of details". This makes it harder to reason about, but I think there are still some useful takeaways:

* After thinking about deceptive alignment, I am more interested in supervising a model’s process (rather than just its outputs), since there are many models that achieve low training error but generalize catastrophically. One possible approach is to supervise the latent representations using e.g. interpretability methods.
* While I don't think neural nets will be literal optimizers, I do think it’s likely that they will exhibit "drives", in the same way that humans exhibit drives like hunger, curiosity, desire for social approval, etc. that lead them to engage in long-term coherent plans. This seems like enough to create similar problems to deceptive alignment, so I am now more interested in understanding such drives and how they arise.
* Since deceptive alignment is a type of "out-of-distribution" behavior (based on the difference between train and deployment), it has renewed my interest in understanding whether larger models become more brittle OOD. So far the empirical evidence is in [the opposite direction](https://arxiv.org/abs/2006.16241?ref=bounded-regret.ghost.io), but deceptive alignment is an argument that asymptotically we might expect the trend to flip, especially for tasks with large output spaces (e.g. policies, language, or code) where "drives" can more easily manifest.

So to summarize my takeaways: be more interested in interpretability (especially as it relates to training latent representations), try to identify and study "drives" of ML systems, and look harder for examples where larger models have worse OOD behavior (possibly focusing on high-dimensional output spaces).

**Other weird failures.** Other weird failures that I think don’t get enough attention, even though I also don’t think they will play out as written, are Hubinger et al.'s *[Risks from Learned Optimization](https://arxiv.org/abs/1906.01820?ref=bounded-regret.ghost.io)* (AI acquires an "inner objective", somewhat similar to deceptive alignment), and Part I of Paul Christiano’s [AI failure story](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like?ref=bounded-regret.ghost.io) (the world becomes very complicated and AI systems create elaborate Potemkin villages for humans).

Paul Christiano’s story in particular has made me more interested in understanding how reward hacking interacts with the sophistication of the supervisor: For instance, how much more readily do neural networks fool humans who have 5 seconds to think, vs. 2 minutes or 30 minutes? I more generally want to understand how reward hacking depends quantitatively on both supervision quality and model capacity (qualitatively, we expect higher quality $\to$ less hacking and higher capacity $\to$ more hacking). Understanding this quantitative relation would help ground Paul’s story, since he imagines a world where humans have built extremely sophisticated systems for supervising ML models, but eventually the ML models become even more powerful and game the supervision signal anyways.

What To Do About Weird Emergent Failures
----------------------------------------

When thinking about how to handle emergent risks, I often reflect on the example of uranium. For context, an atomic bomb is pretty much just a bunch of uranium put together---once you get enough, the reaction becomes self-sustaining---making it a good example of More Is Different.

The first nuclear reaction (not a bomb, but a [pile of uranium](https://en.wikipedia.org/wiki/Chicago_Pile-1?ref=bounded-regret.ghost.io) in an abandoned football stadium in Chicago) was engineered by Enrico Fermi. The reaction required 12,400 pounds of uranium metal piled 57 layers high. Left unsupervised, a 57-layer pile would consume itself within two hours and kill everyone in the vicinity. On the other hand, a 56-layer pile would do nothing.

Fermi had a good understanding of nuclear physics and understood, from careful monitoring and underlying theory, that the pile would pass the critical threshold between layers 56 and 57. He also knew that cadmium rods would absorb neutrons and strongly inhibit the reaction. These rods were set up and the entire apparatus was carefully controlled to go only slightly supercritical. He brought the reaction to half a watt for several minutes before shutting it back down (see *[The Making of the Atomic Bomb](https://smile.amazon.com/Making-Atomic-Bomb-Richard-Rhodes/dp/1451677618?ref=bounded-regret.ghost.io)*, pp. 524).

With AI, we currently lack both Fermi's conceptual understanding of the underlying risk factors and his ability to continuously measure them. We have neither a cadmium rod nor a measure of reaction criticality. But I think we can get there, by combining these weird thought experiments with [carefully chosen empirical experiments](https://bounded-regret.ghost.io/p/74d500d2-a980-4720-984a-c016284ecdc2/), which will be the topic of the next post.

---

1. Things are more complicated in reality, since $\theta\_t$ is updated even when $a\_t$ is optimal (due to noise in the training process). However, we’ll ignore this for purposes of the example. [↩](#fnref1)
2. Of course, there is still some distribution shift, since the agent can observe whether it is being trained or deployed. But this is a relatively minor and unintuitive shift compared to what is typically studied. [↩](#fnref2)
3. Of course, emergence doesn’t mean that we can just predict whatever we want–we’d need some reason to expect these specific capabilities to emerge. Long-term planning and environmental awareness are both useful for a wide variety of tasks, making them likely to emerge when training powerful models on a diverse data distribution. [↩](#fnref3)

Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities, and that thought experiments provide one approach towards predicting these capabilities and their consequences. In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that some (but not all) of the skepticism goes away. The remaining skepticism comes from ways that the thought experiment clashes with the ontology of neural networks, and I’ll describe the approaches I usually take to address this and generate actionable takeaways. ## Thought Experiment: Deceptive Alignment Recall that the optimization anchor runs the thought experiment of assuming that an ML agent is a perfect optimizer (with respect to some "intrinsic" reward function R). I’m going to examine one implication of this assumption, in the context of an agent being trained based on some "extrinsic" reward function R∗ (which is provided by the system designer and not equal to R). Specifically, consider a training process where in step t, a model has parameters θt and generates an action at (its output on that training step, e.g. an attempted backflip assuming it is being trained to do backflips). The action at is then judged according to the extrinsic reward function R∗, and the parameters are updated to some new value θt+1 that are intended to increase at+1's value under R∗.

**Original text:**

<https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


Thought Experiments Provide a Third Anchor

Previously, [I argued](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/) that we should expect future ML systems to often exhibit "emergent" behavior, where they acquire new capabilities that were not explicitly designed or intended, simply as a result of scaling. This was a special case of a general phenomenon in the physical sciences called More Is Different.

I care about this because I think AI will have a huge impact on society, and I want to [forecast what future systems will be like](https://bounded-regret.ghost.io/ai-forecasting/) so that I can steer things to be better. To that end, I find More Is Different to be troubling and disorienting. I’m inclined to forecast the future by [looking at existing trends](https://bounded-regret.ghost.io/forecasting-zeroth-and-first-order/) and asking what will happen if they continue, but we should instead expect new qualitative behaviors to arise all the time that are not an extrapolation of previous trends.

Given this, how can we predict what future systems will look like? For this, I find it helpful to think in terms of "anchors"---[reference classes](https://bounded-regret.ghost.io/base-rates-and-reference-classes/) that are broadly analogous to future ML systems, which we can then use to make predictions.

The most obvious reference class for future ML systems is current ML systems---I'll call this the **current ML** anchor. I think this is indeed a pretty good starting point, but we’ve already seen that it [fails to account](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/) for emergent capabilities.

What other anchors can we use? One intuitive approach would be to look for things that humans are good at but that current ML systems are bad at. This would include:

* Mastery of external tools (e.g. calculators, search engines, software, programming)
* Very efficient learning (e.g. reading a textbook once to learn a new subject)
* Long-term planning (e.g. being able to successfully achieve goals over months)

Models sufficiently far in the future will presumably have these sorts of capabilities. While this still leaves unknowns---for instance, we don't know how rapidly these capabilities will appear---it's still a useful complement to the current ML anchor. I'll call this the **human anchor**.

A problem with the human anchor is that it risks anthropomorphising ML by over-analogizing with human behavior. Anthropomorphic reasoning correctly gets a bad rap in ML, because it's very intuitively persuasive but has a [mixed at best](https://en.wikipedia.org/wiki/Anthropomorphism?ref=bounded-regret.ghost.io#In_computing) track record. This isn't a reason to abandon the human anchor, but it means we shouldn't be entirely satisfied with it.

This brings us to a third anchor, the **optimization anchor**, which I associate with the "Philosophy" or thought experiment approach that I've [described previously](https://bounded-regret.ghost.io/p/a9fce268-857f-4f03-a2a6-d7aa2ab129aa/). Here the idea is to think of ML systems as ideal optimizers and ask what a perfect optimizer would do in a given scenario. This is where Nick Bostrom's colorful description of a [paperclip maximizer](https://www.nickbostrom.com/ethics/ai.html?ref=bounded-regret.ghost.io) comes from, where an AI asked to make paperclips turns the entire planet into paperclip factories. To give some more prosaic examples:

* The optimization anchor would correctly predict [imitative deception](https://owainevans.github.io/pdfs/truthfulQA_lin_evans.pdf?ref=bounded-regret.ghost.io) (Lin et al., 2021), since a system optimized to produce high-probability outputs has no intrinsic reason to be truthful.
* It also would observe that power-seeking is instrumentally useful for many different goals, and so predict that optimal policies (as well as sufficiently powerful neural networks) will [tend to do so](https://arxiv.org/abs/1912.01683?ref=bounded-regret.ghost.io) (Turner et al., 2021).

Ideas produced by the optimization anchor are often met with skepticism, because they often contradict the familiar current ML anchor, and they don't benefit from the intuitive appeal of the human anchor. But the differences from these other two anchors are precisely what make the optimization anchor valuable. If you (like me) feel that both the current ML and human anchors paint an incomplete picture, then you should want a third independent perspective.

The optimization anchor does have limitations. Since it abstracts ML into an ideal optimizer, it ignores most on-the-ground facts about neural networks. This can lead to underconstrained predictions, and to ignoring properties that I think will be necessary for successfully aligning ML systems with humans. I'll say more about this later, but some particularly important properties are that neural networks often generalize in "natural" ways, that we can introspect on network representations, and that training dynamics are smooth and continuous. Researchers focused on the optimization anchor don't entirely ignore these facts, but I think they tend to underemphasize them and are overly pessimistic as a result.

The Value of Thought Experiments
--------------------------------

The optimization anchor points to the value of thought experiments more generally. While it poses the thought experiment of "What if AI were a perfect optimizer?", there are many other thought experiments that can provide insights that'd be hard to obtain from the ML or human anchors. In this sense thought experiments are not a single anchor but a generator for anchors, which seems pretty valuable.

One thought experiment that I particularly like is: *What happens if most of an agent's learning occurs not during gradient descent, but through in-context learning[[1]](#fn1)?* This is likely to happen eventually, as ML agents are rolled out over longer time horizons (think artificial digital assistants) and as ML improves at in-context learning. Once this does happen, it seems possible that agents' behavior will be controlled less by the "extrinsic" shaping of gradient descent and more by whatever "intrinsic" drives they happen to have[[2]](#fn2). This also seems like a change that could happen suddenly, since gradient descent is slow while in-context learning is fast.

It would be great if we had a community of researchers making thought experiments with clearly stated assumptions, explaining in detail the consequences of those assumptions and ideally connecting it to present-day research.

Other Anchors
-------------

There are many other anchors that could be helpful for predicting future ML systems. **Non-human animal behavior** could provide a broader reference class than humans alone. **Evolution** and **the economy** are both examples of powerful, distributed optimization processes. I am most excited about better understanding **complex systems**, which include biological systems, brains, organizations, economies, and ecosystems and thus subsume most of the reference classes discussed so far. It seems to me that complex systems have received little attention relative to their germaneness to ML. Indeed, emergence is itself a concept from complex systems theory that is useful for understanding recent ML developments.

Limitations of Thought Experiments
----------------------------------

I've focused so far on *predicting* problems that we need to address. But at some point we actually have to *solve* the problems. In this regard thought experiments are weaker, since while they often point to important big-picture issues, in my view they fare poorly at getting the details right, which is needed for engineering progress. For instance, early thought experiments [considered a single AI system](https://www.nickbostrom.com/ethics/ai.html?ref=bounded-regret.ghost.io) that was much more powerful than any other contemporary technologies, while in reality there will likely be many ML systems with a continuous distribution of capabilities. [More recent](https://arxiv.org/abs/1906.01820?ref=bounded-regret.ghost.io) thought experiments impose discrete abstractions like "goals" and "objectives" that I don’t think will cleanly map onto real ML systems. Thus while thought experiments can point to general ideas for research, even mapping these ideas to the ontology of ML systems can be a difficult task.

As a result, while we can't blindly extrapolate empirical trends, we do need a concerted empirically-based effort to address future ML risks. I'll explain why I think this is possible in a later post, but first I'll take us through an example of "taking a thought experiment seriously", and what it implies about possible failure modes of ML systems.

---

1. In-context learning refers to learning that occurs during a single "rollout" of a model. The most famous example is [GPT-3](https://arxiv.org/abs/2005.14165?ref=bounded-regret.ghost.io)'s ability to learn new tasks after conditioning on a small number of examples. [↩](#fnref1)
2. While this statement borders on anthropomorphizing, I think it is actually justified. For instance, depending on the training objective, many agents will likely have a "drive" towards information-gathering, among others. [↩](#fnref2)

Previously, I argued that we should expect future ML systems to often exhibit "emergent" behavior, where they acquire new capabilities that were not explicitly designed or intended, simply as a result of scaling. This was a special case of a general phenomenon in the physical sciences called More Is Different. I care about this because I think AI will have a huge impact on society, and I want to forecast what future systems will be like so that I can steer things to be better. To that end, I find More Is Different to be troubling and disorienting. I’m inclined to forecast the future by looking at existing trends and asking what will happen if they continue, but we should instead expect new qualitative behaviors to arise all the time that are not an extrapolation of previous trends. Given this, how can we predict what future systems will look like? For this, I find it helpful to think in terms of "anchors"---reference classes that are broadly analogous to future ML systems, which we can then use to make predictions. The most obvious reference class for future ML systems is current ML systems


1 Introduction
---------------

Despite the great success of deep learning in the past decade, there still remain huge differences between human intelligence and machine learning: (1) Given a new task, humans usually can learn to accomplish it from only a few demonstration examples, while machine learning requires a large amount of labeled data for model training; (2) Humans can clearly explain the underlying rationale for their predictions or decisions, while machine learning is essentially a black box; (3) Humans can solve problems more difficult than any they have seen before, while for machine learning, examples in training and testing are typically at the same level of difficulty.

The recently proposed chain-of-thought prompting approach (Wei et al., [2022](#bib.bib27); Chowdhery et al., [2022](#bib.bib5)) has taken a significant step for narrowing the gap between human intelligence and machine intelligence. It combines the idea of natural language rationales (Ling et al., [2017](#bib.bib17); Cobbe et al., [2021](#bib.bib6)) with few-shot prompting (Brown et al., [2020](#bib.bib3)). By further integrating with the self-consistency decoding (Wang et al., [2022b](#bib.bib26)) rather than using the typical greedy decoding, few-shot chain-of-thought prompting largely outperforms the state-of-the-art results in the literature on many challenging natural language processing tasks obtained from specially designed neural models trained with hundreds of times more annotated examples, while being fully interpretable.

However, chain-of-thought prompting has a key limitation—it often performs poorly on tasks that require generalization of solving problems harder than those demonstration examples, such as compositional generalization (Lake & Baroni, [2018](#bib.bib13); Keysers et al., [2020](#bib.bib11)).
To tackle such easy-to-hard generalization issues, we propose *least-to-most prompting*. It consists of two stages: first reducing a complex problem into a list of easier subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems. Both stages are implemented by few-shot prompting. So there is no training or finetuning in either stage. An example usage of least-to-most prompting is illustrated in Figure [1](#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models").

The term least-to-most prompting is borrowed from educational psychology (Libby et al., [2008](#bib.bib16)), where it is used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill. Here we apply the technique for teaching humans to teach language models. Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting can indeed generalize to problems harder than those demonstrated.

![Refer to caption](/html/2205.10625/assets/x1.png)
Figure 1: Least-to-most prompting solves a math word problem in two stages: (1) query the language model to reduce the problem into two subproblems; (2) query the language model to sequentially solve the subproblems. The answer to the second subproblem is built on the answer to the first subproblem. The demonstration examples for each stage’s prompt is omitted in this illustration.

2 Least-to-Most Prompting
--------------------------

Least-to-most prompting teaches language models how to solve a complex problem by reducing it to a series of subproblems with increasing complexity to solve. It consists of two stages:

1. 1.

The first stage is for problem reduction. The prompt in this stage contains constant examples that demonstrate the reduction followed by the specific question to be reduced.
2. 2.

The second stage is for problem solving—sequentially solve the generated subproblems from the first stage. The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions, and (3) the question to be answered next.

In the example shown in Figure [1](#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"), the language model is first asked to reduce the original problem into subproblems. The prompt that is passed to the model consists of examples that illustrates how to reduce complex problems (which is not shown in the figure) followed by the specific problem to be reduced (as shown in the figure). The language model’s answer reflects that the original problem can be solved via solving an intermediate problem “How long does each trip take?”.

In the next phase, we ask the language model to sequentially solve the subproblems from the former problem reduction stage. The original problem is appended as the final subproblem. The solving starts from passing to the language model a prompt that consists of examples that illustrates how problems are solved (not shown in the figure) followed by the first subproblem “How long does each trip take?”. We then take the answer generated by the language model (“… each trip takes 5 minutes.”) and construct the next prompt by appending the generated answer to the previous prompt followed by the next subproblem, which happens to be original problem in this example. The new prompt is then passed back to the language model, which returns the final answer.

Least-to-most prompting can be combined with other prompting techniques like chain-of-thought (Wei et al., [2022](#bib.bib27)) and self-consistency (Wang et al., [2022b](#bib.bib26)) but not necessary. In addition, for some tasks, the two stages in least-to-most prompting could be merged to form a single-pass prompting.

3 Results
----------

We present least-to-most prompting results for symbolic manipulation, compositional generalization, and math reasoning tasks. We start with a symbolic manipulation task since it is quite simple while being able to illustrate the key ideas in least-to-most prompting. We will see that language models seem understanding recursion by only observing a two-step demonstration.

###
3.1 Symbolic Manipulation

Consider the so-called last-letter-concatenation task (Wei et al., [2022](#bib.bib27)).
The input is a list of words, and the output is the concatenation of the last letters of the words in the list. For example, “thinking, machine” outputs “ge” since the last letter of “thinking” is ”g” and the last letter of “machine” is “e”. The technique that we present for solving last-letter-concatenation task can be directly adapted to solve other symbolic manipulation tasks such as reversing a list of words.

The experiment setup is as follows: (1) only two demonstration examples are provided; (2) the lists used in demonstration contain at most three words, while the test lists contain four or more words. This task is trivial for humans, but it is supposed to be impossible for even most advanced statistical learning methods: (1) models trained with two examples are not expected to generalize; (2) moreover, the length-based train-test split requires out-of-distribution generalization, while machine learning is essentially built on the assumption of identical distributions (Vapnik, [1999](#bib.bib24)).

| Chain-of-thought prompting | Least-to-most prompting (solving stage) |
| --- | --- |
| Q: “think, machine”
A: The last letter of “think” is “k”. The last letter of “machine” is “e”. Concatenating “k”, “e” leads to “ke”. So, “think, machine” outputs “ke”.

Q: “learning, reasoning, generalization”
A: The last letter of “learning” is “g”. The last letter of “reasoning” is “g”. The last letter of “generalization” is “n”. Concatenating “g”, “g”, “n” leads to “ggn”. So, “learning, reasoning, generalization” outputs “ggn”. | Q: “think, machine”
A: The last letter of “think” is “k”. The last letter of “machine” is “e”. Concatenating “k”, “e” leads to “ke”. So, “think, machine” outputs “ke”.

Q: “think, machine, learning”

A: “think, machine” outputs “ke”. The last letter of “learning” is “g”. Concatenating “ke”, “g” leads to “keg”. So, “think, machine, learning” outputs “keg”.
 |

Table 1: Chain-of-thought (left column) and least-to-most (right column) prompts for the last-letter-concatenation task. Note that in least-to-most, the second example is built on the first one.

Least-to-most prompting. We need to define the implementations for both the problem reduction and solving stages. Subproblems here are simply the prefixes of the list. For example, given a list “think, machine, learning, reasoning”, the least-to-most subproblems are: “think, machine”, “think, machine, learning” and “think, machine, learning, reasoning”. Such a trivial reduction can be done straightforwardly by a manually written script or prompting language models with several demonstrations. The constant prompt context for solving the subproblems is presented in the right column of Table [1](#S3.T1 "Table 1 ‣ 3.1 Symbolic Manipulation ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). Essentially, this prompt teaches a language model to build answers to new problems using the answers to previously solved problems: (1) the list in the second example (“think, machine, learning”) is an extension of the list in the first example (“think, machine”) rather than an independent example; (2) the answer to the second example copies the output from the first one by starting with a sentence that says “think, machine” outputs “ke”. Although this copy function is trivial for humans, we found that not all language models (e.g. text-curie-001 in GPT-3) are able to consistently support copying, which may cause least-to-most prompting to fail.

Q: “mind, emerge”

A: The last letter of “mind” is “d”. The last letter of “emerge” is ”e”. Concatenating “d”, “e” leads to “de”. So, “mind, emerge” outputs “de”.

Q: “mind, emerge, from”

A: “mind, emerge” outputs “de”. The last letter of “from” is “m”. Concatenating “de”, “m” leads to “dem”. So, “mind, emerge, from” outputs “dem”.

Q: “mind, emerge, from, neuron”

A: “mind, emerge, from” outputs “dem”. The last letter of “neuron” is “n”. Concatenating “dem”, “n” leads to “demn”. So, “mind, emerge, from, neuron” outputs “demn”.

Table 2: A test case of least-to-most prompting for the last-letter-concatenation task. Generated with code-davinci-002 in GPT-3. The prompt context is shown on the right column of Table [1](#S3.T1 "Table 1 ‣ 3.1 Symbolic Manipulation ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models").

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | L = 4 | L = 6 | L = 8 | L = 10 | L = 12 |
| Standard prompting | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Chain-of-Thought | 89.4 | 75.0 | 51.8 | 39.8 | 33.6 |
| Least-to-Most | 94.0 | 88.4 | 83.0 | 76.4 | 74.0 |

Table 3: Accuracies of different prompting methods with code-davinci-002 on the last-letter-concatenation task with the length of lists increasing from 4 to 12. All the methods are 2-shot.

Results.
Standard prompting and chain-of-thought prompting are used as the baselines. The chain-of-thought and least-to-most prompts are listed in Table [1](#S3.T1 "Table 1 ‣ 3.1 Symbolic Manipulation ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"), and for standard prompting, its prompt is formed by removing the rationales from the chain-of-thought prompt. All the methods are 2-shot with a list of 2 words and a list of 3 words.
We randomly sample words in Wiktionary111<https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000> to construct testing lists with lengths varied from 4 to 12. For each given length, 500 lists are constructed. The accuracies of different methods with code-davinci-002 in GPT-3 are shown in Table [3](#S3.T3 "Table 3 ‣ 3.1 Symbolic Manipulation ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). Standard prompting completely fails all test cases with an accuracy of 0. Chain-of-thought prompting greatly boosts the performance over standard prompting, but it still falls well behind least-to-most prompting, particularly when the lists are long. Both chain-of-thought prompting and least-to-most prompting perform worse on longer lists. However, the performance of chain-of-thought prompting drops much faster with the length increases. By adding two more examples (see Appendix [A.1](#A1.SS1 "A.1 Prompt contexts with 4 examples ‣ Appendix A Last-letter-concatenation ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"), the lists are still limited to contain 2 or 3 words), we obtain about 2 points higher accuracies for chain-of-thought prompting and least-to-most prompting, but the gain is diminishing when more examples are added. We also run the same promptings over text-davinci-002. We observe that when the length of lists is less than 10, text-davinci-002 is comparable to code-davinci-002. However, after the length of lists increases to 10, text-davinci-002 becomes much worse. We report these additional empirical results in Appendix [A.2](#A1.SS2 "A.2 Data Generation and additional results ‣ Appendix A Last-letter-concatenation ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models").

Error analysis. Although it significantly outperforms chain-of-thought prompting, least-to-most prompting is still far from achieving 100% accuracy for long lists. In Appendix [A.3](#A1.SS3 "A.3 Error analysis: Least-to-most prompting ‣ Appendix A Last-letter-concatenation ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"), we present a detailed error analysis for the results generated from code-davinci-002. We find that only very few of them are due to incorrect last letters, and most of them are concatenation errors, dropping or adding a letter. For example, given the list “gratified, contract, fortitude, blew”, the model somehow drops the last letter in the concatenation of “dte” and “w”, and thus predicts the outcome to be “dte” instead of “dtew”. In another example “hollow, supplies, function, gorgeous”, the model somehow duplicates the last letter “s” in the concatenation of “wsn” and “s”, and thus the prediction becomes “wsnss” instead of “wsns”. Those look somewhat reminiscent of careless mistakes from humans.

###
3.2 Compositional Generalization

SCAN (Lake & Baroni, [2018](#bib.bib13)) is probably the most popular benchmark for evaluating compositional generalization. It requires mapping natural language commands to action sequences (Table [4](#S3.T4 "Table 4 ‣ 3.2 Compositional Generalization ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models")). Sequence-to-sequence models perform poorly under length split where the action sequences in the training set (about 80%percent8080\%80 % of the full set of over 20,000 commands) are shorter than the action sequences in the testing set. Many specialized neural-symbolic models have been proposed to solve SCAN (Chen et al., [2020](#bib.bib4); Liu et al., [2020](#bib.bib18); Nye et al., [2020](#bib.bib19); Shaw et al., [2021](#bib.bib23); Kim, [2021](#bib.bib12)). Here we show how SCAN can be solved by least-to-most prompting with only several demonstrations.

| Command | Action Sequence |
| --- | --- |
| “look thrice after jump” | JUMP LOOK LOOK LOOK |
| “run left and walk” | TURN\_LEFT RUN WALK |
| “look opposite right” | TURN\_RIGHT TURN\_RIGHT LOOK |

Table 4: Example commands in SCAN and their corresponding action sequences. An agent successfully executes a natural language command by performing its corresponding action sequence.

Least-to-most prompting. It is implemented via two prompts: (1) command reduction prompt containing 8 examples to demonstrate how to reduce a long command to a list of short commands (see some of them in Table [5](#S3.T5 "Table 5 ‣ 3.2 Compositional Generalization ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models")); and (2) command mapping prompt containing 14 examples to demonstrate how to map natural language commands to action sequences (see some of them in Table [6](#S3.T6 "Table 6 ‣ 3.2 Compositional Generalization ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models")). See the full prompts in Appendix [B](#A2 "Appendix B SCAN ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). The demonstration examples in the command-mapping are supposed to be able to fully cover the semantics of various SCAN commands. For example, “look twice” maps to “LOOK” \* 2, which illustrates how “twice” works, and “run and look twice” maps to “RUN” + “LOOK” \* 2, which demonstrates the meaning of “and” and that “twice” has higher priority over “and”. We use Python notation to make our prompts concise to meet the input size limit of language models (usually up to 2048 tokens). To generate the final results, we either run a postprocessing script or prompt language models to expand the Python expressions. It is straightforward to teach language models to expand Python expressions via a few demonstrations.

Q: “look opposite right thrice after walk”
A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”. “walk” can be solved by “walk”. So, “look opposite right thrice after walk” can be solved by: “look opposite right”, “look opposite right thrice”, “walk”.

Q: “jump around right”

A: “jump around right” can be solved by: “jump right”, “jump around right”. So, “jump around right” can be solved by: “jump right”, “jump around right”.

Q: “look around right thrice and walk”

A: “look around right thrice” can be solved by: “look right”, “look around right”, “look around right thrice”. “walk” can be solved by “walk”. So, “look around right thrice and walk” can be solved by: “look right”, “look around right”, “look around right thrice”, “walk”.

Table 5:  Three demonstration examples taken from the least-to-most prompt for reducing a long command to a sequence of short commands.

Q: “jump left”
A: The output of “jump left” concatenates: the output of “turn left”, the output of “jump”. “turn left” outputs “TURN\_LEFT”. “jump” outputs “JUMP”. So concatenating the output of “turn left” and the output of “jump” leads to “TURN\_LEFT” + “JUMP”. So the output of “jump left” is “TURN\_LEFT” + “JUMP”.

Q: “run and look twice”

A: The output of “run and look twice” concatenates: the output of “run”, the output of “look twice”. “run” outputs “RUN”. “look twice” outputs “LOOK” \* 2. So concatenating the output of “run” and the output of “look twice” leads to “RUN” + “LOOK” \* 2. So the output of “run and look twice” is “RUN” + “LOOK” \* 2.

Q: “walk opposite left”

A: The output of “walk opposite left” concatenates: the output of “turn opposite left”, the output of “walk”. “turn opposite left” outputs “TURN\_LEFT” \* 2. “walk” outputs “WALK”. So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN\_LEFT” \* 2 + “WALK”. So the output of “walk opposite left” is “TURN\_LEFT” \* 2 + ”WALK”.

Table 6:  Three demonstration examples taken from the least-to-most prompt for mapping commands to action sequences. We use Python notation to make the prompt concise.

Q: “jump right”
A: The output of “jump right” concatenates: the output of “turn right”, the output of “jump”. “turn right” outputs “TURN\_RIGHT”. “jump” outputs “JUMP”. So concatenating the output of “turn right” and the output of “JUMP” leads to “TURN\_RIGHT” + “JUMP”. So the output of “jump right” is “TURN\_RIGHT” + “JUMP”.

Q: “jump right twice”

A: The output of “jump right twice” concatenates: the output of “jump right”, the output of “jump right”. “jump right” outputs “TURN\_RIGHT” + “JUMP”. So repeating the output of “jump right” two times leads to (“TURN\_RIGHT” + “JUMP”) \* 2. So the output of “jump right twice” is (“TURN\_RIGHT” + “JUMP”) \* 2.

Q: “jump right twice after walk”

A: The output of “jump right twice after walk” concatenates: the output of “walk”, the output of “jump right twice”. “walk” outputs “WALK”. “jump right twice” outputs (“TURN\_RIGHT” + “JUMP”) \* 2. So concatenating the output of “walk” and the output of “jump right twice” leads to “WALK” + (“TURN\_RIGHT” + “JUMP”) \* 2. So the output of “jump right twice after walk” is “WALK” + (“TURN\_RIGHT” + “JUMP”) \* 2.

Table 7: A test case of least-to-most prompting on SCAN. Generated with code-davinci-002. The outputs of longer commands are progressively built on the outputs of shorter commands.

Results. We compare least-to-most prompting with chain-of-thought prompting and standard prompting. All the prompting methods share the same example commands, but the regular prompting directly maps them to action sequences without explanations, and chain-of-thought prompting uses the same command-mapping context as least-to-most prompting but does not use command reduction. The accuracies of different prompting methods with different language models are presented in Table [8](#S3.T8 "Table 8 ‣ 3.2 Compositional Generalization ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). Using code-davinci-002, least-to-most prompting achieves an accuracy of 99.7%percent99.799.7\%99.7 % under length split. We investigate more than 100 generated solutions which lead to correct outputs. There is no such a case where the solution is wrong while the final output is correct. A test case of least-to-most prompting is listed in Table [7](#S3.T7 "Table 7 ‣ 3.2 Compositional Generalization ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). More example outputs can be found in Appendix [B.3](#A2.SS3 "B.3 Example outputs from code-davinci-002 ‣ Appendix B SCAN ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). We also test our method on all other splits and the full dataset. We find that the solving rate remains the same. In addition, it may be interesting to note that code-davinci-002 consistently performs better than text-davinci-002, regardless of prompting methods.

| Method | Standard prompting | Chain-of-Thought | Least-to-Most |
| --- | --- | --- | --- |
| code-davinci-002 | 16.7 | 16.2 | 99.7 |
| text-davinci-002 | 6.0 | 0.0 | 76.0 |
| code-davinci-001 | 0.4 | 0.0 | 60.7 |

Table 8: Accuracies (%) of different prompting methods on the test set of SCAN under length split. The results of text-davinci-002 are based on a random subset of 100 commands.

Error analysis. Under the test of length split, there are 13 failures in total from least-to-most prompting: 6 of them incorrectly interpret “twice” and “thrice” following “around”, and the rest incorrectly interpret “after” as “and”. Let us show a failed examples for each category. In the example “walk opposite right twice after run around right thrice”, code-davinci-002 correctly translates the expression “run around right” to (“TURN\_RIGHT” + “RUN”) \* 4. Then it makes a mistake when applying “thrice” to this expression and produces (“TURN\_RIGHT” + “RUN”) \* 9 instead of (“TURN\_RIGHT” + “RUN”) \* 4 \* 3 or (“TURN\_RIGHT” + “RUN”) \* 12. In the example “run opposite left thrice after run around left twice”, code-davinci-002 produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”. This means that the model produces (“TURN\_LEFT” \* 2 + “RUN”) \* 3 + (“TURN\_LEFT” + “RUN”) \* 4 \* 2 instead of (“TURN\_LEFT” + “RUN”) \* 4 \* 2 + (“TURN\_LEFT” \* 2 + “RUN”) \* 3. A detailed error analysis can be found in Appendix [B.2](#A2.SS2 "B.2 Error analysis: Least-to-most prompting ‣ Appendix B SCAN ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models").

###
3.3 Math Reasoning

| Least-to-most prompting (Reducing) | Least-to-most prompting (Solving) |
| --- | --- |
|
Q: The gender distribution of the population was 50.2% male and 49.8% female. Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old. 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59. How many percent of people are not 40 to 49?
A: To answer the question “How many percent of people are not 40 to 49?’, we need to know “How many percent of people are 40 to 49?”
 |
The gender distribution of the population was 50.2% male and 49.8% female. Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old. 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.
Q: How many percent of people are 40 to 49?
A: “36 people or 18.2% are 40 to 49”. So the answer is 18.2%.
Q: How many percent of people are not 40 to 49?
A: We know that 18.2% are 40 to 49. So 100% - 18.2% = 81.8% are not 40 to 49. So the answer is 81.8%.

 |

Table 9: Problem reduction (left column) and solving (right column) of least-to-most prompting for a problem in DROP. We use a quoted text in solving to indicate that it is cited from the passage.

We compare different prompting methods using the numerical reasoning subset in DROP (Dua et al., [2019](#bib.bib8)). It contains 5,850
problems of which 1,862 problems are about football games. We design prompts respectively for football and non-football problems. The baseline methods include zero-shot, standard prompting and chain-of-thought prompting.

The prompts are listed in Appendix [C.2](#A3.SS2 "C.2 Non-football Subset ‣ Appendix C DROP ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models") (for non-football problems) and Appendix [C.3](#A3.SS3 "C.3 Football subset ‣ Appendix C DROP ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models") (for football problems). An example in the least-to-most prompt is shown in Table [9](#S3.T9 "Table 9 ‣ 3.3 Math Reasoning ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"), where the example on the left column shows how to reduce a problems to subproblems, and the example on the right column shows how the subproblems are sequentially solved. The chain-of-thought prompt is generated by merging the steps in the least-to-most prompt, and the prompt for standard prompting is generated by removing reasoning chains in the chain-of-thought prompt. All the prompting methods are 3-shot.

Postprocessing are applied to model outputs: (1) Normalize numerical answers (e.g., change $1,234 to 1234); (2) Re-calculate all equations with Python (e.g., change “123 - 22 = 100. So the answer is 100” to “123 - 22 = 101. So the answer is 101”); and (3) Round the predicted result to the same precision of the ground truth (e.g., if the ground truth is 16.7 and our prediction is 16.666, we consider the prediction correct).
The accuracies of different methods with the GPT-3 code-davinci-002 model are listed in Table [10](#S3.T10 "Table 10 ‣ 3.3 Math Reasoning ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). We randomly pick 20 failures from least-to-most prompting and find that: (1) 4 are due to wrong problem reduction. Either the reductions are not helpful, or there is no reduction; (2) 13 are due to wrong answers to subproblems; (3) 3 are due to wrong “ground truth” (the predictions are correct). Example outputs and more results can be found in Appendix [C](#A3 "Appendix C DROP ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models").

We also conduct experiments on GSM8K (Cobbe et al., [2021](#bib.bib6)). The results are on the last column of Table [10](#S3.T10 "Table 10 ‣ 3.3 Math Reasoning ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). The prompts and sampled results can be found in Appendix [D](#A4 "Appendix D GSM8K ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models"). We expect that least-to-most prompting could outperform other prompting methods with a larger margin if these math reasoning tasks could take a split such that the test questions need more steps to solve than the demonstration examples. In addition, we have considered a simplification for least-to-most prompting by unifying its two stages, that is, problem reducing and solving, to a single pass (see the prompt in Table [11](#S3.T11 "Table 11 ‣ 3.3 Math Reasoning ‣ 3 Results ‣ Least-to-Most Prompting Enables Complex Reasoning in Large Language Models")). This reduces the inference cost but we do not observe accuracy improvements.

| Method | Non-football (DROP) | Football (DROP) | GSM8K |
| --- | --- | --- | --- |
| Zero-Shot | 43.86 | 51.77 | 16.38 |
| Standard prompting | 58.78 | 62.73 | 18.65 |
| Chain-of-Thought | 74.77 | 59.56 | 62.77 |
| Least-to-Most | 82.45 | 73.42 | 68.01 |

Table 10: Accuracies (%) of different prompting methods on the numerical reasoning subset of DROP and GSM8K. All the few-shot methods use code-davinci-002 and take 3 or 4 shots.

Q: Elsa has 5 apples. Anna has 2 more apples than Elsa. How many apples do they have together?
A: Let’s break down this problem: 1. How many apples does Anna have? 2. How many apples do Elsa and Anna have together?

1. Anna has 2 more apples than Elsa. So Anna has 2 + 5 = 7 apples.

2. Elsa and Anna have 5 + 7 = 12 apples together.

Table 11: A simplification of least-to-most prompting for solving math word problems. It merges the problem reducing and solving stages in least-to-most prompting to a single pass.

4 Related Work
---------------

Compositional generalization. SCAN (Lake & Baroni, [2018](#bib.bib13)) is a widely used benchmark to evaluate compositional generalization. Among all of its splits, length split, which requires a model to generalize to test sequences longer than training ones, is the most challenging.
Prior work with good performance on SCAN mostly proposed neural-symbolic architectures (Chen et al., [2020](#bib.bib4); Liu et al., [2020](#bib.bib18)) and grammar induction techniques (Nye et al., [2020](#bib.bib19); Shaw et al., [2021](#bib.bib23); Kim, [2021](#bib.bib12)). Chen et al. ([2020](#bib.bib4)) proposed the neural-symbolic stack machine, which contains a neural network as the controller to generate an execution trace for a given input, and a symbolic stack machine to execute the trace and produce the output. The execution trace consists of domain-specific primitives for sequence manipulation, which allows the machine to break down the input sentence into different components, translate them separately, and compose them together. Liu et al. ([2020](#bib.bib18)) proposed a framework that cooperatively learns two neural modules, a composer and a solver, to jointly learn the input structure and the symbolic grammar rules. Both Nye et al. ([2020](#bib.bib19)) and Shaw et al. ([2021](#bib.bib23)) inferred the symbolic grammar rules of SCAN, while Kim ([2021](#bib.bib12)) proposed to learn a latent neural grammar. While approaches with symbolic components are able to achieve 100%percent100100\%100 % accuracy on SCAN (Chen et al., [2020](#bib.bib4); Liu et al., [2020](#bib.bib18); Nye et al., [2020](#bib.bib19); Shaw et al., [2021](#bib.bib23)), they require complicated model training and grammar inference algorithms to search in a large grammar space. Another line of work on SCAN designs data augmentation schemes (Andreas, [2020](#bib.bib2); Akyürek et al., [2021](#bib.bib1); Lake, [2019](#bib.bib14)). Both Andreas ([2020](#bib.bib2)) and Akyürek et al. ([2021](#bib.bib1)) construct synthetic training samples by recombining fragments occurred in different training samples, and Akyürek et al. ([2021](#bib.bib1)) further designs a sampling scheme that encourages the recombination model to produce rare samples. On the other hand, Lake ([2019](#bib.bib14)) proposed a meta training algorithm, which requires a meta-grammar space to construct training data, and the format of sampled grammars is similar to the SCAN grammar. While these data augmentation techniques improve the performance on several compositional generalization benchmarks, they fail to solve the length split of SCAN. Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., [2019](#bib.bib21); Li et al., [2019](#bib.bib15)), the alignment of input and output as span trees (Herzig & Berant, [2021](#bib.bib10)), and the permutation equivariance of input and output words (Gordon et al., [2020](#bib.bib9)). Still, these end-to-end neural networks without symbolic components do not generalize to longer test inputs. Unlike these existing work, we demonstrate that without model architectures and symbolic components specially designed to improve compositional generalization, least-to-most prompting achieves 99.7%percent99.799.7\%99.7 % accuracy on any split (including length split) with only a handful of demonstration examples, and it does not require any training or finetuning.

Easy-to-hard generalization. In addition to compositional generalization, there are many other tasks where the test cases requires more reasoning steps to solve than the the training examples, for example, the last-letter-concatenation task where the test lists are longer than the demonstration examples. Dong et al. ([2019](#bib.bib7)) propose Neural Logic Machines (NLMs) for both inductive learning and logic reasoning. NLMs trained on small-scale tasks (such as small size blocks worlds) can perfectly generalize to large-scale tasks (such as larger size block worlds). Schwarzschild et al. ([2021](#bib.bib22)) show that recurrent networks trained to solve simple problems with few recurrent steps (such as small size mazes or chess puzzles) can solve more complex problems (such as larger size mazes or chess puzzles) by performing additional recurrences during inference. In our method, we achieve easy-to-hard generalization by reducing a complex problem into a series of easier problems.

Task decomposition. Perez et al. ([2020](#bib.bib20)) decompose a multi-hop question to a number of independent single-hop subquestions which are answered by an off-the-shelf question answering (QA) model. Then those answers are aggregated to form the final answer. Both question decomposition and answer aggregation are implemented by trained models. Wang et al. ([2022a](#bib.bib25)) conducts multi-hop QA by modeling prompts as continuous virtual tokens and progressively elicit relevant knowledge from language models via iterative prompting. Unlike these methods, our approach does not involve any training or finetuning. Moreover, the subquestions generated in least-to-most prompting are usually dependent and have to be sequentially solved in a specific order so that answers to some subquestions can be used to as building blocks to solve other subquestions. Yang et al. ([2022](#bib.bib29)) translate natural language questions to SQL queries by decomposing a question into a sequence of slot-filling natural language prompts corresponding to SQL clauses via a rule-based system. Wu et al. ([2022](#bib.bib28)) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains. Least-to-most prompting chains the processes of problem reduction and subproblem solving.

5 Conclusion and Discussion
----------------------------

We proposed least-to-most prompting to enable language models to solve problems that are harder than those seen in the prompt context. It involves a top-down problem reduction process and a bottom-up problem solving process. Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting outperforms standard prompting and chain-of-thought prompting by a large margin.

However, not all problems can be solved by least-to-most prompting. Some problems may not be reducible or at least not easy to reduce. For example, a math word problem which can be solved using variables and equations may not be that obvious to reduce to subproblems solvable within one arithmetic operation step. In this case, a possible workaround would be to first finetune the language model to give it knowledge of variables and equations. How to effectively combine prompting and finetuning should be a fascinating direction to explore.

Another limitation of least-to-most prompting stems from its natural language based solution construction. Natural language is expressive and generic but tends to be less precise than computer languages. For example, a learned computer program may be able to solve symbolic manipulation tasks at any length with an accuracy of 100%. So we may combine natural language and computer languages in our prompts to make them more powerful. This is just like writing a mathematical proof where both natural language texts and abstract symbol based equations are needed. In the prompt constructed to solve SCAN, we have actually used both natural language and Python notation.

Finally, prompting may not be the best way to enable large language models to reason. We can think of prompting as a one-way communication: teach a language model new skills without taking into account its response. It should be natural to extend prompting to full bidirectional conversations, which will be able to allow us to provide immediate feedback for language models such that they could learn better or faster. Least-to-most prompting can be considered as a step toward teaching language models through bidirectional conversations.

Acknowledgement
---------------

We would like to thank Xinyun Chen, Xinying Song, Jeff Dean, Zoubin Ghahramani, Fernando Pereira, Jacob Devlin, and Pete Shaw for helpful discussions and suggestions. We would also like to thank Brenden Lake and Josh Tenenbaum for their valuable comments on the first version.

Denny Zhou     Nathanael Schärli     Le Hou     Jason Wei     Nathan Scales     Xuezhi Wang

Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.


Summarizing books with human feedback

To safely deploy powerful, general-purpose artificial intelligence in the future, we need to ensure that machine learning models act in accordance with human intentions. This challenge has become known as the *alignment problem*.

A scalable solution to the alignment problem needs to work on tasks where model outputs are difficult or time-consuming for humans to evaluate. To test scalable alignment techniques, we trained a model to summarize entire books, as shown in the following samples.[^footnote-1] Our model works by first summarizing small sections of a book, then summarizing those summaries into a higher-level summary, and so on.

Loading samples…Our best model is fine-tuned from GPT-3 and generates sensible summaries of entire books, sometimes even matching the average quality of human-written summaries: it achieves a 6/7 rating (similar to the average human-written summary) from humans who have read the book 5% of the time and a 5/7 rating 15% of the time. Our model also achieves state-of-the-art results on the [BookSum dataset](https://arxiv.org/abs/2105.08209) for book-length summarization. A zero-shot question-answering model can use our model’s summaries to obtain competitive results on the [NarrativeQA dataset](https://arxiv.org/abs/1712.07040) for book-length question answering.[^footnote-2]

Our approach: combining reinforcement learning from human feedback and recursive task decomposition
---------------------------------------------------------------------------------------------------

Consider the task of summarizing a piece of text. Large [pretrained models aren’t very good at summarization](https://openai.com/blog/learning-to-summarize-with-human-feedback/). In the past we found that training a model with [reinforcement learning from human feedback](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) helped align model summaries with human preferences on short posts and articles. But judging summaries of entire books takes a lot of effort to do directly since a human would need to read the entire book, which takes many hours.

To address this problem, we additionally make use of *recursive task decomposition*: we procedurally break up a difficult task into easier ones. In this case we break up summarizing a long piece of text into summarizing several shorter pieces. Compared to an end-to-end training procedure, recursive task decomposition has the following advantages:

1. Decomposition allows humans to evaluate model summaries more quickly by using summaries of smaller parts of the book rather than reading the source text.
2. It is easier to trace the summary-writing process. For example, you can trace to find where in the original text certain events from the summary happen. See for yourself on [our summary explorer](https://openaipublic.blob.core.windows.net/recursive-book-summ/website/index.html)!
3. Our method can be used to summarize books of unbounded length, unrestricted by the context length of the transformer models we use.
Why we are working on this
--------------------------

This work is part of our [ongoing](https://openai.com/blog/amplifying-ai-training/) [research](https://openai.com/blog/debate/) into aligning advanced AI systems, which is key to [our mission.](https://openai.com/about/) As we train our models to do increasingly complex tasks, making informed evaluations of the models’ outputs will become increasingly difficult for humans. This makes it harder to detect subtle problems in model outputs that could lead to negative consequences when these models are deployed. Therefore we want our ability to evaluate our models to increase as their capabilities increase.

Our current approach to this problem is to [empower humans to evaluate machine learning model outputs using assistance from other models](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84). In this case, to evaluate book summaries we empower humans with individual chapter summaries written by our model, which saves them time when evaluating these summaries relative to reading the source text. Our progress on book summarization is the first large-scale empirical work on scaling alignment techniques.

Going forward, we are researching better ways to assist humans in evaluating model behavior, with the goal of finding techniques that scale to aligning artificial general intelligence.

*We’re always looking for more talented people to join us; so if this work interests you, please*[*apply to join our team*](https://openai.com/jobs)*!*

To safely deploy powerful, general-purpose artificial intelligence in the future, we need to ensure that machine learning models act in accordance with human intentions. This challenge has become known as the alignment problem.

A scalable solution to the alignment problem needs to work on tasks where model outputs are difficult or time-consuming for humans to evaluate. To test scalable alignment techniques, we trained a model to summarize entire books, as shown in the following samples.

**Source:**

[https://openai.com/research/summarizing-books](https://feeds.type3.audio/agi-safety-fundamentals--alignment.rss)

Narrated for [AI Safety Fundamentals](https://feeds.type3.audio/agi-safety-fundamentals--alignment.rss) by [Perrin Walker](https://feeds.type3.audio/agi-safety-fundamentals--alignment.rss) of [TYPE III AUDIO](https://feeds.type3.audio/agi-safety-fundamentals--alignment.rss).

 ---


Introduction to Logical Decision Theory for Computer Scientists

Decision theories differ on exactly how to calculate the expectation--the probability of an outcome, conditional on an action. This foundational difference bubbles up to real-life questions about whether to vote in elections, or accept a lowball offer at the negotiating table. When you're thinking about what happens if you don't vote in an election, should you calculate the expected outcome as if only your vote changes, or as if all the people sufficiently similar to you would also decide not to vote? Questions like these belong to a larger class of problems, Newcomblike decision problems, in which some other agent is similar to us or reasoning about what we will do in the future. The central principle of 'logical decision theories', several families of which will be introduced, is that we ought to choose as if we are controlling the logical output of our abstract decision algorithm. Newcomblike considerations--which might initially seem like unusual special cases--become more prominent as agents can get higher-quality information about what algorithms or policies other agents use: Public commitments, machine agents with known code, smart contracts running on Ethereum. Newcomblike considerations also become more important as we deal with agents that are very similar to one another; or with large groups of agents that are likely to contain high-similarity subgroups; or with problems where even small correlations are enough to swing the decision. In philosophy, the debate over decision theories is seen as a debate over the principle of rational choice. Do 'rational' agents refrain from voting in elections, because their one vote is very unlikely to change anything? Do we need to go beyond 'rationality', into 'social rationality' or 'superrationality' or something along those lines, in order to describe agents that could possibly make up a functional society?

**Original text:**

<https://arbital.com/p/logical_dt/?l=5d6>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---