Machine Learning for Humans, Part 2.1: Supervised Learning

The two tasks of supervised learning: regression and classification. Linear regression, loss functions, and gradient descent.
-----------------------------------------------------------------------------------------------------------------------------

[![Vishal Maini](https://miro.medium.com/v2/resize:fill:88:88/1*7qOP1DnjjYcPKMxUG1592g.jpeg)](/@v_maini?source=post_page-----740383a2feab--------------------------------)[![Machine Learning for Humans](https://miro.medium.com/v2/resize:fill:48:48/1*gcWTWMYOzKt37HUnwvv1Sg.png)](https://medium.com/machine-learning-for-humans?source=post_page-----740383a2feab--------------------------------)Listen

Share

*How much money will we make by spending more dollars on digital advertising? Will this loan applicant pay back the loan or not? What’s going to happen to the stock market tomorrow?*

In supervised learning problems, we start with a data set containing **training examples** with associated correct **labels**. For example, when learning to classify handwritten digits, a supervised learning algorithm takes thousands of pictures of handwritten digits along with labels containing the correct number each image represents. The algorithm will then learn the relationship between the images and their associated numbers, and apply that learned relationship to classify completely new images (without labels) that the machine hasn’t seen before. This is how you’re able to deposit a check by taking a picture with your phone!

To illustrate how supervised learning works, let’s examine the problem of **predicting annual income** based on the number of years of higher education someone has completed. Expressed more formally, we’d like to build a model that approximates the relationship ***f*** between the number of years of higher education **X** and corresponding annual income **Y**.

![]()One method for predicting income would be to create a rigid rules-based model for how income and education are related. For example: *“I’d estimate that for every additional year of higher education, annual income increases by $5,000.”*

You could come up with a more complex model by including some rules about degree type, years of work experience, school tiers, etc. For example: *“If they completed a Bachelor’s degree or higher, give the income estimate a 1.5x multiplier.”*

But this kind of explicit rules-based programming doesn’t work well with complex data. Imagine trying to design an image classification algorithm made of if-then statements describing the combinations of pixel brightnesses that should be labeled “cat” or “not cat”.

Supervised machine learning solves this problem by getting the computer to *do the work for you.* By identifying patterns in the data, the machine is able to form heuristics. The primary difference between this and human learning is that machine learning runs on computer hardware and is best understood through the lens of computer science and statistics, whereas human pattern-matching happens in a biological brain (while accomplishing the same goals).

In supervised learning, the machine attempts to learn the relationship between income and education *from scratch*, by running **labeled training data** through a **learning algorithm.** This learned function can be used to estimate the income of people whose income Y is unknown, as long as we have years of education X as inputs. In other words, we can apply our model to the **unlabeled test data** to estimate Y.

The goal of supervised learning is to **predict Y as accurately as possible** when given new examples where X is known and Y is unknown. In what follows we’ll explore several of the most common approaches to doing so.

The two tasks of supervised learning: regression and classification
===================================================================

The rest of this section will focus on regression. In [Part 2.2](/@v_maini/supervised-learning-2-5c1c23f3560d) we’ll dive deeper into classification methods.

Regression: predicting a continuous value
=========================================

Regression predicts a **continuous** **target variable Y**. It allows you to estimate a value, such as housing prices or human lifespan, based on input data X.

Here, **target variable** means the unknown variable we care about predicting, and **continuous** means there aren’t gaps (discontinuities) in the value that Y can take on. A person’s weight and height are continuous values. **Discrete** variables, on the other hand, can only take on a finite number of values — for example, the number of kids somebody has is a discrete variable.

Predicting income is a classic regression problem. Your **input data X** includes all relevant information about individuals in the data set that can be used to predict income, such as years of education, years of work experience, job title, or zip code. These attributes are called **features**, which can be **numerical** (e.g. years of work experience) or **categorical** (e.g. job title or field of study).

You’ll want as many training observations as possible relating these features to the target output Y, so that your model can learn the relationship *f* between X and Y.

The data is split into a **training data set** and a **test data set**. The training set has labels, so your model can learn from these labeled examples. The test set does *not* have labels, i.e. you don’t yet know the value you’re trying to predict. It’s important that your model can generalize to situations it hasn’t encountered before so that it can perform well on the test data.

In our trivially simple 2D example, this could take the form of a .csv file where each row contains a person’s education level and income. Add more columns with more features and you’ll have a more complex, but possibly more accurate, model.

![]()So how do we solve these problems?
==================================

How do we build models that make accurate, useful predictions in the real world? We do so by using **supervised learning algorithms.**

**Now let’s get to the fun part: getting to know the algorithms.** We’ll explore some of the ways to approach regression and classification and illustrate key machine learning concepts throughout.

Linear regression (ordinary least squares)
==========================================

*“Draw the line. Yes, this counts as machine learning.”*

First, we’ll focus on solving the income prediction problem with linear regression, since linear models don’t work well with image recognition tasks (this is the domain of deep learning, which we’ll explore later).

We have our data set X*,* and corresponding target values Y. The goal of **ordinary least squares (OLS)** regression is to learn a linear model that we can use to predict a new *y* given a previously unseen *x* with as little error as possible. We want to guess how much income someone earns based on how many years of education they received.

![]()Linear regression is a **parametric method**, which means it makes an assumption about the form of the function relating X and Y (we’ll cover examples of non-parametric methods later). Our model will be a function that predicts *ŷ* given a specific *x*:

![]()In this case, we make the explicit assumption that there is a **linear relationship** between X and Y — that is, for each one-unit increase in *X*, we see a constant increase (or decrease) in Y.*β0* is the y-intercept and *β1* is the slope of our line, i.e. how much income increases (or decreases) with one additional year of education.

Our goal is to learn the **model parameters** (in this case, *β0* and *β1*) that minimize error in the model’s predictions.

To find the best parameters:

> 1. Define a **cost function**, or **loss function**, that measures how inaccurate our model’s predictions are.
>
> 2. Find the parameters that **minimize loss**, i.e. make our model as accurate as possible.
>
>

Graphically, in two dimensions, this results in a line of best fit. In three dimensions, we would draw a plane, and so on with higher-dimensional hyperplanes.

![]()Mathematically, we look at the difference between each real data point (*y*) and our model’s prediction (ŷ). Square these differences to avoid negative numbers and penalize larger differences, and then add them up and take the average. This is a measure of how well our data fits the line.

![]()*n = # of observations. Using 2\*n instead of n makes the math work out more cleanly when taking the derivative to minimize loss, though some stats people say this is blasphemy. When you start having opinions on this kind of stuff, you’ll know you are all the way in the rabbit hole.*For a simple problem like this, we can compute a closed form solution using calculus to find the optimal beta parameters that minimize our loss function. But as a cost function grows in complexity, finding a closed form solution with calculus is no longer feasible. This is the motivation for an iterative approach called **gradient descent**, which allows us to minimize a complex loss function.

Gradient descent: learn the parameters
======================================

*“Put on a blindfold, take a step downhill. You’ve found the bottom when you have nowhere to go but up.”*

Gradient descent will come up over and over again, especially in neural networks. Machine learning libraries like [scikit-learn](http://scikit-learn.org/stable/) and [TensorFlow](https://www.tensorflow.org/) use it in the background everywhere, so it’s worth understanding the details.

The goal of gradient descent is to find the minimum of our model’s loss function by iteratively getting a better and better approximation of it.

Imagine yourself walking through a valley with a blindfold on. Your goal is to find the bottom of the valley. How would you do it?

A reasonable approach would be to touch the ground around you and move in whichever direction the ground is sloping down most steeply. Take a step and repeat the same process continually until the ground is flat. Then you know you’ve reached the bottom of a valley; if you move in any direction from where you are, you’ll end up at the same elevation or further uphill.

Going back to mathematics, the ground becomes our loss function, and the elevation at the bottom of the valley is the minimum of that function.

Let’s take a look at the loss function we saw in regression:

![]()We see that this is really a function of two variables: *β0* and *β1*. All the rest of the variables are determined, since X, Y, and *n* are given during training. We want to try to minimize this function.

![]()The function is f(β0,β1)=z. To begin gradient descent, you make some guess of the parameters β0 and β1 that minimize the function.

Next, you find the [partial derivatives](https://en.wikipedia.org/wiki/Partial_derivative)of the loss function with respect to each beta parameter: [*dz/dβ0, dz/dβ1*]. A **partial derivative** indicates how much total loss is increased or decreased if you increase *β0* or *β1* by a very small amount.

Put another way, how much would increasing your estimate of annual income assuming zero higher education (*β0*) increase the loss (i.e. inaccuracy) of your model? You want to go in the *opposite* direction so that you end up walking *downhill* and minimizing loss.

Similarly, if you increase your estimate of how much each incremental year of education affects income (*β*1), how much does this increase loss (*z*)? If the partial derivative *dz/β1* is a *negative* number, then *increasing* *β1* is good because it will reduce total loss. If it’s a *positive* number, you want to *decrease* *β1*. If it’s zero, don’t change *β1 because it means you’ve reached an optimum*.

Keep doing that until you reach the bottom, i.e. the algorithm **converged** and loss has been minimized. There are lots of tricks and exceptional cases beyond the scope of this series, but generally, this is how you find the optimal **parameters** for your **parametric** model.

Overfitting
===========

***Overfitting:*** *“Sherlock, your explanation of what just happened is too specific to the situation.”* ***Regularization:*** *“Don’t overcomplicate things, Sherlock. I’ll punch you for every extra word.”* ***Hyperparameter (λ):*** *“Here’s the strength with which I will punch you for every extra word.”*

A common problem in machine learning is **overfitting**: learning a function that perfectly explains the training data that the model learned from, but doesn’t generalize well to unseen test data. Overfitting happens when a model *overlearns* from the training data to the point that it starts picking up idiosyncrasies that aren’t representative of patterns in the real world. This becomes especially problematic as you make your model increasingly complex. *Underfitting* is a related issue where your model is not complex enough to capture the underlying trend in the data.

![]()Source: Coursera’s [ML course](/@v_maini/machine-learning-for-humans-part-1-why-machine-learning-matters-965dfadf19d4), taught by Andrew NgRemember that *the only thing we care about is how the model performs on test data*. You want to predictwhich emails will be marked as spam before they’re marked, not just build a model that is 100% accurate at reclassifying the emails it used to build itself in the first place. Hindsight is 20/20 — the real question is whether the lessons learned will help in the future.

The model on the right has zero loss for the *training data* because it perfectly fits every data point. But the lesson doesn’t generalize. It would do a horrible job at explaining a new data point that isn’t yet on the line.

Two ways to combat overfitting:

**1. Use more training data**. The more you have, the harder it is to overfit the data by learning too much from any single training example.

**2. Use regularization**. Add in a penalty in the loss function for building a model that assigns too much explanatory power to any one feature or allows too many features to be taken into account.

![]()The first piece of the sum above is our normal cost function. The second piece is a **regularization term** that adds a penalty for large beta coefficients that give too much explanatory power to any specific feature. With these two elements in place, the cost function now balances between two priorities: explaining the training data and preventing that explanation from becoming overly specific.

The **lambda** coefficient of the regularization term in the cost function is a **hyperparameter:** a general setting of your model that can be increased or decreased (i.e. **tuned**)in order to improve performance. A higher lambda value will more harshly penalize large beta coefficients that could lead to potential overfitting. To decide the best value of lambda, you’d use a method called **cross-validation** which involves holding out a portion of the training data during training, and then seeing how well your model explains the held-out portion. We’ll go over this in more depth

Woo! We made it.
================

Here’s what we covered in this section:

* How **supervised machine learning** enables computers to learn from labeled training data without being explicitly programmed
* The tasks of supervised learning: **regression** and **classification**
* **Linear regression,** a bread-and-butter **parametric** algorithm
* Learning **parameters** with **gradient descent**
* **Overfitting** and **regularization**

In the next section — [Part 2.2: Supervised Learning II](/@v_maini/supervised-learning-2-5c1c23f3560d) — we’ll talk about two foundational methods of classification: **logistic regression** and **support vector machines**.

Practice materials & further reading
====================================

*2.1a — Linear regression*
--------------------------

*For a more thorough treatment of linear regression, read chapters 1–3 of* [*An Introduction to Statistical Learning*](http://www-bcf.usc.edu/~gareth/ISL/index.html)*. The book is available for free online and is an excellent resource for understanding machine learning concepts with accompanying exercises.*

*For more practice:*

* *Play with the* [*Boston Housing dataset*](http://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)*. You can either use software with nice GUIs like Minitab and Excel or do it the hard (but more rewarding) way with* [*Python*](/@haydar_ai/learning-data-science-day-9-linear-regression-on-boston-housing-dataset-cd62a80775ef) *or* [*R*](https://datascienceplus.com/linear-regression-from-scratch-in-r/)*.*
* *Try your hand at a* [*Kaggle*](http://www.kaggle.com) *challenge, e.g.* [*housing price prediction*](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)*, and see how others approached the problem after attempting it yourself.*

*2.1b — Implementing gradient descent*
--------------------------------------

*To actually implement gradient descent in Python, check out* [*this tutorial*](https://spin.atomicobject.com/2014/06/24/gradient-descent-linear-regression/)*. And* [*here*](http://eli.thegreenplace.net/2016/understanding-gradient-descent/) *is a more mathematically rigorous description of the same concepts.*

*In practice, you’ll rarely need to implement gradient descent from scratch, but understanding how it works behind the scenes will allow you to use it more effectively and understand why things break when they do.*

Enter your email below if you’d like to stay up-to-date with future content 💌
-----------------------------------------------------------------------------

On Twitter? So are we. Feel free to keep in touch — [Vishal](https://twitter.com/v_maini) and [Samer](https://twitter.com/seriousssam) 🙌🏽
-----------------------------------------------------------------------------------------------------------------------------------------

*More from* ***Machine Learning for Humans*** 🤖👶

* [*Part 1: Why Machine Learning Matters*](/machine-learning-for-humans/why-machine-learning-matters-6164faf1df12)✅
* ***Part 2.1: Supervised Learning* ✅**
* [*Part 2.2: Supervised Learning II*](/@v_maini/supervised-learning-2-5c1c23f3560d)
* [*Part 2.3: Supervised Learning III*](/@v_maini/supervised-learning-3-b1551b9c4930)
* [*Part 3: Unsupervised Learning*](/@v_maini/unsupervised-learning-f45587588294)
* [*Part 4: Neural Networks & Deep Learning*](/@v_maini/neural-networks-deep-learning-cdad8aeae49b)
* [*Part 5: Reinforcement Learning*](/@v_maini/reinforcement-learning-6eacf258b265)
* [*Appendix: The Best Machine Learning Resources*](/@v_maini/how-to-learn-machine-learning-24d53bb64aa1)

The two tasks of supervised learning: regression and classification. Linear regression, loss functions, and gradient descent.

How much money will we make by spending more dollars on digital advertising? Will this loan applicant pay back the loan or not? What’s going to happen to the stock market tomorrow?


Biological Anchors: A Trick That Might Or Might Not Work

I've been trying to review and summarize Eliezer Yudkowksy's recent dialogues on AI safety. Previously in sequence: [Yudkowsky Contra Ngo On Agents](https://astralcodexten.substack.com/p/practically-a-book-review-yudkowsky). Now we’re up to Yudkowsky contra Cotra on biological anchors, but before we get there we need to figure out what Cotra's talking about and what's going on.

The [Open Philanthropy Project](https://www.openphilanthropy.org/) ("Open Phil") is a big effective altruist foundation interested in funding AI safety. It's got $20 billion, probably the majority of money in the field, so its decisions matter a lot and it’s very invested in getting things right. In 2020, it asked senior researcher Ajeya Cotra to produce [**a report on when human-level AI would arrive.**](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP) It says the resulting document is "informal" - but it’s 169 pages long and likely to affect millions of dollars in funding, which some might describe as making it *kind* of formal. The report finds a 10% chance of “transformative AI” by 2031, a 50% chance by 2052, and an almost 80% chance by 2100.

Eliezer rejects their methodology and expects AI earlier (he doesn’t offer many numbers, but [here](https://www.econlib.org/archives/2017/01/my_end-of-the-w.html) he gives Bryan Caplan 50-50 odds on 2030, albeit [not totally seriously](https://www.econlib.org/archives/2017/01/my_end-of-the-w.html#comment-166919)). He made the case in his own very long essay, [**Biology-Inspired AGI Timelines: The Trick That Never Works**](https://www.lesswrong.com/posts/ax695frGJEzGxFBK4/biology-inspired-agi-timelines-the-trick-that-never-works), sparking a bunch of arguments and counterarguments and even more long essays.

Source:

https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might

Crossposted from the [Astral Codex Ten](https://sscpodcast.libsyn.com/biological-anchors-a-trick-that-might-or-might-not-work) podcast.

 ---


More Is Different for AI

Machine learning is touching increasingly many aspects of our society, and its effect will only continue to grow. Given this, I and many others care about risks from future ML systems and how to mitigate them.

When thinking about safety risks from ML, there are two common approaches, which I'll call the **Engineering** approach and the **Philosophy** approach:

* The Engineering approach tends to be empirically-driven, drawing experience from existing or past ML systems and looking at issues that either: (1) are already major problems, or (2) are minor problems, but can be expected to get worse in the future. Engineering tends to be bottom-up and tends to be both in touch with and anchored on current state-of-the-art systems.
* The Philosophy approach tends to think more about the limit of very advanced systems. It is willing to entertain thought experiments that would be implausible with current state-of-the-art systems (such as Nick Bostrom's [paperclip maximizer](https://en.wikipedia.org/wiki/Instrumental_convergence?ref=bounded-regret.ghost.io#Paperclip_maximizer)) and is open to considering abstractions without knowing many details. It often sounds more "sci-fi like" and more like philosophy than like computer science. It draws some inspiration from current ML systems, but often only in broad strokes.

I'll discuss these approaches mainly in the context of [ML safety](https://arxiv.org/abs/2109.13916?ref=bounded-regret.ghost.io), but the same distinction applies in other areas. For instance, an Engineering approach to AI + Law might focus on [how to regulate self-driving cars](https://library.oapen.org/handle/20.500.12657/27811?ref=bounded-regret.ghost.io), while Philosophy might ask whether [using AI in judicial decision-making could undermine liberal democracy](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3933648&ref=bounded-regret.ghost.io).

While Engineering and Philosophy agree on some things, for the most part they make wildly different predictions both about what the key safety risks from ML will be and how we should address them:

* Both Engineering and Philosophy would agree on some high-level points: they would agree that [misaligned objectives](https://en.wikipedia.org/wiki/Misaligned_goals_in_artificial_intelligence?ref=bounded-regret.ghost.io) are an important problem with ML systems that is likely to get worse. Engineering believes this because of examples like the Facebook recommender system, while Philosophy believes this based on conceptual arguments like those in *[Superintelligence](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies?ref=bounded-regret.ghost.io)*. Philosophy is more confident that misaligned objectives are a big problem and thinks they could pose an existential threat to humanity if not addressed.
* Engineering and Philosophy would both agree that out-of-distribution robustness is an important issue. However, Philosophy might view most engineering-robustness problems (such as those faced by self-driving cars) as temporary issues that will get fixed once we train on more data. Philosophy is more worried about whether systems can generalize from settings where humans can provide data, to settings where they cannot provide data even in principle.
* Engineering tends to focus on tasks where current ML systems don't work well, weighted by their impact and representativeness. Philosophy focuses on tasks that have a certain abstract property that seems important, such as [imitative deception](https://arxiv.org/abs/2109.07958?ref=bounded-regret.ghost.io).

In my experience, people who strongly subscribe to the Engineering worldview tend to think of Philosophy as fundamentally confused and ungrounded, while those who strongly subscribe to Philosophy think of most Engineering work as misguided and orthogonal (at best) to the long-term safety of ML. Given this sharp contrast and the importance of the problem, I've thought a lot about which—if either—is the "right" approach.

Coming in, I was mostly on the Engineering side, although I had more sympathy for Philosophy than the median ML researcher (who has ~0% sympathy for Philosophy). However, I now feel that:

* **Philosophy is significantly underrated by most ML researchers**.
* The Engineering worldview, taken seriously, actually implies assigning significant weight to thought experiments.

On the other hand, I also feel that:

* Philosophy continues to significantly underrate the value of empirical data.
* Neither of these approaches is satisfying and we actually have **no single good approach** to thinking about risks from future ML systems.

I've reached these conclusions through a combination of thinking, discussing with others, and observing empirical developments in ML since 2011 (when I entered the field). I've distilled my thoughts into a series of blog posts, where I'll argue that:

1. *[Future ML Systems Will be Qualitatively Different](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/)* from those we see today. Indeed, ML systems have historically exhibited qualitative changes as a result of increasing their scale. This is an instance of "More Is Different", which is commonplace in other fields such as physics, biology, and economics (see *[Appendix: More Is Different in Other Domains](https://bounded-regret.ghost.io/p/98db450b-c9bc-4e0d-98ce-1909ba980427/)*). Consequently, we should expect ML to exhibit more qualitative changes as it scales up in the future.
2. Most discussions of ML failures are anchored either on existing systems or on humans. *[Thought Experiments Provide a Third Anchor](https://bounded-regret.ghost.io/p/a2d733a7-108a-4587-97fb-db90f66ce030/)*, and having three anchors is much better than having two, but each has its own weaknesses.
3. If we take thought experiments seriously, we end up predicting that *[ML Systems Will Have Weird Failure Modes](https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/)*. Some important failure modes of ML systems will not be present in any existing systems, and might manifest quickly enough that we can't safely wait for them to occur before addressing them.
4. My biggest disagreement with the Philosophy view is that I think *[Empirical Findings Generalize Surprisingly Far](https://bounded-regret.ghost.io/p/74d500d2-a980-4720-984a-c016284ecdc2/)*, meaning that well-chosen experiments on current systems can tell us a lot about future systems.

This post is the introduction to the series. I'll post the next part each Tuesday, and update this page with links once the post is up. In the meantime, leave comments with any thoughts you have, or contact me if you'd like to preview the upcoming posts and leave feedback.

Machine learning is touching increasingly many aspects of our society, and its effect will only continue to grow. Given this, I and many others care about risks from future ML systems and how to mitigate them. When thinking about safety risks from ML, there are two common approaches, which I'll call the Engineering approach and the Philosophy approach: The Engineering approach tends to be empirically-driven, drawing experience from existing or past ML systems and looking at issues that either: (1) are already major problems, or (2) are minor problems, but can be expected to get worse in the future. Engineering tends to be bottom-up and tends to be both in touch with and anchored on current state-of-the-art systems. The Philosophy approach tends to think more about the limit of very advanced systems. It is willing to entertain thought experiments that would be implausible with current state-of-the-art systems (such as Nick Bostrom's paperclip maximizer) and is open to considering abstractions without knowing many details. It often sounds more "sci-fi like" and more like philosophy than like computer science. It draws some inspiration from current ML systems, but often only in broad strokes. I'll discuss these approaches mainly in the context of ML safety, but the same distinction applies in other areas. For instance, an Engineering approach to AI + Law might focus on how to regulate self-driving cars, while Philosophy might ask whether using AI in judicial decision-making could undermine liberal democracy.

**Original text:**

<https://bounded-regret.ghost.io/more-is-different-for-ai/>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


Future ML Systems Will Be Qualitatively Different

In 1972, the Nobel prize-winning physicist Philip Anderson wrote the essay "[More Is Different](https://science.sciencemag.org/content/177/4047/393?ref=bounded-regret.ghost.io)". In it, he argues that quantitative changes can lead to qualitatively different and unexpected phenomena. While he focused on physics, one can find many examples of More is Different in other domains as well, including biology, economics, and computer science. Some examples of More is Different include:

* **Uranium.** With a bit of uranium, nothing special happens; with a large amount of uranium packed densely enough, you get a nuclear reaction.
* **DNA.** Given only small molecules such as calcium, you can’t meaningfully encode useful information; given larger molecules such as DNA, you can encode a genome.
* **Water.** Individual water molecules aren’t wet. Wetness only occurs due to the interaction forces between many water molecules interspersed throughout a fabric (or other material).
* **Traffic.** A few cars on the road are fine, but with too many you get a traffic jam. It could be that 10,000 cars could traverse a highway easily in 15 minutes, but 20,000 on the road at once could take over an hour.
* **Specialization.** Historically, in small populations, virtually everyone needed to farm or hunt to survive; in contrast, in larger and denser communities, enough food is produced for large fractions of the population to specialize in non-agricultural work.

While some of the examples, like uranium, correspond to a sharp transition, others like specialization are more continuous. I’ll use **emergence** to refer to qualitative changes that arise from quantitative increases in scale, and **phase transitions** for cases where the change is sharp.

In this post, I'll argue that emergence often occurs in the field of AI, and that this should significantly affect our intuitions about the long-term development and deployment of AI systems. We should expect weird and surprising phenomena to emerge as we scale up systems. This presents opportunities, but also poses important risks.

Emergent Shifts in the History of AI
------------------------------------

There have already been several examples of quantitative differences leading to important qualitative changes in machine learning.

**Storage and Learning.** The emergence of machine learning as a viable approach to AI is itself an example of More Is Different. While learning had been discussed since the 1950s, it wasn’t until the 80s-90s that it became a dominant paradigm: for instance, IBM’s [first statistical translation model](https://dl.acm.org/doi/10.3115/991635.991651?ref=bounded-regret.ghost.io) was published in 1988, even though the idea [was proposed](https://aclanthology.org/1952.earlymt-1.1.pdf?ref=bounded-regret.ghost.io) in 1949[[1]](#fn1). Not coincidentally, 1GB of storage cost over $100k in 1981 but only around $9k in 1990 (adjusted to 2021 dollars). The [Hansard corpus](https://catalog.ldc.upenn.edu/LDC95T20?ref=bounded-regret.ghost.io) used to train IBM’s model comprised 2.87 million sentences and would have been difficult to use before the 80s. Even the simple MNIST dataset would have required $4000 in hardware just to store in 1981, but that had fallen to a few dollars by 1998 when it was published. Cheaper hardware thus allowed for a qualitatively new approach to AI: in other words, More storage enabled Different approaches.

**Compute, Data, and Neural Networks.** As hardware improved, it became possible to train neural networks that were very deep for the first time. Better compute enabled bigger models trained for longer, and better storage enabled learning from more data; AlexNet-sized models and ImageNet-sized datasets wouldn’t have been feasible for researchers to experiment with in 1990.

Deep learning performs well with lots of data and compute, but struggles at smaller scales. Without many resources, simpler algorithms tend to outperform it, but with sufficient resources it pulls far ahead of the pack. This reversal of fortune led to qualitative changes in the field. As one example, the field of machine translation moved from [phrase-based models](https://nlp.stanford.edu/phrasal/?ref=bounded-regret.ghost.io) (hand-coded features, complex systems engineering) to [neural sequence-to-sequence models](https://proceedings.neurips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf?ref=bounded-regret.ghost.io) (learned features, specialized architecture and initialization) to simply fine-tuning a [foundation model](https://arxiv.org/abs/2108.07258?ref=bounded-regret.ghost.io) such as BERT or GPT-3. Most work on phrase-based models was obviated by neural translation, and the same pattern held across many other language tasks, where hard-won domain-specific engineering effort was simply replaced by a general algorithm.

**Few-shot Learning.** More recently, [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf?ref=bounded-regret.ghost.io) and [GPT-3](https://arxiv.org/abs/2005.14165?ref=bounded-regret.ghost.io) revealed the emergence of strong few-shot and zero-shot capabilities, via well-chosen natural language prompting.

![](https://bounded-regret.ghost.io/content/images/2022/01/gpt2_only.png)
![](https://bounded-regret.ghost.io/content/images/2022/01/gpt2_and_3.png)

*Top: Few-shot machine translation performance (BLEU score) for GPT-2. Bottom: GPT-3 (trained on more data) has an even starker curve, going from 5 to 25 BLEU between 100M and 400M parameters. Unsupervised baselines, as well as fine-tuned state-of-the-art, are indicated for reference.*

This was an unexpected and qualitatively new phenomenon that only appeared at large scales, and it emerged without ever explicitly training models to have these few-shot capabilities. Comparing GPT-2 to GPT-3 shows that the exact model size needed can vary due to the training distribution or other factors, but this doesn’t affect the basic point that new capabilities can appear without designing or training for them.

**Grokking.** In 2021, [Power et al.](https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdf?ref=bounded-regret.ghost.io) identified a phenomenon they call "grokking", where a network’s generalization behavior improves qualitatively when training it for longer (even though the training loss is already small).

Specifically, for certain algorithmically generated logic/math datasets, neural networks trained for 1,000 steps achieve perfect train accuracy but near-zero test accuracy. However, after around 100,000 steps the test accuracy suddenly increases, achieving near-perfect generalization by 1 million steps.

![grokking](https://bounded-regret.ghost.io/content/images/2021/11/grokking.png)

This shows that even for a single model, we might encounter qualitative phase transitions as we train for longer.

**Other potential examples.** I'll briefly list other examples from recent papers. I don't think these examples are as individually clear-cut, but they collectively paint an interesting picture:

* [McGrath et al. (2021)](https://arxiv.org/abs/2111.09259?ref=bounded-regret.ghost.io) show that AlphaZero acquires many chess concepts at a phase transition near 32,000 training steps.
* [Pan et al. (2021)](https://arxiv.org/abs/2201.03544?ref=bounded-regret.ghost.io) show that reward hacking sometimes occurs via qualitative phase transitions as model size increases.
* DeepMind's recent [Gopher](https://arxiv.org/abs/2112.11446?ref=bounded-regret.ghost.io) model exhibits a phase transition on the FEVER task, acquiring the ability to evaluate evidence provided as side information (Figure 3):

![](https://bounded-regret.ghost.io/content/images/2022/01/fever_plot.png)

* [Wei et al. (2021)](https://arxiv.org/abs/2109.01652?ref=bounded-regret.ghost.io) show that instruction-tuning hurts small models but helps large models (see Figure 6).
* Some few-shot tasks such as arithmetic show phase transitions with model size (see [Brown et al. (2020)](https://arxiv.org/abs/2005.14165?ref=bounded-regret.ghost.io), Figure 3.10).
* [This](https://twitter.com/NaxAlpha/status/1420700413125447683?ref=bounded-regret.ghost.io) researcher shares an anecdote similar to the “grokking” paper.

What This Implies for the Engineering Worldview
-----------------------------------------------

In the [introduction post](https://bounded-regret.ghost.io/more-is-different-for-ai/) to this series, I contrasted two worldviews called Philosophy and Engineering. The Engineering worldview, which is favored by most ML researchers, tends to predict the future by looking at empirical trends and extrapolating them forward. I myself am [quite sympathetic to this view](https://bounded-regret.ghost.io/forecasting-zeroth-and-first-order/), and for this reason I find emergent behavior to be troubling and disorienting. Rather than expecting empirical trends to continue, emergence suggests we should often expect new qualitative behaviors that are not extrapolations of previous trends.

Indeed, in this sense Engineering (or at least pure trend extrapolation) is self-defeating as a tool for predicting the future[[2]](#fn2). The Engineering worldview wants to extrapolate trends, but one trend is that emergent behavior is becoming more and more common. Of the four phase transitions I gave above, the first (storage) occurred around 1995, and the second (compute) occurred around 2015. The last two occurred in 2020 and 2021. Based on past trends, we should expect future trends to break more and more often.[[3]](#fn3)

How can we orient ourselves when thinking about the future of AI despite the probability of frequent deviations from past experience? I'll have a lot more to say about this in the next few posts, but to put some of my cards on the table:

* Confronting emergence will require adopting mindsets that are less familiar to most ML researchers and utilizing more of the Philosophy worldview (in tandem with Engineering and other worldviews).
* Future ML systems will have weird failure modes that don't manifest today, and we should start thinking about and addressing them in advance.
* On the other hand, I don't think that Engineering as a tool for predicting the future is entirely self-defeating. Despite emergent behavior, empirical findings often generalize surprisingly far, at least if we're careful in interpreting them. Utilizing this fact will be crucial to making concrete research progress.

---

1. From the [IBM model authors](https://aclanthology.org/J90-2002.pdf?ref=bounded-regret.ghost.io): “In 1949 Warren Weaver suggested that the problem be attacked with statistical methods and ideas from information theory, an area which he, Claude Shannon, and others were developing at the time (Weaver 1949). Although researchers quickly abandoned this approach, advancing numerous theoretical objections, we believe that the true obstacles lay in the relative impotence of the available computers and the dearth of machine-readable text from which to gather the statistics vital to such an attack. Today, computers are five orders of magnitude faster than they were in 1950 and have hundreds of millions of bytes of storage. Large, machine-readable corpora are readily available.” [↩](#fnref1)
2. This is in contrast to using Engineering to *build capable and impressive systems* today. If anything, recent developments have strongly solidified Engineering’s dominance for this task. [↩](#fnref2)
3. This list is probably subject to selection bias and recency effects, although I predict that my point would still hold up for a carefully curated list (for instance, I didn’t include the several ambiguous examples in my count). I would be happy to bet on more phase transitions in the future if any readers wish to take the other side. [↩](#fnref3)

In 1972, the Nobel prize-winning physicist Philip Anderson wrote the essay "More Is Different". In it, he argues that quantitative changes can lead to qualitatively different and unexpected phenomena. While he focused on physics, one can find many examples of More is Different in other domains as well, including biology, economics, and computer science. Some examples of More is Different include: Uranium. With a bit of uranium, nothing special happens; with a large amount of uranium packed densely enough, you get a nuclear reaction. DNA. Given only small molecules such as calcium, you can’t meaningfully encode useful information; given larger molecules such as DNA, you can encode a genome. Water. Individual water molecules aren’t wet. Wetness only occurs due to the interaction forces between many water molecules interspersed throughout a fabric (or other material).

**Original text:**

<https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


[Week 1] “Intelligence Explosion: Evidence and Import” (Sections 3 to 4.1) by Luke Muehlhauser & Anna Salamon

Intelligence Explosion: Evidence and Import
3.From AI to Machine Superintelligence
It seems unlikely that humans are near the ceiling of possible intelligences, rather than
simply being the first such intelligence that happened to evolve. Computers far outper-
form humans in many narrow niches (e.g. arithmetic, chess, memory size), and there is
reason to believe that similar large improvements over human performance are possible
for general reasoning, technology design, and other tasks of interest. As occasional AI
critic Jack Schwartz (1987) wrote:
If artificial intelligences can be created at all, there is little reason to believe
that initial successes could not lead swiftly to the construction of artificial su-
perintelligences able to explore significant mathematical, scientific, or engi-
neering alternatives at a rate far exceeding human ability, or to generate plans
and take action on them with equally overwhelming speed. Since man’s near-
monopoly of all higher forms of intelligence has been one of the most basic
facts of human existence throughout the past history of this planet, such de-
velopments would clearly create a new economics, a new sociology, and a new
history.
Why might AI “lead swiftly” to machine superintelligence? Below we consider some
reasons.
3.1. AI Advantages
Below we list a few AI advantages that may allow AIs to become not only vastly more
intelligent than any human, but also more intelligent than all of biological humanity
(Sotala 2012; Legg 2008). Many of these are unique to machine intelligence, and that
is why we focus on intelligence explosion from AI rather than from biological cognitive
enhancement (Sandberg 2011).
Increased computational resources. The human brain uses 85–100 billion neurons. This
limit is imposed by evolution-produced constraints on brain volume and metabolism.
In contrast, a machine intelligence could use scalable computational resources (imagine
a “brain” the size of a warehouse). While algorithms would need to be changed in order
to be usefully scaled up, one can perhaps get a rough feel for the potential impact here by
noting that humans have about 3.5 times the brain size of chimps (Schoenemann 1997),
and that brain size and IQ correlate positively in humans, with a correlation coeﬃcient
of about 0.35 (McDaniel 2005). One study suggested a similar correlation between
brain size and cognitive ability in rats and mice (Anderson 1993).15
15. Note that given the definition of intelligence we are using, greater computational resources would
not give a machine more “intelligence” but instead more “optimization power.”
10
Luke Muehlhauser, Anna Salamon
Communication speed. Axons carry spike signals at 75 meters per second or less (Kan-
del, Schwartz, and Jessell 2000). That speed is a fixed consequence of our physiology.
In contrast, software minds could be ported to faster hardware, and could therefore pro-
cess information more rapidly. (Of course, this also depends on the eﬃciency of the
algorithms in use; faster hardware compensates for less eﬃcient software.)
Increased serial depth. Due to neurons’ slow firing speed, the human brain relies on
massive parallelization and is incapable of rapidly performing any computation that re-
quires more than about 100 sequential operations (Feldman and Ballard 1982). Perhaps
there are cognitive tasks that could be performed more eﬃciently and precisely if the
brain’s ability to support parallelizable pattern-matching algorithms were supplemented
by support for longer sequential processes. In fact, there are many known algorithms
for which the best parallel version uses far more computational resources than the best
serial algorithm, due to the overhead of parallelization.16
Duplicability. Our research colleague Steve Rayhawk likes to describe AI as “instant
intelligence; just add hardware!” What Rayhawk means is that, while it will require ex-
tensive research to design the first AI, creating additional AIs is just a matter of copying
software. The population of digital minds can thus expand to fill the available hardware
base, perhaps rapidly surpassing the population of biological minds.
Duplicability also allows the AI population to rapidly become dominated by newly
built AIs, with new skills. Since an AI’s skills are stored digitally, its exact current state
can be copied,17including memories and acquired skills—similar to how a “system state”
can be copied by hardware emulation programs or system backup programs. A human
who undergoes education increases only his or her own performance, but an AI that
becomes 10% better at earning money (per dollar of rentable hardware) than other AIs
can be used to replace the others across the hardware base—making each copy 10% more
eﬃcient.18
Editability. Digitality opens up more parameters for controlled variation than is pos-
sible with humans. We can put humans through job-training programs, but we can’t
perform precise, replicable neurosurgeries on them. Digital workers would be more
editable than human workers are. Consider first the possibilities from whole brain em-
ulation. We know that transcranial magnetic stimulation (TMS) applied to one part of
16. For example see Omohundro (1987).
17. If the first self-improving AIs at least partially require quantum computing, the system states of
these AIs might not be directly copyable due to the no-cloning theorem (Wootters and Zurek 1982).
18. Something similar is already done with technology-enabled business processes. When the phar-
macy chain CVS improves its prescription-ordering system, it can copy these improvements to more than
4,000 of its stores, for immediate productivity gains (McAfee and Brynjolfsson 2008).
11
Intelligence Explosion: Evidence and Import
the prefrontal cortex can improve working memory (Fregni et al. 2005). Since TMS
works by temporarily decreasing or increasing the excitability of populations of neurons,
it seems plausible that decreasing or increasing the “excitability” parameter of certain
populations of (virtual) neurons in a digital mind would improve performance. We
could also experimentally modify dozens of other whole brain emulation parameters,
such as simulated glucose levels, undiﬀerentiated (virtual) stem cells grafted onto par-
ticular brain modules such as the motor cortex, and rapid connections across diﬀerent
parts of the brain.19Secondly, a modular, transparent AI could be even more directly
editable than a whole brain emulation—possibly via its source code. (Of course, such
possibilities raise ethical concerns.)
Goal coordination. Let us call a set of AI copies or near-copies a “copy clan.” Given
shared goals, a copy clan would not face certain goal coordination problems that limit
human eﬀectiveness (J. W. Friedman 1994). A human cannot use a hundredfold salary
increase to purchase a hundredfold increase in productive hours per day. But a copy clan,
if its tasks are parallelizable, could do just that. Any gains made by such a copy clan, or
by a human or human organization controlling that clan, could potentially be invested
in further AI development, allowing initial advantages to compound.
Improved rationality. Some economists model humans as Homo economicus : self-
interested rational agents who do what they believe will maximize the fulfillment of
their goals (M. Friedman 1953). On the basis of behavioral studies, though, Schneider
(2010) points out that we are more akin to Homer Simpson: we are irrational beings
that lack consistent, stable goals (Schneider 2010; Cartwright 2011). But imagine if you
werean instance of Homo economicus . You could stay on a diet, spend the optimal amount
of time learning which activities will achieve your goals, and then follow through on an
optimal plan, no matter how tedious it was to execute. Machine intelligences of many
types could be written to be vastly more rational than humans, and thereby accrue the
benefits of rational thought and action. The rational agent model (using Bayesian prob-
ability theory and expected utility theory) is a mature paradigm in current AI design
(Hutter 2005; Russell and Norvig 2010, ch. 2).
These AI advantages suggest that AIs will be capable of far surpassing the cognitive
abilities and optimization power of humanity as a whole, but will they be motivated to
do so? Though it is diﬃcult to predict the specific motivations of advanced AIs, we can
make some predictions about convergent instrumental goals—instrumental goals useful
for the satisfaction of almost any final goals.
19. Many suspect that the slowness of cross-brain connections has been a major factor limiting the
usefulness of large brains (Fox 2011).
12
Luke Muehlhauser, Anna Salamon
3.2. Instrumentally Convergent Goals
Omohundro (2007, 2008, 2012) and Bostrom (forthcoming) argue that there are several
instrumental goals that will be pursued by almost any advanced intelligence because
those goals are useful intermediaries to the achievement of almost any set of final goals.
For example:
1.An AI will want to preserve itself because if it is destroyed it won’t be able to act
in the future to maximize the satisfaction of its present final goals.
2.An AI will want to preserve the content of its current final goals because if the
content of its final goals is changed it will be less likely to act in the future to
maximize the satisfaction of its present final goals.20
3.An AI will want to improve its own rationality and intelligence because this will
improve its decision-making, and thereby increase its capacity to achieve its goals.
4.An AI will want to acquire as many resources as possible, so that these resources
can be transformed and put to work for the satisfaction of the AI’s final and in-
strumental goals.
Later we shall see why these convergent instrumental goals suggest that the default out-
come from advanced AI is human extinction. For now, let us examine the mechanics of
AI self-improvement.
3.3. Intelligence Explosion
The convergent instrumental goal for self-improvement has a special consequence. Once
human programmers build an AI with a better-than-human capacity for AI design, the
instrumental goal for self-improvement may motivate a positive feedback loop of self-
enhancement.21Now when the machine intelligence improves itself, it improves the
intelligence that does the improving. Thus, if mere human eﬀorts suﬃce to produce
machine intelligence this century, a large population of greater-than-human machine
intelligences may be able to create a rapid cascade of self-improvement cycles, enabling
20. Bostrom (2012) lists a few special cases in which an AI may wish to modify the content of its final
goals.
21. When the AI can perform 10% of the AI design tasks and do them at superhuman speed, the
remaining 90% of AI design tasks act as bottlenecks. However, if improvements allow the AI to perform
99% of AI design tasks rather than 98%, this change produces a much larger impact than when improve-
ments allowed the AI to perform 51% of AI design tasks rather than 50% (Hanson 1998). And when the
AI can perform 100% of AI design tasks rather than 99% of them, this removes altogether the bottleneck
of tasks done at slow human speeds.
13
Intelligence Explosion: Evidence and Import
a rapid transition to machine superintelligence. Chalmers (2010) discusses this process
in some detail, so here we make only a few additional points.
The term “self,” in phrases like “recursive self-improvement” or “when the machine
intelligence improves itself,” is something of a misnomer. The machine intelligence
could conceivably edit its own code while it is running (Schmidhuber 2007; Schaul
and Schmidhuber 2010), but it could also create new intelligences that run indepen-
dently. Alternatively, several AIs (perhaps including WBEs) could work together to
design the next generation of AIs. Intelligence explosion could come about through
“self”-improvement or through other-AI improvement.
Once sustainable machine self-improvement begins, AI development need not pro-
ceed at the normal pace of human technological innovation. There is, however, sig-
nificant debate over how fast or local this “takeoﬀ” would be (Hanson and Yudkowsky
2008; Loosemore and Goertzel 2011; Bostrom, forthcoming), and also about whether
intelligence explosion would result in a stable equilibrium of multiple machine superin-
telligences or instead a machine “singleton” (Bostrom 2006). We will not discuss these
complex issues here.
4.Consequences of Machine Superintelligence
If machines greatly surpass human levels of intelligence—that is, surpass humanity’s
capacity for eﬃcient cross-domain optimization—we may find ourselves in a position
analogous to that of the apes who watched as humans invented fire, farming, writing,
science, guns and planes and then took over the planet. (One salient diﬀerence would be
that no single ape witnessed the entire saga, while we might witness a shift to machine
dominance within a single human lifetime.) Such machines would be superior to us in
manufacturing, harvesting resources, scientific discovery, social aptitude, and strategic
action, among other capacities. We would not be in a position to negotiate with them,
just as neither chimpanzees nor dolphins are in a position to negotiate with humans.
Moreover, intelligence can be applied in the pursuit of any goal. As Bostrom (2012)
argues, making AIs more intelligent will not make them want to change their goal
systems—indeed, AIs will be motivated to preserve their initial goals. Making AIs more
intelligent will only make them more capable of achieving their original final goals,
whatever those are.22
This brings us to the central feature of AI risk: Unless an AI is specifically pro-
grammed to preserve what humans value, it may destroy those valued structures (in-
22. This may be less true for early-generation WBEs, but Omohundro (2007) argues that AIs will
converge upon being optimizing agents, which exhibit a strict division between goals and cognitive ability.
14
Luke Muehlhauser, Anna Salamon
cluding humans) incidentally . As Yudkowsky (2008a) puts it, “the AI does not love you,
nor does it hate you, but you are made of atoms it can use for something else.”
4.1. Achieving a Controlled Intelligence Explosion
How, then, can we give AIs desirable goals before they self-improve beyond our ability to
control them or negotiate with them?23WBEs and other brain-inspired AIs running on
human-derived “spaghetti code” may not have a clear “slot” in which to specify desirable
goals (Marcus 2008). The same may also be true of other “opaque” AI designs, such as
those produced by evolutionary algorithms—or even of more transparent AI designs.
Even if an AI had a transparent design with a clearly definable utility function,24would
we know how to give it desirable goals? Unfortunately, specifying what humans value
may be extraordinarily diﬃcult, given the complexity and fragility of human preferences
(Yudkowsky 2011; Muehlhauser and Helm 2012), and allowing an AI to learndesirable
goals from reward and punishment may be no easier (Yudkowsky 2008a). If this is
correct, then the creation of self-improving AI may be detrimental by default unless we
first solve the problem of how to build an AI with a stable, desirable utility function—a
“Friendly AI” (Yudkowsky 2001).25
But suppose it is possible to build a Friendly AI (FAI) capable of radical self-
improvement. Normal projections of economic growth allow for great discoveries rel-
evant to human welfare to be made eventually—but a Friendly AI could make those
discoveries much sooner. A benevolent machine superintelligence could, as Bostrom
(2003) writes, “create opportunities for us to vastly increase our own intellectual and
emotional capabilities, and it could assist us in creating a highly appealing experiential
world in which we could live lives devoted [to] joyful game-playing, relating to each
other, experiencing, personal growth, and to living closer to our ideals.”
23. Hanson (2012) reframes the problem, saying that “we should expect that a simple continuation of
historical trends will eventually end up [producing] an ‘intelligence explosion’ scenario. So there is little
need to consider [Chalmers’] more specific arguments for such a scenario. And the inter-generational
conflicts that concern Chalmers in this scenario are generic conflicts that arise in a wide range of past,
present, and future scenarios. Yes, these are conflicts worth pondering, but Chalmers oﬀers no reasons
why they are interestingly diﬀerent in a ‘singularity’ context.” We briefly oﬀer just one reason why the
“inter-generational conflicts” arising from a transition of power from humans to superintelligent machines
are interestingly diﬀerent from previous the inter-generational conflicts: as Bostrom (2002) notes, the
singularity may cause the extinction not just of people groups but of the entire human species. For a
further reply to Hanson, see Chalmers (2012).
24. A utility function assigns numerical utilities to outcomes such that outcomes with higher utilities
are always preferred to outcomes with lower utilities (Mehta 1998).
25. It may also be an option to constrain the first self-improving AIs just long enough to develop a
Friendly AI before they cause much damage.
15

It seems unlikely that humans are near the ceiling of possible intelligences, rather than simply being the first such intelligence that happened to evolve. Computers far outperform humans in many narrow niches (e.g. arithmetic, chess, memory size), and there is reason to believe that similar large improvements over human performance are possible for general reasoning, technology design, and other tasks of interest. As occasional AI critic Jack Schwartz (1987) wrote:

"If artificial intelligences can be created at all, there is little reason to believe that initial successes could not lead swiftly to the construction of artificial superintelligences able to explore significant mathematical, scientific, or engi-neering alternatives at a rate far exceeding human ability, or to generate plans and take action on them with equally overwhelming speed. Since man’s near-monopoly of all higher forms of intelligence has been one of the most basic facts of human existence throughout the past history of this planet, such developments would clearly create a new economics, a new sociology, and a new history."

Why might AI “lead swiftly” to machine superintelligence? Below we consider some reasons.


Deceptively Aligned Mesa-Optimizers: It’s Not Funny If I Have To Explain It

Our goal here is to popularize obscure and hard-to-understand areas of AI alignment.

So let’s try to understand the incomprehensible meme!

Our main source will be Hubinger et al 2019, [Risks From Learned Optimization In Advanced Machine Learning Systems](https://arxiv.org/pdf/1906.01820.pdf).

Mesa- is a Greek prefix which means the opposite of meta-. To “go meta” is to go one level up; to “go mesa” is to go one level down (nobody has ever actually used this expression, sorry). So a mesa-optimizer is an optimizer one level down from you.

Consider evolution, optimizing the fitness of animals. For a long time, it did so very mechanically, inserting behaviors like “use this cell to detect light, then grow toward the light” or “if something has a red dot on its back, it might be a female of your species, you should mate with it”. As animals became more complicated, they started to do some of the work themselves. Evolution gave them drives, like hunger and lust, and the animals figured out ways to achieve those drives in their current situation. Evolution didn’t mechanically instill the behavior of opening my fridge and eating a Swiss Cheese slice. It instilled the hunger drive, and I figured out that the best way to satisfy it was to open my fridge and eat cheese.

**Source:**

https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers

Crossposted from the [Astral Codex Ten](https://sscpodcast.libsyn.com/deceptively-aligned-mesa-optimizers-its-not-funny-if-i-have-to-explain-it) podcast.

 ---


[Week 2] “Learning from human preferences” (Blog Post) by Dario Amodei, Paul Christiano & Alex Ray

We present a learning algorithm that uses small amounts of human feedback to solve modern RL environments. Machine learning systems with human feedback [have](https://papers.nips.cc/paper/4805-a-bayesian-approach-for-policy-learning-from-trajectory-preference-queries) [been](https://link.springer.com/chapter/10.1007/978-3-319-02675-6_46) [explored](https://arxiv.org/abs/1208.0984) [before](https://hal.inria.fr/hal-00980839), but we’ve scaled up the approach to be able to work on much more complicated tasks. Our algorithm needed 900 bits of feedback from a human evaluator to learn to backflip—a seemingly simple task which is simple to judge but [challenging](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/#bflip) to specify.

![Humanfeedbackjump](https://images.openai.com/blob/cf6fdf49-ea9e-489d-a1f1-9753291cd09e/humanfeedbackjump.gif)Our algorithm learned to backflip using around 900 individual bits of feedback from the human evaluator.The overall training process is a 3-step feedback cycle between the human, the agent’s understanding of the goal, and the RL training.

![Diagram2x 2](https://images.openai.com/blob/76127a33-15be-4357-aefb-1e509fe1330f/diagram2x-2.png?trim=0,0,0,0&width=10&height=10&quality=50)Our AI agent starts by acting randomly in the environment. Periodically, two video clips of its behavior are given to a human, and the human decides which of the two clips is closest to fulfilling its goal—in this case, a backflip. The AI gradually builds a model of the goal of the task by finding the reward function that best explains the human’s judgments. It then uses RL to learn how to achieve that goal. As its behavior improves, it continues to ask for human feedback on trajectory pairs where it’s most uncertain about which is better, and further refines its understanding of the goal.

Our approach demonstrates promising sample efficiency—as stated previously, the backflip video required under 1000 bits of human feedback. It took less than an hour of a human evaluator’s time, while in the background the policy accumulated about 70 hours of overall experience (simulated at a much faster rate than real-time.) We will continue to work on reducing the amount of feedback a human needs to supply. You can see a sped-up version of the training process in the following video.

![Learning From Human Preferences Poster](https://images.openai.com/blob/0478962c-9211-4f2d-a2da-68c7b5e19d0d/LearningFromHumanPreferencesPoster.jpg?trim=15,0,6,0&width=10&height=10&quality=50)Human Feedback training process00:39We’ve tested our method on a number of tasks in the simulated robotics and Atari domains (without being given access to the reward function: so in Atari, without having access to the game score). Our agents can learn from human feedback to achieve strong and sometimes superhuman performance in many of the environments we tested. In the following animation you can see agents trained with our technique playing a variety of Atari games. The horizontal bar on the right hand side of each frame represent’s each agents prediction about how much a human evaluator would approve of their current behavior. These visualizations indicate that agents trained with human feedback learn to value oxygen in Seaquest (left), anticipate rewards in Breakout and Pong (center), or work out how to recover from crashes in Enduro (right).

![Seaquestsave](https://images.openai.com/blob/66ed060d-eb2b-4526-8ed8-f8c29403b095/seaquestsave.gif)![Spaceinvadersbehavior](https://images.openai.com/blob/66ed060d-eb2b-4526-8ed8-f8c29403b095/spaceinvadersbehavior.gif)![Pong2](https://images.openai.com/blob/66ed060d-eb2b-4526-8ed8-f8c29403b095/pong2.gif)![Enduro1](https://images.openai.com/blob/7b8509cb-549f-4548-bc62-7e1198868fbe/enduro1.gif)Note there’s no need for the feedback to align with the environment’s normal reward function: we can, for example, train our agents to precisely keep even with other cars in Enduro rather than maximizing game score by passing them. We also sometimes find that learning from feedback does better than reinforcement learning with the normal reward function, because the human shapes the reward better than whoever wrote the environment’s reward.

Challenges
----------

Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors *look* correct, so if the human doesn’t have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only *appeared* to be grasping it, as shown below.

![Gifhandlerresized](https://images.openai.com/blob/f12a1b22-538c-475f-b76d-330b42d309eb/gifhandlerresized.gif)We addressed this particular problem by adding in visual cues (the thick white lines in the above animation) to make it easy for the human evaluators to estimate depth.

The research described in this post was done in collaboration with Jan Leike, Miljan Martic, and Shane Legg at DeepMind. Our two organizations plan to continue to collaborate on topics that touch on long-term AI safety. We think that techniques like this are a step towards safe AI systems capable of learning human-centric goals, and can complement and extend existing approaches like reinforcement and imitation learning. This post is representative of the work done by OpenAI’s safety team; if you’re interested in working on problems like this, please [join us](https://openai.com/jobs/)!

Footnote
--------

![Goodgifb1 2](https://images.openai.com/blob/9b63988d-b4dd-4313-bebb-a6e6805891fc/goodgifb1-2.gif)![Badgif3](https://images.openai.com/blob/9b63988d-b4dd-4313-bebb-a6e6805891fc/badgif3.gif)By comparison, we took two hours to write our own reward function (the animation in the above right) to get a robot to backflip, and though it succeeds it’s a lot less elegant than the one trained simply through human feedback (top left). We think there are many cases where human feedback could let us specify a specific goal more intuitively and quickly than is possible by manually hand-crafting the objective.

You can replicate this backflip in [gym](http://github.com/openai/gym) with the following reward function for Hopper:

```
def reward_fn(a, ob):
    backroll = -ob[7]
    height = ob[0]
    vel_act = a[0] * ob[8] + a[1] * ob[9] + a[2] * ob[10]
    backslide = -ob[5]
    return backroll * (1.0 + .3 * height + .1 * vel_act + .05 * backslide)
```
null

One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.


The easy goal inference problem is still hard

INSTRUMENTAL  CON vERGENCE   |  109Instrumental convergence
According to the orthogonality thesis, intelligent agents may have an enormous
range of possible final goals. Nevertheless, according to what we may term the
“instrumental convergence” thesis, there are some instrumental  goals likely to be
pursued by almost any intelligent agent, because there are some objectives that
are useful intermediaries to the achievement of almost any final goal. We can
formulate this thesis as follows:
The instrumental convergence thesis
Several instrumental values can be identified which are convergent in the sense
that their attainment would increase the chances of the agent’s goal being realized
for a wide range of final goals and a wide range of situations, implying that these
instrumental values are likely to be pursued by a broad spectrum of situated intel -
ligent agents.
In the following we will consider several categories where such convergent
instrumental values may be found.8 The likelihood that an agent will recognize
the instrumental values it confronts increases ( ceteris paribus ) with the agent’s
intelligence. We will therefore focus mainly on the case of a hypothetical super -
intelligent agent whose instrumental reasoning capacities far exceed those of
any human. We will also comment on how the instrumental convergence thesis
applies to the case of human beings, as this gives us occasion to elaborate some
essential qualifications concerning how the instrumental convergence thesis
should be interpreted and applied. Where there are convergent instrumental
values, we may be able to predict some aspects of a superintelligence’s behavior
even if we know virtually nothing about that superintelligence’s final goals.
Self-preservation
If an agent’s final goals concern the future, then in many scenarios there will
be future actions it could perform to increase the probability of achieving its
goals. This creates an instrumental reason for the agent to try to be around in the
future—to help achieve its future-oriented goal.
Most humans seem to place some final  value on their own survival. This is not
a necessary feature of artificial agents: some may be designed to place no final
value whatever on their own survival. Nevertheless, many agents that do not care
intrinsically about their own survival would, under a fairly wide range of condi -
tions, care instrumentally about their own survival in order to accomplish their
final goals.
Goal-content integrity
If an agent retains its present goals into the future, then its present goals will
be more likely to be achieved by its future self. This gives the agent a present
110  |  THE  SUPERINTELLIGENT  WILLinstrumental reason to prevent alterations of its final goals. (The argument applies
only to final goals. In order to attain its final goals, an intelligent agent will of
course routinely want to change its subgoals  in light of new information and
insight.)
Goal-content integrity for final goals is in a sense even more fundamental than
survival as a convergent instrumental motivation. Among humans, the opposite
may seem to hold, but that is because survival is usually part of our final goals.
For software agents, which can easily switch bodies or create exact duplicates of
themselves, preservation of self as a particular implementation or a particular
physical object need not be an important instrumental value. Advanced software
agents might also be able to swap memories, download skills, and radically mod -
ify their cognitive architecture and personalities. A population of such agents
might operate more like a “functional soup” than a society composed of distinct
semi-  permanent persons.9 For some purposes, processes in such a system might
be better individuated as teleological threads , based on their values, rather than on
the basis of bodies, personalities, memories, or abilities. In such scenarios, goal-
continuity might be said to constitute  a key aspect of survival.
Even so, there are situations in which an agent can best fulfill its final goals by
intentionally changing them. Such situations can arise when any of the following
factors is significant:
•	Social signaling . When others can perceive an agent’s goals and use that information
to infer instrumentally relevant dispositions or other correlated attributes, it can be
in the agent’s interest to modify its goals to make a favorable impression. For exam -
ple, an agent might miss out on beneficial deals if potential partners cannot trust it to
fulfill its side of the bargain. In order to make credible commitments, an agent might
therefore wish to adopt as a final goal the honoring of its earlier commitments (and
allow others to verify that it has indeed adopted this goal). Agents that could flexibly
and transparently modify their own goals could use this ability to enforce deals.10
•	Social preferences . Others may also have final preferences about an agent’s goals. The
agent could then have reason to modify its goals, either to satisfy or to frustrate
those preferences.
•	Preferences concerning own goal content . An agent might have some final goal con -
cerned with the agent’s own goal content. For example, the agent might have a final
goal to become the type of agent that is motivated by certain values rather than
others (such as compassion rather than comfort).
•	Storage costs . If the cost of storing or processing some part of an agent’s utility func -
tion is large compared to the chance that a situation will arise in which applying that
part of the utility function will make a difference, then the agent has an instrumental
reason to simplify its goal content, and it may trash the bit that is idle.11
We humans often seem happy to let our final values drift. This might often be
because we do not know precisely what they are. It is not surprising that we want
our beliefs  about our final values to be able to change in light of continuing self-
discovery or changing self-presentation needs. However, there are cases in which
we willingly change the values themselves, not just our beliefs or interpretations
INSTRUMENTAL  CON vERGENCE   |  111of them. For example, somebody deciding to have a child might predict that they
will come to value the child for its own sake, even though at the time of the deci -
sion they may not particularly value their future child or like children in general.
Humans are complicated, and many factors might be at play in a situation like
this.12 For instance, one might have a final value that involves becoming the kind
of person who cares about some other individual for his or her own sake, or one
might have a final value that involves having certain experiences and occupying
a certain social role; and becoming a parent—and undergoing the attendant goal
shift—might be a necessary aspect of that. Human goals can also have inconsist -
ent content, and so some people might want to modify some of their final goals to
reduce the inconsistencies.
Cognitive enhancement
Improvements in rationality and intelligence will tend to improve an agent’s
decision-  making, rendering the agent more likely to achieve its final goals. One
would therefore expect cognitive enhancement to emerge as an instrumental goal
for a wide variety of intelligent agents. For similar reasons, agents will tend to
instrumentally value many kinds of information.13
Not all kinds of rationality, intelligence, and knowledge need be instrumentally
useful in the attainment of an agent’s final goals. “Dutch book arguments” can be
used to show that an agent whose credence function violates the rules of probabil -
ity theory is susceptible to “money pump” procedures, in which a savvy bookie
arranges a set of bets each of which appears favorable according to the agent’s
beliefs, but which in combination are guaranteed to result in a loss for the agent,
and a corresponding gain for the bookie.14 However, this fact fails to provide any
strong general instrumental reasons to iron out all probabilistic incoherency.
Agents who do not expect to encounter savvy bookies, or who adopt a general
policy against betting, do not necessarily stand to lose much from having some
incoherent beliefs—and they may gain important benefits of the types mentioned:
reduced cognitive effort, social signaling, etc. There is no general reason to expect
an agent to seek instrumentally useless forms of cognitive enhancement, as an
agent might not value knowledge and understanding for their own sakes.
Which cognitive abilities are instrumentally useful depends both on the agent’s
final goals and on its situation. An agent that has access to reliable expert advice
may have little need for its own intelligence and knowledge. If intelligence and
knowledge come at a cost, such as time and effort expended in acquisition, or
increased storage or processing requirements, then the agent might prefer less
knowledge and less intelligence.15 The same can hold if the agent has final goals
that involve being ignorant of certain facts; and likewise if an agent faces incen -
tives arising from strategic commitments, signaling, or social preferences.16
Each of these countervailing reasons often comes into play for human beings.
Much information is irrelevant to our goals; we can often rely on others’ skill and
expertise; acquiring knowledge takes time and effort; we might intrinsically value
112  |  THE  SUPERINTELLIGENT  WILLcertain kinds of ignorance; and we operate in an environment in which the ability
to make strategic commitments, socially signal, and satisfy other people’s direct
preferences over our own epistemic states is often more important to us than sim -
ple cognitive gains.
There are special situations in which cognitive enhancement may result in an
enormous increase in an agent’s ability to achieve its final goals—in particular,
if the agent’s final goals are fairly unbounded and the agent is in a position to
become the first superintelligence and thereby potentially obtain a decisive strate -
gic advantage, enabling the agent to shape the future of Earth-originating life and
accessible cosmic resources according to its preferences. At least in this special
case, a rational intelligent agent would place a very high instrumental value on
cognitive enhancement.
Technological perfection
An agent may often have instrumental reasons to seek better technology, which
at its simplest means seeking more efficient ways of transforming some given set
of inputs into valued outputs. Thus, a software agent might place an instrumental
value on more efficient algorithms that enable its mental functions to run faster
on given hardware. Similarly, agents whose goals require some form of physi -
cal construction might instrumentally value improved engineering technology
which enables them to create a wider range of structures more quickly and reli -
ably, using fewer or cheaper materials and less energy. Of course, there is a trade -
off: the potential benefits of better technology must be weighed against its costs,
including not only the cost of obtaining the technology but also the costs of learn -
ing how to use it, integrating it with other technologies already in use, and so
forth.
Proponents of some new technology, confident in its superiority to existing
alternatives, are often dismayed when other people do not share their enthusiasm.
But people’s resistance to novel and nominally superior technology need not be
based on ignorance or irrationality. A technology’s valence or normative charac -
ter depends not only on the context in which it is deployed, but also the vantage
point from which its impacts are evaluated: what is a boon from one person’s
perspective can be a liability from another’s. Thus, although mechanized looms
increased the economic efficiency of textile production, the Luddite handloom
weavers who anticipated that the innovation would render their artisan skills
obsolete may have had good instrumental reasons to oppose it. The point here
is that if “technological perfection” is to name a widely convergent instrumental
goal for intelligent agents, then the term must be understood in a special sense—
technology must be construed as embedded in a particular social context, and its
costs and benefits must be evaluated with reference to some specified agents’ final
values.
It seems that a superintelligent singleton —a superintelligent agent that faces no
significant intelligent rivals or opposition, and is thus in a position to determine
INSTRUMENTAL  CON vERGENCE   |  113global policy unilaterally—would have instrumental reason to perfect the tech -
nologies that would make it better able to shape the world according to its pre -
ferred designs.17 This would probably include space colonization technology, such
as von Neumann probes. Molecular nanotechnology, or some alternative still
more capable physical manufacturing technology, also seems potentially very
useful in the service of an extremely wide range of final goals.18
Resource acquisition
Finally, resource acquisition is another common emergent instrumental goal,
for much the same reasons as technological perfection: both technology and
resources facilitate physical construction projects.
Human beings tend to seek to acquire resources sufficient to meet their basic
biological needs. But people usually seek to acquire resources far beyond this
minimum level. In doing so, they may be partially driven by lesser physical desid -
erata, such as increased convenience. A great deal of resource accumulation is
motivated by social concerns—gaining status, mates, friends, and influence,
through wealth accumulation and conspicuous consumption. Perhaps less com -
monly, some people seek additional resources to achieve altruistic ambitions or
expensive non-social aims.
On the basis of such observations it might be tempting to suppose that a super -
intelligence not facing a competitive social world would see no instrumental rea -
son to accumulate resources beyond some modest level, for instance whatever
computational resources are needed to run its mind along with some virtual
reality. Yet such a supposition would be entirely unwarranted. First, the value of
resources depends on the uses to which they can be put, which in turn depends on
the available technology. With mature technology, basic resources such as time,
space, matter, and free energy, could be processed to serve almost any goal. For
instance, such basic resources could be converted into life. Increased computa -
tional resources could be used to run the superintelligence at a greater speed and
for a longer duration, or to create additional physical or simulated lives and civi -
lizations. Extra physical resources could also be used to create backup systems or
perimeter defenses, enhancing security. Such projects could easily consume far
more than one planet’s worth of resources.
Furthermore, the cost of acquiring additional extraterrestrial resources will
decline radically as the technology matures. Once von Neumann probes can be
built, a large portion of the observable universe (assuming it is uninhabited by
intelligent life) could be gradually colonized—for the one-off cost of building and
launching a single successful self-reproducing probe. This low cost of celestial
resource acquisition would mean that such expansion could be worthwhile even if
the value of the additional resources gained were somewhat marginal. For exam -
ple, even if a superintelligence’s final goals only concerned what happened within
some particular small volume of space, such as the space occupied by its original
home planet, it would still have instrumental reasons to harvest the resources of
114  |  THE  SUPERINTELLIGENT  WILLthe cosmos beyond. It could use those surplus resources to build computers to
calculate more optimal ways of using resources within the small spatial region of
primary concern. It could also use the extra resources to build ever more robust
fortifications to safeguard its sanctum. Since the cost of acquiring additional
resources would keep declining, this process of optimizing and increasing safe -
guards might well continue indefinitely even if it were subject to steeply diminish -
ing returns.19
Thus, there is an extremely wide range of possible final goals a superintelli -
gent singleton could have that would generate the instrumental goal of unlimited
resource acquisition. The likely manifestation of this would be the superin -
telligence’s initiation of a colonization process that would expand in all direc -
tions using von Neumann probes. This would result in an approximate sphere
of expanding infrastructure centered on the originating planet and growing in
radius at some fraction of the speed of light; and the colonization of the universe
would continue in this manner until the accelerating speed of cosmic expansion
(a consequence of the positive cosmological constant) makes further procure -
ments impossible as remoter regions drift permanently out of reach (this happens
on a timescale of billions of years).20 By contrast, agents lacking the technology
required for inexpensive resource acquisition, or for the conversion of generic
physical resources into useful infrastructure, may often find it not cost-effective
to invest any present resources in increasing their material endowments. The
same may hold for agents operating in competition with other agents of similar
powers. For instance, if competing agents have already secured accessible cos -
mic resources, there may be no colonization opportunities left for a late-starting
agent. The convergent instrumental reasons for superintelligences uncertain of
the non-existence of other powerful superintelligent agents are complicated by
strategic considerations that we do not currently fully understand but which may
constitute important qualifications to the examples of convergent instrumental
reasons we have looked at here.21
* * *
It should be emphasized that the existence of convergent instrumental reasons,
even if they apply to and are recognized by a particular agent, does not imply
that the agent’s behavior is easily predictable. An agent might well think of ways
of pursuing the relevant instrumental values that do not readily occur to us. This
is especially true for a superintelligence, which could devise extremely clever but
counterintuitive plans to realize its goals, possibly even exploiting as-yet undis -
covered physical phenomena.22 What is predictable is that the convergent instru -
mental values would be pursued and used to realize the agent’s final goals—not
the specific actions that the agent would take to achieve this.

One approach to the AI control problem goes like this:

* Observe what the user of the system says and does.
* Infer the user’s preferences.
* Try to make the world better according to the user’s preference, perhaps while working alongside the user and asking clarifying questions.

This approach has the major advantage that we can begin empirical work today — we can actually build systems which observe user behavior, try to figure out what the user wants, and then help with that. There are many applications that people care about already, and we can set to work on making rich toy models.

It seems great to develop these capabilities in parallel with other AI progress, and to address whatever difficulties actually arise, as they arise. That is, in each domain where AI can act effectively, we’d like to ensure that AI can also act effectively in the service of goals inferred from users (and that this inference is good enough to support foreseeable applications).

This approach gives us a nice, concrete model of each difficulty we are trying to address. It also provides a relatively clear indicator of whether our ability to control AI lags behind our ability to build it. And by being technically interesting and economically meaningful now, it can help actually integrate AI control with AI practice.

Overall I think that this is a particularly promising angle on the AI safety problem.