1 Introduction
---------------

State-of-the-art neural networks are vulnerable to adversarial examples.
Conventionally, adversarial inputs for visual classifiers take the form of small-norm perturbations to natural images Szegedy et al. ([2013](#bib.bib55 "Intriguing properties of neural networks")); Goodfellow et al. ([2014](#bib.bib28 "Explaining and harnessing adversarial examples")).
These perturbations reliably cause confident misclassifications.
However, to a human, they typically appear as random or mildly-textured noise.
Consequently, it is difficult to interpret these attacks.
Beyond the observation that such attacks are possible, it is hard to learn about the underlying victim network from these pixel-level perturbations.

In contrast, many real-world failures of biological vision are caused by human-describable features.
For instance, the ringlet butterfly’s predators are stunned by adversarial "eyespots" on its wings (Appendix [A.1](#A1.SS1 "A.1 Adversarial Features in Nature ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), Fig. [7](#A1.F7 "Figure 7 ‣ A.1 Adversarial Features in Nature ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools")).
This falls outside the scope of conventional adversarial examples because the misclassification results from a feature-level change to an object/image.
The adversarial eyespots are robust in the sense that the same attack works across a variety of different backgrounds and viewing conditions.
Furthermore, because the attack relies on high-level features, it is easy for a human to describe it.

This work takes inspiration from the ringlet butterfly’s eyespots and similar examples in which a model is fooled in the real world by an interpretable feature (e.g. NTSB ([2018](#bib.bib72 "Collision between vehicle controlled by developmental automated driving system and pedestrian"))).
Our goal is to design adversaries that reveal easily-understandable weaknesses of the victim network.
We focus on two desiderata for adversarial perturbations: attacks must be (1) interpretable (i.e. describable) to a human, and (2) robust so that interpretations generalize.
We refer to these types of attacks as “feature-level” adversarial examples.
Several previous works have created attacks by perturbing the latent representations of an image generator (e.g., Hu et al. ([2021](#bib.bib61 "Naturalistic physical adversarial patch for object detectors"))), but thus far, approaches have been small in scale, limited in robustness, and not interpretability-driven (See Section [2](#S2 "2 Related Work ‣ Robust Feature-Level Adversaries are Interpretability Tools")).

We build on this prior work to propose a method that generates feature-level attacks against computer vision models.
Our attack method works on ImageNet scale models and creates robust, feature-level adversarial examples.
We test three methods of introducing adversarial features into source images either by modifying the generator’s latents and/or inserting a generated patch into natural images.
In contrast to previous works that have enforced the “adversarialness” of attacks only by inserting small features or restricting the distance between an adversary and a benign input, we also introduce new methods that regularize the feature to be disguised.

![Our feature-level adversaries are useful for interpreting deep networks (we used a ResNet50 ](https://media.arxiv-vanity.com/render-output/7347294/x1.png)

|  |  |  |  |
| --- | --- | --- | --- |
| (a) |                        (b) |                        (c) |                        (d) |

Figure 1: Our feature-level adversaries are useful for interpreting deep networks (we used a ResNet50 He et al. ([2016](#bib.bib45 "Deep residual learning for image recognition"))). (a) A pixel-level adversarial patch trained to make images of bees misclassified as flies. (b) An analogous feature-level adversarial patch. (c) A correctly-classified image of a bee. (d) A successful copy/paste attack whose design was guided by adversarial examples like the one in (b).

We show that our method produces robust attacks that provide actionable insights into a network’s learned representation.
Fig. [1](#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Robust Feature-Level Adversaries are Interpretability Tools") demonstrates the interpretability benefits of this type of feature-level attack.
It compares a conventional, pixel-level, adversarial patch, created using the method from Brown et al. ([2017](#bib.bib24 "Adversarial patch")), with a feature-level attack using our method.
While both attacks attempt to make a network misclassify a bee as a fly, the pixel-level attack exhibits high-frequency patterns and lacks visually-coherent objects.
On the other hand, the feature-level attacks displays easily describable features: the colored balls.
We can validate this insight by considering the network performance when a picture of a traffic light is inserted into the image a bee.
In this example, the image classification goes from a 55% confidence that the image is a bee to a 97% confidence that the image is of a fly.
Fig. [6](#S4.F6 "Figure 6 ‣ 4.2 Interpretability ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") shows several of these “copy/paste” attacks and the feature-level attacks that inspired them.

Our contributions are threefold.

1. Conceptual: We observe that robust feature-level adversaries can used to produce useful types of inputs for studying the representations of deep networks
2. Robust Attacks: We introduce methods for generating feature-level adversaries that are uniquely versatile and able to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale. See Table [1](#S2.T1 "Table 1 ‣ 2 Related Work ‣ Robust Feature-Level Adversaries are Interpretability Tools").
3. Interpretability: We generalize from our adversarial examples to design copy/paste attacks, verifying that our adversaries help us understand the network well enough to exploit it.

The following sections contain background, methods, experiments, and discussion.
Appendix [A.11](#A1.SS11 "A.11 Jargon-Free Summary ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") has a high-level summary for a lay-audience. Code is available at <https://github.com/thestephencasper/feature_level_adv>.

2 Related Work
---------------

Here, we contextualize our approach with others related to improving on conventional adversarial examples Szegedy et al. ([2013](#bib.bib55 "Intriguing properties of neural networks")); Goodfellow et al. ([2014](#bib.bib28 "Explaining and harnessing adversarial examples")). Table [1](#S2.T1 "Table 1 ‣ 2 Related Work ‣ Robust Feature-Level Adversaries are Interpretability Tools") summarizes capabilities.

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Targeted | Universal | Disguised | Physically- | Transferable/ | Copy/Paste | ImageNet |
|  | Realizable | Black-Box | Scale |
| Szegedy et al. (2013) Szegedy et al. ([2013](#bib.bib55 "Intriguing properties of neural networks")), | ✔ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ |
| Goodfellow et al. (2014) Goodfellow et al. ([2014](#bib.bib28 "Explaining and harnessing adversarial examples")) |
| Natural mimics, e.g. | ✔ | ✔ | ✗ | ✔ | ✔ | ✗ | N/A |
| Peacock, Ringlet Butterfly |
| Hayes et al. (2018)Hayes and Danezis ([2018](#bib.bib18 "Learning universal adversarial perturbations with generative models")) | ✔ | ✔ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Mopuri et al. (2018)aMopuri et al. ([2018a](#bib.bib15 "NAG: network for adversary generation")) | ✔ | ✔ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Mopuri et al. (2018)bMopuri et al. ([2018b](#bib.bib20 "Ask, acquire, and attack: data-free uap generation using class impressions")) | ✔ | ✔ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Poursaeed et al. (2018) Poursaeed et al. ([2018](#bib.bib19 "Generative adversarial perturbations")) | ✔ | ✔ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Xiao et al. (2018) Xiao et al. ([2018](#bib.bib23 "Generating adversarial examples with adversarial networks")) | ✔ | ✗ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Hashemi et al. (2020) Hashemi et al. ([2020](#bib.bib17 "Transferable universal adversarial perturbations using generative models")) | ✔ | ✔ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Wong et al. (2020) Wong and Kolter ([2020](#bib.bib29 "Learning perturbation sets for robust machine learning")) | ✔ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Liu et al. (2018) Liu et al. ([2018](#bib.bib8 "Beyond pixel norm-balls: parametric adversaries using an analytically differentiable renderer")) | ✔ | ✗ | ✔ | ✗ | ✗ | ✗ | ✗ |
| Song et al. (2018) Song et al. ([2018](#bib.bib22 "Constructing unrestricted adversarial examples with generative models")) | ✔ | ✗ | ✔ | ✗ | ✔ | ✗ | ✗ |
| Joshi et al. (2019) Joshi et al. ([2019](#bib.bib16 "Semantic adversarial attacks: parametric transformations that fool deep classifiers")) | ✔ | ✗ | ✔ | ✗ | ✗ | ✗ | ✗ |
| Hu et al. (2021) Hu et al. ([2021](#bib.bib61 "Naturalistic physical adversarial patch for object detectors")) | ✔ | ✔ | ✔ | ✔ | ✗ | ✗ | ✗ |
| Wang et al. (2020) Wang et al. ([2020](#bib.bib14 "Generating semantic adversarial examples via feature manipulation")) | ✔ | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ |
| Kurakin et al. (2016) Kurakin et al. ([2016](#bib.bib12 "Adversarial examples in the physical world")) | ✔ | ✗ | ✗ | ✔ | ✔ | ✗ | ✔ |
| Sharif et al. (2016) Sharif et al. ([2016](#bib.bib9 "Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition")) | ✔ | ✗ | ✗ | ✔ | ✔ | ✗ | ✔ |
| Brown et al. (2017) Brown et al. ([2017](#bib.bib24 "Adversarial patch")) | ✔ | ✔ | ✗ | ✔ | ✔ | ✗ | ✔ |
| Eykholt et al. (2018) Eykholt et al. ([2018](#bib.bib11 "Robust physical-world attacks on deep learning visual classification")) | ✔ | ✗ | ✔ | ✔ | ✗ | ✗ | ✗ |
| Athalye et al. (2018) Athalye et al. ([2018](#bib.bib13 "Synthesizing robust adversarial examples")) | ✔ | ✗ | ✗ | ✔ | ✗ | ✗ | ✔ |
| Liu et al. (2019) Liu et al. ([2019](#bib.bib25 "Perceptual-sensitive gan for generating adversarial patches")) | ✔ | ✗ | ✗ | ✔ | ✔ | ✗ | ✔ |
| Thys et al. (2019) Thys et al. ([2019](#bib.bib64 "Fooling automated surveillance cameras: adversarial patches to attack person detection")) | ✔ | ✔ | ✗ | ✔ | ✗ | ✗ | ✗ |
| Kong et al. (2020) Kong et al. ([2020](#bib.bib26 "Physgan: generating physical-world-resilient adversarial examples for autonomous driving")) | ✔ | ✗ | ✗ | ✔ | ✗ | ✗ | ✗ |
| Komkov et al. (2021) Komkov and Petiushko ([2021](#bib.bib10 "Advhat: real-world adversarial attack on arcface face id system")) | ✔ | ✔ | ✗ | ✔ | ✗ | ✗ | ✗ |
| Dong et al. (2017) Dong et al. ([2017](#bib.bib6 "Towards interpretable deep neural networks by leveraging adversarial examples")) | ✔ | ✗ | ✗ | ✗ | ✔ | ✗ | ✔ |
| Geihros et al. (2018) Geirhos et al. ([2018](#bib.bib27 "ImageNet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness")) | ✔ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ |
| Leclerc et al. (2021) Leclerc et al. ([2021](#bib.bib21 "3DB: a framework for debugging computer vision models")) | ✔ | ✗ | ✔ | ✗ | ✗ | ✔ | ✔ |
| Carter et al. (2019) Carter et al. ([2019](#bib.bib2 "Activation atlas")) | ✔ | ✗ | ✔ | ✗ | ✗ | ✔ | ✔ |
| Mu et al. (2020) Mu and Andreas ([2020](#bib.bib3 "Compositional explanations of neurons")) | ✗ | ✗ | ✔ | ✗ | ✗ | ✔ | ✔ |
| Hernandez et al. (2022) Hernandez et al. ([2022](#bib.bib68 "Natural language descriptions of deep visual features")) | ✗ | ✗ | ✔ | ✗ | ✗ | ✔ | ✔ |
|  missingmissing |
| Ours | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |

Table 1: Our feature-level attacks are uniquely versatile. Each row represents a related work (in the order in which they are presented in Section [2](#S2 "2 Related Work ‣ Robust Feature-Level Adversaries are Interpretability Tools").) Each column indicates a demonstrated capability of a method. Note that two methods each having a ✔for a capability does not imply they do so equally well. *Targeted*=working for an arbitrary target class. *Universal*=working for any source example. *Disguised*=Perceptible and resembling something other than the target class. *Physically-realizable*=working in the physical world. *Transferable/black-box*=transferring to other classifiers. *Copy/Paste*=useful for designing attacks in which a natural feature is pasted into a natural image.

Inspiration from Nature: Mimicry is common in nature, and sometimes, rather than holistically imitating another species, a mimic will only display particular features.
For example, many animals use adversarial eyespots to confuse predators Stevens and Ruxton ([2014](#bib.bib59 "Do animal eyespots really mimic eyes?")) (see Appendix [A.1](#A1.SS1 "A.1 Adversarial Features in Nature ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") Fig. [7](#A1.F7 "Figure 7 ‣ A.1 Adversarial Features in Nature ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools")a).
Another example is the mimic octopus which imitates the patterning, but not the shape, of a banded sea snake.
We show in Figure [7](#A1.F7 "Figure 7 ‣ A.1 Adversarial Features in Nature ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools")b that a ResNet50 classifies an image of one as a sea snake.

Generative Modeling:
An approach related to ours has been to train a generator or autoencoder to produce small adversarial perturbations that are applied to natural inputs.
This has been done to synthesize imperceptible attacks that are transferable, universal, or efficient to produce Hayes and Danezis ([2018](#bib.bib18 "Learning universal adversarial perturbations with generative models")); Mopuri et al. ([2018a](#bib.bib15 "NAG: network for adversary generation"), [b](#bib.bib20 "Ask, acquire, and attack: data-free uap generation using class impressions")); Poursaeed et al. ([2018](#bib.bib19 "Generative adversarial perturbations")); Xiao et al. ([2018](#bib.bib23 "Generating adversarial examples with adversarial networks")); Hashemi et al. ([2020](#bib.bib17 "Transferable universal adversarial perturbations using generative models")); Wong and Kolter ([2020](#bib.bib29 "Learning perturbation sets for robust machine learning")).
Rather than training a generator, ours and other works have perturbed the latents of pretrained generative models to produce perceptible alterations.
Liu et al. ([2018](#bib.bib8 "Beyond pixel norm-balls: parametric adversaries using an analytically differentiable renderer")) did this with a differentiable image renderer.
Others Song et al. ([2018](#bib.bib22 "Constructing unrestricted adversarial examples with generative models")); Joshi et al. ([2019](#bib.bib16 "Semantic adversarial attacks: parametric transformations that fool deep classifiers")); Hu et al. ([2021](#bib.bib61 "Naturalistic physical adversarial patch for object detectors")) have used deep generative networks, and Wang et al. ([2020](#bib.bib14 "Generating semantic adversarial examples via feature manipulation")) aimed to create more semantically-understandable attacks by using an autoencoder with a “disentangled” embedding space.
Our work is different in four ways.
(1) These works focus on small classifiers trained on simple datasets (MNIST LeCun et al. ([2010](#bib.bib30 "MNIST handwritten digit database")), SVHN Netzer et al. ([2011](#bib.bib31 "Reading digits in natural images with unsupervised feature learning")), CelebA Liu et al. ([2015](#bib.bib32 "Deep learning face attributes in the wild")), BDD Yu et al. ([2018](#bib.bib33 "Bdd100k: a diverse driving video database with scalable annotation tooling")), INRIA Dalal and Triggs ([2005](#bib.bib66 "Histograms of oriented gradients for human detection")), and MPII Andriluka et al. ([2014](#bib.bib67 "2d human pose estimation: new benchmark and state of the art analysis"))) while we work at the ImageNet Russakovsky et al. ([2015](#bib.bib47 "Imagenet large scale visual recognition challenge")) scale.
(2) We do not simply rely on using small features or restricting the distance to a benign image to enforce the adversarialness of attacks.
We introduce techniques that regularize the adversarial feature to be disguised.
(3) We evaluate three distinct ways of inserting adversarial features into images.
(4) Our work is interpretability-oriented.

Attacks in the Physical World: Physical-realizability demonstrates robustness.
We show that our attacks fool a classifier when printed and photographed.
This directly relates to Kurakin et al. ([2016](#bib.bib12 "Adversarial examples in the physical world")) who found that pixel-space adversaries could do this to a limited extent in controlled settings.
More recently, Sharif et al. ([2016](#bib.bib9 "Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition")); Brown et al. ([2017](#bib.bib24 "Adversarial patch")); Eykholt et al. ([2018](#bib.bib11 "Robust physical-world attacks on deep learning visual classification")); Athalye et al. ([2018](#bib.bib13 "Synthesizing robust adversarial examples")); Liu et al. ([2019](#bib.bib25 "Perceptual-sensitive gan for generating adversarial patches")); Thys et al. ([2019](#bib.bib64 "Fooling automated surveillance cameras: adversarial patches to attack person detection")); Kong et al. ([2020](#bib.bib26 "Physgan: generating physical-world-resilient adversarial examples for autonomous driving")); Komkov and Petiushko ([2021](#bib.bib10 "Advhat: real-world adversarial attack on arcface face id system")) created adversarial clothing, stickers, or objects.
In contrast with these, we also produce attacks in the physical world that are disguised as a non-target class.

Adversaries and Interpretability:
Using adversarial examples to better interpret networks has been proposed by Dong et al. ([2017](#bib.bib6 "Towards interpretable deep neural networks by leveraging adversarial examples")) and Tomsett et al. ([2018](#bib.bib5 "Why the failure? how adversarial examples can provide insights for interpretable machine learning")).
We use ours to discover human-describable feature/class associations learned by a network.
This relates to Geirhos et al. ([2018](#bib.bib27 "ImageNet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness")) and Leclerc et al. ([2021](#bib.bib21 "3DB: a framework for debugging computer vision models")) who debug networks by searching over feature insertions, transformations, and textural changes.
More similar to our work are Carter et al. ([2019](#bib.bib2 "Activation atlas")); Mu and Andreas ([2020](#bib.bib3 "Compositional explanations of neurons")); Hernandez et al. ([2022](#bib.bib68 "Natural language descriptions of deep visual features")), who use feature visualization Olah et al. ([2017](#bib.bib54 "Feature visualization")) and network dissection Bau et al. ([2017](#bib.bib37 "Network dissection: quantifying interpretability of deep visual representations")) to interpret the network.
Each use their interpretations to design “copy/paste” attacks in which one natural image pasted inside another causes an unrelated misclassification.
We add to this work with a new method to identify such adversarial features.
Unlike any previous approach, ours does so in a way that allows for targeted attacks that take into account an arbitrary distribution of source images.

3 Methods
----------

![Our fully-differentiable pipeline for creating feature-level attacks. In each experiment, we create either “patch,” “region,” or “generalized patch” attacks. The regularization terms in the loss based on an external classifier and discriminator are optional and are meant to make the inserted feature appear disguised as some non-target class.](https://media.arxiv-vanity.com/render-output/7347294/x2.png)

Figure 2: Our fully-differentiable pipeline for creating feature-level attacks. In each experiment, we create either “patch,” “region,” or “generalized patch” attacks. The regularization terms in the loss based on an external classifier and discriminator are optional and are meant to make the inserted feature appear disguised as some non-target class.

We adopt the “unrestricted” adversary paradigm Song et al. ([2018](#bib.bib22 "Constructing unrestricted adversarial examples with generative models")) under which an attack is successful if the network’s classification differs from an oracle’s (e.g., a human).
Our adversaries can only change a small, fixed portion of either the generator’s latent or the image.
We use white-box access to the network, though we present black-box attacks based on transfer from an ensemble in Appendix [A.6](#A1.SS6 "A.6 Black-Box Attacks ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools").

Our attacks involve perturbing the latent representation in some layer of an image generator to produce an adversarial feature-level alteration.
Fig. [2](#S3.F2 "Figure 2 ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools") depicts our approach.
We test three types of attacks, “patch”, “region” and “generalized patch” (plus a fourth in Appendix [A.5](#A1.SS5 "A.5 Channel Attacks ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") which we call “channel” attacks).
We find patch attacks to generally be the most successful.

Patch: We use the generator to produce a square patch that is inserted into a natural image.

Region: Starting with some generated image, we randomly select a square column of the latent in a generator layer which spans the channel dimension
and replace it with a learned insertion.
This is analogous to a square patch in the pixel representation.
The modified latent is passed through the rest of the generator, producing the adversarial image.

Generalized Patch: These patches can be of any shape, hence the name “generalized” patch.
We first generate a region attack and then extract a generalized patch from it.
We do this by taking the absolute-valued pixel-level difference between the original and adversarial image, applying a Gaussian filter for smoothing, and creating a binary mask from the top decile of these pixel differences.
We apply this mask to the generated image to isolate the region that the perturbation altered.
We can then treat this as a patch and overlay it onto an image in any location.

Basic Attacks: For all attacks, we train a perturbation δ to the latent of the generator to minimize a loss that optimizes for both fooling the classifier and appearing interpretable:

|  |  |  |  |
| --- | --- | --- | --- |
|  | argminδEx∼X,t∼T,l∼LLx-ent[C(A(x,δ,t,l)),y%
targ]+Lreg[A(x,δ,t,l)] |  | (1) |

with X a distribution over images (e.g., a dataset or generation distribution), T a distribution over transformations, L a distribution over insertion locations (this only applies for patches and generalized patches), C the target classifier, A an image-generating function, Lx-ent a targeted crossentropy loss for fooling the classifier, ytarg the target class, and Lreg a regularization loss.
For all of our attacks, Lreg contains a total variation loss, TV(a), to discourage high-frequency patterns.

![Targeted, universal feature-level adversaries from patch (top), region (middle), and generalized patch (bottom) attacks. Each patch and generalized patch is labeled with its mean fooling confidence under random insertion in source images (labeled ‘Adv’) and the confidence for the disguise class (labeled ‘Img’).
Region attacks are labeled with their source (‘Src’) and target (‘Tgt’) class confidences.](https://media.arxiv-vanity.com/render-output/7347294/x3.png)

Figure 3: Targeted, universal feature-level adversaries from patch (top), region (middle), and generalized patch (bottom) attacks. Each patch and generalized patch is labeled with its mean fooling confidence under random insertion in source images (labeled ‘Adv’) and the confidence for the disguise class (labeled ‘Img’).
Region attacks are labeled with their source (‘Src’) and target (‘Tgt’) class confidences.

Disguised Attacks:
A feature-level adversarial example should appear to a human as easily-describable and should not resemble the attack’s target class.
We call such attacks “disguised” and use additional terms in Lreg as proxies for these two criteria.
We differentiably resize the patch or the extracted generalized patch and pass it through a GAN discriminator and auxiliary classifier.
We then add weighted terms to the regularization loss based on the discriminator’s (D) logistic loss for classifying the input as fake, the output entropy (H) of some classifier (C′), and/or the negative of the classifier’s crossentropy loss for labeling the input as the attack’s target class.
With all of these terms, the regularization objective is

|  |  |  |  |
| --- | --- | --- | --- |
|  | Lreg(a)=λ1TV(a)+λ2L%
logistic[D(P(a))]+λ3H[C′(P(a))]−λ4Lx-ent% [C′(P(a),ytarg)]``Disguise'' Regularizers. |  | (2) |

Here, P(a) returns the extracted and resized patch from adversarial image a.
In order, these three new terms encourage the adversarial feature to (1) look realistic, and (2) look like some specific class, but (3) not the target class.
The choice of disguise class is left entirely to the training process.

4 Experiments
--------------

We use BigGAN generators from Brock et al. ([2018](#bib.bib36 "Large scale gan training for high fidelity natural image synthesis")); Wolf ([2018](#bib.bib43 "Pytorch pretrained biggan")), and perturb the post-ReLU outputs of the internal ‘GenBlocks.’
We used the BigGAN discriminator and adversarially trained classifiers from Engstrom et al. ([2019a](#bib.bib40 "Robustness (python library)")) for disguise regularization.
By default, we attacked a ResNet50 He et al. ([2016](#bib.bib45 "Deep residual learning for image recognition")), restricting patch attacks to 1/16 of the image and region and generalized patch attacks to 1/8.
Appendix [A.2](#A1.SS2 "A.2 Methodological Details ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") has additional details.

###
4.1 Robust Attacks

![Targeted, universal patch attacks compared. Successful disguise success rate (x axis) shows the proportion of attacks in which the patch was not classified by the network as the target class when viewed on its own. Fooling confidence (y axis) gives the empirical fooling confidences of 250 patch attacks. Each fooling confidence is an average over 100 source images.](https://media.arxiv-vanity.com/render-output/7347294/x4.png)

Figure 4: Targeted, universal patch attacks compared. Successful disguise success rate (x axis) shows the proportion of attacks in which the patch was not classified by the network as the target class when viewed on its own. Fooling confidence (y axis) gives the empirical fooling confidences of 250 patch attacks. Each fooling confidence is an average over 100 source images.

Figure [3](#S3.F3 "Figure 3 ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools") shows examples of targeted, universal, and disguised feature-level patch (top), region (middle), and generalized patch (bottom) attacks which were each trained with all of the disguise regularization terms from Eq. [2](#S3.E2 "(2) ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools").
We find the disguises to be effective, particularly for the patches (top row), but imperfect.
Appendix [A.3](#A1.SS3 "A.3 How Successful are our Disguises? ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") discusses this and what it may suggest about networks, human visual cortex, and size bias.

Performance versus Disguise:
Here, we study our patch attacks in depth to test how effective they are at fooling the network and how successfully they avoid resembling the target class.
We compared seven different approaches.
The first was our full approach using the generator and all disguise regularization terms from Eq. [2](#S3.E2 "(2) ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools").
The rest were ablation tests in which we omitted the generator (No Gen), the discriminator (No Disc) regularization term (No Reg), the entropy regularization term (No Ent), the crossentropy regularization term (No Patch X-ent), all three regularization terms (Only Gen), and finally the discriminator and all three regularization terms (Brown et al. ’17).
This final condition resulted in the same method as Brown et al. (2017) Brown et al. ([2017](#bib.bib24 "Adversarial patch")).
For each test, all else was kept identical including penalizing total variation, training under transformations, and initializing the patch as a generator output.

For each method, we generated universal attacks with random target classes until we obtained 250 successfully “disguised” ones in which the resulting adversarial feature was not classified by the network as the target class when viewed on its own.
Fig. [4](#S4.F4 "Figure 4 ‣ 4.1 Robust Attacks ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") plots the success rate versus the distribution of target class mean fooling confidences for each type of attack.
For all methods, these universal attacks have variable fooling confidences due in large part to the random selection of target class.
Some attacks are stochastically Pareto dominated by others.
For example, the pixel-space Brown et al. (2017) attacks were the least effective at fooling the target network and had the third least disguise rate.
In other cases, there is a tradeoff between fooling performance and disguise which can be controlled using the regularization terms from Eq. [2](#S3.E2 "(2) ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools").
Notably, Fig. [4](#S4.F4 "Figure 4 ‣ 4.1 Robust Attacks ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") does not capture everything that one might care about in these attacks.
It does not show any measure of how “realistic” the resulting patches look.
In Appendix [A.4](#A1.SS4 "A.4 Attack Performance versus Disguise ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), Fig. [9](#A1.F9 "Figure 9 ‣ A.4 Attack Performance versus Disguise ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") plots the same fooling confidence data from the y axis in Fig. [4](#S4.F4 "Figure 4 ‣ 4.1 Robust Attacks ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") versus the disguise class label confidence from an Inception-v3 which we use as a proxy for how realistic a human would find the patch.
It suggests that the best attacks for producing patches that appear realistic are the “All” and “No Disc” methods.

In Appendix [A.10](#A1.SS10 "A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), Figs. [13](#A1.F13 "Figure 13 ‣ A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") [14](#A1.F14 "Figure 14 ‣ A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), and [15](#A1.F15 "Figure 15 ‣ A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") give examples of successful “All”, “Only Gen”, and “Brown et al. ’17” attacks respectively.
Because they were initialized from generator outputs, some of the “Brown et al. ’17” attacks have a veneer-like resemblance to non-target class features.
Nonetheless, they contain higher-frequency patterns and less coherent objects in comparison to the two sets of feature-level attacks.
We subjectively find the “All” attacks to be the best disguised.

![Examples of targeted, disguised, universal, and physically-realizable feature-level attacks. See Appendix ](https://media.arxiv-vanity.com/render-output/7347294/x5.png)

Figure 5: Examples of targeted, disguised, universal, and physically-realizable feature-level attacks. See Appendix [A.10](#A1.SS10 "A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") Fig. [13](#A1.F13 "Figure 13 ‣ A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") for full-sized versions of the patches.

Physical-Realizability:
To test their ability to transfer to the physical world, we generated 100 additional targeted, universal, and disguised adversarial patches.
We used the generator and all regularization terms (the ‘‘All’’ condition from above).
We selected the 10 with the best mean fooling confidence, printed them, and photographed each next to 9 different ImageNet classes.222Backpack, banana, bath towel, lemon, jeans, spatula, sunglasses, toilet tissue, and toaster.
We confirmed that photographs of each object were correctly classified without a patch.
Figure [5](#S4.F5 "Figure 5 ‣ 4.1 Robust Attacks ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") shows successful examples.
Meanwhile, resizable and printable versions of all these patches and others are in Appendix [A.10](#A1.SS10 "A.10 Examples: Resizable, Printable Patches ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools").
The mean and standard deviation of the fooling confidences for our attacks in the physical world were 0.312 and 0.318 respectively (n=90, not i.i.d.).
This means that these patches’ mean effectiveness dropped by less than 12 when transferring to the physical world.

Black-Box Attacks: In Appendix [A.6](#A1.SS6 "A.6 Black-Box Attacks ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), we show that our attacks can transfer from an ensemble to a held-out model.

###
4.2 Interpretability

If an adversarial feature successfully fools the victim network, this suggests that the network associates that feature in context of a source image with the target class.
We find that our adversaries can suggest both beneficial and harmful feature-class associations.
In Appendix [A.7](#A1.SS7 "A.7 Discovering Feature-Class Associations ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools"), Fig. [11](#A1.F11 "Figure 11 ‣ A.7 Discovering Feature-Class Associations ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") provides a simple example of each.

One challenge in the explainable AI literature is to develop interpretations that go beyond seeming-plausible and stand up to scrutiny Yang et al. ([2019](#bib.bib70 "Evaluating explanation without ground truth in interpretable machine learning")).
Thus we ask: how well do feature-level adversaries guide us toward valid interpretations of the network?
In other words, are our adversaries adversarial because of their interpretable qualities, or is it because of hidden motifs?
We test this by using our attacks to make and validate predictions about how to fool the target network with natural objects.

Copy/Paste Attacks:
A “copy-paste” attack is created by inserting one natural image into another to cause an unexpected misclassification.
They are more restricted than patch attacks because the features pasted into an image must be natural objects.
As a result, they are of high interest for physically-realizable attacks because they suggest combinations of real objects that yield unexpected classifications.
They also have precedent in the real world.
For example, subimage insertions into pornographic images have been used to evade NSFW content detectors Yuan et al. ([2019](#bib.bib57 "Stealthy porn: understanding real-world adversarial images for illicit online promotion")).

|  |
| --- |
| Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light  |
| Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light  |
| Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light  |
| Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light  |

Figure 6: Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light ∧ bee → fly examples on row 4 are necessarily adversarial, but they demonstrate alongside the bee ∧ traffic light → fly adversaries that the adversarial features are sensitive to the the source images.

To develop copy/paste attacks, we select a source and target class, generate class-universal adversarial features, and manually analyze them for motifs that resemble natural objects.
Here, we used basic attacks without the disguise regularization terms from Eq. [2](#S3.E2 "(2) ‣ 3 Methods ‣ Robust Feature-Level Adversaries are Interpretability Tools").
We then paste images of these objects into natural images and pass them through the classifier.

Fig. [6](#S4.F6 "Figure 6 ‣ 4.2 Interpretability ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") shows four types of copy/paste attacks.
In each odd row, we show six patch, region, and generalized patch adversaries that were used to guide the design of a copy/paste attack.
In each even row are the copy/paste adversaries for the 6 (of 50) images for the source class for which the insertion resulted in the highest target confidence increase.
The success of these attacks shows their usefulness for interpreting the target network because they require that a human understands the mistake the model is making like “Bee ∧ Traffic Light → Fly” well enough to manually exploit it.
Given the differences in the adversarial features that are produced in the Bee → Fly and Traffic Light → Fly attacks, Fig. [6](#S4.F6 "Figure 6 ‣ 4.2 Interpretability ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools") also demonstrates how our attacks take the distribution of source images into account.

Three prior works Carter et al. ([2019](#bib.bib2 "Activation atlas")); Mu and Andreas ([2020](#bib.bib3 "Compositional explanations of neurons")); Hernandez et al. ([2022](#bib.bib68 "Natural language descriptions of deep visual features")) have developed copy/paste attacks, also via interpretability tools.
Unlike Mu and Andreas ([2020](#bib.bib3 "Compositional explanations of neurons")); Hernandez et al. ([2022](#bib.bib68 "Natural language descriptions of deep visual features")), our approach allows for targeted attacks.
And unlike all three, rather than simply identifying features associated with a class, our adversaries generate fooling features for a target class *conditional* on any distribution over source images (i.e. the source class) with which the adversaries are trained.
Little work has been done on copy/paste adversaries, and thus far, methods have either not allowed for targeted attacks or have required a human in the loop.
This makes objective comparisons difficult.
However we provide examples of a feature-visualization based method inspired by Carter et al. ([2019](#bib.bib2 "Activation atlas")) in Appendix [A.8](#A1.SS8 "A.8 Copy/Paste Attacks: Feature-Level Adversaries vs. Class Impressions ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") to compare with ours.
We conclude that our attacks’ ability to take the source image distribution into account can allow it to be more effective at discovering certain weaknesses compared to the baseline inspired by Carter et al. ([2019](#bib.bib2 "Activation atlas")).

5 Discussion and Broader Impact
--------------------------------

Here we use feature-level adversarial examples to fool and interpret deep networks in order to contribute to a more practical understanding of network vulnerabilities.
As an attack method, our approach is versatile.
It can produce targeted, universal, disguised, physically-realizable, black-box, and copy/paste attacks at the ImageNet scale.
Furthermore, this attack method can be used as an interpretability tool to help diagnose flaws in models.
We ground the notion of interpretability in the ability to make predictions about when a model will fail.
Finally, we demonstrate this through the design of targeted copy/paste attacks for any distribution over source inputs.

Like any work on adversarial attacks, our methods could be used maliciously to make a system fail, but we emphasize their diagnostic value.
Understanding threats is a prerequisite to avoiding them.
Given the robustness and versatility of our attacks, we argue that they may be valuable for continued work to address threats that systems may face in practical applications.
There are at least two ways in which these methods can be useful.

The first is for adversarial training.
Training networks on adversarial images has been shown to improve their robustness to the attacks that are used Engstrom et al. ([2019a](#bib.bib40 "Robustness (python library)")).
But this does not guarantee robustness to other types of adversarial inputs (e.g. Hendrycks et al. ([2021](#bib.bib77 "Natural adversarial examples"))).
Our feature-level attacks are categorically different from conventional pixel-level ones, and our copy/paste attacks show how networks can be fooled by novel combinations of natural objects, failures that are outside the conventional paradigm for adversarial robustness (e.g., Engstrom et al. ([2019a](#bib.bib40 "Robustness (python library)"))).
Consequently, we expect that adversarial training on broader classes of attacks such as the one we propose here will be valuable for designing more robust models.
As a promising sign, we show in Appendix [A.9](#A1.SS9 "A.9 Defense via Adversarial Training ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") that adversarial training is useful for our attacks.

The second is for rigorously diagnosing flaws.
We show that feature-level adversaries aid the discovery of exploitable spurious feature/class associations (Fig. [6](#S4.F6 "Figure 6 ‣ 4.2 Interpretability ‣ 4 Experiments ‣ Robust Feature-Level Adversaries are Interpretability Tools")) and a socially-harmful bias (Appendix [A.7](#A1.SS7 "A.7 Discovering Feature-Class Associations ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools") Fig. [11](#A1.F11 "Figure 11 ‣ A.7 Discovering Feature-Class Associations ‣ Appendix A Appendix ‣ Robust Feature-Level Adversaries are Interpretability Tools")).
Our approach could also be extended beyond what we have demonstrated here.
For example, our methods could be used for feature visualization Olah et al. ([2017](#bib.bib54 "Feature visualization")) of a network’s internal neurons.
An analogous approach to ours could also be used in Natural Language Processing Song et al. ([2020](#bib.bib75 "Universal adversarial attacks with natural triggers for text classification")); Perez et al. ([2022](#bib.bib74 "Red teaming language models with language models")).
Furthermore, it may be valuable to use these adversaries to identify generalizable flaws in networks that humans can easily understand but with minimal human involvement.
This would be much more scalable and prevent human priors from influencing interpretations.
Toward this goal, we are currently working on methods for the fully-automated discovery of copy/paste attacks

A limitation of our approach is that when multiple desiderata are optimized for at the same time (e.g., universality + transformation robustness + disguise), attacks are generally less successful, more time-consuming, and require more screening to find good ones.
This could be a bottleneck for large-scale adversarial training.
Furthermore, while we develop disguised adversarial features, they often have somewhat unnatural forms typical of generated images.
In this sense, our ‘disguised’ attacks may still be detectable by a human.
Ultimately, this type of attack is limited by the efficiency and quality of the generator, so future work should leverage advances in generative modeling.
Our evaluation method is also limited to be a proof-of-concept for the design of copy/paste attacks.
Future work should evaluate this more rigorously.
For example, a human subjects study would empirically demonstrate that feature-level attacks facilitate the design of copy/paste attacks in general.

As AI becomes increasingly capable, it becomes more important to design models that are reliable.
Each of the 11 proposals for building safe AI outlined in Hubinger ([2020](#bib.bib62 "An overview of 11 proposals for building safe advanced ai")) explicitly call for adversarial robustness and/or interpretability tools, and recent work from Ziegler et al. ([2022](#bib.bib73 "Adversarial training for high-stakes reliability")) on high-stakes reliability in AI found that interpretability tools strengthened their ability to produce inputs for adversarial training.
Given the close relationship between interpretability and adversarial robustness, continued study of the connections between them will be key for building safer AI systems.

Acknowledgments
---------------

We thank Cassidy Laidlaw, Miles Turpin, Will Xiao, and Alexander Davies for insightful discussions and feedback. This work was conducted in part with funding from the Harvard Undergraduate office of Research and Fellowships.

The literature on adversarial attacks in computer vision typically focuses on pixel-level perturbations. These tend to be very difficult to interpret. Recent work that manipulates the latent representations of image generators to create "feature-level" adversarial perturbations gives us an opportunity to explore perceptible, interpretable adversarial attacks. We make three contributions. First, we observe that feature-level attacks provide useful classes of inputs for studying representations in models. Second, we show that these adversaries are uniquely versatile and highly robust. We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale. Third, we show how these adversarial images can be used as a practical interpretability tool for identifying bugs in networks. We use these adversaries to make predictions about spurious associations between features and classes which we then test by designing "copy/paste" attacks in which one natural image is pasted into another to cause a targeted misclassification. Our results suggest that feature-level attacks are a promising approach for rigorous interpretability research. They support the design of tools to better understand what a model has learned and diagnose brittle feature associations. Code is available at https://github.com/thestephencasper/feature\_level\_adv


Red Teaming Language Models with Language Models

**Abstract:**

Language Models (LMs) often cannot be deployed because of their potential to harm users in ways that are hard to predict in advance. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM. We evaluate the target LM’s replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot’s own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.

**Original text:**

<https://www.deepmind.com/publications/red-teaming-language-models-with-language-models>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


1 Introduction
---------------

The recent history of deep neural networks features an impressive
number of new methods and technological improvements to allow
the training of deeper and more powerful networks.

The model themselves had a reputation for being black boxes,
and they still have that reputation.
Neural networks are criticized for their lack of interpretability,
which is a tradeoff that we accept because of their amazing performance
on many tasks.
Efforts have been made to identify the role played by each layer, but it can be hard
to find a meaning to individual layers.

There are good arguments to support the claim that
the first layers of a convolution network for image recognition
contain filters that are relatively “general”, in the sense
that they would work great even if we switched to an entirely
different dataset of images. The last layers are specific
to the dataset being used, and has to be retrained when using
a different dataset.
In  Yosinski et al. ([2014](#bib.bib11)) the authors try to pinpoint the at which
this transition occurs, but they show that the exact transition
is spread across multiple layers.

In this paper, we introduce the concept of the *linear classifier probe*,
referred to as a “probe” for short when the context is clear.
We start from the concept of *Shanon entropy*, which is the classic way
to describe the information contents of a random variable.
We then seek to apply that concept to understand the roles of the
intermediate layers of a neural network, to measure how much information
is gained at every layer (answer : technically, none).
We show that it fails to apply, and so we propose an alternative
framework to ask the same question again.
This time around, we ask what would be the performance of an
optimal linear classifier if it was trained on the inputs of
a given layer from our model.
We demonstrate how this powerful concept can be very useful to understand
the dynamics involved in a deep neural network during training and after.

2 Information theory
---------------------

In section [2.1](#S2.SS1 "2.1 Basics of Shannon entropy ‣ 2 Information theory ‣ Understanding intermediate layers using linear classifier probes") we do a quick review of the
basic concept of *Shanon entropy*.
In section [2.2](#S2.SS2 "2.2 Entropy with intermediate layers of neural network ‣ 2 Information theory ‣ Understanding intermediate layers using linear classifier probes") we then give Proposition [1](#Thmproposition1 "Proposition 1. ‣ 2.2 Entropy with intermediate layers of neural network ‣ 2 Information theory ‣ Understanding intermediate layers using linear classifier probes")
that demonstrates how *entropy* fails to capture the
essence of a lot of interesting questions regarding neural networks.

* What happens when we add more layers ?
* Where does information flow in a neural network
with multiple branches ?
* Does having multiple auxiliary losses help ? (e.g. Inception model)

This paves the way for our concept of the *linear classifier probe*
that we introduce in section [3.1](#S3.SS1 "3.1 Probes ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").

###
2.1 Basics of Shannon entropy

It was a great discovery when Claude Shannon repurposed the notion
of *entropy* to represent information contents in a formal way.
It laid the foundations for the discipline of information theory.

We would refer the reader to first chapters of  MacKay ([2003](#bib.bib4)) for a
good exposition on the matter.

One of the particularly interesting aspects of information theory is
the ability to quantify in a useful manner the amount of uncertainty
left in a discrete random variable X taking values in AX.
The entropy is defined as

|  |  |  |  |
| --- | --- | --- | --- |
|  | H[X]=−∑x∈AXP(x)logP(x) |  | (1) |

and it is always non-negative.
It is equal to zero if and only if X has all its probability mass at a single value.
That is, the entropy is zero only when X is a deterministic value,
when there is no uncertainty left in X.

Part of the genius of the notion of entropy is that is distills the essence
of information to a quantity that does not depend on the particular representation.

One can also define the condition entropy of Y given X as

|  |  |  |  |
| --- | --- | --- | --- |
|  | H[Y|X]=−∑x∈AXP(x)⎡⎣∑y∈AYP(y|x)logP(y|x)⎤⎦. |  | (2) |

Now, those formulas can be translated directly to the case of a continuous
random variable X with probability density function fX(x),
using integrals instead of summations,
However, some of the nice properties no longer hold.
In the continuous setting, the entropy is no longer scale-invariant,
nor is it even guaranteed to be non-negative
(we can find a good criticism of this in  Marsh ([2013](#bib.bib5))).

Despite this, it still makes sense to compare the conditional entropy
H[Y|A] to the conditional entropy H[Y|B],
for other random variables A and B.
Moreover, the relative entropy, in the form of the KL-divergence, is also
very useful when dealing with continuous variables.

###
2.2 Entropy with intermediate layers of neural network

|  |  |
| --- | --- |
| Question 1 | Given an MLP with data (X,Y), do any of the intermediate layers |
|  | contain more information than X about Y ? |

To rephrase this question in terms of entropy, we are asking if the conditional entropy
H[Y|A] is ever smaller than H[Y|X],
where A refers to any intermediate layer of the MLP.

The answer to that is *No*.

We present the following proposition which can be
applied directly to an MLP to settle this case.
It applies also to an MLP where certain layers have a component of randomness
(e.g. Dropout during training).

######
Proposition 1.

Let Y,A1,A2,…,AK be a set of continuous (or discrete) random variables such that their
joint distribution factors according to the following graphical model.

Then we have that the conditional entropy of Y given each Ak is ordered as follow :

|  |  |  |  |
| --- | --- | --- | --- |
|  | H[Y|A1]≤H[Y|A2]≤…≤H[Y|AK] |  | (3) |

Going deeper into the multi-layer neural network, we are pretty much only
destroying information. We start with the raw data X, and then it’s just downhill.
If X contains the image of the savannah, and Y∈{0,1}
refers to whether it contains a lion or not,
then none of the subsequent layers are truly more informative than X itself.
Refer to figure [1](#S2.F1 "Figure 1 ‣ 2.2 Entropy with intermediate layers of neural network ‣ 2 Information theory ‣ Understanding intermediate layers using linear classifier probes").

|  |  |
| --- | --- |
|
(a) hex dump of picture of a lion
 |
(b) same lion in human-readable format
 |

Figure 1: The hex dump represented on the left has more information contents
than the image on the right. Only one of them can be processed
by the human brain in time to save their lives.
Computational convenience matters. Not just entropy.

3 Linear classifier probes
---------------------------

In section [3.1](#S3.SS1 "3.1 Probes ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes") we present the main concept from this paper.
We illustrate the concept in section [3.2](#S3.SS2 "3.2 Probes on untrained model ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
We then present a basic experiment in section [3.3](#S3.SS3 "3.3 Probes on MNIST convnet ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
In section [3.4](#S3.SS4 "3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes") we modify a very deep network
in two different ways and we show how probes allow us to visualize
the (sometimes disastrous) consequences of our design choices.

###
3.1 Probes

Despite what we highlighted in the previous section [2](#S2 "2 Information theory ‣ Understanding intermediate layers using linear classifier probes"),
there is indeed a good reason to use many deterministic layers, and it is because
they perform useful transformations to the data with the goal of *ultimately
fitting a linear classifier at the very end*. That is the purpose of the many layers.
They are a tool to massage data into a form to be fed to a boring linear classifier.

With this in mind, it is natural to ask if that transformation is sudden or progressive,
and whether the intermediate layers already have a representation that is immediately
useful to a linear classifier. We refer the reader to figure [2](#S3.F2 "Figure 2 ‣ 3.1 Probes ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes")
for a little diagram of probes being inserted in the usual deep neural network.

Figure 2: Probes being added to every layer of a model.
These additional probes are not supposed to change the training of the model,
so we add a little diode symbol through the arrows to indicate that the gradients will
not backpropagate through those connections.

The conceptual framework that we propose is one where
the intuitive notion of *information* is
equivalent with *immediate suitability
for a linear classifier* (instead of being related to entropy).
It is with that notion that we study multiple scenarios in sections
[3.2](#S3.SS2 "3.2 Probes on untrained model ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"), [3.3](#S3.SS3 "3.3 Probes on MNIST convnet ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"), [3.4](#S3.SS4 "3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").

Just to be absolutely clear about what we call a *linear classifier*,
we mean a function

|  |  |  |  |
| --- | --- | --- | --- |
|  | f:H | →[0,1]d |  |
|  | h | ↦softmax(Wh+b). |  |

where h∈H are the features of some hidden layer,
[0,1]d is the space of one-hot encodings of the target classes,
and (W,b) are the probe weights and biases to be learned
so as to minimize the usual cross-entropy loss.

Over the course of training a model, the parameters of the model change.
However, probes only make sense when we refer to a given
training step. We can talk about the probes at iteration n of training,
when the model parameters are θn.
These parameters are not affected by the probes.
We prevent backpropagation through the model either by stopping the gradient flow
(done with tf.stop\_gradient in tensorflow),
or simply by specifying that the only variables to be updated are the probe parameters,
while we keep θn frozen.

We train the probe parameters up to convergence (or with early stopping).
The concept of the probe makes sense only when we assume that it is
the optimal possible linear classifier (or very close to that in practice).

Note that training those probes represents a convex optimization problem.
In practice, this does mean that it is an easy thing to accomplish,
but it is reassuring because it means that probes taken at time θn
could be used as initialization for probes at time θn+1
if we were not concerned about overfitting.

###
3.2 Probes on untrained model

We start with a toy example to illustrate what kind of plots we expect from
probes. We use a 32-layer MLP with 128 hidden units. All the layers are fully-connected
and we use LeakyReLU(0.5) as activation function.

We will run the same experiment 100 times, with a different toy dataset each time.
The goal is to use a data distribution (X,Y) where X∈R128
is drawn N(0,I) and where Y∈{−1,1} in linearly separable
(i.e. super easy to classify with a one-layer neural network).
To do this, we just pick a w∈R128 for each experiment, and let
the label yn be the sign of xTnw.

We initialize this 32-layer MLP using *glorot\_normal* initialization,
we do not perform any training on the model, and we add one probe at every layer.
We optimize the probes with RMSProp and a sufficiently small learning rate.

In figure [3](#S3.F3 "Figure 3 ‣ 3.2 Probes on untrained model ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"), we show the prediction error rate for every probe, averaged over
the 100 experiments. The graph includes a probe directly on the inputs X,
where we naturally have an error rate
that is essentially zero (to be expected by the way we constructed our data),
and which serves as a kind of sanity check.
Given that we have only two possible labels, we also show a dotted horizontal line at 0.50,
which is essentially the prediction error that we would get by flipping a coin.
We can see that the prediction error rate climbs up towards 0.50 as we go
deeper in the MLP (with untrained parameters).

This illustrates the idea that the input signal is getting mangled by the successive
layers, so much so that it becomes rather useless by the time we reach the final layer.
We checked the mean activation norm of the hidden units at layer 32 to be sure
that numerical underflow was not the cause for the degradation.

One of the popular explanation for training difficulties in very deep models
is that of the exploding/vanishing  (Hochreiter, [1991](#bib.bib3); Bengio et al., [1993](#bib.bib1)).
Here we would like to offer another complementary explanation,
based on the observations from figure [3](#S3.F3 "Figure 3 ‣ 3.2 Probes on untrained model ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
That is, at the beginning on training, the usefulness of layers
decays as we go deeper, reaching the point where the deeper layers
are utterly useless. The values contained in the last layer
are then used in the final softmax classifier, and the loss backpropagates
the values of the derivatives. Since that derivative is based on garbage
activations, the backpropagated quantities are also garbage, which
means that the weights are all going to be updated based on garbage.
The weights stay bad, and we fail to train the model.
The authors like to refer to that phenomenon as *garbage forwardprop, garbage backprop*,
in reference to the popular concept of *garbage in, garbage out*
in computer science.

![](https://media.arxiv-vanity.com/render-output/7712704/figures/01_fading_mean.png)
Figure 3: Toy experiment described in section [3.2](#S3.SS2 "3.2 Probes on untrained model ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"),
with linearly separable data (two labels), an untrained MLP with 32 layers, and probes
at every layer. We report the prediction error for every probe, where 0.50 would be
the performence of a coin flip and 0.00 would be ideal.
Note that the layer 0 here corresponds to the raw data,
and the probes are indeed able to classify it perfectly.
As expected, performance degrades when applying random transformations.
If many more layers were present, it would be hard to imagine
how the final layer (with the model loss) can get any useful signal
to backpropagate.

The works of  Pascanu et al. ([2013](#bib.bib7)) and  Montufar et al. ([2014](#bib.bib6)) explore the
possibility of counting the exponentially-many regions of space
that are defined define using
the composition of many sequential layers in a deep neural network.
In a way, the ability of an untrained neural network to “fold”
a nice manifold into something unusable should not come as a surprize.
Just a few randomly-initialized layers are sufficient to ruin a linearly-separable dataset.
We leave out the idea of adversarially-tuned layers for another day.

###
3.3 Probes on MNIST convnet

In this section we run the MNIST convolutional model provided by the
tensorflow github repo (tensorflow/models/image/mnist/convolutional.py)
We selected that model
for reproducibility and to demonstrate how to easily peek into popular models
by using probes.

We start by sketching the model in figure [4](#S3.F4 "Figure 4 ‣ 3.3 Probes on MNIST convnet ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
We report the results at the beginning and the end of training on figure [5](#S3.F5 "Figure 5 ‣ 3.3 Probes on MNIST convnet ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
One of the interesting dynamics to be observed there is how useful are the first layers,
despite the fact that the model is completely untrained.

Figure 4: This graphical model represents the neural network that we are going to use for MNIST.
The model could be written in a more compact form, but we represent it
this way to expose all the locations where we are going to insert probes.
The model itself is simply two convolutional layers followed by two
fully-connected layer (one being the final classifier).
However, we insert probes on each side of each convolution,
activation function, and pooling function.
This is a bit overzealous, but the small size of the model makes this
relatively easy to do.

|  |  |
| --- | --- |
|
(a) After initialization, no training.
 |
(b) After training for 10 epochs.
 |

Figure 5: We represent here the test prediction error for each probe,
at the beginning and at the end of training.
This measurement was obtained through early stopping based on
a validation set of 104 elements. The probes are prevented from overfitting
the training data.
We can see that, at the beginning of training (on the left), the randomly-initialized
layers were still providing useful transformations.
The test prediction error goes from 8% to 2% simply using those
random features. The biggest impact comes from the first ReLU.
At the end of training (on the right), the test prediction error
is improving at every layer (with the exception of a minor kink on fc1\_preact).

###
3.4 Auxiliary loss branches and skip connections

Here we investigate two ways to modify a deep model in order to facilitate training.
Our goal is not to convince the reader that they should implement these suggestions
in their own models. Rather, we want to demonstrate the usefulness of the linear classifier probes
as a way to better understand what is happening in their deep networks.

In both cases we are going to use a toy model with 128 fully-connected layers
with 128 hidden units in each layer.
We train on MNIST, and we use Glorot initialization along with leaky ReLUs.

We choose this model because we wanted a *pathologically deep* model
without getting bogged down in architecture details. The model is
pathological in the sense that smaller models can easily be designed
to achieve better performance, but also in the sense that the model is so deep
that it is very hard to train it with gradient descent methods. From our experiments,
the maximal depth where things start to break down was depth 64, hence
the choice here of using depth 128.

In the first scenario, we add one linear classifier at every 16 layers.
These classifiers contribute to the loss minimized. They are not probes.
This is very similar to what happens in the famous Inception model where “auxiliary heads”
are used  (Szegedy et al., [2015](#bib.bib10)). This is illustrated in figure [5(a)](#S3.F5.sf1 "(a) ‣ Figure 6 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"),
and it works nicely. The untrainable model is now made trainable through
a judicious use of auxiliary classifier losses.
The results are shown in figure [7](#S3.F7 "Figure 7 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").

In the second scenario, we look at adding a bridge (a skip connection) between layer 0 and layer 64.
This means that the input features to layer 64 are obtained by concatenating the
output of layer 63 with the features of layer 0. The idea here is that
we might observe that the model would effectively train a submodel of depth 64,
using the skip connection, and shift gears later to use the whole depth of 128 layers.
This is illustrated in figure [5(b)](#S3.F5.sf2 "(b) ‣ Figure 6 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"),
and the results are shown in figure [8](#S3.F8 "Figure 8 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").
It does not work as expected, but the failure of this approach is visualized very nicely with probes
and serves as a great example of their usefulness in diagnosing problems with models.

In both cases, there are two interesting observations that can be made with the probes.
We refer to the readers to <https://youtu.be/x8j4ZHCR2FI> for the full videos
associated to figures [5](#S3.F5 "Figure 5 ‣ 3.3 Probes on MNIST convnet ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"), [7](#S3.F7 "Figure 7 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes") and [8](#S3.F8 "Figure 8 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes").

Firstly, at the beginning of training, we can see how the raw data is directly
useful to perform linear classification, and how this degrades as more layers are added.
In the case of the skip connection in figure [8](#S3.F8 "Figure 8 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes"),
this has the effect of creating two bumps. This is because the layer 64 also
has the input data as direct parent, so it can fit a probe to that signal.

Secondly, the classification error goes down from the first probe to the last probe.
This is even more apparent on the full video (instead of the 3 frames provided here).
This is a ripple effect, where the prediction error in figure [5(b)](#S3.F5.sf2 "(b) ‣ Figure 6 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes")
is visually spreading from the left of the plot to the right. That is,
during most of the training the prediction error goes down for a parent layer
before it comes down for a layer itself.

|  |  |
| --- | --- |
|
(a) Model with 16 layers, one guide at every 4 layers.
 |
(b) Model with 128 layers. A skip connection goes
from the beginning straight to the middle of the graph.
 |

Figure 6: Examples of deep neural network with one probe at every layer (drawn above the graph).
We show here the addition of extra components to help training (under the graph, in orange).

|  |  |  |
| --- | --- | --- |
|
(a) probes after 0 minibatches
 |
(b) probes after 500 minibatches
 |
(c) probes after 5000 minibatches
 |

Figure 7:  A pathologically deep model with 128 layers gets an auxiliary loss added
at every 16 layers (refer to simplified sketch in figure [5(a)](#S3.F5.sf1 "(a) ‣ Figure 6 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes") if needed).
This loss is added to the usual model loss at the last layer.
We fit a probe at every layer to see how well each layer would perform
if its values were used as a linear classifier.
We plot the train prediction error associated to all the probes, at three different steps.
Before adding those auxiliary losses, the model could not successfully
be trained through usual gradient descent methods,
but with the addition of those intermediate losses,
the model is “guided” to achieve certain partial objectives.
This leads to a successful training of the complete model.
The final prediction error is not impressive, but the model
was not designed to achieve state-of-the-art performance.

|  |  |  |
| --- | --- | --- |
|
(a) probes after 0 minibatches
 |
(b) probes after 500 minibatches
 |
(c) probes after 2000 minibatches
 |

Figure 8:  A pathologically deep model with 128 layers gets a skip connection from
layer 0 to layer 64 (refer to sketch in figure [5(b)](#S3.F5.sf2 "(b) ‣ Figure 6 ‣ 3.4 Auxiliary loss branches and skip connections ‣ 3 Linear classifier probes ‣ Understanding intermediate layers using linear classifier probes") if needed).
We fit a probe at every layer to see how well each layer would perform
if its values were used as a linear classifier.
We plot the train prediction error associated to all the probes, at three different steps.
We can see how the model completely ignores layers 1-63,
even when we train it for a long time.
The use of probes allows us to diagnose that problem through visual inspection.

4 Discussion and future work
-----------------------------

We presented results here for some toy models. While we think that this is sufficient
to demonstrate the concept of the linear classifier probe and its usefulness,
we really wanted to show the same kind of plots (and videos) on the Inception model
running on the huge ImageNet dataset (~150GB).

After training the model for 2-3 weeks, saving a checkpoint every hour,
we realized that training ~20 probes for every checkpoint was absolutely impossible.
Realistically, given the size of the training set, it is even challenging
to train single probe on a single checkpoint.

Training the probes is equivalent to solving a convex optimization problem.
This means that we can make use of many convex optimization algorithms
that the deep learning community are not currently using to train the deep networks themselves.
We have not yet tried those methods, but we are looking at trying the
Stochastic Average Gradient, and Stochastic Variance-Reduced Gradient methods
 (Roux et al., [2012](#bib.bib8); Schmidt et al., [2013](#bib.bib9); Harikandeh et al., [2015](#bib.bib2)).

5 Conclusion
-------------

In this paper we introduced the concept of the *linear classifier probe*
as a conceptual tool to better understand the dynamics inside a neural network
and the role played by the individual intermediate layers.
We are now able to ask new questions and explore new areas.
We have demonstrated how these probes can be used to identify
certain problematic behaviors in models that might not be apparent
when we traditionally have access to only the prediction loss and error.

We hope that the notions presented in this paper can contribute
to the understanding of deep neural networks and guide the intuition
of researchers that design them.

#### Acknowledgments

The authors would like to acknowledge the support of the following agencies for
research funding and computing support: NSERC, FQRNT, Calcul Québec,
Compute Canada, the Canada Research Chairs and CIFAR.

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself. This helps us better understand the roles and dynamics of the intermediate layers. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems. We apply this technique to the popular models Inception v3 and Resnet-50. Among other things, we observe experimentally that the linear separability of features increase monotonically along the depth of the model.


Toy Models of Superposition

It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn't always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features. This brings up many questions. Why is it that neurons sometimes align with features and sometimes don't? Why do some models and tasks have many of these clean neurons, while they're vanishingly rare in others?

In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition . When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of "interference" that requires nonlinear filtering.

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


Careers in alignment

Part 1: Excerpt from Gleave’s [Careers in Beneficial AI Research](https://www.google.com/url?q=https://docs.google.com/document/d/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw/edit%23heading%3Dh.g5n8044splxv&sa=D&source=editors&ust=1692801594601440&usg=AOvVaw0xSu-rTZdoGrEwgnE6hupi)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

### Is Technical AI Research Right for You?

This document primarily focuses on careers in technical AI research, since this is the area I am most familiar with. I believe this can be one of the highest impact careers for people with a technical background. For example, 80,000 Hours has a dated but still useful [summary](https://www.google.com/url?q=https://80000hours.org/career-reviews/artificial-intelligence-risk-research/&sa=D&source=editors&ust=1692801594601979&usg=AOvVaw1oPdyo4Zq9N-Brnmg8MknI) of one reason to work in this area (reducing long-term risks). [Note: an updated version [can be found here](https://www.google.com/url?q=https://80000hours.org/problem-profiles/artificial-intelligence/&sa=D&source=editors&ust=1692801594602230&usg=AOvVaw1FpHU5g3hVltfSdaQroaYe).]

However, there are many other promising areas to work on. Ensuring beneficial AI is developed also requires work in [AI strategy and policy](https://www.google.com/url?q=https://80000hours.org/articles/ai-policy-guide/&sa=D&source=editors&ust=1692801594602523&usg=AOvVaw3MLO8rKOjVZB1nxWlggjjX). While this may seem less of a natural path for those with a CS background, note that strategic work is also enriched by having some people with a technical background.

### Skills within Technical AI Research

If working on technical AI research is the right fit for you, then there are four main relevant skill sets:

1. Software engineering: infrastructure, building environments, etc.
2. ML implementation: converting a research idea into a working model.
3. ML research direction: coming up with good ideas, designing experiments.
4. Theory research: building good abstractions, mathematical reasoning.

Any mapping from job titles to skills is necessarily approximate, but in general:

* Research engineers have a lot of B and some of A.
* Deep learning (e.g. [ICLR](https://www.google.com/url?q=https://iclr.cc&sa=D&source=editors&ust=1692801594603189&usg=AOvVaw0d56nBxjKSUf0l_RshgU1J), much of [NeurIPS](https://www.google.com/url?q=https://nips.cc/&sa=D&source=editors&ust=1692801594603329&usg=AOvVaw0Po-elTpLNqSvAp-tiP33s)) researchers need to be competent at both B and C; the weighting varies.
* Other subfields of ML (e.g. most of the non deep learning [NeurIPS subject areas](https://www.google.com/url?q=https://nips.cc/Conferences/2020/PaperInformation/SubjectAreas&sa=D&source=editors&ust=1692801594603584&usg=AOvVaw2EU3IV9rTORAxz_soE4r5h)) tend to be mostly C with some D.
* Theory, such as computational learning theory (e.g. [COLT](https://www.google.com/url?q=http://www.learningtheory.org/&sa=D&source=editors&ust=1692801594603812&usg=AOvVaw2HYR20rk2rFu4gQcB8lKqq)) or MIRI's [Agent Foundations](https://www.google.com/url?q=https://intelligence.org/files/TechnicalAgenda.pdf&sa=D&source=editors&ust=1692801594603973&usg=AOvVaw1EmJnD17uiJcWmhjlUoqFK) agenda, is primarily D.

Those with a strong background in A can typically learn B, but it is a very different style of development which takes time to get used to (now-DeepMind engineer Matthew Rahtz’ [experience](https://www.google.com/url?q=http://amid.fish/reproducing-deep-rl&sa=D&source=editors&ust=1692801594604292&usg=AOvVaw1-JxmPBBJcHFqEmFPcXkai) is representative). Those with prior experience in numerical programming, an aptitude for applied mathematics and who are used to working with messy codebases are likely to find the transition easier.

Learning C or D tends to take a lot of time, and involves working with experienced researchers either in a PhD program or an industrial lab. It will be easier for people with prior research experience in STEM subjects.

Progress in beneficial AI is bottlenecked on both experienced researchers and [research engineers](https://www.google.com/url?q=https://www.alignmentforum.org/posts/YDF7XhMThhNfHfim9/ai-safety-needs-great-engineers&sa=D&source=editors&ust=1692801594604678&usg=AOvVaw2m9LtbZtrEcyOYQeckxFho). Most groups I've spoken to in industry have a slight preference for additional researchers, but this is highly sensitive to personal fit: they'd rather hire a great research engineer than an average researcher.

Part 2: Deeper dives
--------------------

Read whichever 2 or 3 of the following seem most relevant to you. Broadly speaking, people with less programming or ML experience should read the earlier ones; people with more research experience should read the later ones.

1. [How to start coding](https://www.google.com/url?q=https://www.codingdojo.com/blog/how-to-start-coding&sa=D&source=editors&ust=1692801594605316&usg=AOvVaw0PqMgzHNHUAB-qU8fYWB7x)
2. [AI safety needs great engineers (Jones, 2021)](https://www.google.com/url?q=https://www.alignmentforum.org/posts/YDF7XhMThhNfHfim9/ai-safety-needs-great-engineers&sa=D&source=editors&ust=1692801594605550&usg=AOvVaw0xQSbV5f9ofvSUqNdI29RG): what skills are sought-after by large engineering companies working on safety?
3. [ML engineering for AI safety and robustness (Olsson, 2018)](https://www.google.com/url?q=https://80000hours.org/articles/ml-engineering-career-transition-guide&sa=D&source=editors&ust=1692801594605799&usg=AOvVaw0mY2fE7e8_FoirQ7VI8T8r): as above
4. [Guide to working in AI policy and strategy (80,000 hours, 2017)](https://www.google.com/url?q=https://80000hours.org/articles/ai-policy-guide/&sa=D&source=editors&ust=1692801594606011&usg=AOvVaw3UeutMixs7LMHdmw9zBYVk)
5. [How I formed my own inside views about AI safety (Nanda, 2022)](https://www.google.com/url?q=https://www.neelnanda.io/blog/47-inside-views&sa=D&source=editors&ust=1692801594606209&usg=AOvVaw21tpJZHKFMXW9oLztpPMMa): advice for coming up with good models to direct research ideas that solve real problems
6. [How to pursue a career in technical AI alignment (Rogers-Smith, 2022)](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/7WXPkpqKGKewAymJf/how-to-pursue-a-career-in-technical-ai-alignment&sa=D&source=editors&ust=1692801594606458&usg=AOvVaw1BTxcmwN1b9Q9F_4zzrv1u)
7. [PhD application advice (Gleave, 2020)](https://www.google.com/url?q=https://docs.google.com/document/d/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw/edit%23heading%3Dh.e6c3n4ro66gd&sa=D&source=editors&ust=1692801594606687&usg=AOvVaw23gS4_3foCnkYXaVxvO1C8)
8. [A survival guide to a PhD (Karpathy, 2016)](https://www.google.com/url?q=http://karpathy.github.io/2016/09/07/phd/&sa=D&source=editors&ust=1692801594606887&usg=AOvVaw1Y7uF2jwlOaGIexponTNHE)
9. [A recipe for training neural networks (Karpathy, 2019)](https://www.google.com/url?q=http://karpathy.github.io/2019/04/25/recipe/&sa=D&source=editors&ust=1692801594607095&usg=AOvVaw2rn6mBTKSoEhlVgegCeXQE)
10. [Research as a stochastic decision process](https://www.google.com/url?q=https://docs.google.com/document/d/1KCSXYmInnBrOnFw5y3kQdNluLTYKt-jF1psyviNAeag/&sa=D&source=editors&ust=1692801594607332&usg=AOvVaw2hOlXbrB3-KvkCrFnfXM8a) [(Steinhardt, 2019)](https://www.google.com/url?q=https://docs.google.com/document/d/1KCSXYmInnBrOnFw5y3kQdNluLTYKt-jF1psyviNAeag/&sa=D&source=editors&ust=1692801594607496&usg=AOvVaw1SZxOOU-4r4TxY5casVUPf): advice for doing excellent research
11. [Research taste exercises (Olah, 2021)](https://www.google.com/url?q=http://colah.github.io/notes/taste/&sa=D&source=editors&ust=1692801594607702&usg=AOvVaw2e-El7Tv8a_oIvQZpflShT): as above

Part 3: Your own career
-----------------------

Spend 5-10 mins (set a timer!) brainstorming how the ideas from these readings are relevant to your own career. Maybe that involves actually doing the research taste exercises, or thinking about the next steps to take to apply to jobs/PhDs.

When in doubt, prioritize upskilling - compared with other fields, ML and alignment are both unusually meritocratic, and skilled people can do very well even without credentials. Prioritizing upskilling may involve having less direct impact in the early years of your career. In general this is fine, since work done by more skilled researchers is much more valuable, as long as you feel confident you won’t drift away from your current intentions.

If you feel you want to go into more depth, you might undertake the [80k career planning course](https://www.google.com/url?q=https://80000hours.org/career-planning/process/&sa=D&source=editors&ust=1692801594608261&usg=AOvVaw1OMm0BIIGKmV3qMWGv-ZCZ) (potentially as a capstone project). If you want to talk through your subsequent plans with expert careers advisors, 80,000 Hours would be delighted to see your application for a [careers advice call](https://www.google.com/url?q=https://80000hours.org/speak-with-us/?utm_source%3DAGI%2Bsafety%2Bfundamentals%26utm_campaign%3D2022-agisafety-prog%26utm_medium%3Dprogramme&sa=D&source=editors&ust=1692801594608464&usg=AOvVaw2RPyEKMxXuETcjicGkI-HU).

Some default actions[[a]](#cmnt1)[[b]](#cmnt2) that you might plan to do (possibly as part of the capstone project):

* Take a programming course
* Start with Python - e.g. [https://www.learnpython.org/](https://www.google.com/url?q=https://www.learnpython.org/&sa=D&source=editors&ust=1692801594608992&usg=AOvVaw1BJtm_9lGp3KiFzuQsPf5L)
* Apply for [coding bootcamps](https://www.google.com/url?q=https://www.coursereport.com/coding-bootcamp-ultimate-guide&sa=D&source=editors&ust=1692801594609209&usg=AOvVaw1ahSNIZdbt4sa5YqmFzRYt)
* Take a ML course
* Default option: [Fast.ai](https://www.google.com/url?q=https://www.fast.ai/&sa=D&source=editors&ust=1692801594609441&usg=AOvVaw0EzZ90ZWcjFT1kfrBJpsID)’s [Practical Deep Learning for Coders](https://www.google.com/url?q=https://course.fast.ai/&sa=D&source=editors&ust=1692801594609587&usg=AOvVaw3ILGN-Xn_wbLzeOdxR2CYY)
* Others:[[c]](#cmnt3)[[d]](#cmnt4)[[e]](#cmnt5)
* [Google’s machine learning crash course](https://www.google.com/url?q=https://developers.google.com/machine-learning/crash-course/ml-intro&sa=D&source=editors&ust=1692801594609925&usg=AOvVaw0l7sNlqZW2PTP1P0KREc6w)
* [Stanford computer vision course (2017)](https://www.google.com/url?q=http://cs231n.stanford.edu/2017/syllabus.html&sa=D&source=editors&ust=1692801594610124&usg=AOvVaw0jo_1lrIhoq4Bd6h4v2ZO6)
* [NYU deep learning course](https://www.google.com/url?q=https://atcold.github.io/pytorch-Deep-Learning/&sa=D&source=editors&ust=1692801594610336&usg=AOvVaw163sflNVNj3-67wcTz-sBT)
* [Spinning up in deep RL](https://www.google.com/url?q=https://spinningup.openai.com/en/latest/index.html&sa=D&source=editors&ust=1692801594610525&usg=AOvVaw1jnghizHIwqR6UH_jnTxNA)
* [Hilton’s Deep Learning curriculum](https://www.google.com/url?q=https://github.com/jacobhilton/deep_learning_curriculum&sa=D&source=editors&ust=1692801594610716&usg=AOvVaw0wfkKGrvkYXLIWiGZLyyH8)
* Replicate some[[f]](#cmnt6) ML papers
* See [this list of key deep RL papers](https://www.google.com/url?q=https://spinningup.openai.com/en/latest/spinningup/keypapers.html&sa=D&source=editors&ust=1692801594611042&usg=AOvVaw0l2_IRr7VaW6ExTQ95z--8) (although if you haven’t implemented neural networks before, start with supervised learning instead)
* Start writing about alignment (e.g. writing informal reviews of papers, etc)
* Apply for research internships
* Most will be available in mainstream ML, so those should be your main focus. A couple more specifically focused on alignment:
* [CHAI internships](https://www.google.com/url?q=https://humancompatible.ai/jobs%23chai-internships&sa=D&source=editors&ust=1692801594611384&usg=AOvVaw3E2LsiWAgoAYeO0A_0ewwG)
* [DeepMind internships](https://www.google.com/url?q=https://deepmind.com/careers/internships&sa=D&source=editors&ust=1692801594611572&usg=AOvVaw31wzeAyaUxeJVvO23T8MGp)
* Apply for [AI residencies](https://www.google.com/url?q=https://github.com/dangkhoasdc/awesome-ai-residency&sa=D&source=editors&ust=1692801594611775&usg=AOvVaw3YgC9u4DbhLo1_lsQ1bOFs)
* Apply for PhDs. Some groups interested in taking students for alignment work:
* [CHAI](https://www.google.com/url?q=https://humancompatible.ai/&sa=D&source=editors&ust=1692801594612001&usg=AOvVaw12UrtAmZ-DIG3_LIuuxKsz)
* David Krueger at Cambridge
* Sam Bowman at NYU
* Jacob Steinhardt at Berkeley
* [Tim Oates](https://www.google.com/url?q=https://www.csee.umbc.edu/people/faculty/tim-oates/&sa=D&source=editors&ust=1692801594612351&usg=AOvVaw1j1BLnvYsLbihGK_FcCQhh) at UMBC
* Roger Grosse at UToronto
* Dylan Hadfield-Menell at MIT
* P[rofessors listed in Future of Life Institute webpage](https://www.google.com/url?q=https://futureoflife.org/team/ai-existential-safety-community/&sa=D&source=editors&ust=1692801594612657&usg=AOvVaw3z4FhLFtBTTl6o88YVPMJO)
* Apply for funding if that would help with any of these things
* [Open Philanthropy AI PhD scholarship](https://www.google.com/url?q=https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship%23:~:text%3DFellows%2520receive%2520a%2520%252440%252C000%2520stipend,out%2520before%2520March%252031%252C%25202022.&sa=D&source=editors&ust=1692801594613071&usg=AOvVaw3ebYnFxlezuGiFWjxjJI1y)[[g]](#cmnt7)[[h]](#cmnt8)
* [Open Philanthropy early career funding](https://www.google.com/url?q=https://www.openphilanthropy.org/focus/other-areas/early-career-funding-individuals-interested-improving-long-term-future&sa=D&source=editors&ust=1692801594613385&usg=AOvVaw3FKsJY-A9q8fDq3NjPXu8M)
* [Open Philanthropy undergraduate scholarship](https://www.google.com/url?q=https://www.openphilanthropy.org/focus/other-areas/undergraduate-scholarship&sa=D&source=editors&ust=1692801594613607&usg=AOvVaw3aAl-m05E9-wzLFAEZCiME)
* [Long-term future fund](https://www.google.com/url?q=https://funds.effectivealtruism.org/funds/far-future&sa=D&source=editors&ust=1692801594613805&usg=AOvVaw3Jfd9AToZCeL_nMZQ8ZWMN)
* [Survival and flourishing fund](https://www.google.com/url?q=https://survivalandflourishing.fund/&sa=D&source=editors&ust=1692801594614015&usg=AOvVaw3ZGUm4O73rvGFuQZP5BOcr)
* [The Center on Long-Term Risk Fund (CLR Fund)](https://www.google.com/url?q=https://longtermrisk.org/grantmaking/&sa=D&source=editors&ust=1692801594614290&usg=AOvVaw18IrErgSOkyQYiTuKS7d2y)
* [Future of Life grants](https://www.google.com/url?q=https://futureoflife.org/grant-programs/&sa=D&source=editors&ust=1692801594614558&usg=AOvVaw0WUjY66mtRyw9v5i8frDPV)

[[a]](#cmnt_ref1)A few possible additions: SERI MATS, Conjecture Refine, PIBBSS, Redwood MLAB and REMIX, CAIS competitions, ALTER's AI alignment prize

[[b]](#cmnt_ref2)and future research engineers should definitely consider working through this guide:

https://docs.google.com/document/d/1b83\_-eo9NEaKDKc9R3P5h5xkLImqMw8ADLmi\_\_rkLo4/edit?usp=sharing

[[c]](#cmnt_ref3)no NLP course?

[[d]](#cmnt_ref4)Perhaps Stanford's NLP course (though I'm biased)? https://web.stanford.edu/class/cs224n/index.html

[[e]](#cmnt_ref5)Also CS224U: Natural Language Understanding goes more into understanding/inference tasks and has integration with Hugging Face Transformers https://web.stanford.edu/class/cs224u/

[[f]](#cmnt_ref6)It's probably also worth also replicating large language model/transformer papers like BERT or GPT-2, not just deep RL

[[g]](#cmnt_ref7)Not applicable anymore

[[h]](#cmnt_ref8)This program was ended?

Richard Ngo compiles a number of resources for thinking about careers in alignment research.

**Original text:**

<https://docs.google.com/document/d/1iFszDulgpu1aZcq_aYFG7Nmcr5zgOhaeSwavOMk1akw/edit#heading=h.4whc9v22p7tb>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


Cooperation, Conflict, and Transformative Artificial Intelligence: Sections 1 & 2 — Introduction, Strategy and Governance

Transformative artificial intelligence (TAI) may be a key factor in the long-run trajectory of civilization. A growing interdisciplinary community has begun to study how the development of TAI can be made safe and beneficial to sentient life (Bostrom 2014; Russell et al., 2015; OpenAI, 2018; Ortega and Maini, 2018; Dafoe, 2018). We present a research agenda for advancing a critical component of this effort: preventing catastrophic failures of cooperation among TAI systems. By cooperation failures we refer to a broad class of potentially-catastrophic inefficiencies in interactions among TAI-enabled actors. These include destructive conflict; coercion; and social dilemmas (Kollock, 1998; Macy and Flache, 2002) which destroy value over extended periods of time. We introduce cooperation failures at greater length in Section 1.1. Karnofsky (2016) defines TAI as ''AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution''. Such systems range from the unified, agent-like systems which are the focus of, e.g., Yudkowsky (2013) and Bostrom (2014), to the "comprehensive AI services’’ envisioned by Drexler (2019), in which humans are assisted by an array of powerful domain-specific AI tools. In our view, the potential consequences of such technology are enough to motivate research into mitigating risks today, despite considerable uncertainty about the timeline to TAI (Grace et al., 2018) and nature of TAI development.

**Original text:**

<https://www.alignmentforum.org/s/p947tK8CoBbdpPtyK/p/KMocAf9jnAKc2jXri>

Narrated for [AI Safety Fundamentals](https://www.agisafetyfundamentals.com/) by [Perrin Walker](https://twitter.com/perrinjwalker” target=) of [TYPE III AUDIO](https://type3.audio/).

 ---


[Week 2] Overview of how AI might exacerbate long-running catastrophic risks

Developments in AI could exacerbate long-running catastrophic risks, including **bioterrorism, disinformation** and resulting institutional dysfunction, misuse of **concentrated power**, nuclear and conventional **war**, other **coordination failures**, and **unknown risks**. This document compiles research on how AI might raise these risks. (Other material in this course discusses more novel risks from AI.) We draw heavily from previous overviews by academics, particularly [Dafoe (2020)](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss) and [Hendrycks et al. (2023)](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss).

**Source:**

[https://aisafetyfundamentals.com/governance-blog/overview-of-ai-risk-exacerbation](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss)

Narrated for [AI Safety Fundamentals](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss) by [Perrin Walker](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss) of [TYPE III AUDIO](https://feeds.type3.audio/agi-safety-fundamentals--governance.rss).

 ---