Deterring Nuclear Proliferation

HTTPSConnectionPool(host='www.osti.gov', port=443): Max retries exceeded with url: /servlets/purl/1525379 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1002)')))


Specification gaming examples


Epoch AI Blog


Visualizing the deep learning revolution

[![Richard Ngo](https://miro.medium.com/v2/resize:fill:88:88/0*69gwDFFNrP6TIq2l.jpg)](/@richardcngo?source=post_page-----722098eb9c5--------------------------------)Listen

Share

The field of AI has undergone a revolution over the last decade, driven by the success of deep learning techniques. This post aims to convey three ideas using a series of illustrative examples:

1. There have been huge jumps in the capabilities of AIs over the last decade, to the point where it’s becoming hard to specify tasks that AIs *can’t* do.
2. This progress has been primarily driven by scaling up a handful of relatively simple algorithms (rather than by developing a more principled or scientific understanding of deep learning).
3. Very few people predicted that progress would be anywhere near this fast; but many of those who did also predict that we might face [existential risk from AGI](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) in the coming decades.

I’ll focus on four domains: vision, games, language-based tasks, and science. The first two have more limited real-world applications, but provide particularly graphic and intuitive examples of the pace of progress.

Vision
======

Image recognition
-----------------

Image recognition has been a focus of AI for many decades. Early research focused on simple domains like handwriting; performance has now improved significantly, beating human performance on many datasets. However, it’s hard to interpret scores on benchmarks in an intuitive sense, so we’ll focus on domains where progress can be visualized more easily.

Image generation
----------------

In 2014, AI image generation advanced significantly with the introduction of Generative Adversarial Networks (GANs). However, the first GANs could only generate very simple or blurry images, like the ones below.

![]()Images with yellow borders are real, all others are GAN-generated.Over the next 8 years, image generation progressed at a very rapid rate; the figure below shows images generated by state-of-the-art systems in each year. Over the last two years in particular, these systems made a lot of progress in generating complex creative scenes in response to language prompts.

![]()This is an astounding rate of progress. What drove it? In part, it was the development of new algorithms — most notably GANs, transformers and diffusion models. However, the key underlying factor was scaling up the amount of compute and data used during training. One demonstration of this comes from the [Parti](https://parti.research.google/) series of image models, which includes four networks of different sizes (with parameter counts ranging from 350 million to 20 billion). Although they were all trained in the same way, for the three prompts below you can clearly see how much better the bigger models are than the smaller ones (e.g. by watching the ability to portray text gradually emerge).

![]()*Prompt: A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says Welcome Friends!*![]()Prompt: A map of the United States made out of sushi. It is on a table next to a glass of red wine.![]()Prompt: A squirrel gives an apple to a birdVideo generation
----------------

An early example of video generation came from DeepMind’s 2019 [DVD-GAN model](/syncedreview/deepmind-dvd-gan-impressive-step-toward-realistic-video-synthesis-12027d942e53). However, although the videos have some realistic features, almost all of the individual videos are noticeably malformed.

![]()More recently, researchers have focused on producing videos in response to text prompts. Here are four sets of examples from different systems released between April and October 2022, showcasing noticeable improvements in that period alone:

![]()[Video diffusion (April 2022)](https://video-diffusion.github.io/)![]()[CogVideo (May 2022)](https://arxiv.org/abs/2205.15868)![]()[Make-A-Video (September 2022)](https://ai.facebook.com/blog/generative-ai-text-to-video/)![]()[Imagen Video (October 2022)](https://imagen.research.google/video/)Games
=====

Video games
-----------

In 2013, the state of the art for using ML to learn to play video games was beating humans at a few very simple Atari games, like Pong and Breakout.

![]()By 2018, this had progressed to beating professionals at the most complex esports, in particular [StarCraft](https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii) and [DOTA2](https://openai.com/five/), in which players take thousands of actions in a row throughout games lasting up to an hour.****

![]()DeepMind’s AlphaStar model, which beat a top professional at StarCraft![]()OpenAI Five, which beat a team of top professionals at DOTA2And by 2022, capabilities had extended from narrow games to open-ended environments like Minecraft, where training on videos of human players allowed AIs to learn to take complex sequences of actions involving many different steps. Below you can see the many subtasks involved in making a diamond pickaxe, and a video of the AI carrying them out.

![]()Strategy games
--------------

2016 and 2017 saw DeepMind’s AlphaGo system beat the world’s best players at Go, a board game far more complex than chess.

![]()Notably, a later version of AlphaGo trained without any human data, only by playing against itself, reached superhuman level (indicated by the green line in the graph below) after less than three days of training, which involved rediscovering many aspects of Go theory and strategy that humans had developed over thousands of years.

![]()However, chess and Go are still quite simple games — players can only interact in very limited ways. 2022 saw [AIs reaching human-level performance at the board game Diplomacy](https://arxiv.org/abs/2210.05492) (a strategy game similar to Risk). Diplomacy requires players to form hidden alliances via complex language-based negotiations, as well as deceiving other players when betraying those alliances.

![]()Language-based tasks
====================

The level of language capabilities used in Diplomacy negotiations were enabled by the development over the last few years of large language models (LLMs) trained on billions of words of text. Despite only being trained to predict the next word, in order to do so very accurately LLMs develop a range of emergent skills, including broad knowledge about the world which can be accessed by giving them the right prompts. The main LLMs we’ll discuss in this section are OpenAI’s [GPT-2](https://openai.com/blog/better-language-models/), [GPT-3](https://en.wikipedia.org/wiki/GPT-3) and [ChatGPT](https://openai.com/blog/chatgpt/) models (the last of which you can [try online here](https://chat.openai.com/)) and several variants of [Google’s PaLM model](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html).

Text generation
---------------

While it’s hard to compare the results from different prompts, some samples from different models can give an intuitive sense for the rate of progress.

[In 2011, state-of-the-art language models](https://icml.cc/2011/papers/524_icmlpaper.pdf) couldn’t even put together grammatical sentences:

`*PROMPT: THE MEANING OF LIFE IS*`

`The meaning of life is the tradition of the ancient human reproduction: it is less favorable to the good boy for when to remove her bigger. In the show’s agreement unanimously resurfaced. The wild pasteured with consistent street forests were incorporated by the 15th century BE. In 1996 the primary rapford undergoes an effort that the reserve conditioning, written into Jewish cities, sleepers to incorporate the .St Eurasia that activates the population. Mar??a Nationale, Kelli, Zedlat-Dukastoe, Florendon, Ptu’s thought is. To adapt in most parts of North America, the dynamic`

`The meaning of life is the tradition of the ancient human reproduction: it is less favorable to the good boy for when to remove her bigger. In the show’s agreement unanimously resurfaced. The wild pasteured with consistent street forests were incorporated by the 15th century BE. In 1996 the primary rapford undergoes an effort that the reserve conditioning, written into Jewish cities, sleepers to incorporate the .St Eurasia that activates the population. Mar??a Nationale, Kelli, Zedlat-Dukastoe, Florendon, Ptu’s thought is. To adapt in most parts of North America, the dynamic`

A big jump in text generation came with the development of [GPT-2 in 2019](https://openai.com/blog/better-language-models/), which was consistently able to generate coherent sentences containing relevant knowledge about the world.

`*PROMPT: MILEY CYRUS WAS CAUGHT SHOPLIFTING FROM ABERCROMBIE AND FITCH ON HOLLYWOOD BOULEVARD TODAY*`

`The 19-year-old singer was caught on camera being escorted out of the store by security guards.
The singer was wearing a black hoodie with the label ‘Blurred Lines’ on the front and ‘Fashion Police’ on the back
The singer was also wearing a pair of black-rimmed glasses, a black jacket, black jeans and black sandals.
She was carrying a pair of black and white striped gloves and a small black bag.`

A further jump came with [GPT-3 in 2020](https://www.gwern.net/GPT-3#indiana-jones-tenure-denial), which was trained in a very similar way to GPT-2, but with around 100 times more parameters.

`*PROMPT: UNITED METHODISTS AGREE TO HISTORIC SPLIT — THOSE WHO OPPOSE GAY MARRIAGE WILL FORM THEIR OWN DENOMINATION*`

`After two days of intense debate, the United Methodist Church has agreed to a historic split — one that is expected to end in the creation of a new denomination, one that will be “theologically and socially conservative,” according to the Washington Post. The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will “discipline” clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination.
The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the “largest Protestant denomination in the US”, but that it has been shrinking in recent decades. The new split will be the second in the church’s history. The first occurred in 1968, when roughly 10% of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split “comes at a critical time for the church, which has been losing members for years,” which has been “pushed toward the brink of a schism over the role of LGBTQ people in the church.” Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.`

The next model in the series was [ChatGPT, released in 2022](https://openai.com/blog/chatgpt/). ChatGPT significantly surpasses GPT-3 on a range of tasks; as one example, see its creative writing response to the prompt below:

![]()Common-sense question-answering
-------------------------------

Historically, one of the key bottlenecks in AI has been common-sense reasoning. Over the last few years, though, large language models have become capable of giving straightforwardly correct common-sense answers to a wide range of questions. (Note, however, that whether or not a LLM responds correctly to a question often depends sensitively on how the question is asked.)

Here are some explanations of jokes from [Google’s 2022 PaLM model](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html) (with the top box containing examples given to the model, and then the boxes below containing the model’s own responses):

![]()Here are a range of further capabilities showcased by PaLM:

![]()The most impressive question-answering results come from measurements of the performance of GPT-3.5 (the model behind ChatGPT) and GPT-4 on a range of AP high school exams. GPT-3.5 already beats most humans on many exams; GPT-4 does much better on many of its weaker exams, especially quantitative exams — e.g. going from worse than almost all humans to around median on the AP Calculus exam. The jump is particularly striking given that these models were trained only a few months apart.

![]()Planning and strategic thinking
-------------------------------

LLMs have also developed the capability to describe long-term plans, and break them down into a series of tasks. For example, here’s ChatGPT:

![]()While their ability to act on these plans is currently limited, a range of work aims to adapt LLMs to perform computer-based tasks. Here’s an example of the [ACT-1](https://www.adept.ai/act) model carrying out a task using a computer interface:

And here’s a video of Google’s [PaLM-SayCan](https://sites.research.google/palm-saycan) robot generating and carrying out a multi-step plan in response to user instructions:

Science
=======

While success at the language-based tasks described above might allow AIs to automate much white-collar work, in the long term the most impactful and concerning capabilities will likely involve AIs capable of automating the process of scientific and technological advancement (including progress in AI development itself). Over the last few years we’ve seen rapid progress towards AIs which are useful in scientific research.

Coding
------

An early step towards automating coding came with GPT-2, where the researchers who trained it noticed a surprising phenomenon: it was capable of autocompleting realistic (although usually buggy) code, because its training data had included a range of code from across the internet.

That was in 2019. In 2020, GPT-3 developed much more sophisticated coding capabilities, with the Codex language model (based on GPT-3) being capable of following language instructions to write whole functions. Here’s an example of Codex writing code to implement a feature in a simple game, based on language instructions.

![]()The full video can be found [here](https://openai.com/blog/openai-codex/#spacegame).Here’s an example of Codex writing code to plot a graph:

![]()The full video can be found [here](https://openai.com/blog/openai-codex/#datascience).[DeepMind’s AlphaCode system](https://www.deepmind.com/blog/competitive-programming-with-alphacode) was trained to solve competitive programming problems, and would have placed in the top 54% of humans who participated in the coding competitions it was tested on.

![]()ChatGPT is also capable of answering high-level coding questions, including some at a comparable level to interview questions for hiring software engineers:

![]()![]()Math
----

The last few years have also seen incredible progress on applying AI to mathematics. AI had already been used to [help prove some mathematical theorems](https://en.wikipedia.org/wiki/Four_color_theorem#Proof_by_computer), but they required humans to manually specify what parts of the solution to search for. By contrast, modern language models are capable of answering free-form mathematical questions, including some questions used in math competitions.

Here are some examples of Google’s MINERVA model (a variant of PaLM) answering math questions:

![]()![]()Here’s an example of [a model from OpenAI](https://openai.com/blog/formal-math) which is capable of both informal proofs and formal proofs in the LEAN programming language:

![]()![]()Although none of the examples above show AIs doing novel research, DeepMind’s [AlphaTensor](https://www.deepmind.com/blog/discovering-novel-algorithms-with-alphatensor) model discovered a new algorithm for matrix multiplication which was faster than any designed by humans. While AlphaTensor was specifically developed for this purpose (as opposed to the more general systems discussed above), the result is notable because matrix multiplication is the key step in training neural networks.

Life sciences
-------------

The biggest result thus far for AI in the life sciences has been the success of [AlphaFold 2](https://www.deepmind.com/blog/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology), DeepMind’s AI for predicting protein structures, which was described by experts in the field as having solved the protein folding problem (one of the core open problems in biology). While this claim may be somewhat overblown, it was nevertheless a huge leap forward compared with previous attempts:

![]()Some examples of AlphaFold 2’s predictions, compared with the experimental findings:

![]()A more worrying advance comes from [a team of drug development researchers](https://www.chemistryworld.com/news/drug-discovery-ai-that-developed-new-nerve-agents-raises-difficult-questions/4015462.article), who reversed their algorithm for finding harmless drugs to show how it could potentially generate chemical weapons. The chart below shows the predicted toxicity of compounds generated by their AI; many of them have a lower lethal dose than VX (one of the most lethal toxins known).

![]()Source: [Dual use of artificial-intelligence-powered drug discovery (Urbina et al., 2022)](http://Dual use of artificial-intelligence-powered drug discovery)Looking forward
===============

Just to highlight how absurd the rate of progress in AI has been, note that around half of the breakthroughs I’ve mentioned happened in 2022 alone! Overall we’re seeing AI becoming increasingly good at taking actions in complex environments, doing strategic thinking, and then connecting it to the real world. You might worry that showcasing individual examples is a haphazard way to track this progress, compared with measuring more quantitative metrics. However, it’s becoming difficult to design benchmark tasks which remain unsolved for more than a few years, even when they’re intended to track long-term progress. The chart below shows that, while it used to take decades to reach human level on benchmarks, many new benchmarks are being saturated in only a couple of years.

The pace of progress is surprising even to experts in the field. For example, consider the MATH dataset of challenging competition mathematics problems, released in 2021. [Professional forecasters predicted](https://bounded-regret.ghost.io/ai-forecasting-one-year-in/) that the best accuracy achieved by mid-2022 would be 12.7%; in fact, it was over 50%, well outside their confidence intervals. They also significantly underestimated progress on the 2020 Massive Multitask Language Understanding dataset, which covers elementary mathematics, US history, computer science, law, and more. These forecasters were not experts in ML, but predicted much faster progress than most experts did, and *still* dramatically underestimated progress on these two benchmarks (although on two other benchmarks their predictions were more accurate).

![]()Source: [AI Forecasting: One Year In (Steinhardt, 2022)](https://bounded-regret.ghost.io/ai-forecasting-one-year-in/)One reason that it’s hard to make accurate forecasts is because the compute used in training machine learning systems has been scaling up so dramatically. To get a sense for this, see the graph below showing the computational resources used to train over 100 cutting-edge machine learning systems. Note in particular that the y-axis is on a logarithmic scale — the most compute-intensive system pictured on the graph was trained using over 10 million times more compute than the biggest system from ten years earlier (around the start of the deep learning era).

![]()Source: [Compute Trends Across Three Eras of Machine Learning (Sevilla et al., 2022)](https://arxiv.org/abs/2202.05924)The people who have come closest to correctly forecasting how fast progress would be have typically done so by focusing on compute scaling, even when it led to absurd-sounding consequences. In the 1990s, [Hans Moravec predicted](https://web.archive.org/web/20060615031852/http://transhumanist.com/volume1/moravec.htm) human-level AI by the 2030s, based on forecasting compute scaling. [In 2005, Ray Kurzweil used a similar method](https://en.wikipedia.org/wiki/The_Singularity_Is_Near) to predict a “profound and disruptive transformation in human capability” (which he called the “singularity”) by 2045. In 2010, before the deep learning revolution, DeepMind co-founder [Shane Legg predicted](http://www.vetta.org/2010/12/goodbye-2010/) human-level AI by 2028 using compute-based estimates. OpenAI co-founder Ilya Sutskever, whose AlexNet paper sparked the deep learning revolution, was also an early proponent of the idea that scaling up deep learning would be transformative. Whether or not the specifics of these predictions are right, they all seem much more plausible now than they did a decade ago, especially when compared to the mainstream consensus that progress in AI would be nowhere near this fast. More recently, in 2020 a team at OpenAI developed [scaling laws](https://arxiv.org/abs/2001.08361) which claimed that scaling up the amount of compute used in large language models would predictably improve their performance, a claim which has largely been borne out since then (although there have been [recent corrections](https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training) to the specific scaling laws they proposed).

Another thing that many of these forecasters share is serious concern about the possibility that AGI won’t be aligned with human values, and that this might lead to catastrophic outcomes. Shane started and led the [DeepMind technical AGI safety team](https://deepmindsafetyresearch.medium.com/); more recently, [Ilya has been spending](https://www.alignmentforum.org/posts/3S4nyoNEEuvNsbXt8/common-misconceptions-about-openai#Incorrect__OpenAI_leadership_is_dismissive_of_existential_risk_from_AI_) 50% of his time on alignment research; and the core team behind the scaling laws paper founded [Anthropic](https://www.anthropic.com/), an alignment-focused research lab. The possibility that we might be in the run-up to AGI naturally focuses the mind on [the risks](https://www.agisafetyfundamentals.com/) of bringing such a powerful technology into the world.


How ChatGPT Works Technically


Goal Misgeneralisation examples


AI Risk Discussions


Career Resources on AI Strategy Research


12 Tentative Ideas for US AI Policy

Published: April 17, 2023  | by [Luke Muehlhauser](/about/team/luke-muehlhauser)
About two years ago, I [wrote](https://www.openphilanthropy.org/research/our-ai-governance-grantmaking-so-far/) that “it’s difficult to know which ‘intermediate goals’ [e.g. policy goals] we could pursue that, if achieved, would clearly increase the odds of eventual good outcomes from transformative AI.” Much has changed since then, and in this post I give an update on 12 ideas for US policy goals[[1]](#f+13628+2+1)Many of these policy options would plausibly also be good to implement in other jurisdictions, but for most of them the US is a good place to start (the US is plausibly the most important jurisdiction anyway, given the location of leading companies, and many other countries sometimes follow the … [Continue reading](#f+13628+2+1) that I tentatively think would increase the odds of good outcomes from [transformative AI](https://docs.google.com/document/d/15siOkHQAoSBl_Pu85UgEDWfmvXFotzub31ow3A11Xvo/edit).[[2]](#f+13628+2+2)For more on intermediate goals, see [Survey on intermediate goals in AI governance](https://forum.effectivealtruism.org/posts/g4fXhiJyj6tdBhuBK/survey-on-intermediate-goals-in-ai-governance).

I think the US generally over-regulates, and that most people underrate the [enormous benefits of rapid innovation](https://www.vox.com/future-perfect/2018/11/8/18052076/human-history-in-one-chart-industrial-revolution). However, when 50% of the (survey-responding) experts on a specific technology [think](https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/) there is a reasonable chance it will result in outcomes that are “extremely bad (e.g. human extinction),” I think ambitious and thoughtful regulation is warranted.[[3]](#f+13628+2+3)This paragraph was added on April 18, 2023.

First, some caveats:

* These are my own tentative opinions, not Open Philanthropy’s.[[4]](#f+13628+2+4)Besides my day job at Open Philanthropy, I am also a Board member at [Anthropic](https://www.anthropic.com/), though I have no shares in the company and am not compensated by it. Again, these opinions are my own, not Anthropic’s. I might easily change my opinions in response to further analysis or further developments.
* My opinions are premised on a strategic picture similar to the one outlined in my colleague Holden Karnofsky’s [Most Important Century](https://www.cold-takes.com/most-important-century-series-roadmap/) and [Implications of…](https://www.cold-takes.com/tag/implicationsofmostimportantcentury/) posts. In other words, I think transformative AI could bring enormous benefits, but I also take full-blown existential risk from transformative AI as a plausible and urgent concern, and I am more agnostic about this risk’s likelihood, shape, and tractability than e.g. [a recent *TIME* op-ed](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/).
* None of the policy options below have gotten sufficient scrutiny (though they have received far more scrutiny than is presented here), and there are many ways their impact could turn out — upon further analysis or upon implementation — to be net-negative, even if my basic picture of the strategic situation is right.
* To my knowledge, none of these policy ideas have been worked out in enough detail to allow for immediate implementation, but experts have begun to draft the potential details for most of them (not included here). None of these ideas are original to me.
* This post doesn’t explain much of my *reasoning* for tentatively favoring these policy options. All the options below have complicated mixtures of pros and cons, and many experts oppose (or support) each one. This post isn’t intended to (and shouldn’t) *convince* anyone. However, **in the wake of recent AI advances and discussion, many people have been asking me for these kinds of policy ideas, so I am sharing my opinions here**.
* Some of these policy options are more politically tractable than others, but, as I think we’ve seen recently, the political landscape sometimes shifts rapidly and unexpectedly.

Those caveats in hand, below are some of my current personal guesses about US policy options that would reduce existential risk from AI in expectation (in no order).[[5]](#f+13628+2+5)There are many other policy options I have purposely not mentioned here. These include:

Hardware export controls. The US has already implemented major export controls on semiconductor manufacturing equipment and high-end chips. These controls have both pros and cons from my perspective, though … [Continue reading](#f+13628+2+5)

1. **Software export controls****.** Control the export (to anyone) of “frontier AI models,” i.e. models with highly general capabilities over some threshold, or (more simply) models trained with a compute budget over some threshold (e.g. as much compute as $1 billion can buy today). This will help limit the proliferation of the models which probably pose the greatest risk. Also restrict API access in some ways, as API access can potentially be used to generate an optimized dataset sufficient to train a smaller model to reach performance similar to that of the larger model.
2. **Require hardware security features on cutting-edge chips****.** Security features on chips can be leveraged for many useful compute governance purposes, e.g. to verify compliance with export controls and domestic regulations, monitor chip activity without leaking sensitive IP, limit usage (e.g. via interconnect limits), or even intervene in an emergency (e.g. remote shutdown). These functions can be achieved via firmware updates to already-deployed chips, though some features would be more tamper-resistant if implemented on the silicon itself in future chips.
3. **Track stocks and flows of cutting-edge chips, and license big clusters.** Chips over a certain capability threshold (e.g. the one used for the October 2022 export controls) should be tracked, and a license should be required to bring together large masses of them (as required to cost-effectively train frontier models). This would improve government visibility into potentially dangerous clusters of compute. And without this, other aspects of an effective compute governance regime can be rendered moot via the use of undeclared compute.
4. **Track and require a license to develop frontier AI models****.** This would improve government visibility into potentially dangerous AI model development, and allow more control over their proliferation. Without this, other policies like the information security requirements below are hard to implement.
5. **Information security requirements****.** Require that frontier AI models be subject to extra-stringent information security protections (including cyber, physical, and personnel security), including during model training, to limit unintended proliferation of dangerous models.
6. **Testing and evaluation requirements****.** Require that frontier AI models be subject to extra-stringent safety testing and evaluation, including some evaluation by an independent auditor meeting certain criteria.[[6]](#f+13628+2+6)See e.g. p. 15-16 of the [GPT-4 system card report for an illustration.](https://cdn.openai.com/papers/gpt-4-system-card.pdf)
7. **Fund specific genres of alignment, interpretability, and model evaluation R&D****.** Note that if the genres are not specified well enough, such funding can effectively widen (rather than shrink) the gap between cutting-edge AI capabilities and available methods for alignment, interpretability, and evaluation. See e.g. [here](https://www.openphilanthropy.org/request-for-proposals-for-projects-in-ai-alignment-that-work-with-deep-learning-systems/) for one possible model.
8. **Fund defensive information security R&D**, again to help limit unintended proliferation of dangerous models. Even the broadest funding strategy would help, but there are many ways to target this funding to the development and deployment pipeline for frontier AI models.
9. **Create a narrow antitrust safe harbor for AI safety & security collaboration****.** Frontier-model developers would be more likely to collaborate usefully on AI safety and security work if such collaboration were more clearly allowed under antitrust rules. Careful scoping of the policy would be needed to retain the basic goals of antitrust policy.
10. **Require certain kinds of AI incident reporting**, similar to incident reporting requirements in other industries (e.g. aviation) or to data breach reporting requirements, and similar to some vulnerability disclosure regimes. Many incidents wouldn’t need to be reported publicly, but could be kept confidential within a regulatory body. The goal of this is to allow regulators and perhaps others to track certain kinds of harms and close-calls from AI systems, to keep track of where the dangers are and rapidly evolve mitigation mechanisms.
11. **Clarify the liability of AI developers for concrete AI harms****,** especially clear physical or financial harms, including those resulting from negligent security practices. A new framework for AI liability should in particular address the risks from frontier models carrying out actions. The goal of clear liability is to incentivize greater investment in safety, security, etc. by AI developers.
12. **Create means for rapid shutdown of large compute clusters and training runs****.** One kind of “off switch” that may be useful in an emergency is a non-networked power cutoff switch for large compute clusters. As far as I know, most datacenters don’t have this.[[7]](#f+13628+2+7)E.g. the lack of an off switch exacerbated the fire that destroyed a datacenter in Strasbourg; see section VI.2.1 – iv of [this report](https://www.igedd.developpement-durable.gouv.fr/IMG/pdf/rapport_ovh_67_vdif_cle01cf13.pdf). Remote shutdown mechanisms on chips (mentioned above) could also help, though they are vulnerable to interruption by cyberattack. Various additional options could be required for compute clusters and training runs beyond particular thresholds.

Of course, even if one agrees with some of these high-level opinions, I haven’t provided enough detail in this short post for readers to know what, exactly, to advocate for, or how to do it. If you have useful skills, networks, funding, or other resources that you might like to direct toward further developing or advocating for one or more of these policy ideas, please indicate your interest in [this short Google Form](https://forms.gle/4EZmFbxnJ8pp6Qip8). (The information you share in this form will be available to me [Luke Muehlhauser] and some other Open Philanthropy employees, but we won’t share your information beyond that without your permission.)

 Footnotes[+]Footnotes[−]

Footnotes| [1](#r+13628+2+1 "Alt+ ←") | Many of these policy options would plausibly also be good to implement in other jurisdictions, but for most of them the US is a good place to start (the US is plausibly the most important jurisdiction anyway, given the location of leading companies, and many other countries sometimes follow the US), and I know much less about politics and policymaking in other countries. |
| [2](#r+13628+2+2 "Alt+ ←") | For more on intermediate goals, see [Survey on intermediate goals in AI governance](https://forum.effectivealtruism.org/posts/g4fXhiJyj6tdBhuBK/survey-on-intermediate-goals-in-ai-governance). |
| [3](#r+13628+2+3 "Alt+ ←") | This paragraph was added on April 18, 2023. |
| [4](#r+13628+2+4 "Alt+ ←") | Besides my day job at Open Philanthropy, I am also a Board member at [Anthropic](https://www.anthropic.com/), though I have no shares in the company and am not compensated by it. Again, these opinions are my own, not Anthropic’s. |
| [5](#r+13628+2+5 "Alt+ ←") | There are many other policy options I have purposely not mentioned here. These include:* Hardware export controls. The US has already implemented [major export controls](https://en.wikipedia.org/wiki/United_States_New_Export_Controls_on_Advanced_Computing_and_Semiconductors_to_China) on semiconductor manufacturing equipment and high-end chips. These controls have both pros and cons from my perspective, though it’s worth noting that they may be a necessary complement to some of the policies I tentatively recommend in this post. For example, the controls on semiconductor manufacturing equipment help to preserve a unified supply chain to which future risk-reducing compute governance mechanisms can be applied. These hardware controls will likely need ongoing maintenance by technically sophisticated policymakers to remain effective.
* “US boosting” interventions, such as semiconductor manufacturing subsidies or AI R&D funding. One year ago I was weakly in favor of these policies, but recent analyses have nudged me into weakly expecting these interventions are net-negative given e.g. the likelihood that they shorten AI timelines. But more analysis could flip me back. “US boosting” by increasing high-skill immigration may be an exception here because it relocates rather than creates a key AI input (talent), but I’m unsure, e.g. because skilled workers may accelerate AI faster in the US than in other jurisdictions. As with all the policy opinions in this post, it depends on the magnitude and certainty of multiple effects pushing in different directions, and those figures are difficult to estimate.
* AI-slowing regulation that isn’t “directly” helpful beyond slowing AI progress, e.g. a law saying that the “fair use” doctrine doesn’t apply to data used to train large language models. Some things in this genre might be good to do for the purpose of buying more time to come up with needed AI alignment and governance solutions, but I haven’t prioritized looking into these options relative to the options listed in the main text, which simultaneously buy more time and are “directly” useful to mitigating the risks I’m most worried about. Moreover, I think creating the ability to slow AI progress during the most dangerous period (in the future) is more important than slowing AI progress now, and most of the policies in the main text help with slowing AI progress in the future, whereas some policies that slow AI today don’t help much with slowing AI progress in the future.
* Launching new multilateral agreements or institutions to regulate AI globally. Global regulation is needed, but I haven’t yet seen proposals in this genre that I expect to be both feasible and effective. My guess is that the way to work toward new global regulation is similar to how the October 2022 export controls have [played out](https://www.nytimes.com/2023/01/28/business/economy/netherlands-japan-china-chips.html): the US can move first with an effective policy on one of the topics above, and then persuade other influential countries to join it.
* A [national research cloud](https://hai.stanford.edu/policy/national-ai-research-resource). I’d guess this is unhelpful because it accelerates AI R&D broadly and creates a larger number of people who can train dangerously large models, though the implementation details matter.

 |
| [6](#r+13628+2+6 "Alt+ ←") | See e.g. p. 15-16 of the [GPT-4 system card report for an illustration.](https://cdn.openai.com/papers/gpt-4-system-card.pdf) |
| [7](#r+13628+2+7 "Alt+ ←") | E.g. the lack of an off switch exacerbated the fire that destroyed a datacenter in Strasbourg; see section VI.2.1 – iv of [this report](https://www.igedd.developpement-durable.gouv.fr/IMG/pdf/rapport_ovh_67_vdif_cle01cf13.pdf). |


Opportunities for Funding


Charting a Path to AI Accountability