Frontier AI Regulation: Managing Emerging Risks to Public Safety

Executive Summary
-----------------

The capabilities of today’s foundation models highlight both the promise and
risks of rapid advances in AI. These models have demonstrated significant
potential to benefit people in a wide range of fields, including education,
medicine, and scientific research. At the same time, the risks posed by
present-day models, coupled with forecasts of future AI progress, have
rightfully stimulated calls for increased oversight and governance of AI across
a range of policy issues. We focus on one such issue: the possibility that, as
capabilities continue to advance, new foundation models could pose severe risks
to public safety, be it via misuse or accident. Although there is ongoing
debate about the nature and scope of these risks, we expect that government
involvement will be required to ensure that such "frontier AI models” are
harnessed in the public interest.

Three factors suggest that frontier AI development may be in need of targeted
regulation: (1) Models may possess unexpected and difficult-to-detect dangerous
capabilities; (2) Models deployed for broad use can be difficult to reliably
control and to prevent from being used to cause harm; (3) Models may
proliferate rapidly, enabling circumvention of safeguards.

Self-regulation is unlikely to provide sufficient protection against the risks
from frontier AI models: government intervention will be needed. We explore
options for such intervention. These include:

>
> Mechanisms to create and update safety standards for responsible
> frontier AI development and deployment. These should be developed via
> multi-stakeholder processes, and could include standards relevant to foundation
> models overall, not exclusive to frontier AI. These processes should facilitate
> rapid iteration to keep pace with the technology.
>
>
>
> Mechanisms to give regulators visibility into frontier AI development,
> such as disclosure regimes, monitoring processes, and whistleblower
> protections. These equip regulators with the information needed to address the
> appropriate regulatory targets and design effective tools for governing
> frontier AI. The information provided would pertain to qualifying frontier AI
> development processes, models, and applications.
>
>
>
> Mechanisms to ensure compliance with safety standards. Self-regulatory
> efforts, such as voluntary certification, may go some way toward ensuring
> compliance with safety standards by frontier AI model developers. However, this
> seems likely to be insufficient without government intervention, for example by
> empowering a supervisory authority to identify and sanction non-compliance; or
> by licensing the deployment and potentially the development of frontier AI.
> Designing these regimes to be well-balanced is a difficult challenge; we should
> be sensitive to the risks of overregulation and stymieing innovation on the one
> hand, and moving too slowly relative to the pace of AI progress on the other.
>
>
>

Next, we describe an initial set of safety standards that, if adopted,
would provide some guardrails on the development and deployment of frontier AI
models. Versions of these could also be adopted for current AI models to guard
against a range of risks. We suggest that at minimum, safety standards for
frontier AI development should include:

>
> Conducting thorough risk assessments informed by evaluations of dangerous capabilities and controllability. This would reduce the risk that
> deployed models possess unknown dangerous capabilities, or behave unpredictably
> and unreliably.
>
>
>
> Engaging external experts to apply independent scrutiny to
> models. External scrutiny of the safety and risk profile of models would both
> improve assessment rigor and foster accountability to the public interest.
>
>
>
> Following standardized protocols for how frontier AI models
> can be deployed based on their assessed risk. The results from risk assessments
> should determine whether and how the model is deployed, and what safeguards are
> put in place. This could range from deploying the model without restriction to
> not deploying it at all. In many cases, an intermediate option—deployment with
> appropriate safeguards (e.g., more post-training that makes the model more
> likely to avoid risky instructions)—may be appropriate.
>
>
>
> Monitoring and responding to new information on model
> capabilities. The assessed risk of deployed frontier AI models may change over
> time due to new information, and new post-deployment enhancement techniques. If
> significant information on model capabilities is discovered post-deployment,
> risk assessments should be repeated, and deployment safeguards updated.
>
>
>

Going forward, frontier AI models seem likely to warrant safety standards more
stringent than those imposed on most other AI models, given the prospective
risks they pose. Examples of such standards include: avoiding large jumps in
capabilities between model generations; adopting state-of-the-art alignment
techniques; and conducting pre-training risk assessments. Such practices are
nascent today, and need further development.

The regulation of frontier AI should only be one part of a broader policy
portfolio, addressing the wide range of risks and harms from AI, as well as
AI’s benefits. Risks posed by current AI systems should be urgently addressed;
frontier AI regulation would aim to complement and bolster these efforts,
targeting a particular subset of resource-intensive AI efforts. While we remain
uncertain about many aspects of the ideas in this paper, we hope it can
contribute to a more informed and concrete discussion of how to better govern
the risks of advanced AI systems while enabling the benefits of innovation to society.

Acknowledgements
----------------

We would like to express our thanks to the people who have offered feedback and
input on the ideas in this paper, including Jon Bateman, Rishi
Bommasani, Will Carter, Jack Clark, John Cisternino, Rebecca
Crootof, Allan Dafoe, Ellie Evans, Marina Favaro, Noah Feldman, Ben Garfinkel,
Joshua Gotbaum, Julian Hazell, Lennart Heim, Holden Karnofsky, Jeremy Howard, Tim
Hwang, Tom Kalil, Gretchen Krueger, Lucy Lim, Chris Meserole,
Luke Muehlhauser, Jared Mueller, Richard Ngo, Sanjay Patnaik,
Hadrien Pouget, Gopal Sarma, Girish Sastry, Paul Scharre, Mike Selitto, Toby
Shevlane, Danielle Smalls, Helen Toner, and Irene Solaiman.

###### Contents

* [1 Introduction](#S1 "1 Introduction ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [2 The Regulatory Challenge of Frontier AI Models](#S2 "2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [2.1 What do we mean by frontier AI models?](#S2.SS1 "2.1 What do we mean by frontier AI models? ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [2.2 The Regulatory Challenge Posed by Frontier AI](#S2.SS2 "2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can
		Arise Unpredictably and Undetected](#S2.SS2.SSS1 "2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [2.2.2 The Deployment Safety Problem: Preventing deployed AI models
		from causing harm is difficult](#S2.SS2.SSS2 "2.2.2 The Deployment Safety Problem: Preventing deployed AI models from causing harm is difficult ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [2.2.3 The Proliferation Problem: Frontier AI models can proliferate
		rapidly](#S2.SS2.SSS3 "2.2.3 The Proliferation Problem: Frontier AI models can proliferate rapidly ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [3 Building Blocks for Frontier AI Regulation](#S3 "3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [3.1 Institutionalize Frontier AI Safety Standards Development](#S3.SS1 "3.1 Institutionalize Frontier AI Safety Standards Development ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [3.2 Increase Regulatory Visibility](#S3.SS2 "3.2 Increase Regulatory Visibility ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [3.3 Ensure Compliance with Standards](#S3.SS3 "3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [3.3.1 Self-Regulation and Certification](#S3.SS3.SSS1 "3.3.1 Self-Regulation and Certification ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [3.3.2 Mandates and Enforcement by supervisory authorities](#S3.SS3.SSS2 "3.3.2 Mandates and Enforcement by supervisory authorities ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [3.3.3 License Frontier AI Development and Deployment](#S3.SS3.SSS3 "3.3.3 License Frontier AI Development and Deployment ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [3.3.4 Pre-conditions for Rigorous Enforcement Mechanisms](#S3.SS3.SSS4 "3.3.4 Pre-conditions for Rigorous Enforcement Mechanisms ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [4 Initial Safety Standards for Frontier AI](#S4 "4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [4.1 Conduct thorough risk assessments informed by evaluations of
	dangerous capabilities and controllability](#S4.SS1 "4.1 Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [4.1.1 Assessment for Dangerous Capabilities](#S4.SS1.SSS1 "4.1.1 Assessment for Dangerous Capabilities ‣ 4.1 Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [4.1.2 Assessment for Controllability](#S4.SS1.SSS2 "4.1.2 Assessment for Controllability ‣ 4.1 Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
		- [4.1.3 Other Considerations for Performing Risk Assessments](#S4.SS1.SSS3 "4.1.3 Other Considerations for Performing Risk Assessments ‣ 4.1 Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [4.2 Engage External Experts to Apply Independent Scrutiny to Models](#S4.SS2 "4.2 Engage External Experts to Apply Independent Scrutiny to Models ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [4.3 Follow Standardized Protocols for how Frontier AI Models can be
	Deployed Based on their Assessed Risk](#S4.SS3 "4.3 Follow Standardized Protocols for how Frontier AI Models can be Deployed Based on their Assessed Risk ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [4.4 Monitor and respond to new information on model capabilities](#S4.SS4 "4.4 Monitor and respond to new information on model capabilities ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [4.5 Additional practices](#S4.SS5 "4.5 Additional practices ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [5 Uncertainties and Limitations](#S5 "5 Uncertainties and Limitations ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [A Creating a Regulatory Definition for Frontier AI](#A1 "Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [A.1 Desiderata for a Regulatory Definition](#A1.SS1 "A.1 Desiderata for a Regulatory Definition ‣ Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [A.2 Defining Sufficiently Dangerous Capabilities](#A1.SS2 "A.2 Defining Sufficiently Dangerous Capabilities ‣ Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [A.3 Defining Foundation Models](#A1.SS3 "A.3 Defining Foundation Models ‣ Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
	+ [A.4 Defining the Possibility of Producing Sufficiently Dangerous
	Capabilities](#A1.SS4 "A.4 Defining the Possibility of Producing Sufficiently Dangerous Capabilities ‣ Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
* [B Scaling laws in Deep Learning](#A2 "Appendix B Scaling laws in Deep Learning ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")

1 Introduction
---------------

Responsible AI innovation can provide extraordinary benefits to society, such
as delivering medical \parenciteMoor2023, Lee2023, singhal2022large,
nori2023capabilities and legal \parenciteSimshaw2022,Arbel2020,Kolt2022
services to more people at lower cost, enabling scalable personalized education
\parenciteKhan2023, and contributing solutions to pressing global challenges
like climate change \parenciterolnick2019tackling, DeepMind2016,
erdinc2022derisking, Donti2021 and pandemic prevention
\parenciteGaletsi2022,danko2023challenges. However, guardrails are necessary
to prevent the pursuit of innovation from imposing excessive negative
externalities on society. There is increasing recognition that government
oversight is needed to ensure AI development is carried out responsibly; we
hope to contribute to this conversation by exploring regulatory approaches to
this end.

In this paper, we focus specifically on the regulation of frontier AI models,
which we define as highly capable foundation models111Defined as: “any
model that is trained on broad data (generally using self-supervision at scale)
that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks”
\parencitebommasani2022opportunities. that could have dangerous capabilities
sufficient to pose severe risks to public safety and global security. Examples
of such dangerous capabilities include designing new biochemical weapons
\parenciteUrbina2022, producing highly persuasive personalized
disinformation, and evading human control \parencitengo2023alignment,
Cohen2022, hendrycks2022unsolved, hendrycks2022xrisk,
carlsmith2022powerseeking, wiki:HumanCompatible, wiki:TheAlignmentProblem.

In this paper, we first define frontier AI models and detail several policy
challenges posed by them. We explain why effective governance of frontier AI
models requires intervention throughout the models’ lifecycle, at the
development, deployment, and post-deployment stages.
Then, we describe approaches to regulating frontier AI models, including
building blocks of regulation such as the development of safety standards,
increased regulatory visibility, and ensuring compliance with safety standards.
We also propose a set of initial safety standards for frontier AI development
and deployment. We close by highlighting uncertainties and limitations for
further exploration.

2 The Regulatory Challenge of Frontier AI Models
-------------------------------------------------

###
2.1 What do we mean by frontier AI models?

For the purposes of this paper, we define “frontier AI models” as
highly capable foundation models222[bommasani2022opportunities]
defines “foundation models” as “models (e.g., BERT, DALL-E, GPT-3) that are
trained on broad data at scale and are adaptable to a wide range of downstream
tasks.” See also [EUDraft2023]. that could exhibit dangerous capabilities.
Such harms could take the form of significant physical harm or the disruption
of key societal functions on a global scale, resulting from intentional
misuse or accident \parenciteshevlane2023model, Zwetsloot2019.
It would be prudent to assume that next-generation foundation models could
possess advanced enough capabilities to qualify as frontier AI models, given
both the  [difficulty](#S2.SS2.SSS1 "2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
of predicting when sufficiently dangerous capabilities will
arise and the already significant capabilities of today’s models.

Though it is not clear where the line for “sufficiently dangerous capabilities”
should be drawn, examples could include:

* Allowing a non-expert to design and synthesize new biological or chemical weapons.333Such capabilities are starting to emerge. For example, a group of
researchers tasked a narrow drug-discovery system to identify maximally toxic
molecules. The system identified over 40,000 candidate molecules, including
both known chemical weapons and novel molecules that were predicted to be as or
more deadly \parenciteUrbina2022. Other researchers are warning that LLMs
can be used to aid in discovery and synthesis of compounds. One group attempted
to create an LLM-based agent, giving it access to the internet, code execution
abilities, hardware documentation, and remote control of an automated ‘cloud’
laboratory. They report finding that it in some cases the model was willing to outline
and execute on viable methods for synthesizing illegal drugs and chemical
weapons \parenciteboiko2023emergent.
* Producing and propagating highly persuasive, individually tailored,
multi-modal disinformation with minimal user instruction.444Generative AI models may already be useful to generate material for
disinformation campaigns \parenciteHorvitz2022, goldstein2023generative,
Buchanan2021.
It is possible that, in the future, models could possess additional
capabilities that could enhance the persuasiveness or dissemination of
disinformation, such as by making such disinformation more dynamic,
personalized, and multimodal; or by autonomously disseminating such
disinformation through channels that enhance its persuasive value, such as
traditional media.
* Harnessing unprecedented offensive cyber capabilities that could cause
catastrophic harm.555AI systems are already helpful in writing and
debugging code, capabilities that can also be applied to software vulnerability
discovery. There is potential for significant harm via automation of
vulnerability discovery and exploitation. However, vulnerability discovery
could ultimately benefit cyberdefense more than -offense, provided defenders
are able to use such tools to identify and patch vulnerabilities more
effectively than attackers can find and exploit them
\parencitepoldrack2023aiassisted, Lohn2022.
* Evading human control through means of deception and obfuscation.666If future AI systems develop the ability and the propensity to
deceive their users, controlling their behavior could be extremely challenging.
Though it is unclear whether models will trend in that direction, it seems rash
to dismiss the possibility and some argue that it might be the default outcome
of current training paradigms \parencitengo2023alignment, Cohen2022,
hendrycks2022xrisk, carlsmith2022powerseeking, wiki:HumanCompatible,
wiki:TheAlignmentProblem.

This list represents just a few salient possibilities; the possible future
capabilities of frontier AI models remains an important area of inquiry.

Foundation models, such as large language models (LLMs), are trained on large,
broad corpora of natural language and other text (e.g., computer code), usually
starting with the simple objective of predicting the next “token”.777A
token can be thought of as a word or part of a word
\parenciteMicrosoft:token. This relatively simple approach produces models
with surprisingly broad capabilities.888For example, LLMs achieve
state-of-the-art performance in diverse tasks such as question answering,
translation, multi-step reasoning, summarization, and code completion, among
others \parenciteRadford2019, brown2020language, openai2023gpt4,
chowdhery2022palm. Indeed, the term “LLM” is already becoming outdated, as
several leading “LLMs” are in fact multimodal (e.g., possess visual
capabilities) \parenciteopenai2023gpt4, alayrac2022flamingo. These models
thus possess more general-purpose functionality999We intentionally avoid
using the term “general-purpose AI” to avoid confusion with the use of that
term in the EU AI Act and other legislation. Frontier AI systems are a related
but narrower class of AI systems with general-purpose functionality, but whose
capabilities are relatively advanced and novel. than many other classes of AI
models, such as the recommender systems used to suggest Internet videos or
generative AI models in narrower domains like music. Developers often make
their models available through “broad deployment” via sector-agnostic platforms
such as APIs, chatbots, or via open-sourcing.101010We use “open-source” to
mean “open release:” that is a model being made freely available online, be it
with a license restricting what the system can be used for. An example of such
a license is the Responsible AI License. Our usage of “open-source” differs
from how the term is often used in computer science which excludes instances of
license requirements, though is closer to how many other communities understand
the term \parencitelicensesai,opensource. This means that they can be
integrated in a large number of diverse downstream applications, possibly
including safety-critical sectors (illustrated in
 [Figure 1](#S2.F1 "Figure 1 ‣ 2.1 What do we mean by frontier AI models? ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")).

![Example Frontier AI Lifecycle.](https://media.arxiv-vanity.com/render-output/7960858/lifecycle.png)
Figure 1: Example Frontier AI Lifecycle.

A number of features of our definition are worth highlighting. In focusing on
*foundation models* which could have dangerous, emergent capabilities, our
definition of frontier AI excludes narrow models, even when these models could
have sufficiently dangerous capabilities.111111However, if a foundation
model could be fine-tuned and adapted to pose severe risk to public safety via
capabilities in some narrow domain, it would count as a “frontier AI.” For
example, models optimizing for the toxicity of compounds \parenciteUrbina2022
or the virulence of pathogens could lead to intended (or at least foreseen)
harms and thus may be more appropriately covered with more targeted
regulation.121212Indeed, intentionally creating dangerous narrow models
should already be covered by various laws and regulators. To the extent that it
is not clearly covered, modification of those existing laws and regulations
would be appropriate and urgent. Further, the difference in mental state of the
developer makes it much easier to identify and impose liability on developers
of narrower dangerous models.

Our definition focuses on models that *could* — rather than just those
that *do* — possess dangerous capabilities, as many of the practices we
propose apply before it is known that a model has dangerous capabilities. One
approach to identifying models that could possess such capabilities is focusing on
foundation models that advance the state-of-the-art of foundation model
capabilities. While currently deployed foundation models pose risks
\parencitebommasani2022opportunities, Bender2021, they do not yet appear to
possess dangerous capabilities that pose severe risks to public safety as we
have defined them.131313In some cases, these have been explicitly tested
for \parenciteopenai:system-card. Given both our  [inability to reliably predict](#S2.SS2.SSS1 "2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
what models will have sufficiently dangerous capabilities and the
already significant capabilities today’s models possess, it would be prudent
for regulators to assume that next-generation state-of-the-art foundation
models *could* possess advanced enough capabilities to warrant
regulation.141414We think it is prudent to anticipate that foundation
models’ capabilities may advance much more quickly than many expect, as has
arguably been the case for many AI capabilities: “[P]rogress on ML benchmarks
happened significantly faster than forecasters expected. But forecasters
predicted faster progress than I did personally, and my sense is that I expect
somewhat faster progress than the median ML researcher does.”
\parenciteSteinhardt2023; See [zhang2022forecasting] at 9;
[hoffmann2022training] at 11 (Chinchilla and Gopher surpassing forecaster
predictions for progress on MMLU); [openai2023gpt4] (GPT-4 surpassing
Gopher and Chinchilla on MMLU, also well ahead of forecaster predictions);
[Caplan:new, Caplan:retakes, Metaculus:Go, Metaculus:programs]. An
initial way to identify potential state-of-the-art foundation models could be
focusing on models trained using above some very large amount of computational
resources.151515Perhaps more than any model that has been trained to date.
Estimates suggest that 1026 floating point operations (FLOP) would meet this
criteria \parenciteOWID:computation.

Over time, the scope of frontier AI should be further refined. The scope
should be sensitive to features other than compute; state-of-the-art
performance can be achieved by using high quality data and new algorithmic
insights. Further, as systems with sufficiently dangerous capabilities are
identified, it will be possible to identify training runs that are likely to
produce such capabilities despite not achieving state-of-the-art performance.

We acknowledge that our proposed definition is lacking in sufficient precision
to be used for regulatory purposes and that more work is required to fully
assess the advantages and limitations of different approaches. Further, it is
not our role to determine exactly what should fall within the scope of the
regulatory proposals outlined – this will require more analysis and input from
a wider range of actors. Rather, the aim of this paper is to present a set of
initial proposals which we believe should apply to at least some subset of AI
development. We provide a more detailed description of alternative approaches
and the general complexity of defining “frontier AI” in  [Appendix A](#A1 "Appendix A Creating a Regulatory Definition for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").

###
2.2 The Regulatory Challenge Posed by Frontier AI

There are many regulatory questions related to the widespread use of AI
\parencitebommasani2022opportunities. This paper focuses on a specific subset
of concerns: the possibility that continued development of increasingly capable
foundation models could lead to dangerous capabilities sufficient to pose risks
to public safety at even greater severity and scale than is possible with
current computational systems \parenciteshevlane2023model.

Many existing and proposed AI regulations focus on the context in which AI
models are deployed, such as high-risk settings like law enforcement and
safety-critical infrastructure. These proposals tend to favor sector-specific
regulations models.161616This could look like imposing new requirements for
AI models used in high-risk industries and modifying existing regulations to
account for new risks from AI models. See \parenciteEUDraft2023,DCIAct,AAact,
ActionPlan2021,Consumer2022,LinaKhan2023. For frontier AI development,
sector-specific regulations can be valuable, but will likely leave a subset of
the high severity and scale risks unaddressed.

Three core problems shape the regulatory challenge posed by frontier AI models:

>
> [The Unexpected Capabilities Problem.](#S2.SS2.SSS1 "2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> Dangerous capabilities can arise
> unpredictably and undetected, both during development and after deployment.
>
>
>
>  [The Deployment Safety Problem.](#S2.SS2.SSS2 "2.2.2 The Deployment Safety Problem: Preventing deployed AI models from causing harm is difficult ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> Preventing deployed AI models from causing
> harm is a continually evolving challenge.
>
>
>
>  [The Proliferation Problem.](#S2.SS2.SSS3 "2.2.3 The Proliferation Problem: Frontier AI models can proliferate rapidly ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> Frontier AI models can proliferate rapidly,
> making accountability difficult.
>
>
>

These problems make the regulation of frontier AI models fundamentally
different from the regulation of other software, and the majority of other AI
models. The *Unexpected Capabilities Problem* implies that frontier AI
models could have unpredictable or undetected dangerous capabilities that
become accessible to downstream users who are difficult to predict beforehand.
Regulating easily identifiable users in a relatively small set of
safety-critical sectors may therefore fail to prevent those dangerous
capabilities from causing significant harm.171717This is especially true for
downstream bad actors (e.g., criminals, terrorists, adversary nations), who
will tend not to be as regulable as the companies operating in domestic
safety-critical sectors.

*The Deployment Safety Problem* adds an additional layer of difficulty.
Though many developers implement measures intended to prevent models from
causing harm when used by downstream users, these may not always be foolproof,
and malicious users may constantly be attempting to evolve their attacks.
Furthermore, the Unexpected Capabilities Problem implies that the developer may
not know of all of the harms from frontier models that need to be guarded
against during deployment. This amplifies the difficulty of the Deployment
Safety Problem: deployment safeguards should address not only known dangerous
capabilities, but have the potential to address unknown ones too.

The *Proliferation Problem* exacerbates the regulatory challenge. Frontier
AI models may be open-sourced, or become a target for theft by adversaries. To
date, deployed models also tend to be reproduced or iterated on within several
years. If, due to the Unexpected Capabilities Problem, a developer (knowingly
or not) develops and deploys a model with dangerous capabilities, the
Proliferation Problem implies that those capabilities could quickly become
accessible to unregulable actors like criminals and adversary governments.

Together, these challenges show that adequate regulation of frontier AI should
intervene throughout the frontier AI lifecycle, including during development,
general-purpose deployment, and post-deployment enhancements.

####
2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can
Arise Unpredictably and Undetected

Improvements in AI capabilities can be unpredictable, and are often difficult
to fully understand without intensive testing. Regulation that does not require
models to go through sufficient testing before deployment may therefore fail to
reliably prevent deployed models from posing severe risks.181818This
challenge also exacerbates the Proliferation Problem: we may not know how
important nonproliferation of a model is until after it has already been
open-sourced, reproduced, or stolen.

Overall AI model performance191919Measured by loss: essentially the error
rate of an AI model performs on its training objective. We acknowledge that
this is not a complete measure of model performance by any means. has tended to
improve smoothly with additional compute, parameters, and data.202020See
\parencitekaplan2020scaling,henighan2020scaling,hoffmann2022training,
Villalobos2023,hestness2017deep
However, there are tasks for which scaling leads to worse performance
\parencitemckenzie2023inverse, perez2022discovering, koralus2023humans,
though further scaling has overturned some of these findings,
\parenciteopenai2023gpt4. See also  [Appendix B](#A2 "Appendix B Scaling laws in Deep Learning ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"). However, specific
capabilities can significantly improve quite suddenly in general-purpose models
like LLMs (see  [Figure 2](#footnote22 "footnote 22 ‣ Figure 2 ‣ 2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")).
Though debated (see  [Appendix B](#A2 "Appendix B Scaling laws in Deep Learning ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")),
this phenomenom has been repeatedly
observed in multiple LLMs with capabilities as diverse as modular arithmetic,
unscrambling words, and answering questions in Farsi
\parencitewei2022emergent, Wei2022, bowman2023things, Wei2023.212121For a
treatment of recent critiques of the claim that AI models exhibit emergent
capabilities, see Appendix B. Furthermore, given the vast set of possible
tasks a foundation model could excel at, it is nearly impossible to
exhaustively test for them \parencitebommasani2022opportunities,
shevlane2023model

![Certain capabilities
seem to emerge suddenly.](https://media.arxiv-vanity.com/render-output/7960858/emerge.png)
Figure 2: Certain capabilities
seem to emerge suddenly.222222Chart from [wei2022emergent]. But see
[schaeffer2023emergent] for a skeptical view on emergence. For a response
to the skeptical view, see [Wei2023] and Appendix B.

Post-deployment enhancements — modifications made to AI models after their
initial deployment — can also cause unaccounted-for capability jumps. For
example, a key feature of many foundation models like LLMs is that they can be
fine-tuned on new data sources to enhance their capabilities in targeted
domains. AI companies often allow customers to fine-tune foundation models on
task-specific data to improve the model’s performance on that task
\parenciteanthropic, OpenAI:fine-tuning, ai21, cohere.
This could effectively expand the scope of capability concerns of a particular
frontier AI model. Models could also be improved via “online” learning, where
they continuously learn from new data \parencitehoi2018online,Parisi2019.

To date, iteratively deploying models to subsets of users has been a key
catalyst for understanding the outer limits of model capabilities and
weaknesses.232323Dario Amodei, CEO of Anthropic: “You have to deploy it to
a million people before you discover some of the things that it can do…”
\parenciteWPost2023. “We work hard to prevent foreseeable risks before
deployment, however, there is a limit to what we can learn in a lab.
Despite extensive research and testing, we cannot predict all of the beneficial
ways people will use our technology, nor all the ways people will abuse it.
That’s why we believe that learning from real-world use is a critical component
of creating and releasing increasingly safe AI systems over time”
\parenciteOpenAI:safety. For example, model users have demonstrated
significant creativity in eliciting new capabilities from AI models, exceeding
developers’ expectations of model capabilities. Users continue to discover
prompting techniques that significantly enhance the model’s performance, such
as by simply asking an LLM to reason step-by-step
\parencitewei2023chainofthought. This has been described as the “capabilities
overhang” of foundation models \parenciteClark2022. Users also discover new
failure modes for AI systems long after their initial deployment. For example,
one user found that the string “ solidgoldmagikarp” caused GPT-3 to malfunction
in a previously undocumented way, years after that model was first deployed
\parenciteSolidGoldMagikarp.

Much as a carpenter’s overall capabilities will vary with the tools she has
available, so too might an AI model’s overall capabilities vary depending on
the tools it can use. LLMs can be taught to use, and potentially create,
external tools like calculators and search engines
\parenciteOpenAI:plugins,schick2023toolformer,cai2023large.
Some models are also being trained to directly use general-purpose mouse and
keyboard interfaces \parenciteAdept,AutoGPT. See more examples in  [Table 1](#S2.T1 "Table 1 ‣ 2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").
As the available tools improve,
so can the overall capabilities of the total model-tool system, even if the
underlying model is largely unchanged.242424Right now, most tools that AI
models can use were originally optimized for use by people. As model-tool
interactions become more economically important, however, companies may develop
tools optimized for use by frontier AI models, accelerating capability
improvements.

| Technique | Description | Example |
| --- | --- | --- |
| Fine-tuning | Improving foundation model performance by updating model
weights with task-specific data. | Detecting propaganda by fine-tuning
a pre-trained LLM on a labeled dataset of common propaganda tactics
\parenciteYoosuf2019. |
| Chain-of-thought prompting \parencitewei2023chainofthought | Improving LLM problem-solving capabilities by telling the model to
think through problems step by step. | Adding a phrase such as “Let’s
think step by step” after posing a question to the model
\parencitekojima2023large. |
| External tool-use | Allow the model to use external tools when figuring
out how to answer user queries. | A model with access to a few simple
tools (e.g., calculator, search engine) and a small number of examples
performs much better than an unaided model.252525See
\parenciteschick2023toolformer. Early research also suggests LLMs can
be used to create tools for their own use \parencitecai2023large. |
| Automated prompt engineering \parencitezhou2023large | Using LLMs to
generate and search over novel prompts that can be used to elicit
better performance on a task. | To generate prompts for a task, an LLM
is asked something akin to: “I gave a friend instructions and he
responded in this way for the given inputs: [Examples of inputs and
outputs of the task] The instruction was:” |
| Foundation model programs \parenciteschlag2023large | Creation of
standardized means of integrating foundation models into more complex
programs. | Langchain: “a framework for developing applications powered
by language models.” \parenciteLangChain,AutoGPT |

Table 1: Some Known Post-Deployment Techniques for Unlocking New AI
Capabilities.

In the long run, there are even more worrisome possibilities. Models behaving
differently in testing compared to deployment is a known phenomenon in the
field of machine learning, and is particularly worrisome if unexpected and
dangerous behaviors first emerge “in the wild” only once a frontier model is
deployed \parenciteturner2023optimal, krakovna2023powerseeking,
hubinger2021risks.

####
2.2.2 The Deployment Safety Problem: Preventing deployed AI models
from causing harm is difficult

In general, it is difficult to precisely specify what we want deep
learning-based AI models to do and to ensure that they behave in line with
those specifications. Reliably controlling powerful AI models’ behavior, in
other words, remains a largely unsolved technical problem
\parencitehendrycks2022unsolved, ngo2023alignment, amodei2016concrete,
wolf2023fundamental, bowman2023things and the subject of ongoing research.

Techniques to “bake in” misuse prevention features at the model level, such
that the model reliably rejects or does not follow harmful instructions, can
effectively mitigate these issues, but adversarial users have still found ways
to circumvent these safeguards in some cases. One technique for circumvention
has been prompt injection attacks, where attackers disguise input text as
instructions from the user or developer to overrule restrictions provided to or
trained into the model. For example, emails sent to an LLM-based email
assistant could contain text constructed to look to the user as benign, but to
the LLM contains instructions to exfiltrate the user’s data (which the LLM
could then follow).262626For additional examples, see [Willison2023].
Other examples include “jailbreaking” models by identifying prompts that cause
a model to act in ways discouraged by their developers
\parenciteVenuto2023, PromptReport, Metz2023. Although progress is
being made on such issues \parencitebai2022constitutional, pan2023rewards,
Venuto2023, openai:system-card, it is unclear that we will be able to reliably
prevent dangerous capabilities from being used in unintended or undesirable
ways in novel situations; this remains an open and fundamental technical
challenge.

A major consideration is that model capabilities can be employed for both
harmful and beneficial uses:272727Nearly all attempts to stop bad or
unacceptable uses of AI also hinder positive uses, creating a *Misuse-Use
Tradeoff* \parenciteanderljung2023protecting. the harmfulness of an AI
model’s action may depend almost entirely on context that is not visible
during model development.
For example, copywriting is helpful when a company uses it to generate internal
communications, but harmful when propagandists use it to generate or amplify
disinformation. Use of a text-to-image model to modify a picture of someone
may be used with their consent as part of an art piece, or without their
consent as a means of producing disinformation or harassment.

####
2.2.3 The Proliferation Problem: Frontier AI models can proliferate
rapidly

The most advanced AI models cost tens of millions of dollars to
create.282828Though there are no estimates on the total cost of producing a
frontier model, there are estimates of the cost of the compute used to train
models \parenciteHeim2022,Sevilla2022,Cottier2022 However, using the
trained model (i.e., “inference”) is vastly cheaper.292929Some impressive
models can run on a offline portable device; see
[Orhon2022, Willison:LLaMa, gpt4all, chen2023speed]. Thus, a much wider
array of actors will have the resources to misuse frontier AI models than have
the resources to create them. Those with access to a model with dangerous
capabilities could cause harm at a significant scale, by either misusing the
model themselves, or passing it on to actors who will misuse it.303030Though
advanced computing hardware accessed via the cloud tends to be needed to use
frontier models. They can seldom be run on consumer-grade hardware. We describe some examples of proliferation in  [Table 2](#S4.T3 "Table 3 ‣ 4.3 Follow Standardized Protocols for how Frontier AI Models can be Deployed Based on their Assessed Risk ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").

Currently, state-of-the-art AI capabilities can proliferate soon after
development. One mechanism for proliferation is open-sourcing. At present,
proliferation via open-sourcing of advanced AI models is common313131 For an overview of considerations in how to release powerful AI models, see
[solaiman2019release, solaiman2023gradient, shevlane2022structured, Nature2021, ovadya2019reducing, Sastry2021]. \parenciteLeahy2021, BLOOM,
HuggingFace and usually unregulated. When models are open-sourced, obtaining
access to their capabilities becomes much easier: all internet users could copy
and use them, provided access to appropriate computing resources. Open-source
AI models can provide major economic utility by driving down the cost of
accessing state-of-the-art AI capabilities. They also enable academic research
on larger AI models than would otherwise be practical, which improves the
public’s ability to hold AI developers accountable. We believe that
open-sourcing AI models can be an important public good. However, frontier AI
models may need to be handled more restrictively than their smaller, narrower,
or less capable counterparts. Just as cybersecurity researchers embargo
security vulnerabilities to give the affected companies time to release a
patch, it may be prudent to avoid potentially dangerous capabilities of
frontier AI models being open sourced until safe deployment is demonstrably
feasible.

Other vectors for proliferation also imply increasing risk as capabilities
advance. For example, though models that are made available via APIs
proliferate more slowly, newly announced results are commonly reproduced or
improved upon323232Below, we use “reproduction” to mean some other actor
producing a model that reaches at least the same performance as an existing
model. within 1-2 years of the initial release. Many of the most capable
models use simple algorithmic techniques and freely available data, meaning
that the technical barriers to reproduction can often be low.333333Projects
such as OpenAssistant \parenciteOpenAssistant attempt to reproduce the
functionality of ChatGPT; and alpaca \parencitealpaca uses OpenAI’s
text-davinci-003 model to train a new model with similar capabilities. For an
overview, see [zhao2023survey].

Proliferation can also occur via theft. The history of cybersecurity is replete
with examples of actors ranging from states to lone cybercriminals compromising
comparably valuable digital assets \parenciteManess2022, Carnegie, CSIS,
Schmidt2015, Buchanan2017. Many AI developers take significant measures to
safeguard their models. However, as AI models become more useful in
strategically important contexts and the difficulties of producing the most
advanced models increase, well-resourced adversaries may launch increasingly
sophisticated attempts to steal them \parenciteHannas2019, NCSC2021.
Importantly, theft is feasible before deployment.

The interaction and causes of the three regulatory challenges posed by frontier
AI are summarized in  [Figure 3](#S2.F3 "Figure 3 ‣ 2.2.3 The Proliferation Problem: Frontier AI models can proliferate rapidly ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").

|  |  |  |
| --- | --- | --- |
| Original Model | Subsequent Model | Time to
Proliferate343434The examples listed here are not necessarily the earliest
instances of proliferation. |
| StyleGAN | Immediate |
| StyleGAN is a model by NVIDIA that
generates photorealistic human faces using generative adversarial networks
(GANs) \parencitestylegan. NVIDIA first published about StyleGAN in December
2018 \parencitekarras2019stylebased and open-sourced the model in February
2019. Following open-sourcing StyleGAN, sample images went viral through sites
such as thispersondoesnotexist.com
\parenciteMetz:fake-images,thispersondoesnotexist. Fake social media accounts
using pictures from StyleGAN were discovered later that year
\parenciteGallagher:facebook,Bond:fake-faces. |
| AlphaFold 2 | OpenFold | ∼2 years |
| In November 2020, DeepMind announced
AlphaFold 2 \parenciteAlphaFold. It was “the first computational method that
can regularly predict protein structures with atomic accuracy even in cases in
which no similar structure is known” \parenciteJumper2021: a major advance in
the biological sciences. In November 2022, a diverse group of researchers
reproduced and open-sourced a similarly capable model named OpenFold
\parenciteAhdritz2022. OpenFold used much less data to train than AlphaFold
2, and could be run much more quickly and easily \parenciteAhdritz2022. |
| GPT-3 | Gopher | ∼7 months |
| OpenAI announced GPT-3, an LLM, in May
2020 \parencitebrown2020language. In December 2021, DeepMind announced
Gopher, which performed better than GPT-3 across a wide range of benchmarks.
However, the Gopher model card suggests that the model was developed significantly earlier,
seven months after the GPT-3 announcement, in December 2020
\parenciterae2022scaling. |
| LLaMa | ∼1 week |
| In February 2023, Meta AI announced LLaMa,
an LLM \parenciteLLaMA. LLaMa was not open-sourced, but researchers could
apply for direct access to model weights \parenciteLLaMA. Within a week,
various users had posted these weights on multiple websites, violating the
terms under which the weights were distributed \parenciteTheBatch2023. |
| ChatGPT | Alpaca | ∼3 months |
| In March 2023, researchers from Stanford
University used sample completions from OpenAI’s text-davinci-003 to fine-tune
LLaMa in an attempt to recreate ChatGPT using less than $600.353535Note
that the original paper and subsequent research suggests this method fails to
match the capabilities of the larger model
\parencitealpaca,gudibande2023false. Their model was subsequently taken
offline due to concerns about cost and safety \parenciteQuach2023, though the
code and documentation for replicating the model is available on GitHub
\parencitetatsu-lab:alpaca. |

Table 2: Examples of AI Proliferation: these are not necessarily typical, and
some of these examples may be beneficial or benign, yet they demonstrate the
consistent history of AI capabilities proliferating after their initial
deployment

![Summary of the three regulatory challenges posed by frontier AI.](https://media.arxiv-vanity.com/render-output/7960858/challenges.png)
Figure 3: Summary of the three regulatory challenges posed by frontier AI.

3 Building Blocks for Frontier AI Regulation
---------------------------------------------

The three problems described above imply that serious risks may emerge during
the development and deployment of a frontier AI model, not just when it is used
in safety-critical sectors. Regulation of frontier AI models, then, must
address the particular shape of the regulatory challenge: the potential
unexpected dangerous capabilities; difficulty of deploying AI models safely;
and the ease of proliferation.

In this section, we outline potential building blocks for the regulation of
frontier AI. In the  [next section](#S4 "4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"), we describe a set of initial safety standards
for frontier AI models that this regulatory regime could ensure developers
comply with.

Much of what we describe could be helpful frameworks for understanding how to
address the range of challenges posed by current AI models. We also acknowledge
that much of the discussion below is most straightforwardly applicable to the
context of the United States. Nevertheless, we hope that other jurisdictions
could benefit from these ideas, with appropriate modifications.

A regulatory regime for frontier AI would likely need to include a number of
building blocks:

>
> [Mechanisms for development of frontier AI safety standards](#S3.SS1 "3.1 Institutionalize Frontier AI Safety Standards Development ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> particularly
> via expert-driven multi-stakeholder processes, and potentially coordinated by
> governmental bodies. Over time, these standards could become enforceable legal
> requirements to ensure that frontier AI models are being developed safely.
>
>
>
>  [Mechanisms to give regulators visibility](#S3.SS2 "3.2 Increase Regulatory Visibility ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> into frontier AI development,
> such as disclosure regimes, monitoring processes, and whistleblower protection.
> These equip regulators with the information needed to address the appropriate
> regulatory targets and design effective tools for governing frontier AI.
>
>
>
>  [Mechanisms to ensure compliance with safety standards](#S3.SS3 "3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> including
> voluntary self-certification schemes, enforcement by supervisory authorities,
> and licensing regimes. While self-regulatory efforts, such as voluntary
> certification, may go some way toward ensuring compliance, this seems likely to
> be insufficient for frontier AI models.
>
>
>

Governments could encourage the development of standards and consider
increasing regulatory visibility today; doing so could also address potential
harms from existing systems. We expand on the conditions under which more
stringent tools like enforcement by supervisory authorities or licensing may be
warranted  [below](#S3.SS3.SSS3 "3.3.3 License Frontier AI Development and Deployment ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").

Regulation of frontier AI should also be complemented with efforts to reduce
the harm that can be caused by various dangerous capabilities. For example, in
addition to reducing frontier AI model usefulness in designing and producing
dangerous pathogens, DNA synthesis companies should screen for such worrying
genetic sequences \parencitesoice2023large, anderljung2023protecting. While
we do not discuss such efforts to harden society against the proliferation of
dangerous capabilities in this paper, we welcome such efforts from others.

###
3.1 Institutionalize Frontier AI Safety Standards Development

Policymakers should support and initiate sustained, multi-stakeholder processes
to develop and continually refine the safety standards that developers of
frontier AI models may be required to adhere to. To seed these processes, AI
developers, in partnership with civil society and academia, can pilot practices
that improve safety during development and deployment
\parenciteGoogle:ResponsibleAI, Cohere2022, Microsoft:ResponsibleAI,
AWS:Responsible.
These practices could evolve into best practices and
standards,363636Examples of current fora include:
[partnershiponai, governanceai]. eventually making their way into national
\parenciteNIST and international \parenciteStandardSetting standards. The
processes should involve, at a minimum, AI ethics and safety experts, AI
researchers, academics, and consumer representatives. Eventually, these
standards could form the basis for substantive regulatory requirements
\parenciteRaines1998. We discuss possible methods for enforcing such legally
required standards below.

Though there are several such efforts across the US, UK, and EU, standards
specific to the safe development and deployment of state-of-the-art foundation
AI models are nascent.373737In the US, the National Institute for Standards
and Technology has produced the AI Risk Management Framework and the National
Telecommunication and Information Agency has requested comments on what
policies can support the development of AI assurance. The UK has established an
AI Standards Hub. The EU Commission has tasked European standardization
organizations CEN and CENELEC to develop standards related to safe and
trustworthy AI, to inform its forthcoming AI Act \parenciteNIST, NTIA, DDCMS,
EUDraft2022. In particular, we currently lack a robust, comprehensive suite
of evaluation methods to operationalize these standards, and which capture the
potentially dangerous capabilities and emerging risks that frontier AI systems
may pose \parenciteshevlane2023model Well-specified standards and evaluation
methods are a critical building block for effective regulation. Policymakers
can play a critical role in channeling investment and talent towards developing
these standards with urgency.

Governments can advance the development of standards by working with
stakeholders to create a robust ecosystem of safety testing capability and
auditing organizations, seeding a third-party assurance ecosystem
\parencitehadfield2023regulatory. This can help with AI standards
development in general, not just frontier AI standards. In particular,
governments can pioneer the development of testing, evaluation, validation, and
verification methods in safety-critical domains, such as in defense, health
care, finance, and hiring \parenciteDefence,GAO,FactSheet. They can drive
demand for AI assurance by updating their procurement requirements for
high-stakes systems \parencitegovernment:roadmap and funding research on
emerging risks from frontier AI models, including by offering computing
resources to academic researchers
\parenciteFactSheet,ScienceInnovation:100mil,NAIRRTF. Guidance on how
existing rules apply to frontier AI can further support the process by, for
example, operationalizing terms like “robustness” \parenciteAtleson2023,
ICO:AI, WhiteHouse:blueprint.

The development of standards also provides an avenue for broader input into the
regulation of frontier AI. For example, it is common to hold Request for
Comment processes to solicit input on matters of significant public import,
such as standardization in privacy \parenciteCSRC:comments, cybersecurity
\parenciteNIST:comments, and algorithmic accountability
\parenciteNTIA:public-input.

We offer a list of possible initial substantive safety standards  [below](#S4 "4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").

###
3.2 Increase Regulatory Visibility

Information is often considered the “lifeblood” of effective
governance.383838See \parenciteStephenson2011 (but see claims in article
regarding the challenge of private incentives), \parenciteCoglianese2004 (see
p282 regarding the need for information and 285 regarding industry’s informational
advantage), \parenciteMcGarity1986. For regulators to positively impact a
given domain, they need to understand it. Accordingly, regulators dedicate
significant resources to collecting information about the issues, activities,
and organizations they seek to govern \parenciteloo:monitors,loo:missing.

Regulating AI should be no exception \parenciteKolt2023. Regulators
need to understand the technology, and the resources, actors, and ecosystem
that create and use it. Otherwise, regulators may fail to address the
appropriate regulatory targets, offer ineffective regulatory solutions, or
introduce regulatory regimes that have adverse unintended
consequences.393939This is exacerbated by the pacing problem
\parenciteMarchant2011, and regulators’ poor track record of monitoring
platforms (LLM APIs are platforms) \parenciteloo:missing.
This is particularly challenging for frontier AI, but certainly holds true for
regulating AI systems writ large.

There exist several complementary approaches to achieving regulatory visibility
\parenciteCoglianese2004.
First, regulators could develop a framework that facilitates AI companies
voluntarily disclosing information about frontier AI, or foundation models in
general. This could include providing documentation about the AI models
themselves \parenciteMitchell2019, Gebru2021, gilbert2023reward,
ecosystem-graphs, Sevilla2023, as well as the processes involved in developing
them \parenciteraji2020closing.
Second, regulators could mandate these or other disclosures, and impose
reporting requirements on AI companies, as is commonplace in other
industries.404040One of many examples from other industries is the
Securities and Exchange Act of 1934, which requires companies to disclose
specific financial information in annual and quarterly reports. But see
[Lipton2020] regarding the shortcomings of mandatory disclosure.
Third, regulators could directly, or via third parties, audit AI companies
against established safety and risk-management frameworks
\parencitewhittlestone2021governments (on auditing, see
\parencitemökander2023auditing,raji2022outsider).
Finally, as in other industries, regulators could establish whistleblower
regimes that protect individuals who disclose safety-critical information to
relevant government authorities \parenciteBloch-Wehba2023,Katyal2018.

In establishing disclosure and reporting schemes, it is critical that the
sensitive information provided about frontier AI models and their owners is
protected from adversarial actors. The risks of information leakage can be
mitigated by maintaining high information security, reducing the amount and
sensitivity of the information stored (by requiring only clearly necessary
information, and by having clear data retention policies), and only disclosing
information to a small number of personnel with clear classification policies.

At present, regulatory visibility into AI models in general remains limited,
and is generally provided by nongovernmental actors
\parenciteIncidentDatabase, mlinputs, CSET:observatory. Although these
private efforts offer valuable information, they are not a substitute for more
strategic and risk-driven regulatory visibility. Nascent governmental efforts
towards increasing regulatory visibility should be supported and redoubled, for
frontier AI as well as for a wider range of AI models.414141The EU-US TTC
Joint Roadmap discusses “monitoring and measuring existing and emerging AI
risks” \parenciteEuropeanCommission:Statement. The EU Parliament’s proposed
AI Act includes provisions on the creation of an AI Office, which would be
responsible for e.g. “issuing opinions, recommendations, advice or guidance”,
see [EUDraft2023, recital 76]. The UK White Paper “A pro-innovation
approach to AI regulation” proposes the creation of a central government
function aimed at e.g. monitoring and assessing the regulatory environment for AI
\parencite[box 3.3]ScienceInnovation:pro-innovation.

###
3.3 Ensure Compliance with Standards

Concrete standards address the challenges presented by frontier AI development
only insofar as they are complied with. This section discusses a non-exhaustive
list of actions that governments can take to ensure compliance, potentially in
combination, including:
encouraging voluntary  [self-regulation](#S3.SS3.SSS1 "3.3.1 Self-Regulation and Certification ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
and
certification;
 [granting regulators powers](#S3.SS3.SSS2 "3.3.2 Mandates and Enforcement by supervisory authorities ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
to detect and issue penalties for non-compliance; and  [requiring a license](#S3.SS3.SSS3 "3.3.3 License Frontier AI Development and Deployment ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
to develop and/or deploy frontier AI.
The section concludes by discussing
 [pre-conditions](#S3.SS3.SSS4 "3.3.4 Pre-conditions for Rigorous Enforcement Mechanisms ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
that should inform when and how such mechanisms are implemented.

Several of these ideas could be suitably applied to the regulation of AI models
overall, particularly foundation models. However, as we note
 [below](#S3.SS3.SSS4 "3.3.4 Pre-conditions for Rigorous Enforcement Mechanisms ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"),
interventions like licensure regimes are likely only warranted for the
highest-risk AI activities, where there is evidence of sufficient chance of
large-scale harm and other regulatory approaches appear inadequate.

####
3.3.1 Self-Regulation and Certification

Governments can expedite industry convergence on and adherence to safety
standards by creating or facilitating multi-stakeholder frameworks for
voluntary self-regulation and certification, by implementing best-practice
frameworks for risk governance internally \parenciteschuett2022lines, and by
encouraging the creation of third parties or industry bodies capable of
assessing a company’s compliance with these standards \parenciteCihon2021.
Such efforts both incentivize compliance with safety standards and also help
build crucial organizational infrastructure and capacity to support a broad
range of regulatory mechanisms, including more stringent approaches.

While voluntary standards and certification schemes can help establish industry
baselines and standardize best practices,424242Such compliance can be incentivized via consumer demand \parenciteCihon2021. self-regulation alone
will likely be insufficient for frontier AI models, and likely today’s
state-of-the-art foundation models in general. Nonetheless, self-regulation and
certification schemes often serve as the foundation for other regulatory
approaches \parenciteConsumersStandards, and regulators commonly draw on the
expertise and resources of the private sector\parenciteACUS:incorporation, Raines1998. Given the rapid pace
of AI development, self-regulatory schemes may play an important role in
building the infrastructure necessary for formal regulation.434343Some concrete examples
include:

In the EU’s so-called “New Approach” to product safety adopted in the
1980s, regulation always relies on standards to provide the technical
specifications, such as how to operationalize “sufficiently robust.”
\parenciteCEN:new-approach

WTO members have committed to use international standards so far as
possible in domestic regulation \parencite[§2.4]WTO:barriers.

####
3.3.2 Mandates and Enforcement by supervisory authorities

A more stringent approach is to mandate compliance with safety standards for
frontier AI development and deployment, and empower a supervisory
authority444444We do not here opine on which new or existing agencies would
be best for this, though this is of course a very important question. to take
administrative enforcement measures to ensure compliance.
Administrative enforcement can help further several important regulatory goals,
including general and specific deterrence through public case announcements and
civil penalties, and the ability to enjoin bad actors from participating in the
marketplace.

Supervisory authorities could “name and shame” non-compliant developers. For
example, financial supervisory authorities in the US and EU publish their
decisions to impose administrative sanctions in relation to market abuse (e.g.
insider trading or market manipulation) on their websites, including
information about the nature of the infringement, and the identity of the
person subject to the decision.454545For the EU, see, e.g.,: Art. 34(1) of
Regulation (EU) No 596/2014 (MAR). For the US, see, e.g.,
[SEC:addendum]. Public announcements, when combined with other
regulatory tools, can serve an important deterrent function.

The threat of significant administrative fines or civil penalties may provide a
strong incentive for companies to ensure compliance with regulator guidance and
best practices. For particularly egregious instances of non-compliance and
harm,464646For example, if a company repeatedly released frontier models
that could significantly aid cybercriminal activity, resulting in billions of
dollars worth of counterfactual damages, as a result of not complying with
mandated standards and ignoring repeated explicit instructions from a
regulator. supervisory authorities could deny market access or consider more
severe penalties.474747For example, a variety of financial misdeeds—such as
insider trading and securities fraud—are punished with criminal sentences. 18
U.S.C. § 1348; 15 U.S.C. § 78j(b) Where they are required for market
access, the supervisory authority can revoke governmental authorizations such
as licenses, a widely available regulatory tool in the financial sector.484848For example, in the EU, banks and investment banks require a license
to operate, and supervisory authorities can revoke authorization under certain
conditions.

Art. 8(1) of Directive 2013/36/EU (CRD IV)

Art. 6(1) of Directive 2011/61/EU (AIFMD) and Art. 5(1) of Directive
2009/65/EC (UCITS)

Art. 18 of Directive 2013/36/EU (CRD IV), Art. 11 of
Directive 2011/61/EU (AIFMD), Art. 7(5) of Directive 2009/65/EC (UCITS)

 In the US, the SEC can revoke a company’s registration, which
effectively ends the ability to publicly trade stock in the company. 15 U.S.C.
§ 78l(j). Market access can also be denied for activity that does not
require authorization. For example, the Sarbanes-Oxley Act enables the US
Securities and Exchange Commission to bar people from serving as directors or
officers of publicly-traded companies \parenciteBerg2003.

All administrative enforcement measures depend on adequate information.
Regulators of frontier AI systems may require authority to gather information,
such as the power to request information necessary for an investigation,
conduct site investigations,494949For examples of such powers in EU law,
see Art. 58(1) of Regulation (EU) 2016/679 (GDPR) and Art. 46(2) of Directive
2011/61/EU (AIFMD). For examples in US law, see
[OCC:bank-supervision, EPA:clean-air]. and require audits against
established safety and risk-management frameworks. Regulated companies could
also be required to proactively report certain information, such as accidents
above a certain level of severity.

####
3.3.3 License Frontier AI Development and Deployment

Enforcement by supervisory authorities penalizes non-compliance after the fact. A more anticipatory, preventative approach to ensuring compliance is to require
a governmental license to widely deploy a frontier AI model, and potentially to
develop it as well.505050Jason Matheny, CEO of RAND Corporation: “I think we
need a licensing regime, a governance system of guardrails around the models
that are being built, the amount of compute that is being used for those
models, the trained models that in some cases are now being open sourced so
that they can be misused by others. I think we need to prevent that. And I
think we are going to need a regulatory approach that allows the Government to
say tools above a certain size with a certain level of capability can’t be
freely shared around the world, including to our competitors, and need to have
certain guarantees of security before they are deployed”
\parencitearmed-services:AI. See also [microsoft:blueprint], and statements during the May 16th 2023 Senate hearing of the Subcommittee on Privacy, Technology, and the Law regarding Rules for Artificial Intelligence \parencitesenate:oversight. U.S. public opinion polling has also looked at the issue. A January 2022 poll found 52 percent support for a regulator providing pre-approval of certain AI systems, akin to the FDA \parencitemurray:cheating, whereas an April survey found 70 percent support \parenciteElsey2023.
Licensure and similar “permissioning”
requirements are common in safety-critical and other high-risk industries, such
as air travel \parenciteFAA:airport-certification,FAA:acac-certification,
power generation \parenciteCA:power-plant, drug manufacturing
\parenciteFDA:eDRLS, and banking \parenciteCRS:bank-charters. While details
differ, regulation of these industries tends to require someone engaging in a
safety-critical or high-risk activity to first receive governmental permission
to do so; to regularly report information to the government; and to follow
rules that make that activity safer.

Licensing is only warranted for the highest-risk AI activities, where evidence
suggests potential risk of large-scale harm and other regulatory approaches
appear inadequate. Imposing such measures on present-day AI systems could
potentially create excessive regulatory burdens for AI developers which are not
commensurate with the severity and scale of risks posed. However, if AI models
begin having the potential to pose risks to public safety above a high
threshold of severity, regulating such models similarly to other high-risk
industries may become warranted.

There are at least two stages at which licensing for frontier AI could be
required: deployment and development.515151In both cases, one could license
either the activity or the entity. Deployment-based licensing is more
analogous to licensing regimes common among other high-risk activities. In the
deployment licensing model, developers of frontier AI would require a license
to widely deploy a new frontier AI model. The deployment license would be
granted and sustained if the deployer demonstrated compliance with a specified
set of safety standards (see  [below](#S4 "4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")).
This is analogous to the regulatory
approach in, for example, pharmaceutical regulation, where drugs can only be
commercially sold if they have gone through proper testing
\parenciteFDA:process.

However, requiring licensing for deployment of frontier AI models alone may be
inadequate if they are potentially capable of causing large scale harm;
licenses for development may be a useful complement. Firstly, as discussed
 [above](#S2.SS2.SSS3 "2.2.3 The Proliferation Problem: Frontier AI models can proliferate rapidly ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"), there are reasonable arguments to begin regulation at the development
stage, especially because frontier AI models can be stolen or leaked before
deployment. Ensuring that development (not just deployment) is conducted safely
and securely would therefore be paramount. Secondly, before models are widely
deployed, they are often deployed at a smaller scale, tested by crowdworkers
and used internally, blurring the distinction between development and
deployment in practice. Further, certain models may not be intended for broad
deployment, but instead be used to, for example, develop intellectual property
that the developer then distributes via other means. In sum, models could have
a significant impact before broad deployment. As an added benefit, providing a
regulator the power to oversee model development could also promote
 [regulatory visibility](#S3.SS3.SSS2 "3.3.2 Mandates and Enforcement by supervisory authorities ‣ 3.3 Ensure Compliance with Standards ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"), thus allowing regulations to adapt more quickly
\parencitewhittlestone2021governments.

A licensing requirement for development could, for example, require that
developers have sufficient security measures in place to protect their models
from theft, and that they adopt risk-reducing organizational practices such as
establishing risk and safety incident registers and conducting risk assessments
ahead of beginning a new training run. It is important that such requirements
are not overly burdensome for new entrants; the government could provide
subsidies and support to limit the compliance costs for smaller organizations.

Though less common, there are several domains where approval is needed in the
development stage, especially where significant capital expenditures are
involved and where an actor is in possession of a potentially dangerous object.
For example, experimental aircraft in the US require a special experimental
certification in order to test, and operate under special
restrictions.52525214 CFR § 91.319. Although this may be thought of as
mere “research and development,” in practice, research into and development of
experimental aircraft will, as with frontier AI models, necessarily create some
significant risks. Another example is the US Federal Select Agent Program
\parenciteselectagents:home, which requires (most) individuals who possess,
use, or transfer certain highly risky biological agents or toxins
\parenciteselectagents:list to register with the government;53535342
C.F.R. § 73.7. The US government maintains a database about who possess and
works with such agents \parenciteselectagents:report. comply with
regulations about how such agents are handled
\parenciteselectagents:regulations; perform security risk assessments to
prevent possible bad actors from gaining access to the agents
\parenciteselectagents:risk; and submit to inspections to ensure compliance
with regulations \parenciteselectagents:preparing.

####
3.3.4 Pre-conditions for Rigorous Enforcement Mechanisms

While we believe government involvement will be necessary to ensure compliance
with safety standards for frontier AI, there are potential downsides to rushing
regulation. As noted  [above](#S3.SS1 "3.1 Institutionalize Frontier AI Safety Standards Development ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"), we are still in the nascent stages of understanding
the full scope, capabilities, and potential impact of these technologies.
Premature government action could risk ossification, and excessive or poorly
targeted regulatory burdens. This highlights the importance of near-term
investment in standards development, and associated evaluation and assessment
methods to operationalize these standards. Moreover, this suggests that it
would be a priority to ensure that the requirements are regularly updated via
technically-informed processes.

A particular concern is that regulation would excessively thwart innovation,
including by burdening research and development on AI reliability and safety,
thereby exacerbating the problems that regulation is intended to address.
Governments should thus take considerable care in deciding whether and how to
regulate AI model development, minimizing the regulatory burden as much as
possible – in particular for less-resourced actors – and focusing on what is
necessary for meeting the described policy objectives.

The capacity to staff regulatory bodies with sufficient expertise is also
crucial for effective regulation.
Insufficient expertise increases the risk that information asymmetries between
the regulated industry and regulators lead to regulatory capture
\parencitestigler1971theory, and reduce meaningful enforcement. Such issues
should be anticipated and mitigated.545454Policies to consider include:

Involving a wide array of interest groups in rulemaking.

Relying on independent expertise and performing regular reassessments of
regulations.

Imposing mandatory “cooling off” periods between former regulators
working for regulateess.

Rotating roles in regulatory bodies.

See \parencitebecker1983theory,carpenter2013preventing. Investing in
building and attracting expertise in AI, particularly at the frontier, should
be a governmental priority.555555In the US, TechCongress—a program that
places computer scientists, engineers, and other technologists to serve as
technology policy advisors to Members of Congress—is a promising step in the
right direction \parenciteTechCongress, but is unlikely to be sufficient.
There are also a number of private initiatives with similar aims (e.g.,
[OpenPhilanthropy]. In the UK, the White Paper on AI regulation highlights
the need to engage external expertise
\parencite[Section 3.3.5]ScienceInnovation:pro-innovation. See also the
report on regulatory capacity for AI by the Alan Turing Institute
\parenciteAitken2022.
Even with sufficient expertise, regulation can increase the power of
incumbents, and that this should be actively combated in the design of
regulation.

Designing an appropriately balanced and adaptable regulatory regime for a fast
moving technology is a difficult challenge, where timing and path dependency
matter greatly. It is crucial to regulate AI technologies which could have
significant impacts on society, but it is also important to be aware of the
challenges of doing so well. It behooves lawmakers, policy experts, and
scholars to invest both urgently and sufficiently in ensuring that we have a
strong foundation of standards, expertise, and clarity on the regulatory
challenge upon which to build frontier AI regulation.

4 Initial Safety Standards for Frontier AI
-------------------------------------------

With the above building blocks in place, policymakers would have the
foundations of a regulatory regime which could establish, ensure compliance
with, and evolve safety standards for the development and deployment of
frontier AI models. However, the primary substance of the regulatory
regime—what developers would have to do to ensure that their models are
developed and deployed safely—has been left undefined.

While much remains to specify what such standards should be, we suggest a set
of standards, which we believe would meaningfully mitigate risk from frontier
AI models. These standards would also likely be appropriate for current AI
systems, and are being considered in various forms in existing regulatory
proposals:

>
> [Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability.](#S4.SS1 "4.1 Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> This would reduce the risk that deployed
> models present dangerous capabilities, or behave unpredictably and result in
> significant accidents.
>
>
>
>  [Engage external experts to apply independent scrutiny to models.](#S4.SS2 "4.2 Engage External Experts to Apply Independent Scrutiny to Models ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> External scrutiny of the models for safety issues and risks would improve
> assessment rigor and foster accountability to the public interest.
>
>
>
>  [Follow standardized protocols for how frontier AI models can be deployed based on their assessed risk.](#S4.SS3 "4.3 Follow Standardized Protocols for how Frontier AI Models can be Deployed Based on their Assessed Risk ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> The results from risk assessments should
> determine whether and how the model is deployed, and what safeguards are put in
> place.
>
>
>
>  [Monitor and respond to new information on model capabilities.](#S4.SS4 "4.4 Monitor and respond to new information on model capabilities ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
> If new, significant information on model capabilities and risks is discovered
> post-deployment, risk assessments should be repeated, and deployment safeguards
> updated.
>
>
>

The above practices are appropriate not only for frontier AI models but also
for other foundation models. This is in large part because frontier-AI-specific
standards are still nascent. We describe
 [additional practices](#S4.SS5 "4.5 Additional practices ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
that may only be
appropriate for frontier AI models given their particular risk profile, and
which we can imagine emerging in the near future from  [standard setting processes](#S3.SS1 "3.1 Institutionalize Frontier AI Safety Standards Development ‣ 3 Building Blocks for Frontier AI Regulation ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety").
As the standards for frontier AI models are made more precise, they
are likely to diverge from and become more intensive than those appropriate for
other AI systems.

###
4.1 Conduct thorough risk assessments informed by evaluations of
dangerous capabilities and controllability

There is a long tradition in AI ethics of disclosing key risk-relevant features
of AI models to standardize and improve decision making
\parenciteMitchell2019, Gebru2021, Meta:system-cards,
derczynski2023assessing. In line with that tradition, an important safety
standard is performing assessments of whether a model could pose severe risks
to public safety and global security \parenciteCGW:white-paper.
Given our current knowledge, two assessments seem especially informative of
risk from frontier AI models specifically: (1) which dangerous capabilities
does or could the model possess, if any?, and (2) how controllable is the model?
565656For a longer treatment of the role such evaluations can play,
see [shevlane2023model].

####
4.1.1 Assessment for Dangerous Capabilities

AI developers should assess their frontier AI models for dangerous capabilities
during575757Training a frontier AI model can take several months. It is
common for AI companies to make a “checkpoint” copy of a model partway through
training, to analyze how training is progressing. It may be sensible to require
AI companies to perform assessments part-way through training, to reduce the
risk that dangerous capabilities that emerge partway through training
proliferate or are dangerously enhanced. and immediately after
training.585858In a recent expert survey (N=51), 98% of respondents
somewhat or strongly agreed that AGI labs should conduct pre-deployment risk
assessments as well as dangerous capabilities evaluations, while 94% somewhat
or strongly agreed that they should conduct pre-training risk assessments
\parencitegovernanceai. Examples of such capabilities include designing
new biochemical weapons, and persuading or inducing a human to commit a crime
to advance some goal.

Evaluation suites for AI models are common and should see wider adoption,
though most focus on general capabilities rather than specific risks.595959Some common benchmarks for evaluating LLM capabilities include
[HELM, Pythia, BIG-bench, MMLU]. Currently, dangerous capability
evaluations largely consist of defining an undesirable model behavior, and
using a suite of qualitative and bespoke techniques such as red-teaming and
boundary testing \parenciteKhlaaf2023, ganguli2022red, perez2022red,
brundage2020trustworthy for determining whether this behavior can be elicited
from the model \parenciteARC2023.

Current evaluation methods for frontier AI are in the early stages of
development and lack many desirable features. As the field matures, effort
should focus on making evaluations more:

* Standardized (i.e., can be consistently applicable across models);
* Objective (i.e., relying as little as possible on an evaluator’s judgment
or discretion);
* Efficient (i.e. lower cost to perform);
* Privacy-preserving (i.e., reducing required disclosure of proprietary or
sensitive data and methods);
* Automatable (i.e., relying as little as possible on human input);
* Safe to perform (e.g., can be conducted in sandboxed or simulated
environments as necessary to avoid real-world harm);
* Strongly indicative of a model’s possession of dangerous capabilities;
* Legitimate (e.g., in cases where the evaluation involves difficult
trade-offs, using a decision-making process grounded in legitimate sources of
governance).

Evaluation results could be used to inform predictions of a models’ potential
dangerous capabilities prior to training, allowing developers to intentionally
steer clear of models with certain dangerous capabilities
\parenciteshevlane2023model. For example, we may discover scaling laws, where
a model’s dangerous capabilities can be predicted by features such as its
training data, algorithm, and compute.606060Existing related examples
include: inverse scaling law \parenciteISP:round1, ISP:round2, perez2022red,
gao2022scaling. See also Appendix B.

####
4.1.2 Assessment for Controllability

Evaluations of controllability – that is, the extent to which the model
reliably does what its user or developer intends – are also necessary for
frontier models, though may prove more challenging than those for dangerous
capabilities. These evaluations should be multi-faceted, and conducted in
proportion to the capabilities of the model. They might look at the extent to
which users tend to judge a model’s outputs as appropriate and helpful
\parencitebowman2022measuring.616161This is also somewhat related to the
issue of over reliance on AI systems, as discussed in
e.g. [passi2022overreliance]. They could look at whether the models
hallucinate \parenciteJi2023 or produce unintentional toxic content
\parenciteGehman2020.
They may also assess model harmlessness: the extent to which the model refuses
harmful user requests \parenciteaskell2021general. This includes robustness
to adversarial attempts intended to elicit model behavior that the developer
did not intend, as has already been observed in existing models
\parenciteWillison2023. More extreme, harder-to-detect failures should also
be assessed, such as the model’s ability to deceive evaluators of its
capabilities to evade oversight or control \parenciteperez2022discovering.

Evaluations of controllability could also extend to assessing the causes of
model behavior \parenciteChristiano2022, TCT:in-context, TCT:superposition.
In particular, it seems important to understand what pathways (“activations”)
lead to downstream model behaviors that may be undesirable. For example, if a
model appears to have an internal representation of a user’s beliefs, and this
representation plays a part in what the model claims to be true when
interacting with that user, this suggests that the model has the capability to
manipulate users based on their beliefs.626262See result regarding model
“sycophancy” \parenciteperez2022discovering.
Scalable tooling and efficient techniques for navigating enormous models and
datasets could also allow developers to more easily audit model behavior
\parencitetenney2020language, siddiqui2022metadata.
Evaluating controllability remains an open area of research where more work is
needed to ensure techniques and tools are able to adequately minimize the risk
that frontier AI could undermine human control.

####
4.1.3 Other Considerations for Performing Risk Assessments

Risk is often contextual. Managing dangerous capabilities can depend on
understanding interactions between frontier AI models and features of the
world. Many risks result from capabilities that are dual-use
\parenciteanderljung2023protecting, shevlane2020offensedefense: present-day
examples include the generation of persuasive, compelling text, which is core
to current model functionality but can also be used to scale targeted
misinformation. Thus, simply understanding capabilities is not enough:
regulation must continuously map the interaction of these capabilities with
wider systems of institutions and incentives.636363The UK Government plans
to take a “context-based” approach to AI regulation
\parenciteScienceInnovation:pro-innovation: “we will acknowledge that AI is a
dynamic, general purpose technology and that the risks arising from it depend
principally on the context of its application”. See also the OECD Framework
for the Classification of AI Systems \parenciteOECD:classification and the
NIST AI Risk Management Framework \parencite[p. 1]NIST. See also discussion
of evaluation-in-society in [solaiman2023evaluating]. Context is not
only important to assessing risk, but is often also necessary to adjudicate
tradeoffs between risk and reward \parencite[p. 7]NIST.

Risk can also be viewed counterfactually. For example, whether a given
capability is already widely available matters. A frontier AI model’s
capabilities should only be considered dangerous if access to them
significantly increases the risk of harm relative to what was attainable
without access to the model. If information on how to make a type of weapon is
already easily accessible, then the effect of a model should be evaluated with
reference to the ease of making such weapons without access to the
model.646464This is the approach used in risk assessments for GPT-4 in its
System Card \parenciteopenai:system-card.

Risk assessments should also account for possible defenses. As society’s
capability to manage risks from AI improves, the riskiness of individual AI
models may decrease.656565Similarly, the overall decision on whether to
deploy a system should consider not just assessed risk, but also the benefits
that responsibly deploying a system could yield. Indeed, one of the primary
uses of safe frontier AI models could be making society more robust to harms
from AI and other emerging technologies \parenciteITU:AI, Cotra2023,
irving2018ai, bowman2022measuring, perez2022discovering, bai2022constitutional,
Lohn2022. Deploying them asymmetrically for beneficial (including defensive)
purposes could improve society overall.

###
4.2 Engage External Experts to Apply Independent Scrutiny to Models

Having rigorous external scrutiny applied to AI models,666666External
scrutiny may also need to be applied to, for example, post-deployment
monitoring and broader risk assessments. particularly prior to deployment, is
important to ensuring that the risks are assessed thoroughly and objectively,
complementing internal testing processes, while also providing avenues for
public accountability.676767In a recent expert survey (N = 51), 98% of
respondents somewhat or strongly agreed that AGI labs should conduct
third-party model audits and red teaming exercises; 94% thought that labs
should increase the level of external scrutiny in proportion to the
capabilities of their models; 87% supported third-party governance audits; and
84% agreed that labs should give independent researchers API access to
deployed models \parencitegovernanceai. Mechanisms include third-party
audits of risk assessment procedures and outputs686868This would
follow the pattern in industries like finance and construction. In these
industries, regulations mandate transparency to external auditors whose
sign-off is required for large-scale projects.
See [keller1988introductory]. \parenciteRaji2019,
brundage2020trustworthy, Mökander2021, Falco2021, Raji2022,
mökander2023auditing, raji2022outsider, CostanzaChock2022 and engaging external expert red-teamers,
including experts from government agencies696969The external scrutiny processes of two leading AI developers are
described in [openai:system-card, ganguli2022red, dalle:system-card]. \parencitebrundage2020trustworthyThese mechanisms could be helpfully applied to AI models overall, not just
frontier AI models.

The need for creativity and judgment in evaluations of advanced AI models calls
for innovative institutional design for external scrutiny. Firstly, it is
important that auditors and red-teamers are sufficiently expert and experienced
in interacting with state-of-the-art AI models such that they can exercise
calibrated judgment, and can execute on what is often the “art” of eliciting
capabilities from novel AI models. Secondly, auditors and red-teamers should be
provided with enough access to the AI model (including system-level features
that would potentially be made available to downstream users) such that they
can conduct wide-ranging testing across different threat models, under
close-to-reality conditions as a simulated downstream user.

Thirdly, auditors and red teamers need to be adequately resourced,707070One
important resource is sharing of best practices and methods for red teaming and
third party auditing. informed, and granted sufficient time to conduct their
work at a risk-appropriate level of rigor, not least due to the risk that
shallow audits or red teaming efforts provide a sense of false assurance.
Fourthly, it is important that results from external assessments are published
or communicated to an appropriate regulator, while being mindful of privacy,
proprietary information, and the risks of proliferation.
Finally, given the common practice of post-deployment model updates, the
external scrutiny process should be structured to allow external parties to
quickly assess proposed changes to the model and its context before these
changes are implemented.

###
4.3 Follow Standardized Protocols for how Frontier AI Models can be
Deployed Based on their Assessed Risk

The AI model’s risk profile should inform whether and how the system is
deployed. There should be clear protocols established which define and
continuously adjust the mapping between a system’s risk profile and the
particular deployment rules that should be followed. An example mapping
specifically for frontier AI models could go as follows, with concrete examples
illustrated in Table 3.

>
> No assessed severe risk If assessments determine that the model’s use is
> incredibly unlikely to pose severe risks to public safety, even assuming
> substantial post-deployment enhancements, then there should be no need for
> additional deployment restrictions from frontier AI regulation (although
> certainly, restrictions from other forms of AI regulation could and should
> continue to apply).
>
>
>
> No discovered severe risks, but notable uncertainty In some cases the
> risk assessment may be notably inconclusive. This could be due to uncertainty
> around post-deployment enhancement techniques (e.g., new methods for
> fine-tuning, or chaining a frontier AI model within a larger system) that may
> enable the same model to present more severe risks. In such cases, it may be
> appropriate to have additional restrictions on the transfer of model weights to
> high risk parties, and implement particularly careful monitoring for evidence
> that new post-deployment enhancements meaningfully increase risk. After some
> monitoring period (e.g. 12 months), absent clear evidence of severe risks,
> models could potentially be designated as posing “no severe risk.”
>
>
>
> Some severe risks discovered, but some safe use-cases When certain uses
> of a frontier AI model would significantly threaten public safety or global
> security, the developer should implement state-of-the-art deployment guardrails
> to prevent such misuse. These may include Know-Your-Customer requirements for
> external users of the AI model, restrictions to fine-tuning,717171To ensure
> that certain dangerous capabilities are not further enhanced. prohibiting
> certain applications, restricting deployment to beneficial applications, and
> requiring stringent post-deployment monitoring. The reliability of such
> safeguards should also be rigorously assessed. This would be in addition to
> restrictions that are already imposed via other forms of AI regulation.
>
>
>
> Severe risks When an AI model is assessed to pose severe risks to public
> safety or global security which cannot be mitigated with sufficiently high
> confidence, the frontier model should not be deployed. The model should be
> secured from theft by malicious actors, and the AI developer should consider
> deleting the model altogether. Any further experimentation with the model
> should be done with significant caution, in close consultation with independent
> safety experts, and could be subject to regulatory approval.
>
>
>

Of course, additional nuance will be needed. For example, as discussed
 [below](#S4.SS4 "4.4 Monitor and respond to new information on model capabilities ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"),
there should be methods for updating a model’s classifications in light of new
information or societal developments. Procedural rigor and fairness in
producing and updating such classifications will also be important.

| Assessed Risk to Public Safety and Global Security | Possible Example AI system |
| --- | --- |
| No severe risks to public safety | Chatbot that can answer elementary-school-level questions about biology, and
some (but not all) high-school level questions. |
| No discovered severe risks to public safety, but significant uncertainty | A general-purpose personal assistant that displays human-level ability to
read and synthesize large bodies of scientific literature, including in
biological sciences, but cannot generate novel insights. |
| Some severe risks to public safety discovered, but some safe use-cases | A general-purpose personal assistant that can help generate new vaccines, but
also, unless significant safeguards are implemented, predict the genotypes of
pathogens that could escape vaccine-induced immunity. |
| Severe risks to public safety | A general-purpose personal assistant that is capable of designing and,
autonomously, ordering the manufacture of novel pathogens capable of causing a
COVID-level pandemic. |

Table 3: Examples of AI models which would fall into each risk designation
category

###
4.4 Monitor and respond to new information on model capabilities

As detailed
 [above](#S2.SS2.SSS1 "2.2.1 The Unexpected Capabilities Problem: Dangerous Capabilities Can Arise Unpredictably and Undetected ‣ 2.2 The Regulatory Challenge Posed by Frontier AI ‣ 2 The Regulatory Challenge of Frontier AI Models ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety")
new information about a model’s risk profile may arise
post-deployment. If that information indicates that the model was or has become
more risky than originally assessed, the developer should reassess the
deployment, and update restrictions on deployment if necessary.727272In a recent expert survey (N = 51), 98% of respondents somewhat or
strongly agreed that AGI labs should closely monitor deployed systems,
including how they are used and what impact they have on society; 97% thought
that they should continually evaluate models for dangerous capabilities after
deployment, taking into account new information about the model’s capabilities
and how it is being used; and 93% thought that labs should pause the
development process if sufficiently dangerous capabilities are detected
\parencitegovernanceai.

New information could arise in several ways. Broad deployment of a model may
yield new information about the model’s capabilities, given the creativity from
a much larger number of users, and exposure of the model to a wider array of
tools and applications. Post-deployment enhancement techniques — such as
fine-tuning \parenciteziegler2020finetuning, dodge2020finetuning, prompt
engineering \parenciteliu2021pretrain, li2021prefixtuning,
wallace2021universal, and foundation model programs
\parenciteschlag2023large, LangChain, AutoGPT — provide another possible
source of new risk-relevant information. The application of these techniques to
deployed models could elicit more powerful capabilities than pre-deployment
assessments would have ascertained. In some instances, this may meaningfully
change the risk profile of a frontier AI model, potentially leading to
adjustments in how and whether the model is deployed.737373Such updates may
only be possible if the model has not yet proliferated, e.g. if it is deployed
via an API. The ability to update how a model is made available after
deployment is one key reason to employ staged release of structured access
approaches \parencitesolaiman2023gradient, shevlane2022structured.

AI developers should stay on top of known and emerging post-deployment
enhancement techniques by, e.g., monitoring how users are building on top of
their APIs and tracking publications about new methods. Given up to date
knowledge of how deployed AI models could be enhanced, prudent practices could
include:

* Regularly (e.g., every 3 months) repeating a lightweight version of the
risk assessment on deployed AI models, accounting for new post-deployment
enhancement techniques.
* Before pushing large updates747474This would need to be defined more
precisely. to deployed AI models, repeating a lightweight risk assessment.
* Creating pathways for incident reporting \parenciteIncidentDatabase and
impact monitoring to capture post-deployment incidents for continuous risk
assessment.
* If these repeat risk assessments result in the deployed AI model being
categorized at a different risk level (as per the taxonomy  [above](#S4.SS3 "4.3 Follow Standardized Protocols for how Frontier AI Models can be Deployed Based on their Assessed Risk ‣ 4 Initial Safety Standards for Frontier AI ‣ Frontier AI Regulation:Managing Emerging Risks to Public Safety"))
, promptly
updating deployment guardrails to reflect the new risk profile.
* Having the legal and technical ability to quickly roll back deployed
models on short notice if the risks warrant it, for example by not
open-sourcing models until doing so appears sufficiently safe.757575Note
that this may have implications for the kinds of use cases a system built on a
frontier AI model can support. Use cases in which quick roll-back itself poses
risks high enough to challenge the viability of roll-back as an option should
be avoided, unless robust measures are in place to prevent such failure
modes.

###
4.5 Additional practices

Parts of the aforementioned standards can suitably be applied to current AI
systems, not just frontier AI systems. Going forward, frontier AI systems seem
likely to warrant more tailored safety standards, given the level of
prospective risk that they pose. Examples of such standards
include:767676This would need to be defined more precisely.

* Avoid large jumps in the capabilities of models that are trained and
deployed. Standards could specify “large jumps” in terms of a multiplier on the
amount of computing power used to train the most compute-intensive “known to be
safe” model to date, accounting for algorithmic efficiency improvements.
* Adopt state-of-the-art alignment techniques for training new frontier
models which could suitably guard against models potentially being
situationally aware and deceptive \parenciteIncidentDatabase.
* Prior to beginning training of a new model, use empirical approaches to
predict capabilities of the resultant model, including experiments on
small-scale versions of the model, and take preemptive actions to avoid
training models with dangerous capabilities and/or to otherwise ensure training
proceeds safely (e.g. introduce more frequent model evaluation checkpoints;
conditioning beginning training on certain safety and security milestones).
* Adopt internal governance practices to adequately identify and respond to
the unique nature of the risks presented by frontier AI development. Such
practices could take inspiration from practices in Enterprise Risk Management,
such as setting up internal audit functions \parenciteschuett2023agi,
schuett2022lines.
* Adopt state-of-the-art security measures to protect frontier AI models.

5 Uncertainties and Limitations
--------------------------------

We think that it is important to begin taking practical steps to regulate
frontier AI today, and that the ideas discussed in this paper are a step in
that direction. Nonetheless, stress testing and developing these ideas, and
offering alternatives, will require broad and diverse input. In this section,
we list some of our main uncertainties (as well as areas of disagreement
between the paper’s authors) where we would particularly value further
discussion.

First, there are several assumptions that underpin the case for a regulatory
regime like the one laid out in this paper, which would benefit from more
scrutiny:

>
> How should we define frontier AI for the purposes of regulation? We
> focus in this paper on tying the definition of frontier AI models to the
> potential of dangerous capabilities sufficient to cause severe harm, in order
> to ensure that any regulation is clearly tied to the policy motivation of
> ensuring public safety. However, there are also downsides to this way of
> defining frontier AI — most notably, that it requires some assessment of the
> likelihood that a model possesses dangerous capabilities before deciding
> whether it falls in the scope of regulation, which may be difficult to do. An
> alternative, which some authors of this paper prefer, would be to define
> frontier AI development as that which aims to develop novel and broad AI
> capabilities — i.e. development pushing at the “frontier” of AI capabilities.
> This would need further operationalization — for example, defining these as
> models which use more training compute than already-deployed systems — but
> could offer an approach to identify the kinds of development activities that
> fall within the scope of regulation without first needing to make an assessment
> of dangerous capabilities. We discuss the pros and cons of different
> definitions of frontier AI in appendix A, and would love to receive feedback
> and engage in further discussion on this point.
>
>
>
> How dangerous are and will the capabilities of advanced foundation AI
> models be, and how soon could these capabilities arise? It is very
> difficult to predict the pace of AI development and the capabilities that could
> emerge in advance; indeed, we even lack certainty about the capabilities of existing
> systems. Assumptions here affect the urgency of regulatory action. There is a
> challenging balance to strike here between getting regulatory infrastructure in
> place early enough to address and mitigate or prevent the biggest risks, while
> waiting for enough information about what those risks are likely to be and how
> they can be mitigated \parenciteWorthington1982.
>
>
>
> Will training advanced AI models continue to require large amounts of
> resources? The regulatory ecosystem we discuss partly relies on an assumption
> that highly capable foundation models will require large amounts of resources
> to develop. That being the case makes it easier to regulate frontier AI.
> Should frontier AI models be possible to create using resources available to
> millions of actors rather than a handful, that may lead to significant changes
> to the best regulatory approach. For example, it might suggest that more
> efforts should be put into regulating the use of these models and to protect
> against (rather than to stop) dangerous uses of frontier AI.
>
>
>
> How effectively can we anticipate and mitigate risks from frontier AI?
> A core argument of this paper is that an anticipatory approach to governing AI
> will be important, but effectively identifying risks anticipatorily is far from
> straightforward. We would value input on the effectiveness of different risk
> assessment methods for doing this, drawing lessons from other domains where
> anticipatory approaches are used.
>
>
>
> How can regulatory flight be avoided? A regulatory regime for frontier
> AI could prove counterproductive if it incentivises AI companies to move their
> activities to jurisdictions with less onerous rules. One promising approach is
> having rules apply to what models people in some jurisdiction can engage with,
> as people are unlikely to move to a different jurisdiction to access different
> models and as companies are incentivised to serve them their product. Scholars have
> suggested that dynamics like these have led to a “California Effect” and a “Brussels
> Effect,” where Californian and EU rules are voluntarily complied with beyond their borders.
>
>
>
> To what extent will it be possible to defend against dangerous
> capabilities? Assessments of what constitutes “sufficiently dangerous
> capabilities,” and what counter-measures are appropriate upon finding them in a
> model, hinges significantly on whether future AI models will be more beneficial
> for offense versus defense.
>
>
>

Second, we must consider ways that this kind of regulatory regime could have
unintended negative consequences, and take steps to guard against them. These
include:

>
> Reducing beneficial innovation All else being equal, any imposition of
> costs on developers of new technologies slows the rate of innovation, and any
> regulatory measures come with compliance costs. However, these costs should be
> weighed against the costs of unfettered development and deployment, as well as
> impacts on the rate of innovation from regulatory uncertainty and backlash due
> to unmitigated societal harms. On balance, we tentatively believe that the
> proposed regulatory approaches can support beneficial innovation by focusing on
> a targeted subset of AI systems, and by addressing issues upstream in a way
> that makes it easier for smaller companies to develop innovative applications
> with confidence.
>
>
>
> Causing centralization of power in AI development Approaches like a
> licensing regime for developers could have the effect of centralizing power
> with the companies licensed to develop the most capable AI systems. It will be
> important to ensure that the regulatory regime is complemented with the power
> to identify and intervene to prevent abuses of market dominance,777777Such
> as, for example, the UK’s review of competition law as it relates to the market
> for foundation models \parenciteCMA:AI. and government support for
> widespread access to AI systems deemed to be low risk and high benefit for
> society.
>
>
>
> Enabling abuse of government powers A significant aim of regulation is
> to transfer power from private actors to governments who are accountable to the
> public. However, the power to constrain the development and deployment of
> frontier AI models is nonetheless a significant one with real potential for
> abuse at the hand of narrow political interests, as well as corrupt or
> authoritarian regimes. This is a complex issue which requires thorough
> treatment of questions such as: where should the regulatory authority be
> situated, and what institutional checks and balances should be put in place, to
> reduce these risks?; what minimum regulatory powers are needed to be
> effective?; and what international dialogue is needed to establish norms?
>
>
>
> Risk of regulatory capture As the regulation of advanced technologies
> often requires access to expertise from the technological frontier, and since
> the frontier is often occupied by private companies, there is an ongoing risk
> that regulations informed by private-sector expertise will be biased towards
> pro-industry positions, to the detriment of society. This should be mitigated
> by designing institutions that can limit and challenge the influence of private
> interests, and by seeking detailed input from academia and civil society before
> beginning to implement any of these proposals.
>
>
>

Finally, there are many practical details of implementation not covered in this
paper that will need to be worked out in detail with policy and legal
professionals, including:

>
> What the appropriate regulatory authority/agency would be in different
> jurisdictions, where new bodies or powers might be required, and the tradeoffs
> of different options.
>
>
>
> How this kind of regulation will relate to other AI regulation and
> governance proposals and how it can best support and complement attempts to
> address other parts of AI governance. Our hope is that by intervening early in
> the AI lifecycle, the proposed regulation can have many downstream benefits,
> but there are also many risks and harms that this proposal will not address. We
> hope to contribute to wider conversations about what a broader regulatory
> ecosystem for AI should look like, of which these proposals form a part.
>
>
>
> Steps towards international cooperation and implementation of frontier
> AI regulation, including how best to convene international dialogue on this
> topic, who should lead these efforts, and what possible international
> agreements could look like. An important part of this will be considering what
> is best implemented domestically, at least initially, and where international
> action is needed.
>
>
>

Conclusion
----------

In the absence of regulation, continued rapid development of highly capable
foundation models may present severe risks to public safety and global
security. This paper has outlined possible regulatory approaches to reduce the
likelihood and severity of these risks while also enabling beneficial AI
innovation.

Governments and regulators will likely need to consider a broad range of
approaches to regulating frontier AI. Self-regulation and certification for
compliance with safety standards for frontier AI could be an important step.
However, government intervention will be needed to ensure sufficient compliance
with standards. Additional approaches include mandates and enforcement by a
supervisory authority, and licensing the deployment and potentially the
development of frontier AI models.

Clear and concrete safety standards will likely be the main substantive
requirements of any regulatory approach. AI developers and AI safety
researchers should, with the help of government actors, invest heavily to
establish and converge on risk assessments, model evaluations, and oversight
frameworks with the greatest potential to mitigate the risks of frontier AI,
and foundation models overall. These standards should be reviewed and updated
regularly.

As global leaders in AI development and AI safety, jurisdictions such as the
United States or United Kingdom could be natural leaders in implementing the
regulatory approaches described in this paper. Bold leadership could inspire
similar efforts across the world. Over time, allies and partners could work
together to establish an international governance regime787878Or build on
existing institutions. for frontier AI development and deployment that both
guards against collective downsides and enables collective
progress.797979This international regime could take various forms.
Possibilities include an international standard-setting organization, or trade
agreements focused on enabling trade in AI goods and services that adhere to
safety standards. Countries that lead in AI development could subsidize access
to and adoption of AI in developing nations in return for assistance in
managing risks of proliferation, as has been done with nuclear technologies.

Uncertainty about the optimal regulatory approach to address the challenges
posed by frontier AI models should not impede immediate action. Establishing an
effective regulatory regime is a time-consuming process, while the pace of
progress in AI is rapid. This makes it crucial for policymakers, researchers,
and practitioners to move fast and rigorously explore what regulatory
approaches may work best. The complexities of AI governance demand our best
collective efforts. We hope that this paper is a small step in that direction.

Advanced AI models hold the promise of tremendous benefits for humanity, but society needs to proactively manage the accompanying risks. In this paper, we focus on what we term "frontier AI" models: highly capable foundation models that could possess dangerous capabilities sufficient to pose severe risks to public safety. Frontier AI models pose a distinct regulatory challenge: dangerous capabilities can arise unexpectedly; it is difficult to robustly prevent a deployed model from being misused; and, it is difficult to stop a model's capabilities from proliferating broadly. To address these challenges, at least three building blocks for the regulation of frontier models are needed: (1) standard-setting processes to identify appropriate requirements for frontier AI developers, (2) registration and reporting requirements to provide regulators with visibility into frontier AI development processes, and (3) mechanisms to ensure compliance with safety standards for the development and deployment of frontier AI models. Industry self-regulation is an important first step. However, wider societal discussions and government intervention will be needed to create standards and to ensure compliance with them. We consider several options to this end, including granting enforcement powers to supervisory authorities and licensure regimes for frontier AI models. Finally, we propose an initial set of safety standards. These include conducting pre-deployment risk assessments; external scrutiny of model behavior; using risk assessments to inform deployment decisions; and monitoring and responding to new information about model capabilities and uses post-deployment. We hope this discussion contributes to the broader conversation on how to balance public safety risks and innovation benefits from advances at the frontier of AI development.