Transformative AI Governance:

A Literature Review

Legal Priorities Project – AI Foundations Report 2

April 2023

Matthijs Maas[[1]](#ftnt1)

The transformative AI (TAI) governance community has developed rapidly as the technology’s stakes have become increasingly clear.[[2]](#ftnt2) However, this field still faces a lack of legibility and clarity in different lines of work. Accordingly, this report aims to provide a taxonomy and literature review of TAI governance, in order to contribute to strategic clarity and enable more focused and productive research and debate. This report:

* Discusses the aims & scope of this review, and discusses inclusion criteria;
* Reviews work aimed at understanding the problem of transformative AI governance, mapping TAI’s technical parameters, deployment parameters, governance parameters, and indirect lessons from history, models, and theory;
* Reviews work aimed at understanding possible solutions to the problem of TAI governance, mapping key actors, their levers of governance of TAI, pathways to influence on these levers’ deployment, policy proposals, and assets;
* Reviews work aimed at prioritization among these solutions, based on different theories of action;

Reading guide:

* For most readers I recommend you identify a theme or area of interest and jump to that section, rather than read from the top.
* This list is particularly useful when you already have a specific research question and want to see what work has been done, and how a particular line of work would fit into the larger landscape. It can also be useful for generating syllabi for reading groups or courses.

---

Contents

[Reading guide        3](#h.xrtx23vqdl6t)

[Aims        3](#h.u6l1st1nf0ji)

[Scope        4](#h.xuvj5it4qnum)

[Problem-focused work: strategic parameters of the AI governance challenge        6](#h.n3epc55q1hdm)

[Technical parameters        7](#h.2yhqi7wcito7)

[TAI technical landscape        7](#h.ugp35xcxsjwm)

[TAI development paths        7](#h.itg757rlafis)

[TAI timelines        11](#h.l2modyukrqpo)

[Outside view analyses of timelines        11](#h.31waf92ca0ob)

[Judgment-based analyses of timelines        14](#h.pf654o723d5a)

[Inside view models on AI timelines        15](#h.irg0n6j6vbhz)

[Methodological debates on AI timelines analysis        16](#h.22t9tea9ool0)

[TAI trajectories and early warning signals        17](#h.gbj1avtwe9ba)

[Threat models for extreme risks from advanced AI        19](#h.g5iwq2y5vrcu)

[Indirect arguments for risks from AI        19](#h.rcbnkjqtsbwd)

[Threat models for direct catastrophe from AI        22](#h.9yg2vksj2ss6)

[Various scenarios for direct catastrophe        23](#h.l8e4aoklk69i)

[Other work: vignettes, surveys, historiography, critiques        27](#h.f2v9q0zhs2s7)

[Threat models for indirect AI contributions to existential risk factors        29](#h.3beeaoi5tqbz)

[Profile of technical alignment problem        30](#h.y462xislojgv)

[Deployment parameters        32](#h.j8xk0v1bun5c)

[Current state of AI development ecosystem        32](#h.h5c3x9ripvbr)

[Future trajectory of TAI development ecosystem        33](#h.sr8g0k15v92c)

[Governance parameters        35](#h.x87a3scvnjaz)

[Current conditions for AI governance efforts today        35](#h.4bmnxfqs4nq9)

[Future trajectory of conditions for AI governance efforts        38](#h.d7eugcfpuy2p)

[Other lenses on the TAI governance problem        40](#h.kboekb3wuc7m)

[Lessons derived from theory        40](#h.e8tmmuugainm)

[Lessons derived from models and wargames        44](#h.bn7u5842nw5h)

[Lessons derived from historical cases        45](#h.ak2kj76t4o81)

[Cases of technology development programs        45](#h.3um6u32wh85n)

[Cases of sudden technology breakthroughs        47](#h.4do8y5touktu)

[Cases of technological proliferation & societal impacts        48](#h.6l2scln4ef7r)

[Cases of key actor perceptions- and regulation of emerging technologies        52](#h.pig15hg80ry2)

[Lessons from ethics and political theory: desiderata for ideal AI governance        0](#h.aft668v9bahv)

[Solution-focused work: key actors, levers, pathways, and proposals        0](#h.83jbkwqh2ym)

[Potential Key Actors shaping TAI        0](#h.a100wrz3ygjn)

[AI developer (lab & tech company) actors        0](#h.f2lv9n7dcnlr)

[AI industry, hardware supply chain, and academic actors        0](#h.qtapto9yloz0)

[State & governmental actors        0](#h.hfo65rb6yjky)

[International organizations        0](#h.z7zm54qlgw0y)

[Public, Civil Society, & media actors        0](#h.sd33lors8sb0)

[Levers of governance        0](#h.5romymfdade3)

[AI developer levers        0](#h.hbekt0bwx3mq)

[AI industry & academia levers        0](#h.6xvv7tw1daho)

[Compute supply chain industry levers        0](#h.58tjevyt25hn)

[Governmental levers        0](#h.o4tazsvok05a)

[General governmental levers        0](#h.g0lb4wu2beai)

[Specific key governments levers        0](#h.qy6l6dxrg6tq)

[Public, civil society & media actor levers        0](#h.tszb1bbymv79)

[International organizations & regime levers        0](#h.7kb28oqqk0m7)

[New or future institutions and levers        0](#h.2w8ytcutjqsr)

[Pathways to influence        0](#h.d3h4oirmaame)

[Direct shaping of TAI systems’ decisions through law        0](#h.nmzeyuawx31)

[Shaping governmental decisions        0](#h.kmgr0zi4c9if)

[Shaping court decisions        0](#h.pa56zz9lpwr7)

[Shaping lab & industry decisions        0](#h.upkrkxy0xgxh)

[Shaping AI research community decisions        0](#h.93apiwjzqgif)

[Shaping international institutions’ decisions        0](#h.d9pd8q9r3cyd)

[Other pathways to shape various actors’ decisions        0](#h.rf37bj6t2eh)

[Existing and Applicable Law        0](#h.xtzugu6j7dxo)

[General regulation        0](#h.tojympm397a)

[US law        0](#h.pcc1og2g183c)

[EU law        0](#h.vno7qwezv9j1)

[International law        0](#h.miloaff1xadg)

[Proposals        0](#h.rdfr7i4xo7m)

[General heuristics for key actors in crafting TAI policies        0](#h.ft3j9r3eemu9)

[TAI Policy Proposals        0](#h.13k9jc79o5ll)

[Prioritization-focused work: comparing theories of action        0](#h.rq37u2khotvo)

[Technical AI alignment theories of impact        0](#h.5xp4e06v71ac)

[AI governance theories of impact        0](#h.fcetqlfaelx8)

[Comparing theories of action        0](#h.n0583g907g2g)

---

Reading guide
=============

This document aims to review, structure, and organize existing work in the field of Transformative AI Governance.

Despite being a fairly young and interdisciplinary field, it offers a wealth of productive work to draw on and is increasingly structured through various research agendas[[3]](#ftnt3) and syllabi.[[4]](#ftnt4) However, while technical alignment research has been mapped in various literature reviews and distillations,[[5]](#ftnt5) few attempts have been made to map research on the governance of transformative AI.[[6]](#ftnt6)

Aims
----

This document aims to provide an overview and taxonomy of work in this field, in order to:

1. Disentangle the field, to promote greater clarity and legibility regarding the range of research, connections between different research streams and directions, and open gaps or underexplored questions.
2. Help newcomers to the field understand the existing literature, pay back some of the field’s ‘research debt’,[[7]](#ftnt7) and reduce duplication of effort or work;
3. Enable clearer conversations between researchers exploring different questions or lines of research, discussing how and where their insights intersect;
4. Enable better comparative analysis between different approaches to transformative AI governance; and
5. Contribute to overall strategic clarity or coherence, improving the quality of interventions.

Scope
-----

What is the scope of this review? As discussed in Foundations Report 1,[[8]](#ftnt8) I define the TAI governance field as “the study and shaping of local and global governance systems—including norms, policies, laws, processes, and institutions—that affect the research, development, deployment, and use of existing and future AI systems in ways that help navigate the transition to a world with transformative AI systems.”

However, my aim in this document is not to engage in restrictive boundary policing of which research constitutes the core TAI governance field. My guiding heuristic here is not whether a given piece of research is directly, explicitly and exclusively focused on certain ‘right’ problems (e.g. extreme risks from transformative AI), nor whether it is motivated by certain normative frameworks (e.g. longtermist philosophy), nor even whether it explicitly uses certain terminology (e.g. ‘Transformative AI’ or ‘AGI’). Rather, the heuristic is simply whether the research helps answer a part of the TAI governance puzzle.

Accordingly, this review aims to cast a fairly broad net, to cover work that:[[9]](#ftnt9)

* Explicitly focuses on the governance of extreme risks from future, potentially transformative AI systems;
* Focuses on the governance of today’s AI systems, where (at least some of) the authors are interested in the implications of the analysis for the governance of future AI systems; and
* Focuses on today’s AI systems or even on non-AI topics (such as historical case studies of technology governance), where the work is not directly motivated by extreme risks but offers lessons that drawn upon by the TAI governance community to inform insights for the governance of future AI systems.

With this in mind, there are some types of work that I mostly omit from this review:

* In-progress, unpublished, or draft work: By and large, this review focuses on public and published analyses.[[10]](#ftnt10) Given that a significant portion of relevant and key work in this field is still unpublished, this means that this review may not capture all research directions in this field.
* Possibly relevant but underexplored work: There is a wide range of work that would be valuable for the TAI governance community but which has not been widely recognized yet, such as certain historical case studies or theoretical lenses.[[11]](#ftnt11)

A project like this will inevitably miss key work, so it should not be considered exhaustive, but rather as a living document that will be occasionally updated as the field develops. To that end, I welcome feedback, suggestions, and criticism.

---

Problem-focused work: strategic parameters of the AI governance challenge
=========================================================================

Most object-level work in the field of TAI governance has sought to disambiguate and reduce uncertainties around relevant strategic parameters.[[12]](#ftnt12)

|  |
| --- |
| TAI strategic parameters can be defined as ‘features of the world, such as the future AI development trajectory, the prevailing deployment landscape, and applicable policy conditions, which significantly determine the strategic nature of the TAI governance challenge.’[[13]](#ftnt13) |

Strategic parameters serve as highly decision-relevant or even crucial considerations, determining which interventions or solutions are appropriate, necessary, viable, or beneficial for addressing the TAI governance challenge. Different views of these parameters constitute underlying cruxes for different theories of actions and approaches.

This review discusses three types of parameters:[[14]](#ftnt14)

* Technical parameters of the TAI challenge (i.e. what are the future technical developments in AI; on what timelines and on what trajectory will progress occur; why or how might such systems pose risks; and how difficult is the alignment challenge);
* Deployment parameters of who is most likely to develop TAI systems and how they are likely to develop and use these (i.e. whose development decisions are to be governed); and
* Governance parameters of how, when, and why governance interventions to shape TAI development and deployment are most likely to be viable, effective, or productive.

Accordingly, research in this sub-field includes:

* Empirical and theoretical work aiming to identify or get better estimates of these parameters, as they apply to TAI.
* Work applying other lenses to the TAI governance problem, drawing on other fields (existing theories, models, historical case studies, political and ethical theory) in order to derive actionable lessons.

Technical parameters
--------------------

An initial body of work focuses on mapping the relevant technical parameters of the challenge for TAI governance. This includes work on a range of topics relating to understanding the future technical landscape, understanding the likelihood of catastrophic risks given various specific threat models, and understanding the profile of the technical alignment problem, and the prospects of it being solved by existing technical alignment research agendas.[[15]](#ftnt15)

### TAI technical landscape

Work to map technical paths includes work to understand the future paths, timelines, and trajectories of TAI development.

#### TAI development paths

One subfield involves research to chart the future technical landscape of TAI.[[16]](#ftnt16) This includes research on a range of domains:

* Exploring distinct possible forms of advanced AI:[[17]](#ftnt17) e.g. ‘human-level AI (HLAI)’,[[18]](#ftnt18) ‘AI+’,[[19]](#ftnt19) Artificial General Intelligence (AGI),[[20]](#ftnt20) superintelligence,[[21]](#ftnt21) CAIS,[[22]](#ftnt22) General-Purpose AI Systems (GPAIS),[[23]](#ftnt23) ‘autonomous machine intelligence’,[[24]](#ftnt24) ‘simulators’,[[25]](#ftnt25) etc. […]
* Analyzing hypothesized paradigms towards achieving advanced AI:[[26]](#ftnt26) ‘scaling’ approaches (e.g. ‘prosaic AGI’[[27]](#ftnt27)); ‘de novo AGI’,[[28]](#ftnt28) AGI from evolutionary algorithms;[[29]](#ftnt29) ‘seed AI’;[[30]](#ftnt30) ‘brain-like AGI’,[[31]](#ftnt31) ‘whole-brain emulation’;[[32]](#ftnt32) ‘digital people’,[[33]](#ftnt33) brain-computer-interfaces; embodiment;[[34]](#ftnt34) [...]
* Analyzing direct pathways to TAI, specifically the question of whether or not current approaches will take us to advanced capabilities;
* Arguments in favour of a direct path: ‘scaling hypothesis’,[[35]](#ftnt35) ‘prosaic AGI’;[[36]](#ftnt36) ‘Human feedback on diverse tasks (HFDT)’;[[37]](#ftnt37) [...]
* Arguments against a direct path, highlighting various limits and barriers; ‘deep limitations’;[[38]](#ftnt38) ‘the limits of machine intelligence’,[[39]](#ftnt39) ‘why AI is harder than we think’;[[40]](#ftnt40) other skeptical arguments;[[41]](#ftnt41) [...]
* Discussion of the possible components of ‘engineering roadmaps’ for AGI-like systems.[[42]](#ftnt42)

#### TAI timelines

A core aim of the field is to chart the timelines for TAI development across the future technical development landscape.[[43]](#ftnt43) This research focuses on various lines of evidence[[44]](#ftnt44) (here listed in order from more abstract to more concrete and empirical; and from relying more on outside views arguments, to relying more on inside view arguments[[45]](#ftnt45));

##### Outside view analyses of timelines

Outside view analyses of AI development timelines, including:

* Estimates based on philosophical arguments & anthropic reasoning:
* Prima facie likelihood that we (of all generations) are the ones to find ourselves living in the ‘most important’ century, one that we can expect to contain things such as transformative technologies;[[46]](#ftnt46)
* Estimates based on extrapolating historical (growth) trends:
* Insights from endogenous growth theory on AI takeoff dynamics;[[47]](#ftnt47)
* Likelihood of explosive economic growth occurring this century, for some reason (plausibly technological, plausibly AI[[48]](#ftnt48)), given analyses of long-run economic history;[[49]](#ftnt49)
* The accelerating historical rate of development in the development of new technologies;[[50]](#ftnt50)
* The historical patterns of barriers to technology development, including delays,[[51]](#ftnt51) as well as instances of deployment/diffusion lags;[[52]](#ftnt52)
* Estimates based on extrapolating from historical trends in efforts dedicated to creating advanced AI:
* external ‘semi-informative priors’ (i.e. only basic information regarding how long people have attempted to build transformative AI, and what resources they have used, and comparing it to how long it has taken other, comparable research fields to achieve their goals, given certain levels of funding and effort);[[53]](#ftnt53)
* arguments extrapolating from ‘significantly increased near-future investments in AI progress’, given that (comparatively) moderate past investments already yielded significant progress;[[54]](#ftnt54)
* Estimates based on meta-induction from the track record of past predictions:
* the general historical track record of past technological predictions, especially those made by futurists;[[55]](#ftnt55) and the frequency of past periods of excessive optimism (hype) or excessive pessimism (counterhype);[[56]](#ftnt56)
* the specific historical track record of past predictions around AI development;[[57]](#ftnt57) and the frequency of past periods excessive optimism (hype) or excessive pessimism (counterhype);[[58]](#ftnt58)

##### Judgment-based analyses of timelines

Judgment-based analyses of timelines, including:

* Estimates based on (specialist) expert opinions:
* expert opinion surveys;[[59]](#ftnt59)
* expert elicitation techniques (e.g. Delphi method)[[60]](#ftnt60)
* Estimates based on (generalist) estimates from information aggregation mechanisms (financial markets; forecaster prediction markets):[[61]](#ftnt61)
* forecasters’ predictions of further AI progress on prediction platforms,[[62]](#ftnt62) or forecasting competitions;[[63]](#ftnt63)
* Current financial markets real interest rates, assuming the efficient market hypothesis, suggesting that markets reject short timelines;[[64]](#ftnt64)

##### Inside view models on AI timelines

Inside view models-based analyses of timelines, including:

* Estimates based on first-principles analysis:
* Comparison with human biological cognition (‘biological anchors’ framework):[[65]](#ftnt65) comparing projected trends in falling costs of training AI models to the expected minimum amount of computation needed to train an AI model as large as the human brain;[[66]](#ftnt66)
* Analysis of performance scaling laws in current large language models,[[67]](#ftnt67) compared against likely barriers in the (future) availability of the data and compute needed to train them;[[68]](#ftnt68)
* Estimates based on direct evaluation:
* debates over the significance and implications of specific ongoing AI breakthroughs for further development;[[69]](#ftnt69)
* operationalizing and measuring the generality of existing AI systems;[[70]](#ftnt70)

##### Methodological debates on AI timelines analysis

Various methodological debates around AI timelines analysis:

* On the potential pitfalls in many of the common methods (forecasting methods,[[71]](#ftnt71) extrapolation, expert predictions[[72]](#ftnt72)) in forecasting AI.
* On the risk of misinterpreting forecasters depending on poor operationalization;[[73]](#ftnt73)
* On the risk of ‘deference cycles’ in debates over AI timelines,[[74]](#ftnt74) because the opinions and analyses of a small number of people end up tacitly informing the evaluations of a wide range of others, in ways that create the impression of many people independently achieving similar conclusions.[[75]](#ftnt75)
* On the (potentially) limited utility of further discourse over/research into AGI timelines: arguments that all low-hanging fruit may already have been plucked;[[76]](#ftnt76) counterarguments that specific timelines remain relevant to prioritization of strategies;[[77]](#ftnt77)

#### TAI trajectories and early warning signals

A third technical subfield aims at charting the trajectories of advanced AI development, especially the potential for rapid and sudden capability gains, and whether there will be advanced warning signs:

* Exploring likely AGI ‘takeoff speeds’:[[78]](#ftnt78)
* From first principles: arguments in favour of ‘fast takeoff’;[[79]](#ftnt79) vs. arguments for slow(er), more continuous development;[[80]](#ftnt80)
* By analogy: exploring historical precedents for sudden disjunctive leaps in technological capabilities.[[81]](#ftnt81)
* Mapping the epistemic texture of the AI development trajectory, in terms of possible advance warning signs of capability breakthroughs,[[82]](#ftnt82) or the lack of any such fire alarms;[[83]](#ftnt83)

### Threat models for extreme risks from advanced AI

A second subcluster of work focuses on understanding the threat models of advanced AI risk,[[84]](#ftnt84) both on the basis of indirect arguments for risks, specific threat models for direct catastrophe or takeover,[[85]](#ftnt85) or specific threat models for indirect risks.

#### Indirect arguments for risks from AI

Analyses that aim to explore indirect arguments (from an outside view) over whether or why we might have grounds to be concerned about advanced AI.

* Analogies with historical cases:
* Cases of intelligence enabling control: emergence of human dominion over the natural world: ‘second species argument’;[[86]](#ftnt86) ‘the human precedent as indirect evidence of danger’.[[87]](#ftnt87)
* Historical cases where actors were able to achieve large shifts in power, despite only wielding relatively minor technological advantages: conquistadors.[[88]](#ftnt88)
* Historical cases of ‘lock-in’ of suboptimal or bad societal trajectories, based on earlier choices and exacerbated by various mechanisms for lock-in; climate change, the agricultural revolution, colonial projects.[[89]](#ftnt89)
* Analogies with known ‘control problems’ observed in other domains:
* Analogies with economics principal-agent problems;[[90]](#ftnt90) -and to constitutional law ‘incomplete contracting’ theorems;[[91]](#ftnt91)
* Analogies to economic systems;[[92]](#ftnt92) to bureaucratic systems and markets, and their accordant failure modes and externalities;[[93]](#ftnt93)
* Analogies to ‘Goodhart’s Law’, where a proxy target metric used to improve a system is used so far that further optimization becomes ineffective or harmful;[[94]](#ftnt94)
* Analogies to the ‘political control problem’--the problem of the alignment and control of powerful social entities (corporations, militaries, political parties) with (the interests of) their societies: a problem that remains somewhat unsolved, with societal solutions relying on patchwork and fallible responses that cannot always prevent misalignment (e.g. corporate malfeasance, military coups, or unaccountable political corruption);[[95]](#ftnt95)
* Analogies with animal behaviour, such as cases of animals responding to incentives in ways that demonstrate specification gaming;[[96]](#ftnt96)
* Illustration with thought experiments and well-established narrative tropes: ‘sorcerer’s apprentice’;[[97]](#ftnt97) ‘King Midas problem’;[[98]](#ftnt98) ‘paperclip maximizer’;[[99]](#ftnt99)
* Theoretical arguments to expect misalignment risks from existing ML architectures:
* Arguments based on the workings of modern deep learning systems;[[100]](#ftnt100)
* Empirical evidence of unsolved alignment failures in existing ML systems, which are expected to persist or scale in more advanced AI systems:
* ‘faulty reward functions in the wild’;[[101]](#ftnt101) ‘specification gaming’;[[102]](#ftnt102) reward model overoptimization;[[103]](#ftnt103)
* ‘instrumental convergence’[[104]](#ftnt104); goal misgeneralization and ‘inner misalignment’ in reinforcement learning,[[105]](#ftnt105)
* language model misalignment,[[106]](#ftnt106) and other unsolved safety problems in modern ML;[[107]](#ftnt107) the harms from increasingly agentic algorithmic systems.[[108]](#ftnt108)
* Arguments from the competitive environment that will shape the evolutionary development of AIs:
* Hendrycks’s ‘Natural Selection Favors AIs over Humans’, suggesting that competitive pressures amongst developers may lead the most successful AI agents to likely have (or be given) undesirable traits, which creates risks.[[109]](#ftnt109)

#### Threat models for direct catastrophe from AI

Work focused at understanding direct existential threat models.[[110]](#ftnt110) This includes:

* Other overviews and taxonomies of different accounts of AI risk: Barrett & Baum’s ‘model of pathways to risk’;[[111]](#ftnt111) Clarke et al.’s Modelling Transformative AI Risks (MTAIR);[[112]](#ftnt112) Clarke & Martin on Distinguishing AI takeover scenarios,[[113]](#ftnt113) Clarke & Martin’s ‘Investigating AI Takeover Scenarios’;[[114]](#ftnt114) Clarke’s Classifying Sources of AI X-risk,[[115]](#ftnt115) Vold & Harris ‘How Does Artificial Intelligence Pose an Existential Risk?’;[[116]](#ftnt116) Ngo ‘disentangling arguments for the importance of AI safety’;[[117]](#ftnt117) Grace’s overview of arguments for existential risk from AI;[[118]](#ftnt118) Nanda’s ‘threat models’,[[119]](#ftnt119) Kenton et al.;[[120]](#ftnt120) [...]

##### Various scenarios for direct catastrophe

Other lines of work have moved from providing indirect arguments of risk, to instead sketching specific scenarios in- and through which advanced AI systems could directly inflict existential catastrophe.

* Scenario: existential disaster because of misaligned superintelligence or power-seeking AI
* Older accounts: Yudkowsky 2009;[[121]](#ftnt121) Bostrom 2014;[[122]](#ftnt122) Sotala 2018;[[123]](#ftnt123) Sotala and Yampolskiy 2017,[[124]](#ftnt124) Alexander 2016;[[125]](#ftnt125)
* Newer accounts: Cotra & Karnofsky on AI takeover analysis;[[126]](#ftnt126) Christiano’s account of ‘What Failure Looks Like, (Part 2)’;[[127]](#ftnt127) Carlsmith on existential risks from power-seeking AI,[[128]](#ftnt128) Ngo on AGI Safety From First Principles;[[129]](#ftnt129) AGI Ruin: a list of lethalities;[[130]](#ftnt130) ‘Minimal accounts’ of AI takeover scenarios;[[131]](#ftnt131)
* Skeptical accounts: critiques of AI takeover scenarios;[[132]](#ftnt132)
* Scenario: dystopian trajectory lock-in because of gradual, irretrievable ceding of human power over the future to AI systems
* Christiano’s account of ‘What Failure Looks Like, (1)’;[[133]](#ftnt133)
* Scenario: extreme ‘suffering risks’ because of misaligned system
* Various accounts of ‘worst-case AI safety’;[[134]](#ftnt134)
* Potential for a ‘suffering explosion’ experienced by AI systems.[[135]](#ftnt135)
* Scenario: existential disaster because of conflict between AI systems, multi-system interactions
* Work on ‘cooperation failure’[[136]](#ftnt136) and ‘Multipolar failure’[[137]](#ftnt137)
* Scenario: dystopian trajectory lock-in because of misuse of TAI to establish and/or maintain totalitarian regimes;
* Use of advaned AI to establish robust totalitarianism,[[138]](#ftnt138) or other lock-in of the future;[[139]](#ftnt139)
* Scenario: failures in- or misuse of intermediary (non-AGI) AI systems, resulting in catastrophe
* deployment of ‘prepotent’ AI systems capable of outperforming human collective efforts on various dimensions;[[140]](#ftnt140)
* militarization of AI enabling in massive attacks using swarms of lethal autonomous weapons systems;[[141]](#ftnt141)
* use of machine learning systems in nuclear command and control, resulting in (intentional or unintentional) nuclear escalation;[[142]](#ftnt142)
* Conventional AI-enabled systems (e.g. autonomous ships) being deployed in ways that result in inadvertent escalation to the nuclear level;[[143]](#ftnt143)
* nuclear arsenals as an arsenal ‘overhang’ for advanced AI;[[144]](#ftnt144)
* use of AI to accelerate research into catastrophically dangerous weapons (e.g. bioweapons).[[145]](#ftnt145)

##### Other work: vignettes, surveys, historiography, critiques

* Work to sketch vignettes reflecting on potential threat models:
* AI Impacts’ AI Vignettes project;[[146]](#ftnt146)
* FLI Worldbuilding competition;[[147]](#ftnt147)
* Wargaming exercises;[[148]](#ftnt148)
* Other vignettes: Nintil;[[149]](#ftnt149) Hilton;[[150]](#ftnt150) Gwern;[[151]](#ftnt151)
* Surveys of how researchers in the community rate different AI existential risk scenarios:
* Clarke et al.;[[152]](#ftnt152) Bensinger;[[153]](#ftnt153) Ross;[[154]](#ftnt154)
* Historiographic accounts of changes in AI risk arguments and debates over time:
* Overall history of concerns of AI risk (1950s to present);[[155]](#ftnt155)
* Early history of the Rationalist and AI risk communities (1990s-2010);[[156]](#ftnt156)
* Recent shifts in arguments (e.g. 2014-present);[[157]](#ftnt157)
* Critical investigations and counterarguments to the case for extreme AI risks:
* Object-level counterarguments: Garfinkel’s ‘how sure are we about this AI stuff?[[158]](#ftnt158); Grace’s ‘counterarguments to the basic AI X-Risk Case’;[[159]](#ftnt159) ‘AI will change the world, but won’t take it over by playing ‘3-dimensional chess’;[[160]](#ftnt160)
* Epistemic arguments, arguments about community dynamics and argument selection effects; ‘Against a General Factor of Doom’;[[161]](#ftnt161) ‘my highly personal skepticism’;[[162]](#ftnt162) ‘but have they engaged with the arguments?’[[163]](#ftnt163)

#### Threat models for indirect AI contributions to existential risk factors

Work focused at understanding indirect ways in which AI could contribute to existential threats, shaping societal ‘turbulence’[[164]](#ftnt164) and other existential risk factors.[[165]](#ftnt165) This covers various long-term impacts on societal parameters such as science, cooperation, power, epistemics, and values;[[166]](#ftnt166)

* Destabilising political impacts from AI systems in areas such as:
* domestic politics (e.g. polarization, legitimacy of elections);
* international political economy;
* international security;[[167]](#ftnt167)
* speed and character of war
* technology races and international stability
* balance of power
* Hazardous malicious uses;[[168]](#ftnt168)
* Impacts on ‘epistemic security’ & information environment;[[169]](#ftnt169)
* Erosion of international law and global governance architectures;[[170]](#ftnt170)
* Other diffuse societal harms.[[171]](#ftnt171)

### Profile of technical alignment problem

* Work mapping different geographical or institutional hubs active on AI alignment:
* Overview of AI safety community & problem;[[172]](#ftnt172) databases of active research institutions,[[173]](#ftnt173) and of research;[[174]](#ftnt174)
* Work mapping current technical alignment approaches: (selection)
* Various summaries of alignment approaches: Everitt et al.[[175]](#ftnt175); Christiano,[[176]](#ftnt176) Neel Nanda,[[177]](#ftnt177) Victoria Krakovna,[[178]](#ftnt178) Kirchner et al.,[[179]](#ftnt179) Rohin Shah,[[180]](#ftnt180) Even R. Murphy;[[181]](#ftnt181) Evan Hubinger,[[182]](#ftnt182) etc. (disclaimer: this is a limited overview).
* Work aiming to assess the (relative) efficacy or promise of different approaches to alignment, insofar possible:[[183]](#ftnt183) Cotra,[[184]](#ftnt184) Soares,[[185]](#ftnt185) Leike.[[186]](#ftnt186) (...)
* Mapping the relative contributions to technical AI safety by different communities:
* Leech on the relative contributions by academic work on AI safety;[[187]](#ftnt187)
* Shah on the chance that AI safety gets ‘solved by default’;[[188]](#ftnt188)

Deployment parameters
---------------------

Another major part of the field aims to understand the parameters of the TAI deployment landscape, by mapping the size and configuration of the ‘gameboard’ of relevant TAI developers–the actors whose (ability to take) key decisions (e.g. around whether or how to deploy particular TAI systems, and how much to invest in alignment research, etc.) may be key in determining risks and outcomes from TAI.

### Current state of AI development ecosystem

Mapping the current state and disposition of the AI development ecosystem (the ‘gameboard’):

* Mapping the current size, activity, and productivity of the AI research field;[[189]](#ftnt189)
* Mapping the current global distribution of active AGI programs,[[190]](#ftnt190) including across key players such as the US or China;[[191]](#ftnt191)
* Mapping the current distribution of relevant inputs in AI development, such as the distribution of computation,[[192]](#ftnt192) semiconductor manufacturing,[[193]](#ftnt193) AI talent;[[194]](#ftnt194) Open-Source Machine Learning Software,[[195]](#ftnt195) etc.
* Mapping the current shape of the AI supply chain;[[196]](#ftnt196)
* Mapping developments in adjacent, potentially relevant technologies, such as cryptography,[[197]](#ftnt197) nanotechnology’;[[198]](#ftnt198) etc.

### Future trajectory of TAI development ecosystem

Mapping the likely future trajectories of the TAI development ecosystem:

* Mapping & forecasting trends in relevant inputs in AI, such as various metrics in machine learning research:[[199]](#ftnt199)
* Trends in compute inputs scaling;[[200]](#ftnt200) and in the training costs and GPU price-performance of machine learning systems over time;[[201]](#ftnt201)
* Trends in dataset scaling, and potential ceilings;[[202]](#ftnt202)
* Trends in algorithmic progress, including their effect on the ability to leverage other inputs, e.g. affecting the relative importance of CPUs versus specialized hardware;[[203]](#ftnt203)
* Mapping & forecasting trends in input criticality for AI, such as trends in data efficiency,[[204]](#ftnt204) and the degree to which data becomes the operative constraint on language model performance;[[205]](#ftnt205)
* Mapping & forecasting dominant actors in the future AI ecosystem, in terms of:
* different actors’ control of- & access to key inputs, and/or -chokepoints;
* shape of the AI supply chain (e.g. level of integration; monopoly structure);[[206]](#ftnt206)
* shape of AI deployment landscape (e.g. dominance of key operators of generative models vs. copycat models);
* Anticipating the likely behaviour or attitude of key TAI actors, with regards to their caution about- and investment in safety research: expecting AI companies to ‘race forward’ and dedicate ‘naive safety effort’;[[207]](#ftnt207)

Governance parameters
---------------------

Work on governance parameters aims to map (1) how AI systems are likely to be governed, as well as (2) the conditions for developing and implementing productive governance interventions on TAI risk, both today and in the future.

### Current conditions for AI governance efforts today

Mapping the prevailing (political) conditions that affect (T)AI governance efforts initiated today, such as:

* current perceptions of AI among different relevant actors;
* public perceptions of the future of AI;[[208]](#ftnt208) -of AI’s societal impacts;[[209]](#ftnt209) - of the need for AI regulation;[[210]](#ftnt210) - of the rights or standing of AI entities;[[211]](#ftnt211)
* public trust in different actors to responsibly develop AI;[[212]](#ftnt212)
* AI practitioners trust in different actors to responsibly develop AI;[[213]](#ftnt213)
* policymaker perceptions of AI;[[214]](#ftnt214) the prominence of different memes, rhetorical frames, or narratives around AI;[[215]](#ftnt215)
* the prevailing (i.e. default, ‘business-as-usual’) governance landscape of regulations that will be applied to AI in the near term, which will directly affect the development landscape for advanced AI, and indirectly bracket the space for any new (TAI-specific) governance proposals proposed later:[[216]](#ftnt216)
* Existing industry norms and practices applied to AI: in areas such as release practices around generative AI systems;[[217]](#ftnt217)
* Existing laws & governance regimes which may be extended to or affect AI development: e.g. anticompetition law;[[218]](#ftnt218) national and international standards;[[219]](#ftnt219) international law norms, treaties and regimes;[[220]](#ftnt220) existing global governance institutions;[[221]](#ftnt221) etc.
* AI-specific governance regimes currently under development (see also [Existi](#h.xtzugu6j7dxo)[ng and Applicable Norms, Principles and Laws](#h.xtzugu6j7dxo)); formulation of US AI policy agenda;[[222]](#ftnt222) [...]
* Identifying barriers to effective AI governance;
* definitional complexities around AI;[[223]](#ftnt223)
* complexity of analysing the best governance responses;[[224]](#ftnt224)
* difficulty of building consensus, given geopolitical stakes and tensions;[[225]](#ftnt225)
* Identifying the impacts of current and ongoing events on the future political trajectory for TAI governance:
* 2022’s US AI chip export controls;[[226]](#ftnt226) the release of DALLE-2, and the proliferation of open-source image generation models;[[227]](#ftnt227) 2023 Google AI strategy shifts in reaction to ChatGPT,[[228]](#ftnt228) etc.

### Future trajectory of conditions for AI governance efforts

Mapping the future conditions that will apply and affect TAI governance initiatives, closer to TAI development:

* Predicting shifts in future perceptions of AI among relevant actors, given:
* The spread of ongoing academic conversations concerned about TAI risk;[[229]](#ftnt229)
* The effects of ‘warning shots’;[[230]](#ftnt230)
* The effect of motivated misinformation or politicized AI risk skepticism;[[231]](#ftnt231)
* Predicting the impact of future technologies on governance, and the ways these could shift the possibility frontier of what kind of regimes will be politically viable and enforceable;
* Effects of AI on general cooperative capabilities;[[232]](#ftnt232)
* Effects of AI on international law creation & enforcement;[[233]](#ftnt233)
* Effects of AI on arms control monitoring;[[234]](#ftnt234)
* [...]

Note, the above is not exhaustive of all potentially relevant strategic parameters.[[235]](#ftnt235)

Other lenses on the TAI governance problem
------------------------------------------

Other work aims to derive key strategic lessons for TAI governance, not by aiming to empirically map or estimate first-order facts about the key (technical, deployment or governance) strategic parameters, but rather by drawing indirect (empirical, strategic, and/or normative) lessons from abstract models, historical cases, and/or political theory.

### Lessons derived from theory

Work characterising the features of TAI technology and of its governance challenge, drawing on existing literatures or bodies of theory:

Mapping clusters and taxonomies of AI’s governance problems

* AI creating distinct types of risk deriving from (1) accidents, (2) misuse, (3) structure;[[236]](#ftnt236)
* AI creating distinct problem logics across domains: (1) ethical challenges, (2) safety risks, (3) security threats, (4) structural shifts, (5) common goods, (6) governance disruption;[[237]](#ftnt237)
* AI driving four risk clusters: (1) inequality, turbulence, authoritarianism, (2) great-power war, (3) the problems of control, alignment and political order, (4) value erosion from competition;[[238]](#ftnt238)

Mapping the political features of TAI technology:

* AI as general-purpose technology, highlighting radical impacts on economic growth, disruption to existing socio-political relations, and potential for backlash and social conflict;[[239]](#ftnt239)
* AI as industry-configured general-purpose tech (low fixed costs & private sector dominance), highlighting challenges of rapid proliferation (compared to ‘prestige-’, ‘public’, or ‘strategic’ technologies);[[240]](#ftnt240)
* AI as information technology, highlighting challenges of increasing returns to scale driving greater income inequality; impacts on broad collective identities as well as community fragmentation, increased centralization of (cybernetic) control;[[241]](#ftnt241)
* AI as intelligence technology, highlighting challenges of bias, alignment and control of the principal over the agent;[[242]](#ftnt242)
* AI as regulation-resistant technology, rendering coordinated global regulation difficult;[[243]](#ftnt243)

Mapping the structural features of the TAI governance challenge:

* As global public good;[[244]](#ftnt244)
* As collective action problem;[[245]](#ftnt245)
* As a challenge akin to ‘racing through a minefield’[[246]](#ftnt246)
* As arms race;[[247]](#ftnt247) – but see also critiques of the arms race framing, on definitional grounds,[[248]](#ftnt248) on empirical grounds,[[249]](#ftnt249) or on grounds of rhetorical- or framing risks.[[250]](#ftnt250)
* As technology race;[[251]](#ftnt251) whether motivated by security concerns, or by prestige motivations;[[252]](#ftnt252)
* As wicked problem;[[253]](#ftnt253)
* As political economy problem;[[254]](#ftnt254)
* [...]

Identifying design considerations for international institutions & regimes;

* From general theory on the rational design of international institutions;[[255]](#ftnt255)
* From work on the orchestration and organization of regime complexes of many institutions, norms, conventions, etc.[[256]](#ftnt256)
* [...]

### Lessons derived from models and wargames

Work to derive or construct abstract models for AI governance, to derive lessons for on AI’s intermediate proliferation and societal impacts. Including models of:

* International strategic dynamics in risky technology races;[[257]](#ftnt257) theoretical models of the role of information sharing,[[258]](#ftnt258) agreement or incentive modelling;[[259]](#ftnt259)
* AI competition, and whether and how AI safety insights will be applied, under different AI safety-performance tradeoffs;[[260]](#ftnt260) collaboration on safety as a social dilemma;[[261]](#ftnt261)
* The offense-defense balance of increasing investments in technologies;[[262]](#ftnt262)
* The offense-defense balance of scientific knowledge in AI with potential for misuse;[[263]](#ftnt263)
* Lessons from the ‘epistemic communities’ lens, on how coordinated expert networks can shape policy;[[264]](#ftnt264)
* Lessons from wargames and role-playing exercises;[[265]](#ftnt265)
* [...]

### Lessons derived from historical cases

Work to identify and study relevant historical precedents, analogies or cases, and to derive lessons for (AI) governance.[[266]](#ftnt266) This includes both studies where historical cases have been directly applied to TAI governance, as well as studies where the link has not been drawn, but which might nevertheless offer productive insights for the governance of advanced AI.

#### Cases of technology development programs

Historical cases of actors pursuing large-scale scientific or technological programs for different rationales:

* Development of major transformative technologies during wartime: development of the atom bomb;[[267]](#ftnt267)
* Pursuit of scientific and strategically valuable megaprojects: Apollo Program & Manhattan Project;[[268]](#ftnt268)
* Technologies pursued for prestige reasons: the US/USSR space race;[[269]](#ftnt269) the French nuclear program;[[270]](#ftnt270)
* Risk of races being started by possibly incorrect perceptions that a rival is actively pursuing a technology: the Manhattan Project (1939-1945), spurred by the Einstein Letter; the ‘missile gap’ project to build up a US ICBM capability (1957-1962);[[271]](#ftnt271)
* Different potential strategies for pursuing advanced strategic technologies: the distinct nuclear proliferation strategies (‘hedging, sprinting, sheltered pursuit, hiding’) taken by different countries in pursuing nuclear weapons.[[272]](#ftnt272)

Historical cases of joint or coalitional global collaboration on large-scale scientific programs:

* Cases where shocks or shifts in interests enabled the establishment of ‘big science’ collaborations: CERN, ITER, International Space Station; Human Genome Project;[[273]](#ftnt273) attempted collaborations on Apollo-Soyuz between the US and Soviet space programs;[[274]](#ftnt274)

Historical cases of excessive ‘hype’ leading to premature regulatory attention:

* Cases of premature legal focus on technologies that eventually (for various reasons) took much longer to develop or implement than had been anticipated: Weather modification technology,[[275]](#ftnt275) deep seabed mining;[[276]](#ftnt276) self-driving cars;[[277]](#ftnt277) virtual and augmented reality;[[278]](#ftnt278) other technologies charted under the Gartner Hype Cycle reports;[[279]](#ftnt279)

#### Cases of sudden technology breakthroughs

Historical cases of rapid, historically discontinuous breakthroughs in technological performance on key metrics:

* Cases of ‘large robust discontinuities’ in historical technology performance trends:[[280]](#ftnt280)
* the Pyramid of Djoser (2650 BC—structure height trends),
* the SS Great Eastern (1858—ship size trends),
* the first and second telegraph (1858, 1866—speed of sending a message across the Atlantic Ocean),
* the first non-stop transatlantic flight (1919—speed of passenger or military payload travel),
* first nuclear weapons (1945—relative effectiveness of explosives),
* first ICBM (1958—average speed of military payload),
* the discovery of YBa2Cu3O7 as a superconductor (1987—warmest temperature of superconduction).[[281]](#ftnt281)

Historical cases of sudden, unexpected technology breakthroughs that were held to be unlikely or impossible even shortly before they happened:

* Invention of flight;[[282]](#ftnt282)
* Invention of penicillin, nuclear fission, nuclear bombs, or space flight;[[283]](#ftnt283)
* Invention of internet hyperlinks and effective internet search (debatable);[[284]](#ftnt284)

#### Cases of technological proliferation & societal impacts

Historical cases of technological proliferation and penetration:[[285]](#ftnt285)

* Patterns in the development, dissemination and impacts of major technological advancements: flight, the telegraph, nuclear weapons, the laser, penicillin, and the transistor, and others;[[286]](#ftnt286)
* Proliferation and penetration rates of other technologies, in terms of time between invention and widespread use: steam engine (80 years), electricity (40 years), IT (20 years);[[287]](#ftnt287) mobile phones.
* Role of espionage in facilitating critical technology diffusion: early nuclear proliferation;[[288]](#ftnt288) numerous information leaks in modern IT systems.[[289]](#ftnt289)
* Supply chain evolution of previous general-purpose technologies: studies of how railroads, electricity, and cloud computing industries, where supply chains were initially vertically integrated but then evolved into a fully disintegrated natural monopoly structure, with a handful of primary ‘upstream’ firms selling services to many ‘downstream’ application sectors;[[290]](#ftnt290)

Historical cases of large-scale societal impacts from other technologies:[[291]](#ftnt291)

* Impacts of previous narrowly transformative technologies: impact of nuclear weapons on warfare; electrification of militaries as driver of ‘general-purpose military transformation’.[[292]](#ftnt292)
* Impacts of previous general-purpose technologies: general electrification;[[293]](#ftnt293) printing, steam engines, rail transport, motor vehicles, aviation, computing;[[294]](#ftnt294)
* Impacts of previous ‘revolutionary’ / ‘radically transformative’[[295]](#ftnt295) technologies: domesticated crops and the steam engine;[[296]](#ftnt296)
* Impacts of previous information technologies: speech and culture, writing, and the printing press; digital services; communications technologies[[297]](#ftnt297)
* Impacts of previous intelligence technologies: “price mechanisms in a free market, language, bureaucracy, peer review in science, evolved institutions like the justice system and law;[[298]](#ftnt298)
* Impacts of previous labor-substitution technologies to understanding the possible societal impacts of large language models;[[299]](#ftnt299)

Historical precedents for particular types of dangers or threat models from technologies:

* Cases of human-machine interface risks and failures: various ‘normal accidents’ in diverse industries and domains[[300]](#ftnt300)
* Cases of misuse risk: the proliferation of easily available hacking tools, such as the ‘Blackshades Remote Access Tool’;[[301]](#ftnt301)
* counter-example of non-use of possible Decisive Strategic Advantage: brief US nuclear monopoly;[[302]](#ftnt302)
* Cases of structural risks:
* Role of technologies in lowering the threshold for war initiation: the alleged role of railways in inducing swift, all-or-none military mobilization schedules and precipitating escalation to World War I;[[303]](#ftnt303)
* Cases of technologically-induced value shifts & -erosion:
* Shared values eroded by pressures of global economic competition: ”sustainability, decentralized technological development, privacy, and equality”;[[304]](#ftnt304)
* Technological progress biasing the development of states towards welfare-degrading (inegalitarian and autocratic) forms: agriculture, bronzeworking, chariots, cavalry;[[305]](#ftnt305)
* Technological progress biasing the development of states towards welfare-promoting forms: ironworking, ramming warships; industrial revolution;[[306]](#ftnt306)
* Technological progress leading to gradual shifts in societal values:
* Changes in the prevailing technology of energy capture driving changes in societal views on violence, equality and fairness;[[307]](#ftnt307)
* demise of duelling and honour culture after (low-skill) pistols replaced (high-skill) swords; changes in sexual morality after the appearance of contraceptive technology; changes in attitudes towards farm animals after the rise of meat replacements; the rise of the plough as driver of diverging gender norms;[[308]](#ftnt308)

#### Cases of key actor perceptions- and regulation of emerging technologies

Historical precedents for how key actors are likely to view, treat, or regulate AI;

* The relative roles of various US actors in shaping the development of past strategic general-purpose technologies: biotech, aerospace tech, and cryptography;[[309]](#ftnt309)
* US government policy towards strategic assets: oil;[[310]](#ftnt310) early development of US nuclear power regulation;[[311]](#ftnt311)
* The historical use of US antitrust law motivated by national security considerations; various cases over the last century;[[312]](#ftnt312)
* Early regulation of an emerging general-purpose technology: electricity regulation in the US;[[313]](#ftnt313)
* Public and policy reactions to ‘warning shots’: Sputnik launch; Hiroshima bombing;[[314]](#ftnt314)
* Public and policy reactions to publicly visible accidents: Three Mile Island meltdown;[[315]](#ftnt315) COVID-19 pandemic.[[316]](#ftnt316)
* Regulatory backlash and path dependency: case of GMOs regulations in US vs. EU;[[317]](#ftnt317)
* Indeliberate factors contributing to technological delay or restraint; various examples of technologies being abandoned;[[318]](#ftnt318)
* ‘Regulatory capture’ and/or influence of industry actors on tech policy; role of US military industrial complex in perpetuating the ‘bomber gap’ and ‘missile gap’ myths[[319]](#ftnt319) undue corporate influence in the World Health Organisation during the 2009 H1N1 pandemic;[[320]](#ftnt320)
* Cases of national norm ‘antipreneurship’, aiming to preserve the prevailing global normative status quo at the global level, against proposals for new regulation or norm-setting: US resistance to proposed global restraints on space weapons, between 2000-present, utilizing a range of diplomatic strategies and tactics to preserve a permissive international legal framework governing outer space;[[321]](#ftnt321)

Historical precedents for effects of new technology on governance tools:

* Technological changes disrupting or eroding the legal integrity of earlier (treaty) regimes: submarine warfare; [...][[322]](#ftnt322)
* Technologies strengthening international law: satellites, communications technology;[[323]](#ftnt323)

Historical precedents for efficacy of various governance mechanisms at shaping technology responsibly:

* Scientific secrecy around early development of powerful new technologies: nuclear weapons;[[324]](#ftnt324) early computers (Collossus and ENIAC).[[325]](#ftnt325)
* The successes or failures in the oversight of various safety-critical technologies: both failures (1984 Bhopal industrial chemical leak; US space shuttle Challenger explosion; Chernobyl accident), as well as successes of ‘high reliablity organisations’ (e.g. aircraft carriers)[[326]](#ftnt326)
* Regulateability of different types of digital technologies; relative successes and failures of global regulation of different digital technologies that are (1) centralized and clearly material (e.g. submarine cables), (2) decentralized and clearly material (e.g. Smart speakers); (3) centralized and seemingly immaterial (e.g. search engines), and (4) decentralized and seemingly immaterial (e.g. Bitcoin protocol);[[327]](#ftnt327)
* CBM measures to stabilize relations: 1972 Incidents at Sea Agreement;[[328]](#ftnt328) 12th-19th century development of Maritime Prize Law;[[329]](#ftnt329)
* International access & benefit distribution mechanisms: IAEA;[[330]](#ftnt330)
* Monitoring & Verification mechanisms in arms control regimes; M&V implementation across three types of nuclear arms control treaties: nonproliferation treaties; U.S.-U.S.S.R./Russia arms limitation treaties; and nuclear test bans;[[331]](#ftnt331)
* Scientific community (temporary) moratoria on research: Asilomar Conference;[[332]](#ftnt332) H5N1 gain-of-function debate;[[333]](#ftnt333)
* Cases of scientific and industry self-regulation: Asilomar Conference; Second International Conference on Synthetic Biology; 2004-2007 failed efforts to develop guidelines for nanoparticles;[[334]](#ftnt334)
* Cases where treaty comittments, institutional infighting or bureaucratic politics contributed to technological restraint: a range of cases resulting in cancellation of weapon systems development, including: nuclear-ramjet powered cruise missiles; ‘continent killer’ nuclear warheads; nuclear-powered aircraft, the majority of ~31-38 nuclear weapons programs undertaken; ‘death dust’ radiological weapons; various types of anti-ballistic-missile defense, and many others.[[335]](#ftnt335)
* International institutional design lessons from successes and failures in other areas: global governance successes and failures in the regime complexes for environment, security, and/or trade issues;[[336]](#ftnt336)
* Use of soft law governance for shaping emerging technologies: historical governance of Environmental Technologies; Nanotechnology; Information and Communication Technologies (ICT); and the Life Sciences.[[337]](#ftnt337)
* Participatory mechanisms: use of scenario methods & risk assessments in climate research;[[338]](#ftnt338)

Historical precedents for the efficacy of various pathways to influencing key actors to deploy regulatory levers:

* Historical ‘prescient actions’ in urging early action against risky new technologies: Leo Szilard’s warning of the dangers of nuclear weapons;[[339]](#ftnt339) Alexander Fleming’s 1945 warning of the risk of antibiotic resistance; [...] [[340]](#ftnt340)
* Historical cases of governmental reaction and response to new risks as they emerge: 1973 Oil Crisis, the 1929-1933 Great Depression,[[341]](#ftnt341) 2007-2009 financial crisis;[[342]](#ftnt342) the COVID 19 Pandemic[[343]](#ftnt343)
* How effectively other global risks motivated action in response, how cultural and intellectual orientations influence their perceptions: biotechnology, nuclear weapons, global warming, and asteroid collission;[[344]](#ftnt344)
* The impact of cultural media (film, etc) on priming policymakers to risks:[[345]](#ftnt345) the role of The Day After in motivating efforts towards nuclear arms control;[[346]](#ftnt346) of the movies Deep Impact and Armageddon in shaping perceptions of the importance of asteroid defense;[[347]](#ftnt347) of Ghost Fleet (2015) in shaping Pentagon perceptions of the importance of emerging technologies to war;[[348]](#ftnt348) of Contagion in priming early UK policy responses to COVID-19;[[349]](#ftnt349)
* The role of epistemic communities in achieving international regulation: anti-ballistic missile systems;[[350]](#ftnt350) nuclear weapons;[[351]](#ftnt351) other arms control agreements;[[352]](#ftnt352)
* Attempted efforts towards international control of new technology: early momentum but ultimate failure of Baruch Plan for world control of nuclear weapons;[[353]](#ftnt353) failure of world control of aviation in 1920s;[[354]](#ftnt354)
* Policy responses to past scientific breakthroughs, and the role of geopolitics vs. expert engagement: the 1967 UN OUter Space Treaty; the UK’s Warnock Committee and the Human Embryology Act 1990; the establishment of the Internet Corporation for assigned Names and Numbers (ICANN), and the European ban on GMO crops;[[355]](#ftnt355)
* The role of activism in spurring nuclear nonproliferation agreements,[[356]](#ftnt356) and nuclear test bans;
* The role of philanthropy and scientists in fostering Track-II diplomacy: Pugwash conferences;[[357]](#ftnt357)
* [...]

### Lessons from ethics and political theory: desiderata for ideal AI governance

Mapping the space of desiderata for ‘ideal AI governance’;[[358]](#ftnt358)

* Mapping broad normative desiderata for good governance regimes for advanced AI;[[359]](#ftnt359)
* Understanding how to weigh different good outcomes post-TAI-deployment;[[360]](#ftnt360)
* Understanding the different functional goals and tradeoffs in good international institutional design;[[361]](#ftnt361)
* […]

---

Solution-focused work: key actors, levers, pathways, and proposals
==================================================================

Strategic clarity requires an understanding not just of the landscape of the problem, but also the full range of possible levers that could be used in response. Critically, this is not just about speculating about what governance tools we may want to put in place for future transformative AI systems mid-transition (after they have arrived). Rather, there might be actions we could take in the ‘pre-emergence’ stage, to adequately prepare ourselves.[[362]](#ftnt362)

Within the field, there has been extensive work on options and areas of intervention. Yet there is no clear, integrated map of the Transformative AI Governance landscape, and its gaps. Sam Clarke proposes that there are different ways of carving up the landscape, such as based on different types of interventions, different geographic hubs, or Theories of Victory.[[363]](#ftnt363) To extend this, I suggest we can segment the TAI Governance solution space along work which aims to identify and understand:[[364]](#ftnt364)

* Key actors that will likely (be in a strong position to) shape TAI;
* Levers of influence (by which these actors might shape TAI);
* Pathways towards influencing these actors to deploy their levers well;[[365]](#ftnt365)
* Concrete proposals for policy products or broad heuristics.[[366]](#ftnt366)

Potential Key Actors shaping TAI
--------------------------------

I.e. whose decisions might especially affect the development and deployment of TAI, directly or indirectly, such that these decisions should be shaped to be as beneficial as possible?

|  |
| --- |
| Key actors can be defined as ‘actors whose key decisions will have significant impact on shaping the outcomes from TAI, either directly (first-order), or by strongly affecting such decisions made by other actors (second-order).’[[367]](#ftnt367)Key decisions can be further defined as ‘a choice or series of choices by a key actor to use its levers of governance, in ways that directly affect beneficial TAI outcomes, and which are hard to reverse.’[[368]](#ftnt368) |

Some work in this space explores the relative importance of (the decisions of) different types of key actors:

* The roles of state vs. firms, vs. AI researchers in shaping AI policy;[[369]](#ftnt369)
* Role of ‘epistemic communities’ of scientific experts;[[370]](#ftnt370) especially members of the AI research community;[[371]](#ftnt371)
* The role of different potentially relevant stakeholders for responsible AI systems across its development chain, from individual stakeholders, to organisational stakeholders, to national/international stakeholders;[[372]](#ftnt372)
* The relative role of expert advice vs. public pressure in shaping policymakers’ approach to AI;[[373]](#ftnt373)
* Role of different actors in and around the corporation, in shaping lab policy;[[374]](#ftnt374)
* Within the lab: senior management, shareholders, AI lab employees and activists,[[375]](#ftnt375) …
* Outside the lab: corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, governments;[[376]](#ftnt376)

Other work focuses more specifically on mapping particular key actors whose decisions may be particularly important in shaping TAI outcomes, depending on one’s view of strategic parameters.

The following list aims at being relatively comprehensive: however, the (absolute and relative) importance of any of these actors obviously differs hugely between worldviews and approaches.

### AI developer (lab & tech company) actors

Leading AI firms pursuing AGI:

* OpenAI, DeepMind, Anthropic, Aleph Alpha, Adept, Cohere, Inflection, Keen.[[377]](#ftnt377)
* Known Chinese institutions researching ‘general AI’;[[378]](#ftnt378)

Large tech companies[[379]](#ftnt379) that may take an increasingly significant role in AGI research:

* Microsoft
* Google
* Facebook
* Amazon
* […]

Future AGI labs, currently not known but to be established/achieve prominence;

* "Magma"[[380]](#ftnt380)

### AI industry, hardware supply chain, and academic actors

Industry bodies

* Partnership on AI
* ML Commons[[381]](#ftnt381)
* IEEE (Institute of Electrical and Electronics Engineers) + IEEE-SA (standards body)
* ISO (and IEC)
* US standard-setting organisations:
* NIST
* European Standards Organizations (ESOs) (tasked with setting standards for the EU AI Act):
* the European Committee for Standardisation (CEN),
* European Committee for Electrotechnical Standardisation (CENELEC),
* European Telecommunications Standards Institute (ETSI).[[382]](#ftnt382)
* [...]

Hardware supply chain industry[[383]](#ftnt383)

* Providers of optical components to photolithography machines:
* Carl Zeiss AG [Germany] - key ASML supplier of optical lenses;[[384]](#ftnt384)
* Producers of extreme ultraviolet (EUV) photolithography machines:
* ASML [The Netherlands][[385]](#ftnt385)
* Photoresist processing providers:
* Asahi Kasei and Tokyo Ohka Kogyo Co. [Japan];[[386]](#ftnt386)
* Advanced chip manufacturing:
* TMSC [Taiwan],
* Intel [US],
* Samsung [South Korea]
* Semiconductor intellectual property owners and chip designers:
* Arm [UK]
* Graphcore [UK]
* DRAM integrated circuit chips:
* Samsung (market share 44%) [South Korea],
* SK Hynix (27%) [South Korea],
* Micron (22%) [US].
* GPU providers:
* Intel (market share 62%) [US],
* AMD (18%) [US]
* NVIDIA (20%) [US].
* Cloud computing providers:[[387]](#ftnt387)
* Globally: Amazon Web Services (32%), Microsoft Azure (21%), and Google Cloud (8%)
* Chinese market: Alibaba, Huawei and Tencent.

Software tools & community service providers

* arXiv
* GitHub
* Colab
* HuggingFace
* [...]

Academic communities:

* Scientific ML community;[[388]](#ftnt388)
* Conferences: NeurIPS, AAAI/ACM, ICLR, IJCAI-ECAI. AIES and FAccT
* AI ethics community and various subcommunities
* Numerous national-level academic or research institutes;
* [...]

Other active tech community actors

* Open-source machine learning software community;[[389]](#ftnt389)
* ‘Open’ / diffusion-encouraging[[390]](#ftnt390) AI community (e.g. Stability.ai, Eleuther.ai)[[391]](#ftnt391)
* Hacker communities
* Cybersecurity & information security expert communities;[[392]](#ftnt392)
* […]

### State & governmental actors

Various states, and their constituent agencies or bodies, that are, plausibly will be, or potentially could be moved to be in powerful positions to shape the development of TAI.

The US;[[393]](#ftnt393) –and, within it (in no particular order):[[394]](#ftnt394)

* Congress
* Intelligence committees
* Appropriations committees
* Commerce committees
* Congressional AI Caucus[[395]](#ftnt395)
* House Committee on Science, Space, and Technology
* Supreme Court
* Executive Branch
* Office of Science and Technology Policy (OSTP), especially its National AI Initiative Office (NAIIO)
* National Security Council
* National Science Foundation (basic and applied grants)[[396]](#ftnt396)
* Department of Commerce
* National Institute of Standards and Technology (NIST)[[397]](#ftnt397)
* Office of Management and Budget (OMB)
* Bureau of Industry and Security
* Department of Defense (DoD)
* CDAO (Chief Digital and Artificial Intelligence Office)
* DARPA
* Emerging Capabilities Policy Office
* Office of Net Assessment
* National Security Commission on Artificial Intelligence[[398]](#ftnt398)
* Department of Homeland Security[[399]](#ftnt399)
* FEMA
* U.S. Customs and Border Protection (use of facial recognition)
* Intelligence Community
* IARPA
* ODNI
* In-Q-Tel
* Independent agencies:
* Federal Trade Commission (FTC) (including their new Office of Technology[[400]](#ftnt400)), able to regulate a broad range of harms of AI systems;[[401]](#ftnt401) to seek injunctions to order a company to cease certain practices;[[402]](#ftnt402) to interpret Section 230 (which affects whether digital platforms or social media companies would be liable for unlawful speech produced by AI tools or search engines).
* Food and Drug Administration, for approving medical AI systems;
* Securities and Exchange Commission, for shaping financial applications;
* Committee on Foreign Investment in the United States (CFIUS); a federal interagency committee able to review foreign investments in U.S. companies on national security grounds;
* [...]
* Relevant state legislatures;
* State of California (potentially significant extraterritorial regulatory effects);[[403]](#ftnt403)
* State of Illinois & State of Texas (among first states to place restrictions on biometrics);
* […]

China;[[404]](#ftnt404) –and within it:

* 20th Central Committee of the Chinese Communist Party;
* China’s State Council;
* Bureaucratic actors: Cyberspace Administration of China (CAC), Ministry of Industry and Information Technology, the Ministry of Science and Technology,[[405]](#ftnt405) National science and technology ethics committee;[[406]](#ftnt406)
* Standardization Administration of China (SAC)
* Key research labs; Baidu Research, Alibaba DAMO Academy, Tencent AI Lab, Huawei, JD Research Institute, Beijing Institute for General Artificial Intelligence; Beijing Academy of Artificial Intelligence, etc.[[407]](#ftnt407)
* Actors and institutions engaged in track-II diplomacy on AI: Institute for AI International Governance (I-AIIG) and Center for International Security and Strategy (CISS) at Tsinghua University, overseen by Madam Fu Ying (current Chairperson of the National People’s Congress Foreign Affairs Committee).[[408]](#ftnt408)
* [...]

The EU,[[409]](#ftnt409) and within it:

* European Commission;[[410]](#ftnt410)
* European Parliament
* Joint Research Centre (JRC)
* International Outreach for Human-Centric Artificial Intelligence Initiative (joint initiative by the European Commission’s Service for Foreign Policy Instruments (FPI) and the Directorate General for Communications Networks, Content and Technology (DG CONNECT), in collaboration with the European External Action Services (EEAS));[[411]](#ftnt411)
* (Proposed) European Artificial Intelligence Board and notified bodies;[[412]](#ftnt412)
* CEN-CENELEC

The UK,[[413]](#ftnt413) and within it:

* The Cabinet Office;[[414]](#ftnt414)
* The Office of the Chief Scientific Advisor for National Security
* Office for Science and Technology Strategy
* Foreign Commonwealth and Development Office (FCDO)
* Ministry of Defence (MoD)
* Defence AI and Autonomy Unit (DAU) - strategy level policy across UK Defence
* Defence AI Centre (DAIC) - unit of excellence re AI best practices and guidance across UK Defence
* Defence Science and Technology Laboratory (Dstl)
* Department for Science, Innovation and Technology (DSIT), within which sit:
* Government Office for Science
* Office for Science and Technology Strategy
* Office for Artificial Intelligence
* The Centre for Data Ethics and Innovation (CDEI) - focus on enabling the trustworthy use of data and AI;
* UK Parliament
* The House of Lords Select Committee on AI (former)
* Commons Science and Technology Committee
* AI APPG
* The Digital Regulators Cooperation Forum, comprised of four key regulators for digital/tech:
* Information Commissioner's Office (ICO),
* Competition and Markets Authority (CMA); in particularly the Digital Markets Unit (DMU)
* Office of Communications (Ofcom),
* Financial Conduct Authority (FCA)
* Advanced Research and Invention Agency (ARIA);
* Funding bodies:
* UKRI (funder of science)
* National Security Strategic Investment Fund (NSSIF) - joint initiative by UK Government and the British Business Bank.
* [...]

Key states hosting the global hardware supply chain:[[415]](#ftnt415)

* US (Nvidia)
* Taiwan (TSMC)
* South Korea (Samsung)
* The Netherlands (ASML)
* Japan (photoresist processing);
* UK (Arm)
* Germany (Carl Zeiss AG)
* [...]

Other states that may play various roles:

* Singapore (potential neutral hub);[[416]](#ftnt416)
* India
* Russia
* France (significant AI research talent)
* Canada (significant AI research talent)
* Germany
* Various Global South actors;[[417]](#ftnt417)
* Smaller countries as digital norm setters; Estonia, Norway; [...][[418]](#ftnt418)
* [...]

### International organizations

International standard-setting institutions[[419]](#ftnt419)

* ISO
* IEC
* IEEE
* CEN/CENELEC
* VDE (Association for Electrical, Electronic & Information Technologies), and its AI Quality & Testing Hub[[420]](#ftnt420)
* [...]

Various United Nations bodies[[421]](#ftnt421)

* ITU;[[422]](#ftnt422)
* UNESCO[[423]](#ftnt423)
* Office of the UN Tech Envoy (conducting the process leading to the Global Digital Compact in 2024)
* UN Science, Technology, and Innovation (STI) Forum
* UN Executive Office of the Secretary-General
* UN General Assembly
* UN Security Council
* UN Human Rights Council[[424]](#ftnt424)
* Office of the High Commissioner on Human Rights[[425]](#ftnt425)
* UN Chief Executives Board for Coordination[[426]](#ftnt426)
* Secretary-General’s High-Level Advisory Board on Effective Multilateralism (HLAB)
* [...]

Other international institutions already engaged on AI in some capacity[[427]](#ftnt427) (in no particular order);

* OECD
* Global Partnership on AI
* G7[[428]](#ftnt428)
* G20[[429]](#ftnt429)
* Council of Europe (Ad Hoc Committee on Artificial Intelligence (CAHAI));[[430]](#ftnt430)
* NATO[[431]](#ftnt431)
* AI Partnership for Defense[[432]](#ftnt432)
* Global Road Traffic Forum[[433]](#ftnt433)
* International Maritime Organisation
* EU-US Trade and Technology Council (TTC)[[434]](#ftnt434)
* EU-India Trade and Technology Council
* Multi-stakeholder fora: World Summit on the Information Society (WSIS); Internet Governance Forum (IGF); Global Summit on AI for Good;[[435]](#ftnt435) World Economic Forum (Centre for Trustworthy Technology)
* [...]

Other international institutions not yet engaged on AI;

* International & regional courts: International Criminal Court (ICC); International Court of Justice (ICJ); European Court of Justice;
* [...]

Future AI governance institutions that will be established;[[436]](#ftnt436)

* …?

### Public, Civil Society, & media actors

Civil society organisations[[437]](#ftnt437)

* Gatekeepers engaged in AI-specific norm-setting & advocacy: Human Rights Watch; Campaign to Stop Killer Robots;[[438]](#ftnt438) AlgorithmWatch;[[439]](#ftnt439)
* Civilian Open Source Intelligence (OSINT) actors engaged in monitoring state violations of human rights / international humanitarian law:[[440]](#ftnt440) Bellingcat; NYT Visual Investigation Unit; CNS (Arms Control Wonk), Middlesbury Institute, Forensic Architecture, BBC Africa Eye, Syrian Archive, …
* Military AI mediation: Centre for Humanitarian Dialogue; Geneva Centre for Security Policy[[441]](#ftnt441)
* [...]

Media actors

* Mass media;[[442]](#ftnt442)
* Tech media;
* ‘Para-scientific media’[[443]](#ftnt443)
* Social media (e.g. Twitter)
* [...]

Cultural actors

* Film industry (Hollywood, etc.)
* Influential & widely read (science fiction) authors[[444]](#ftnt444)
* [...]

Levers of governance
--------------------

I.e. how might each key actor shape the development of TAI?

|  |
| --- |
| A ‘Lever (of governance)’ can be defined as ‘a tool or intervention that can be used by key actors to shape or affect (1) the primary outcome of TAI development; (2) key strategic parameters of TAI governance; (3) other key actors’ choices or key decisions.’[[445]](#ftnt445) |

Research in this field includes analysis of different types of tools (key levers or interventions) available to each actor to shape TAI.[[446]](#ftnt446)

### AI developer levers

Developer (intra-lab)-level levers:[[447]](#ftnt447)

* Other overviews of levers: alignment research; strong security; standards and monitoring; successful, careful AI projects.[[448]](#ftnt448)
* Levers for technical research safety testing:[[449]](#ftnt449) decoding; limiting systems; adversarial training; throughout-lifecycle TEVV policies;[[450]](#ftnt450) safety evals regimes;[[451]](#ftnt451)
* Levers for safe risk management in development process: the Three Lines of Defense (3LoD) model;[[452]](#ftnt452) organizational and operational criteria for adequately safe development;[[453]](#ftnt453) ‘Defence in depth’ risk management procedures;[[454]](#ftnt454)
* Levers to ensure cautious decision-making: oversight boards;[[455]](#ftnt455) corporate governance policies that support and enable caution,[[456]](#ftnt456) such as incorporating as a Delaware Public Benefit Corporation (in a way that allows the Board of Directors to balance stockholders’ pecuniary interests against their social mission);
* Levers to ensure operational security: information security best practices;[[457]](#ftnt457) structured access mechanisms[[458]](#ftnt458) at the level of cloud-based AI service interfaces;
* Policies for responsibly sharing safety-relevant information: information-providing policies to increase legibility and compliance: model cards;[[459]](#ftnt459)
* Policies to ensure organization can pace and/or pause capability research:[[460]](#ftnt460) Board authority to pause research; channels to invite external AI scientists to review alignment of systems;[[461]](#ftnt461) …

Developer external (unilateral) levers:

* Voluntary safety committments;[[462]](#ftnt462)
* Norm entrepreneurship (i.e. lobbying, public statements, or initiatives that signal public dissatisfaction with an existing state of affairs, potentially alerting others to the existence of a shared complaint, facilitating potential ‘norm cascades’ towards new expectations or collective solutions);[[463]](#ftnt463)
* [...]

### AI industry & academia levers

Industry-level (coordinated inter-lab) levers:

* Self-regulation[[464]](#ftnt464)
* Codes of conduct
* AI ethics principles[[465]](#ftnt465)
* Professional norms[[466]](#ftnt466)
* AI ethics advisory committees[[467]](#ftnt467)
* Incident databases[[468]](#ftnt468)
* Institutional, software & hardware mechanisms for enabling developers to make verifiable claims[[469]](#ftnt469)
* Bug bounties[[470]](#ftnt470)
* Other inter-lab cooperation mechanisms:[[471]](#ftnt471)
* Assist Clause[[472]](#ftnt472)
* Windfall Clause[[473]](#ftnt473)
* Mutual monitoring agreements (red-teaming, incident-sharing, compute accounting, seconding engineers)
* Communications and heads-up
* Third-party auditing
* Bias and safety bounties
* Secure compute enclaves
* Standard benchmarks & audit trails
* Publication norms;[[474]](#ftnt474)
* […]

Third-party industry actors levers:

* Publication reviews[[475]](#ftnt475)
* Certification schemes;[[476]](#ftnt476)
* Auditing schemas;[[477]](#ftnt477)
* [...]

Scientific community levers

* Institutional Review Boards (IRBs)[[478]](#ftnt478)
* Pre-publication impact assessment requirements;[[479]](#ftnt479) academic conference practices;[[480]](#ftnt480)
* Publication & model sharing & release norms[[481]](#ftnt481)
* Benchmarks;[[482]](#ftnt482)
* Differential technological development (innovation prizes)[[483]](#ftnt483)
* (Temporary) moratoria;[[484]](#ftnt484)
* […]

### Compute supply chain industry levers

Global compute industry-level levers;[[485]](#ftnt485)

* Stock-and-flow accounting;
* Operating licenses;
* Supply chain chokepoints;[[486]](#ftnt486)
* Inspections
* Passive architectural on-chip constraints (eg performance caps)
* Active architectural on-chip constraints (e.g. shutdown mechanisms)
* […]

### Governmental levers

We can distinguish between general governmental levers, and the specific levers available to particular key states.

#### General governmental levers

General governmental levers:[[487]](#ftnt487)

Legislatures’ levers:[[488]](#ftnt488)

* Create new AI-specific regimes:
* Horizontal risk regulation;[[489]](#ftnt489)
* Industry-specific risk regulatory regimes;
* Permitting & market gatekeeping;[[490]](#ftnt490)
* Bans or moratoria;
* [...]
* Amend laws to extend or apply existing regulations to AI:[[491]](#ftnt491)
* Domain/industry-specific risk regulations;
* Competition / antitrust law,[[492]](#ftnt492) including doctrines around:
* Merger control
* Abuse of dominance
* Cartels and Collusion & agreements on hardware security
* State aid
* Liability law[[493]](#ftnt493)
* Insurance law;[[494]](#ftnt494)
* Contract law[[495]](#ftnt495)
* IP law[[496]](#ftnt496)
* Copyright law, impact on data scraping practices;[[497]](#ftnt497)
* Criminal law;[[498]](#ftnt498)
* Privacy & data protection law
* Procurement processes & public procurement law;[[499]](#ftnt499)
* [...]

Executive levers:

* Executive orders
* Foreign investment restrictions
* AI R&D funding strategies;[[500]](#ftnt500)
* Nationalisation of firms;
* Certification schemes;
* Various tools of Differential technology development:
* preferential advancement of safer AI architectures: funding and direct development programs, government prizes, advanced market commitments, regulatory requirements, tax incentives;[[501]](#ftnt501)
* slowing down research lines towards dangerous AI architectures: moratoria, bans, defunding, divestment, and/or ‘stage-gating’ review processes;[[502]](#ftnt502)
* foreign policy decisions, e.g. initiate multilateral treaty negotiations;
* [...]

Judiciaries’ levers:

* Judicial decisions handed down on cases involving AI, which extend or apply existing doctrines to AI,[[503]](#ftnt503) shaping economic incentives & setting precedent for regulatory treatment of advanced AI; e.g. US Supreme Court ruling on Gonzalez v. Google, which has implications for whether algorithmic recommendations will receive full Section 230 protections;[[504]](#ftnt504)
* Judicial review, especially of drastic executive actions taken in response to AI risk scenarios;[[505]](#ftnt505)
* Judicial policymaking, through discretion in evaluating proportionality or balancing tests;[[506]](#ftnt506)

Expert agencies’ levers:

* [mix of features of other actors, from setting policies, to adjudicating disputes, to enforcing decisions][[507]](#ftnt507)
* Create or propose soft law[[508]](#ftnt508)
* [...]

Ancillary institutions

* Improved monitoring infrastructures[[509]](#ftnt509)
* Provide services in terms of training, insurance, procurement, identification, archiving, etc.[[510]](#ftnt510)

Foreign Ministries/State Department

* Set activities and agenda in global AI governance institutions;
* Bypass or challenge existing institutions, and fragment regimes by engaging in ‘competitive regime creation’[[511]](#ftnt511) or ‘forum shopping’;[[512]](#ftnt512)
* Initiate multilateral treaty negotiations;
* Advice policymakers about the existence and meaning of international law, and which obligations these impose;[[513]](#ftnt513)
* Conduct state behaviour around AI issues (in terms of state policy, and through discussion of AI issues in national legislation, diplomatic correspondence, etc), in such a way as to contribute to the establishment of binding Customary International Law (CIL);[[514]](#ftnt514)
* [...]

#### Specific key governments levers

Levers my specific key governments:

US-specific levers;

* AI-specific regulations: AI Bill of Rights,[[515]](#ftnt515) Algorithmic Accountability Act;
* General levers,[[516]](#ftnt516) such as: federal R&D funding, foreign investment restrictions, export controls,[[517]](#ftnt517) visa vetting, expanded visa pathways, secrecy orders, voluntary screening procedures, use of Defense Production Act,[[518]](#ftnt518) Antitrust enforcement, the ‘Born Secret’ Doctrine; nationalization of companies or compute hardware;
* NIST
* Standard-setting (AI Risk Management Framework);[[519]](#ftnt519)
* FTC:
* Antitrust actions;[[520]](#ftnt520)
* Injunctions to cease harmful practices;[[521]](#ftnt521)
* [...]
* [...]

EU-specific levers:

* AI-specific regulations:
* AI Act, both in direct terms;[[522]](#ftnt522) and through its extraterritorial impacts as part of a ‘Brussels Effect’;[[523]](#ftnt523)
* Standard-setting by European Standards Organizations (ESOs)[[524]](#ftnt524)
* AI Liability Directive[[525]](#ftnt525)
* [...]

China-specific levers:

* AI-specific regulations
* Standards;[[526]](#ftnt526)
* Activities in global AI governance institutions[[527]](#ftnt527)
* [...]

UK-specific levers;[[528]](#ftnt528)

* National Security and Investment Act 2021
* Competition Law: 1998 Competition Act
* Export Control legislation
* Secrecy orders
* [...]

### Public, civil society & media actor levers

Civil Society/activist movement levers;[[529]](#ftnt529)

* Lab-level (internal) levers:
* Shareholder activism, voting out CEOs;
* Unions and intra-organisational advocacy; strikes; walkouts;[[530]](#ftnt530)
* Capacity-building of employee activism via recruitment, political education, training, legal advice;
* Lab-level (external) levers:
* Stigmatization of irresponsible practices;[[531]](#ftnt531)
* Investigative journalism; awareness-raising of scandals & incidents; hacking & leaks; whistleblowing;
* Impact litigation;[[532]](#ftnt532) class-action lawsuits;[[533]](#ftnt533)
* Public protest;[[534]](#ftnt534) direct action (eg sit-ins)
* Industry-level levers:
* Norm advocacy & lobbying;
* Open letters & statements
* Mapping and highlighting (compliance) performance of companies; establishing metrics, indexes, prizes; certification schemes;[[535]](#ftnt535)
* Public-focused levers:
* Media content creation;[[536]](#ftnt536)
* Boycott & divestment;
* Shaming of state noncompliance with international law;[[537]](#ftnt537)
* [...]
* Creating alternatives:
* Public interest technology research;
* Creating alternative (types of) institutions,[[538]](#ftnt538) new labs;
* State-focused levers:
* Monitor compliance with international law;[[539]](#ftnt539)
* [...]

### International organizations & regime levers

International standards bodies’ levers:

* Set technical safety & reliability standards;[[540]](#ftnt540)
* [...]

International regime levers,[[541]](#ftnt541) such as:

* Setting or shaping norms & expectations;
* Setting, affirming and/or clarifying states’ obligations under existing international law principles ([see section on existing law](#h.xtzugu6j7dxo));
* Set fora and/or agenda for negotiation of new treaties or regimes, in various formats, such as;
* Broad framework conventions;[[542]](#ftnt542)
* Nonproliferation & arms control agreements;[[543]](#ftnt543)
* Export control regimes;[[544]](#ftnt544)
* [...]
* Create (technical) benchmarks and focal points for decisionmaking by both states and nonstate actors;[[545]](#ftnt545)
* Organize training & workshops with national officials;
* [...]
* Coordinating behaviour; reducing uncertainty, improving trust:
* Confidence-Building Measures;[[546]](#ftnt546)
* Review Conferences (e.g. BWC)
* Conferences of Parties (e.g. UNFCCC)
* Establishing information & benefit-sharing mechanisms:
* [...]
* Create common knowledge or shared perceptions of problems; establish ‘fire alarms’:
* Intergovernmental scientific bodies (e.g. Intergovernmental Panel on Climate Change (IPCC), Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES))
* International warning systems (e.g. WHO's ‘public health emergency of international concern’ mechanism).
* Adjudicate and arbitrate state disagreements over application of policies, resolving tensions or crises for regimes;
* Arbitral bodies (e.g. WTO Appellate Body);
* Adjudicatory tribunals (e.g. ICJ)
* Treaty bodies (e.g. Human Rights Committee).
* Other dispute resolution mechanisms (e.g. BWC or Res 1540 allowing complaints to be lodged at the UNSC).
* Establishing material constraints:
* Supply-side material proliferation controls (e.g. stock-and-flow accounting; trade barriers);
* Fair and equitable treatment standards in international investment law;
* Monitoring state compliance:
* Inspection regimes;
* Safeguards
* National contributions
* Network of national contact points;
* Sanctioning noncompliance:
* Sanctions
* Inducing reputational costs;[[547]](#ftnt547) in particular through shaming;[[548]](#ftnt548)
* [...]

### New or future institutions and levers

Novel governance institutions and innovations:

* ‘Regulatory markets’ and private regulatory authorities;[[549]](#ftnt549)
* New monitoring institutions and information markets;[[550]](#ftnt550)
* Quadratic voting & radical markets[[551]](#ftnt551)
* Blockchain smart contracts;[[552]](#ftnt552)
* […]

Pathways to influence
---------------------

I.e. how might concerned stakeholders ensure that key actors use their levers to shape TAI development in appropriate ways?

|  |
| --- |
| In this context, a ‘pathway (to influence)’ can be defined as ‘a tool or intervention by which other actors (that may not themselves be key actors) can affect, persuade, induce, incentivize, or require key actors to make certain key decisions. This can include interventions that ensure that certain levers of control are (not) used, or used in particular ways’.[[553]](#ftnt553) |

This includes research on the different pathways by which the use of these above levers might be enabled, advocated for, and implemented (i.e. the tools available to affect the decisions by key actors).

This can draw on mappings and taxonomies: ‘A Map to Navigate AI Governance’;[[554]](#ftnt554) ‘the longtermist AI Governance Landscape’;[[555]](#ftnt555) [...]

### Direct shaping of TAI systems’ decisions through law

Directly shaping TAI actions through law (i.e. legal systems & norms as technical alignment anchor or lodestar):

* "Law-following AI"[[556]](#ftnt556)
* "Incomplete Contracting"[[557]](#ftnt557)
* Negative human rights as technical safety constraint for minimal alignment;[[558]](#ftnt558)
* Human rights norms as benchmark for maximal alignment;[[559]](#ftnt559)
* Encoded ‘fiduciary duties towards users’[[560]](#ftnt560)
* Mandatory on-chip controls (monitoring & remote shutdown)
* ‘Legal informatics’ approach to alignment;[[561]](#ftnt561)
* [...]

### Shaping governmental decisions

Shaping governmental decisions around AI levers, at the level of:

* Legislature:
* Engage in the legislative AI policymaking process;[[562]](#ftnt562)
* [...]
* Executive:
* Serve as high-bandwidth policy advisor;[[563]](#ftnt563)
* Provide actionable technical information;[[564]](#ftnt564)
* Shape, provide or spread narratives,[[565]](#ftnt565) ideas, ‘memes’,[[566]](#ftnt566) framings or (legal) analogies[[567]](#ftnt567) for AI governance;
* Clarify or emphasize established principles within national law (e.g. precautionary principle; cost-benefit analysis[[568]](#ftnt568)) and/or state obligations under international law (e.g. Customary International Law;[[569]](#ftnt569) International Human Rights Law;[[570]](#ftnt570) etc.).
* (see section on ‘[Existing and Applicable Principles, Norms, and Laws](#h.xtzugu6j7dxo)’)

### Shaping court decisions

Shaping court decisions around AI systems, that set critical precedent for the application of AI policy to TAI;

* Advance legal scholarship to advance new arguments or interpretations;
* Clarifying the ‘ordinary meaning’ of legal terms;[[571]](#ftnt571)
* Identifying precedent;[[572]](#ftnt572)
* Judge seminars and training courses[[573]](#ftnt573)
* Online information repositories[[574]](#ftnt574)
* [...]

### Shaping lab & industry decisions

Shaping individual lab decisions:

* Governmental regulations (as above: e.g. industry risk-, liability-, criminal-, etc);
* Educate researchers about AI risk, making AI safety research more concrete and legible, and/or creating common knowledge about researchers’ attitudes on risk;[[575]](#ftnt575)
* Rules around board of directors;[[576]](#ftnt576)
* Employee activism;[[577]](#ftnt577)
* Human rights norms: both those generally applicable to business activities under the Ruggie Principles;[[578]](#ftnt578) as well as those shaping decisions by tech company oversight bodies;[[579]](#ftnt579)
* Industry standards & decisions;

Shaping industry-wide decisions:

* (Governmental regulation, as above)
* Ensure competition law framework enables cooperation on safety;[[580]](#ftnt580)
* […]

### Shaping AI research community decisions

Shaping AI research community decisions;

* Develop clear guidelines and toolsets to facilitate pre-publication impact assessment;[[581]](#ftnt581)
* Framing & stigmatization;[[582]](#ftnt582)
* Participatory technology assessment processes;[[583]](#ftnt583)

Shaping civil society decisions:

* Work with ‘gatekeeper’ organisations to put issues on the advocacy agenda;[[584]](#ftnt584)
* [...]

### Shaping international institutions’ decisions

Shaping international institutional decisions:

* Clarify global administrative law obligations[[585]](#ftnt585)
* Influence on domestic policy processes on transnational legal processes;[[586]](#ftnt586)
* Scientific expert bodies’ role in informing multilateral treaty-making, by preparing evidence for treaty-making bodies, scientifically advising these bodies, and directly exchanging with them at intergovernmental body sessions or dialogical events;[[587]](#ftnt587)
* [...]

Shaping standards bodies’ decisions:

* Technical experts’ direct participation in standards development;[[588]](#ftnt588)
* Advancing standardization of TAI-relevant safety best practices;[[589]](#ftnt589)

### Other pathways to shape various actors’ decisions

Shaping [various actors] decisions:

* Influential fiction; novels, etc.
* Compelling narratives or depictions of good outcomes;[[590]](#ftnt590)
* [...]

Direct work (e.g. employment):

* Influence through specific career paths in AI governance;[[591]](#ftnt591)
* Calls for social scientists;[[592]](#ftnt592) calls for technical work & careers[[593]](#ftnt593)

### Existing and Applicable Law

I.e. which existing (technology-neutral) principles, laws, or doctrines can be invoked or applied to AI, in order to convince key actors to use their levers to shape TAI in responsible ways?

The following lists are not exhaustive. Moreover, they gather a range of laws, principles and norms which (especially in international law) have significantly differing relative status or weight.

#### General regulation

Existing law affecting AI development:

* Inputs-focused:
* Compute
* People
* Data (copyright, etc);
* Labour laws (applicable to human feedback and data annotation labour).
* General safety laws
* Tort law (‘risk-taking behaviour’) & duty of care
* Specific product safety laws
* Impact assessment laws;
* Risk laws
* Abstract endangerment
* [...]
* Whistleblower protection laws

Existing laws affecting AI release & application:

* Illegal content (e.g. hate speech, indecent content, instructions for producing weapons)
* Fraud (e.g. fraudulent trading strategies)
* Patents;
* Freedom of Speech exceptions (e.g. defamation and libel, holocaust denial, verbal assault)
* [...]

#### US law

Relevant principles, obligations or doctrines in US law, whether in legislation[[594]](#ftnt594) or in case law;

* Section 230 (determining whether digital platforms or social media companies would be liable for unlawful speech produced by AI tools or search engines);
* Major Questions Doctrine;[[595]](#ftnt595) (which holds that courts should not defer to agency statutory interpretations that concern questions of “vast economic or political significance.”)
* Global Catastrophic Risk Management Act;[[596]](#ftnt596)
* [...]

#### EU law

Relevant principles, obligations and doctrines in EU law:[[597]](#ftnt597)

* Non-discrimination law,
* Unfair commercial practices law,
* Data protection law (GDPR),
* Contract and tort law;
* Competition law;[[598]](#ftnt598)
* [...]

#### International law

Relevant existing principles and obligations under international law,[[599]](#ftnt599) whether based in treaty law,[[600]](#ftnt600) or under established norms of Customary International Law (CIL).[[601]](#ftnt601) Note: as these are technology-neutral, the specific implications of many of these norms to TAI policy issues needs clarification.

* Norms of international peace and security law:
* prohibitions on the use of force and intervention in the domestic affairs of other states;
* Existing export control and nonproliferation agreements;
* Principles of international humanitarian law, such as:
* distinction and proportionality in wartime;
* Prohibition on weapons that are by nature indiscriminate or cause unnecessary suffering;
* the requirements of humanity,
* the obligation to conduct legal reviews of new weapons or means of war (Article 36 under Additional Protocol I to the Geneva Conventions);
* Norms of International Human Rights Law,[[602]](#ftnt602) human rights and freedoms, including the right to life, freedom from cruel, inhuman and degrading treatment: among others the rights to freedom of expression and association; security of the person, among others; the principle of human dignity;[[603]](#ftnt603)
* Norms of international environmental law, including the no-harm principle, the principle of prevention and precaution, etc.
* International Criminal Law (ICL), with regards to war crimes, crimes against humanity; and with regards to case law of international criminal courts regarding questions of effective control;[[604]](#ftnt604)
* Rules on state responsibility,[[605]](#ftnt605) including liability for harm;
* Peremptory norms of jus cogens, outlawing e.g. genocide, maritime piracy, slavery, wars of aggression, torture;
* International Economic Law;[[606]](#ftnt606) security exception measures under international trade law, and non-precludement measures under international investment law, amongst others;[[607]](#ftnt607)
* International Disaster Law: obligations regarding disaster preparedness, including forecasting and pre-disaster risk assessment, multi-sectoral forecasting and early warning systems, disaster risk and emergency communication mechanisms, etc. (Sendai Framework);
* Legal protections for the rights of future generations: existing national constitutional protections for the rights of future generations;[[608]](#ftnt608) potential forthcoming UN Declaration on Future Generations;[[609]](#ftnt609) …
* [...]

Proposals
---------

I.e. what are specific assets, proposals or resources that can be used in acting on pathways to influence on key actors?

|  |
| --- |
| In this context, a ‘(Decision-relevant) Asset’ can be defined as: ‘resources that can be used by other actors in pursuing pathways to influence on key actors, that aim to induce how these key actors make key decisions (e.g. about whether or how to use their levers). This includes new technical research insights, worked-out policy products; networks of direct advocacy, memes or narratives.’A ‘(Policy) product’ can be defined as ‘a subclass of assets; specific legible proposals that can be presented to key actors.’ |

### General heuristics for key actors in crafting TAI policies

General heuristics: for making policies relevant or actionable to TAI.

* Heuristics for crafting good AI regulation:
* Articulating good terms for drafting and scoping AI regulations,[[610]](#ftnt610)
* Optimal regulatory approaches for prioritizing AI regulations;[[611]](#ftnt611)
* Establishing monitoring infrastructures to provide governments with actionable information;[[612]](#ftnt612)
* Heuristics for institutional design:
* Desiderata for international institutional design;[[613]](#ftnt613)
* Requirements for a TAI regime to avoid ‘perpetual risk’: exclusivity, benevolence, stability, success at alignment;[[614]](#ftnt614)
* Heuristics for future-proofing governance regimes: desiderata and systems for making existing regulations more adaptive, scalable or resilient;[[615]](#ftnt615)
* Traditional (treaty) amendment mechanisms:
* The formal treaty amendment process;[[616]](#ftnt616)
* Evolutionary interpretation of treaties;[[617]](#ftnt617)
* Unilateral state actions (explanatory memoranda, treaty reservations) or multilateral responses (Working Party Resolution; Amendment);[[618]](#ftnt618)
* the development of lex scripta treaties through the lex posterior of customary international law, spurred by new state behaviour;[[619]](#ftnt619)
* Instrument choices that promote flexibility:
* the use of framework conventions;[[620]](#ftnt620)
* use of informal governance institutions;[[621]](#ftnt621)
* the subsequent layering of soft law on earlier hard law regimes;[[622]](#ftnt622)
* Use of uncorrelated governance instruments to enable legal resilience;[[623]](#ftnt623)
* Regime design choices that promote flexibility:
* Scope: include ‘general purpose AI systems’ in the remit of regulated systems;[[624]](#ftnt624)
* Phrasing: in-text technological neutrality or deliberate ambiguity;[[625]](#ftnt625)
* Flexibility provisions: textual flexibility provisions;[[626]](#ftnt626)
* Approaches beyond the regime:
* Pragmatic and informal ‘emergent flexibility’ about the meaning of norms and rules during crises;[[627]](#ftnt627)
* [...]

### TAI Policy Proposals

Specific proposals for transformative AI-relevant policies (presented without comparison or prioritization):

Early proposals

* ‘Long-term strategies for ending existential risk’, including: international coordination, sovereign AI, AI-empowered project, decisive technological advantage;[[628]](#ftnt628)
* Lists of possible high-level ‘responses’ to AI risk, in particular ‘societal proposals’, including: ‘Do nothing’; ‘Integrate with society’; ‘Regulate Research’; ‘Enhance Human Capabilities’; ‘Relinquish Technology’[[629]](#ftnt629)

Recent proposed policy ‘products’ (selected, very incomplete):[[630]](#ftnt630)

* Proposals for near-term policy:
* A ‘pause’ on large-scale system training;[[631]](#ftnt631) giving time for near-term policy-setting in domains such as robust third-party auditing and certification, regulation of access to computational power, establishment of capable national AI agencies; establishing liability for AI-caused harms, etc.[[632]](#ftnt632)
* Proposals for establishing new AI research efforts:
* New AI research organizations: UN AI Research Organization;[[633]](#ftnt633) a ‘politically neutral hub’ for AI research;[[634]](#ftnt634)
* Proposals for establishing new governance mechanisms or institutions;
* New domestic institutions: US ‘AI Control Council’;[[635]](#ftnt635)
* New institutions or regimes: European AI Agency;[[636]](#ftnt636) global AI treaty[[637]](#ftnt637)
* [...]
* Proposals for establishing corporate governance and soft law
* Corporate governance: establish corporate governance & AI certification schemes;[[638]](#ftnt638) set up trusted AI auditing organisation with expertise on existential risks
* Proposals for AI resource governance:
* Compute governance regimes: establish on-chip firmware mechanisms, inspection regimes, and supply chain monitoring and custody mechanisms to ensure no actor can use large quantities of specialized chips to execute ML training runs in violation of established rules;[[639]](#ftnt639)
* Data governance: establish public data trusts to assert control over public training data for foundation models;[[640]](#ftnt640)
* Proposals for engagement in existing international processes;
* International policy engagement: support the campaign to ban lethal autonomous weapons;[[641]](#ftnt641) | input to the UN High-level Panel on Digital Cooperation to establish a UN Intergovernmental Panel on AI[[642]](#ftnt642) | orchestrate soft law policy under G20[[643]](#ftnt643) | engage in debate about digital technology governance at UN Summit for the Future[[644]](#ftnt644) | ‘pre-deployment agreement’[[645]](#ftnt645)
* Lists of more concrete policy ideas for GCR mitigation; overview list by GCRP;[[646]](#ftnt646)

---

Prioritization-focused work: comparing theories of action
=========================================================

These mappings are useful, but by themselves they are not enough to deliver strategic clarity about which of these actors should be approached; which of these levers should be developed. For that, it is necessary to develop more systematic accounts of different (currently held; possible) theories of victory. The idea of exploring and comparing such theories of action is not new. There have been various accounts that aim to articulate the linkages between near-term actions and longer-term goals.

Technical AI alignment theories of impact
-----------------------------------------

Theories of impact in the technical AI alignment space, such as:

* Wei Dai’s ‘AI Safety Success Stories’,[[647]](#ftnt647)
* Neel Nanda’s ‘Overview of the AI Alignment Landscape’,[[648]](#ftnt648)
* ‘A Longlist of Theories of Impact for Interpretability’,[[649]](#ftnt649)
* Evan Hubinger’s ‘A positive case for how we might succeed at prosaic AI alignment’,[[650]](#ftnt650)
* [...]

AI governance theories of impact
--------------------------------

Theories of impact in the AI governance space:

* Asset-Decision-model: Allan Dafoe has sketched a two-stage model for impact, which involves the direction of research activities to help (1) create assets (“technical solutions; strategic insights; shared perception of risks; a more cooperative worldview; well-motivated and competent advisors; credibility, authority, and connections for those experts”), which can eventually inform (2) impactful decisions (“by AI researchers, activists, public intellectuals, CEOs, generals, diplomats, or heads of state”);[[651]](#ftnt651)
* Jade Leung’s (2019) ‘how can we see the impact of AI strategy research?’ follows in the same lines, to argue that:
* there are key decisions we want to influence, made by (1) those developing and deploying AI, and (2) those actors shaping the environments in which it is developed and deployed (i.e. research lab environment; legislative environment; market environment).[[652]](#ftnt652)
* When approaching these decision-makers, we can aim to influence their (1) priorities, (2) strategies, and (3) tactics;
* In doing so, we should (1) filter for making a case on a few tractable good things; (2) translate these into digestible memes; (3) ensure your work reaches the key circle of influence;[[653]](#ftnt653)
* Ben Garfinkel’s ‘AI Strategy: Pathways for Impact’[[654]](#ftnt654) highlights three distinct “main (non-technical) pathways for positively influencing the development of advanced AI systems.”
* Become a decision-maker (or close enough to influence one)
* Spread good memes that are picked up by decision-makers
* Think of good memes to spread, and make them credible;
* Seth Baum’s ‘affecting the future of AI governance’ has distinguished several avenues by which AI policy could shape the long-term:[[655]](#ftnt655)
* (1) improve current AI governance;
* (2) support AI governance communities;
* (3) advance research on future AI governance;
* (4) advance CS design of AI safety and ethics to create solutions;
* (5) improve underlying governance conditions.

Specific scenarios for success:

* ‘What success looks like’:[[656]](#ftnt656) scenarios for success:
* “Scenario 1: Alignment is much easier than expected
* Scenario 2: The combination of many technical and governance strategies works
* Scenario 3: Accident and regulation
* Scenario 4: Alignment by chance
* Scenario 5: US-China driven global cooperation
* Scenario 6: Apollo Project for AI”
* Karnofsky AI strategy nearcasting;[[657]](#ftnt657)

Comparing theories of action
----------------------------

There has been limited public work on this, though it includes, amongst others, Campos (2022): ‘AGI Timelines in Governance: Different Strategies for Different Timeframes’;[[658]](#ftnt658) Stein-Perlman’s discussion of ten ‘Frames’ for AI strategy;[[659]](#ftnt659) this Foundation Report series;  as well as a follow-up report on Approaches to Transformative AI Governance.

Legal Priorities Project | www.legalpriorities.org

---

[[1]](#ftnt_ref1) Senior Research Fellow, Legal Priorities Project | Research Affiliate, Centre for the Study of Existential Risk, University of Cambridge. ORCID iD: 0000-0002-6170-9393. Email: [matthijs.maas@legalpriorities.org](mailto:matthijs.maas@legalpriorities.org). For comments and input on this document, I thank Di Cooke, Jeffrey Ding, Christoph Winter, Jonas Schuett, Oliver Guest, Gavin Leech, Peter Cihon, Jess Whittlestone, Eugenio Vargas Garcia, Jacob Arbeid, José Jaime Villalobos, Sam Clarke, Shin-Shin Hua, Richard Ngo, Liang Fang, Aishwarya Saxena, Moritz Kleinalterkamp, Carlos Ignacio Gutierrez, Charlotte Stix, Beth Barnes, Sascha Simon, Lewin Schmitt, Christian Ruhl, Zach Stein-Perlman, Lalitha Sundaram, Lara Thurnherr, and Anthony Barrett. Any remaining errors are my own.

[[2]](#ftnt_ref2) For different definitions of the scope of this field, see also: Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023.

[[3]](#ftnt_ref3) For a number of influential research agendas, see: Dafoe, Allan. ‘AI Governance: A Research Agenda’. Oxford: Center for the Governance of AI, Future of Humanity Institute, 2018.[https://www.fhi.ox.ac.uk/govaiagenda/](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/govaiagenda/&sa=D&source=editors&ust=1692801434996046&usg=AOvVaw2K2yufHturshLcRPzytRA1). Winter, Christoph, Jonas Schuett, Eric Martínez, Suzanne Van Arsdale, Renan Araújo, Nick Hollman, Jeff Sebo, Andrew Stawasz, Cullen O’Keefe, and Giuliana Rotola. ‘Legal Priorities Research: A Research Agenda’. Legal Priorities Project, January 2021.[https://www.legalpriorities.org/research\_agenda.pdf](https://www.google.com/url?q=https://www.legalpriorities.org/research_agenda.pdf&sa=D&source=editors&ust=1692801434996584&usg=AOvVaw0zW2xkRYGVTpGkuTVLa4Qn). (Chapter 4); Clifton, Jesse. ‘Cooperation, Conflict, and Transformative Artificial Intelligence - A Research Agenda’. Center on Long-Term Risk, March 2020.[https://longtermrisk.org/files/Cooperation-Conflict-and-
Transformative-Artificial-Intelligence-A-Research-Agenda.pdf](https://www.google.com/url?q=https://longtermrisk.org/files/Cooperation-Conflict-and-Transformative-Artificial-Intelligence-A-Research-Agenda.pdf&sa=D&source=editors&ust=1692801434997321&usg=AOvVaw0lWgFVKm6ab_Gk56W4cr7r); Dafoe, Allan, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. ‘Open Problems in Cooperative AI’. ArXiv:2012.08630 [Cs], 15 December 2020.[http://arxiv.org/abs/2012.08630](https://www.google.com/url?q=http://arxiv.org/abs/2012.08630&sa=D&source=editors&ust=1692801434997889&usg=AOvVaw1nabwixG6edxZSgnJfYLhp); Gruetzemacher, Ross, Florian E. Dorner, Niko Bernaola-Alvarez, Charlie Giattino, and David Manheim. ‘Forecasting AI Progress: A Research Agenda’. Technological Forecasting and Social Change 170 (1 September 2021): 120909.[https://doi.org/10.1016/j.techfore.2021.120909](https://www.google.com/url?q=https://doi.org/10.1016/j.techfore.2021.120909&sa=D&source=editors&ust=1692801434998482&usg=AOvVaw34h_syB9npB4AyusckjaAz).

[[4]](#ftnt_ref4) See recently: BlueDot Impact. ‘AI Governance Curriculum’. AGI Safety Fundamentals, 2022.[https://www.agisafetyfundamentals.com/ai-governance-curriculum](https://www.google.com/url?q=https://www.agisafetyfundamentals.com/ai-governance-curriculum&sa=D&source=editors&ust=1692801434999273&usg=AOvVaw3H4tG-jWuoH9FTN2dfYSwA); and for older syllabi: Dafoe, Allan. ‘Reading Guide for the Global Politics of Artificial Intelligence’, 2017.[https://www.allandafoe.com/aireadings](https://www.google.com/url?q=https://www.allandafoe.com/aireadings&sa=D&source=editors&ust=1692801434999814&usg=AOvVaw1VYg3_ZbHO8ydPfdddqHbi).; Zwetsloot, Remco. ‘Syllabus: Artificial Intelligence and International Security’, 2018, 19. [https://www.fhi.ox.ac.uk/wp-content/uploads/Artificial-
Intelligence-and-International-Security-Syllabus.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Artificial-Intelligence-and-International-Security-Syllabus.pdf&sa=D&source=editors&ust=1692801435000117&usg=AOvVaw0OIpmOpcqMlthVgJTJv7XG)

[[5]](#ftnt_ref5) See the section on the [profile of technical alignment challenge](#h.y462xislojgv), below; also see: Christiano, Paul. ‘Current Work in AI Alignment’. Effective Altruism, 3 April 2020.[https://www.effectivealtruism.org/articles/paul-christiano-current-work-in-ai-alignment](https://www.google.com/url?q=https://www.effectivealtruism.org/articles/paul-christiano-current-work-in-ai-alignment&sa=D&source=editors&ust=1692801435071494&usg=AOvVaw1dDXSZmncpHHPcYfL7E2Af).; for an older literature review of the technical AI risk field, see: Everitt, Tom, Gary Lea, and Marcus Hutter. ‘AGI Safety Literature Review’. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018.[http://arxiv.org/abs/1805.01109](https://www.google.com/url?q=http://arxiv.org/abs/1805.01109&sa=D&source=editors&ust=1692801435071968&usg=AOvVaw1deSQ61FmBG56rkZ9ncS4b).

[[6]](#ftnt_ref6) Clarke, Sam. ‘The Longtermist AI Governance Landscape: A Basic Overview’. EA Forum, 18 January 2022.[https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-
governance-landscape-a-basic-overview](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview&sa=D&source=editors&ust=1692801435062191&usg=AOvVaw3Kk_yTTCSY75mbRp9THxzg); BlueDot Impact. ‘AI Governance Curriculum’. AGI Safety Fundamentals, 2022.[https://www.agisafetyfundamentals.com/ai-governance-curriculum](https://www.google.com/url?q=https://www.agisafetyfundamentals.com/ai-governance-curriculum&sa=D&source=editors&ust=1692801435062572&usg=AOvVaw0Hbof4eAS21medh81tHPnb).

[[7]](#ftnt_ref7) Olah, Chris, and Shan Carter. ‘Research Debt’. Distill 2, no. 3 (22 March 2017): e5.[https://doi.org/10.23915/distill.00005](https://www.google.com/url?q=https://doi.org/10.23915/distill.00005&sa=D&source=editors&ust=1692801435123224&usg=AOvVaw0aeS2trpZ0N2hop2w19yWO).

[[8]](#ftnt_ref8) Maas, Matthijs, “Disentangling Definitions in Transformative AI Governance’. Legal Priorities Project Foundations Report #1. Forthcoming 2023. (discussing various technical, policy, and strategy-focused definitions of this field).

[[9]](#ftnt_ref9) Inclusion does not imply endorsement of a given article’s conclusions.

[[10]](#ftnt_ref10) A small number of references to in-progress or forthcoming work are also included, with the authors’ consent.

[[11]](#ftnt_ref11) Possible new lenses and case studies are instead discussed in a forthcoming report: Maas, Matthijs. Handbook of Approaches in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023.

[[12]](#ftnt_ref12) Note, these are not exhaustive of all relevant key parameters for AI governance. For previous mappings of relevant (technical and governance) parameters for TAI governance, see also: Avin, Shahar. ‘Exploring AGI Scenarios’. Presented at the FLI, 2019.[https://futureoflife.org/wp-
content/uploads/2019/02/avin\_friday\_am.pdf?x76795](https://www.google.com/url?q=https://futureoflife.org/wp-content/uploads/2019/02/avin_friday_am.pdf?x76795&sa=D&source=editors&ust=1692801434935446&usg=AOvVaw2KGmAVVJqbmqJzVnbBCtfT).; Seth Baum on AI Governance, 2021.[https://www.youtube.com/watch?v=G-8uEg7mCdA](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DG-8uEg7mCdA&sa=D&source=editors&ust=1692801434935980&usg=AOvVaw13Q4iU2HAFVUWJeWD52rIk).; Hua, Shin-Shin, and Haydn Belfield. ‘Effective Enforceability of EU Competition Law Under Different AI Development Scenarios: A Framework for Legal Analysis’. Verfassungsblog (blog), 18 August 2022.[https://verfassungsblog.de/effective-enforceability-of-eu-competition-law-under-different-ai-development-scenarios/](https://www.google.com/url?q=https://verfassungsblog.de/effective-enforceability-of-eu-competition-law-under-different-ai-development-scenarios/&sa=D&source=editors&ust=1692801434936587&usg=AOvVaw2TkixMrgCH4SOcoRz1N6Ir). (sketching six ‘dimensions’ for AI development scenarios–capability, speed of development, key inputs into AI development, model of AI system, number of actors, nature of actor); Hobbhahn, Marius, Max Räuker, Yannick Mühlhäuser, Jasper Götting, and Simon Grimm. ‘What Success Looks Like’. Effective Altruism Forum, 28 June 2022.[https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like&sa=D&source=editors&ust=1692801434937128&usg=AOvVaw37A4q3MLbUn0sNyj1rrADA). (proposing a range of ‘scenario variables’); Kilian, Kyle A., Christopher J. Ventura, and Mark M. Bailey. ‘Examining the Differential Risk from High-Level Artificial Intelligence and the Question of Control’. arXiv, 10 November 2022.[https://doi.org/10.48550/arXiv.2211.03157](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2211.03157&sa=D&source=editors&ust=1692801434937585&usg=AOvVaw2RTDnLGe8i0FbUbBjEX-HH). Pg. 7 (sketching 14 primary ‘dimensions’ of AI technology transitions, which can see a total of 47 different ‘conditions’ (future outcomes).

[[13]](#ftnt_ref13) For discussion of these terms, see also Maas, Matthijs, “Disentangling Definitions in Transformative AI Governance’. Legal Priorities Project Foundations Report #1. Forthcoming 2023.

[[14]](#ftnt_ref14) In other contexts, these have also been called ‘risk factors’. See also Winter, Christoph; Matthijs Maas, José Jaime Villalobos, Alfredo Parra. ‘Theory of Change’. Legal Priorities Project. (2023 Forthcoming).

[[15]](#ftnt_ref15) For a recent summary and introduction to much of this field, see also: Hilton, Benjamin. ‘Preventing an AI-Related Catastrophe - Problem Profile’. 80,000 Hours, 25 August 2022.[https://80000hours.org/problem-profiles/artificial-intelligence/](https://www.google.com/url?q=https://80000hours.org/problem-profiles/artificial-intelligence/&sa=D&source=editors&ust=1692801434974832&usg=AOvVaw3Q_frxatYs6h6sMuHDhN47).

[[16]](#ftnt_ref16) For a previous model and description of the ‘technical landscape’, see Dafoe, Allan. ‘AI Governance: A Research Agenda’. Oxford: Center for the Governance of AI, Future of Humanity Institute, 2018.[https://www.fhi.ox.ac.uk/govaiagenda/](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/govaiagenda/&sa=D&source=editors&ust=1692801434961243&usg=AOvVaw2vCc9NkkD5yiKhiT17GYb0). Pg. 15-33 (distinguishing the subfields ‘Mapping Technical Possibilities’, ‘Assessing AI Progress’, and ‘AI Safety’).

[[17]](#ftnt_ref17) See also the discussion and definitions of these terms, with sources, in [Appendix 1](https://www.google.com/url?q=https://docs.google.com/document/d/1nhiad5KHLsxMX6eU1AzXz_Z_q5eq3s1VxkBraYf2XPE/edit%23heading%3Dh.qeqmq8k8erst&sa=D&source=editors&ust=1692801434976444&usg=AOvVaw3V6m_XxpFpltDMAsYz_eU4).

[[18]](#ftnt_ref18) McCarthy, John. ‘From Here to Human-Level AI’. In Proceedings of the Fifth International Conference on Principles of Knowledge Representation and Reasoning, 640–46. KR’96. Cambridge, Massachusetts, USA: Morgan Kaufmann Publishers Inc., 1996. [https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.384.8219&rep=rep1&type=pdf](https://www.google.com/url?q=https://citeseerx.ist.psu.edu/viewdoc/download?doi%3D10.1.1.384.8219%26rep%3Drep1%26type%3Dpdf&sa=D&source=editors&ust=1692801434976976&usg=AOvVaw3xJxpS20ySs78lsvUHDLpE) ; Nilsson, Nils J. ‘Human-Level Artiﬁcial Intelligence? Be Serious!’ AI Magazine, 2005. [https://ai.stanford.edu/~nilsson/OnlinePubs-Nils/General%20Essays/AIMag26-04-HLAI.pdf](https://www.google.com/url?q=https://ai.stanford.edu/~nilsson/OnlinePubs-Nils/General%2520Essays/AIMag26-04-HLAI.pdf&sa=D&source=editors&ust=1692801434977380&usg=AOvVaw0TGJqVGMXSR7YcQ0huIwUh) ; AI Impacts. ‘Human-Level AI’. AI Impacts, 23 January 2014.[https://aiimpacts.org/human-level-ai/](https://www.google.com/url?q=https://aiimpacts.org/human-level-ai/&sa=D&source=editors&ust=1692801434977858&usg=AOvVaw2XQqZyMXjuxWQD3TKZ99rj).

[[19]](#ftnt_ref19) Chalmers, David J. ‘The Singularity: A Philosophical Analysis’. Journal of Consciousness Studies 17 (2010): pg. 11.

[[20]](#ftnt_ref20) Goertzel, Ben. ‘Artificial General Intelligence: Concept, State of the Art, and Future Prospects’. Journal of Artificial General Intelligence 5, no. 1 (1 December 2014): 1–48.[https://doi.org/10.2478/jagi-2014-0001](https://www.google.com/url?q=https://doi.org/10.2478/jagi-2014-0001&sa=D&source=editors&ust=1692801434978554&usg=AOvVaw2GcfPzRn4Fh1EtJPBtz1Sh). (pg 2); and see generally Goertzel, Ben, and Cassio Pennachin, eds. Artificial General Intelligence. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007.[https://doi.org/10.1007/978-3-540-68677-4\_5](https://www.google.com/url?q=https://doi.org/10.1007/978-3-540-68677-4_5&sa=D&source=editors&ust=1692801434979135&usg=AOvVaw2qYwl6LAeHeTFoh7_F0-2y).; Ngo, Richard. ‘AGI Safety From First Principles’, 2020.[https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ](https://www.google.com/url?q=https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ&sa=D&source=editors&ust=1692801434979644&usg=AOvVaw0aoxirks2rwkEHSRGRk6s8). Pg. 5.; see Muelhauser, Luke. ‘What Is AGI?’ Machine Intelligence Research Institute, 11 August 2013.[https://intelligence.org/2013/08/11/what-is-agi/](https://www.google.com/url?q=https://intelligence.org/2013/08/11/what-is-agi/&sa=D&source=editors&ust=1692801434980196&usg=AOvVaw2JvCUoA94bUMunhrX7AqgN).; Barnett, Matthew. ‘When Will the First General AI System Be Devised, Tested, and Publicly Announced?’ Metaculus, 23 August 2020.[https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/](https://www.google.com/url?q=https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/&sa=D&source=editors&ust=1692801434980779&usg=AOvVaw0DFHPUy4NhhxZXSAM5Hb1O).

[[21]](#ftnt_ref21) Bostrom, Nick. ‘How Long Before Superintelligence?’ International Journal of Futures Studies 2 (1998).[https://nickbostrom.com/superintelligence](https://www.google.com/url?q=https://nickbostrom.com/superintelligence&sa=D&source=editors&ust=1692801434981494&usg=AOvVaw3R0WlQLKI_Xh9WfrkCvCCt).; Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. Pg. 22.

[[22]](#ftnt_ref22) Drexler, K Eric. ‘Reframing Superintelligence: Comprehensive AI Services as General Intelligence’. Future of Humanity Institute, 2019. [https://www.fhi.ox.ac.uk/wp-content/uploads/
Reframing\_Superintelligence\_FHI-TR-2019-1.1-1.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Reframing_Superintelligence_FHI-TR-2019-1.1-1.pdf&sa=D&source=editors&ust=1692801434982075&usg=AOvVaw0l1P1OOZGufywRyXxSVhaR). Pg. 1.

[[23]](#ftnt_ref23) Gutierrez, Carlos Ignacio, Anthony Aguirre, Risto Uuk, Claire C. Boine, and Matija Franklin. ‘A Proposal for a Definition of General Purpose Artificial Intelligence Systems’. Future of Life Institute - Working Paper, 5 October 2022.[https://doi.org/10.2139/ssrn.4238951](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4238951&sa=D&source=editors&ust=1692801435102605&usg=AOvVaw1zeqKvSK5Bw1ofb39IC3mK).

[[24]](#ftnt_ref24) LeCun, Yann. ‘A Path Towards Autonomous Machine Intelligence’. OpenReview, 27 June 2022.[https://openreview.net/forum?id=BZ5a1r-kVsf](https://www.google.com/url?q=https://openreview.net/forum?id%3DBZ5a1r-kVsf&sa=D&source=editors&ust=1692801434982806&usg=AOvVaw38PhCHeQ8iB3gCKVgVpKXY).

[[25]](#ftnt_ref25) janus. ‘Simulators’. LessWrong, 2 September 2022.[https://www.lesswrong.com/posts/
vJFdjigzmcXMhNTsx/simulators](https://www.google.com/url?q=https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators&sa=D&source=editors&ust=1692801435268590&usg=AOvVaw2DyHp_5UMyW3Q37XtQvcU7). However, for a critique of this framing, see: Barnes, Beth. ‘“simulator” Framing and Confusions about LLMs’. AI Alignment Forum, 1 January 2023.[https://www.alignmentforum.org/posts/dYnHLWMXCYdm9xu5j/simulator-framing-and-confusions-about-llms](https://www.google.com/url?q=https://www.alignmentforum.org/posts/dYnHLWMXCYdm9xu5j/simulator-framing-and-confusions-about-llms&sa=D&source=editors&ust=1692801435269023&usg=AOvVaw3BorUtC20zS2-gjTU9BVKv).

[[26]](#ftnt_ref26) Note: while many of these approaches have been discussed over the last decades, many have not featured significantly in modern ML research. For other taxonomies of development paradigms, see also: Hannas, William, Huey-Meei Chang, Daniel Chou, and Brian Fleeger. ‘China’s Advanced AI Research: Monitoring China’s Paths to “General” Artificial Intelligence’. Center for Security and Emerging Technology, July 2022.[https://cset.georgetown.edu/publication
/chinas-advanced-ai-research/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/chinas-advanced-ai-research/&sa=D&source=editors&ust=1692801435135812&usg=AOvVaw0ITi4311DwKtXCaeHPltck). (pg 12-16).

[[27]](#ftnt_ref27) Christiano, Paul. ‘Prosaic AI Alignment’. Medium, 28 March 2017.[https://ai-alignment.com/
prosaic-ai-control-b959644d79c2](https://www.google.com/url?q=https://ai-alignment.com/prosaic-ai-control-b959644d79c2&sa=D&source=editors&ust=1692801435136399&usg=AOvVaw2gwob4Q8ikRgklK30b4pQ2).

[[28]](#ftnt_ref28) Eth, Daniel. ‘The Technological Landscape Affecting Artificial General Intelligence and the Importance of Nanoscale Neural Probes’. Informatica 41, no. 4 (27 December 2017).[https://www.informatica.si/index.php/informatica/article/view/1874](https://www.google.com/url?q=https://www.informatica.si/index.php/informatica/article/view/1874&sa=D&source=editors&ust=1692801435431911&usg=AOvVaw2O7tTsrLVFU9UUFVNfG5hS).

[[29]](#ftnt_ref29) Carl Shulman and Nick Bostrom, “How Hard Is Artificial Intelligence? Evolutionary Arguments and Selection Effects,” Journal of Consciousness Studies 19.7-8, 2012. [https://nickbostrom.com/aievolution.pdf](https://www.google.com/url?q=https://nickbostrom.com/aievolution.pdf&sa=D&source=editors&ust=1692801435136798&usg=AOvVaw1UHJKj6zCya7Mkq2UMkRx3)

[[30]](#ftnt_ref30) Yudkowsky, Eliezer. ‘Levels of Organization in General Intelligence’. In Artificial General Intelligence, edited by Ben Goertzel and Cassio Pennachin, 389–501. Cognitive Technologies. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007.[https://doi.org/10.1007/978-3-540-68677-4
\_12](https://www.google.com/url?q=https://doi.org/10.1007/978-3-540-68677-4_12&sa=D&source=editors&ust=1692801435137441&usg=AOvVaw0zzbEfJwtnOsPGopcj9q6A). Pg. 96. See also Yudkowsky, Eliezer. ‘General Intelligence and Seed AI’. Singularity Institute, 2001.[https://web.archive.org/web/20120805130100/singularity.org/files/GISAI.html](https://www.google.com/url?q=https://web.archive.org/web/20120805130100/singularity.org/files/GISAI.html&sa=D&source=editors&ust=1692801435137935&usg=AOvVaw3yVSFowfuZHBmedchQYPTv).

[[31]](#ftnt_ref31) Byrnes, Steven. ‘[Intro to Brain-like-AGI Safety] 1. What’s the Problem & Why Work on It Now?’ AI Alignment Forum, 26 January 2022.[https://www.alignmentforum.org/posts/
4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why](https://www.google.com/url?q=https://www.alignmentforum.org/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why&sa=D&source=editors&ust=1692801435138675&usg=AOvVaw1GgNvdqpvI8GzxtpT2hfDP).

[[32]](#ftnt_ref32) Bostrom, Nick, and Anders Sandberg. ‘Whole Brain Emulation: A Roadmap’. Technical Report. Future of Humanity Institute, 2008.[http://www.fhi.ox.ac.uk/reports/2008-3.pdf](https://www.google.com/url?q=http://www.fhi.ox.ac.uk/reports/2008-3.pdf&sa=D&source=editors&ust=1692801435139265&usg=AOvVaw2DSWCL_eshteyy4hNOuzKn). Pg 7.

[[33]](#ftnt_ref33) Karnofsky, Holden. ‘Digital People Would Be An Even Bigger Deal’. Cold Takes, 27 July 2021.[https://www.cold-takes.com/how-digital-people-could-change-the-world/](https://www.google.com/url?q=https://www.cold-takes.com/how-digital-people-could-change-the-world/&sa=D&source=editors&ust=1692801435139879&usg=AOvVaw05eUpXwP-5fIiWIVdZ5XQF).

[[34]](#ftnt_ref34) See for instance: Gopalakrishnan, Keerthana. ‘Embodiment Is Indispensable for AGI’, 7 June 2022.[https://keerthanapg.com/tech/embodiment-agi/](https://www.google.com/url?q=https://keerthanapg.com/tech/embodiment-agi/&sa=D&source=editors&ust=1692801435140425&usg=AOvVaw3Hz6HK4arYiMSAnUmeirVE) or [https://www.lesswrong.com/posts/
vBBxKBWn4zRXwivxC/embodiment-is-indispensable-for-agi](https://www.google.com/url?q=https://www.lesswrong.com/posts/vBBxKBWn4zRXwivxC/embodiment-is-indispensable-for-agi&sa=D&source=editors&ust=1692801435140686&usg=AOvVaw29BBJqJVg8_V1pnESJwvf1)

[[35]](#ftnt_ref35) Among many others: Silver, David, Satinder Singh, Doina Precup, and Richard S. Sutton. ‘Reward Is Enough’. Artificial Intelligence 299 (1 October 2021): 103535.[https://doi.org/10.1016/j.artint.2021.103535](https://www.google.com/url?q=https://doi.org/10.1016/j.artint.2021.103535&sa=D&source=editors&ust=1692801434938353&usg=AOvVaw05yqEbtvF9nQPsGl8HRHYW). And famously: Sutton, Rich. ‘The Bitter Lesson’. Incomplete Ideas (blog), 2019.[http://www.incompleteideas.net/IncIdeas/BitterLesson.html](https://www.google.com/url?q=http://www.incompleteideas.net/IncIdeas/BitterLesson.html&sa=D&source=editors&ust=1692801434938900&usg=AOvVaw2gK7LhRuBN5_tSfc8BUe0q). For an articulation of the scaling hypothesis, see also: Branwen, Gwern. ‘The Scaling Hypothesis’, 28 May 2020.[https://www.gwern.net/Scaling-hypothesis](https://www.google.com/url?q=https://www.gwern.net/Scaling-hypothesis&sa=D&source=editors&ust=1692801434939866&usg=AOvVaw0kiESJG02OYaN2uYRaSze0).; and for an overview of literature, see: Branwen, Gwern. ‘Machine Learning Scaling’, 24 April 2021.[https://www.gwern.net/notes/
Scaling](https://www.google.com/url?q=https://www.gwern.net/notes/Scaling&sa=D&source=editors&ust=1692801434940297&usg=AOvVaw1nWgjJK6j4YW08xOPMjga-).

[[36]](#ftnt_ref36) Christiano, Paul. ‘Prosaic AI Alignment’. Medium, 28 March 2017.[https://ai-alignment.com/
prosaic-ai-control-b959644d79c2](https://www.google.com/url?q=https://ai-alignment.com/prosaic-ai-control-b959644d79c2&sa=D&source=editors&ust=1692801434959825&usg=AOvVaw3s9RjydnkP7zaYIRGH0bp0).

[[37]](#ftnt_ref37) Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/
posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801434983640&usg=AOvVaw1QoTmK7Z1cFw-KRXTK_HfX).

[[38]](#ftnt_ref38) Cremer, Carla Zoe. ‘Deep Limitations? Examining Expert Disagreement over Deep Learning’. Progress in Artificial Intelligence, 26 June 2021.[https://doi.org/10.1007/s13748-021-00239-1](https://www.google.com/url?q=https://doi.org/10.1007/s13748-021-00239-1&sa=D&source=editors&ust=1692801434941012&usg=AOvVaw0qRwHxG0wgd3XSOip3mJaO).

[[39]](#ftnt_ref39) Shevlin, Henry, Karina Vold, Matthew Crosby, and Marta Halina. ‘The Limits of Machine Intelligence’. EMBO Reports 20, no. 10 (4 October 2019): e49177.[https://doi.org/10.15252/embr.
201949177](https://www.google.com/url?q=https://doi.org/10.15252/embr.201949177&sa=D&source=editors&ust=1692801435141320&usg=AOvVaw0H_Z7xPzG88YmlTPu2ck0Y).

[[40]](#ftnt_ref40) Mitchell, Melanie. ‘Why AI Is Harder Than We Think’. ArXiv:2104.12871 [Cs], 26 April 2021.[http://arxiv.org/abs/2104.12871](https://www.google.com/url?q=http://arxiv.org/abs/2104.12871&sa=D&source=editors&ust=1692801435141931&usg=AOvVaw0ud_hklqtraYFLi2BHwc67).;

[[41]](#ftnt_ref41) Long, Robert, and Asya Bergal. ‘Evidence against Current Methods Leading to Human Level Artificial Intelligence’. AI Impacts (blog), 12 August 2019.[https://aiimpacts.org/evidence-against-
current-methods-leading-to-human-level-artificial-intelligence/](https://www.google.com/url?q=https://aiimpacts.org/evidence-against-current-methods-leading-to-human-level-artificial-intelligence/&sa=D&source=editors&ust=1692801435142696&usg=AOvVaw0-jaVXMaL0RuXdmu8j-Mml); Kirk, Robert, and David Krueger. ‘Causal Confusion as an Argument against the Scaling Hypothesis’. AI Alignment Forum, 20 June 2022.[https://www.alignmentforum.org/posts/FZL4ftXvcuKmmobmj/causal-confusion-as-an-argument-against-the-scaling](https://www.google.com/url?q=https://www.alignmentforum.org/posts/FZL4ftXvcuKmmobmj/causal-confusion-as-an-argument-against-the-scaling&sa=D&source=editors&ust=1692801435143272&usg=AOvVaw2oo7lTtJuvPfLQkamgiAB-). See also Barak, Boaz. ‘Injecting Some Numbers into the AGI Debate’. Windows On Theory (blog), 27 June 2022.[https://windowsontheory.org/2022/06/27/injecting-
some-numbers-into-the-agi-debate/](https://www.google.com/url?q=https://windowsontheory.org/2022/06/27/injecting-some-numbers-into-the-agi-debate/&sa=D&source=editors&ust=1692801435143855&usg=AOvVaw0qAH3T9hFb7utAqK6S_jOb). See also: Marcus, Gary. ‘What “Game over” for the Latest Paradigm in AI Might Look Like’. Substack newsletter. The Road to AI We Can Trust (blog), 29 October 2022.[https://garymarcus.substack.com/p/what-game-over-for-the-latest-paradigm](https://www.google.com/url?q=https://garymarcus.substack.com/p/what-game-over-for-the-latest-paradigm&sa=D&source=editors&ust=1692801435144401&usg=AOvVaw04ugIUu5KLihVw0LAOG5za) (arguing that there will be three limits to ‘scaling maximalism’: insufficient data, insufficient compute, and insufficient task scaling).

[[42]](#ftnt_ref42) Levin, John-Clark, and Matthijs M. Maas. ‘Roadmap to a Roadmap: How Could We Tell When AGI Is a “Manhattan Project” Away?’, 7. Santiago de Compostela, Spain, 2020.[http://dmip.webs.upv.es/EPAI2020/papers/EPAI\_2020\_paper\_11.pdf](https://www.google.com/url?q=http://dmip.webs.upv.es/EPAI2020/papers/EPAI_2020_paper_11.pdf&sa=D&source=editors&ust=1692801434959207&usg=AOvVaw1DE66pYXNwpdmEI7B3Fp_8).

[[43]](#ftnt_ref43) The taxonomy of approaches presented in this section follows and extends a framework by: Karnofsky, Holden. ‘AI Timelines: Where the Arguments, and the “Experts,” Stand’. Cold Takes, 7 September 2021.[https://www.cold-takes.com/where-ai-forecasting-stands-today/](https://www.google.com/url?q=https://www.cold-takes.com/where-ai-forecasting-stands-today/&sa=D&source=editors&ust=1692801435304532&usg=AOvVaw1PR7Kx0IjrKHFt8S5mvsyq). See also previously Muelhauser, Luke. ‘What Do We Know about AI Timelines?’ Open Philanthropy. Open Philanthropy, 12 October 2015.[https://www.openphilanthropy.org/research/what-do-we-know-
about-ai-timelines/](https://www.google.com/url?q=https://www.openphilanthropy.org/research/what-do-we-know-about-ai-timelines/&sa=D&source=editors&ust=1692801435305155&usg=AOvVaw18KfOPtK31wwAHOwPj1idB).

[[44]](#ftnt_ref44) For a recent literature review of work to estimate TAI timelines, see: Wynroe, Keith, David Atkinson, and Jaime Sevilla. ‘Literature Review of Transformative Artificial Intelligence Timelines’. Epoch, 17 January 2023.[https://epochai.org/blog/literature-review-of-transformative-
artificial-intelligence-timelines](https://www.google.com/url?q=https://epochai.org/blog/literature-review-of-transformative-artificial-intelligence-timelines&sa=D&source=editors&ust=1692801435204231&usg=AOvVaw2vuEJiRJSNp5wO6NU_o0NR).

[[45]](#ftnt_ref45) The distinction between an ‘inside view’ and an ‘outside view’ analysis used here, derives from the classical psychological work by Kahnemann, Tverskey, and Lovallo on planning and forecasting biases. In this model;

“[a]n inside view forecast is generated by focusing on the case at hand, by considering the plan and the obstacles to its completion, by constructing scenarios of future progress, and by extrapolating current trends. The outside view [...] essentially ignores the details of the case at hand, and involves no attempt at detailed forecasting of the future history of the project. Instead, it focuses on the statistics of a class of cases chosen to be similar in relevant respects to the present one. The case at hand is also compared to other members of the class, in an attempt to assess its position in the distribution of outcomes for the class.”

Kahneman, Daniel, and Dan Lovallo. ‘Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking’. Management Science 39, no. 1 (1993): 17–31. [https://econpapers.repec.org/article/
inmormnsc/v\_3a39\_3ay\_3a1993\_3ai\_3a1\_3ap\_3a17-31.htm](https://www.google.com/url?q=https://econpapers.repec.org/article/inmormnsc/v_3a39_3ay_3a1993_3ai_3a1_3ap_3a17-31.htm&sa=D&source=editors&ust=1692801435204951&usg=AOvVaw1ipTWYXsIuywHBJSPzDRtr). The insight that outside view forecasts can in some circumstances be more accurate than inside view evaluations underlies the methodology called ‘reference class forecasting’. In more recent work on ‘superforecasting’ of various events, there is an emphasis on first taking the outside view, and only then to modify the conclusion using the inside view. Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. Reprint edition. Broadway Books, 2016. For an application of these lessons to AI forecasting, see also: Kokotajlo, Daniel. ‘Evidence on Good Forecasting Practices from the Good Judgment Project: An Accompanying Blog Post’. AI Impacts, 7 February 2019.[https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/](https://www.google.com/url?q=https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/&sa=D&source=editors&ust=1692801435205579&usg=AOvVaw1XNKESs7i6MYiJqemS5KtT). For an argument that integrates inside- and outside view arguments on AI risk, see: Armstrong, Stuart. ‘Is AI an Existential Threat? We Don’t Know, and We Should Work on It’. York University, Toronto, 30 November 2020.[https://www.youtube.com/watch?
v=WLXuZtWoRcE](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DWLXuZtWoRcE&sa=D&source=editors&ust=1692801435205990&usg=AOvVaw2xTzak-RKxaOuL2kzgvq5b). For a discussion of the pitfalls of unreflexive appeals to ‘outside view’ evaluation, see: Kokotajlo, Daniel. ‘Taboo “Outside View”’. EA Forum, 17 June 2021.[https://forum.effectivealtruism.org/posts/wYpARcC4WqMsDEmYR/taboo-outside-view](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/wYpARcC4WqMsDEmYR/taboo-outside-view&sa=D&source=editors&ust=1692801435206425&usg=AOvVaw3L8CG3VyTJIKtP6wbjmwYT).

[[46]](#ftnt_ref46) MacAskill, William. ‘Are We Living at the Hinge of History?’ Global Priorities Institute, September 2020.[https://globalprioritiesinstitute.org/wp-content/uploads/William-MacAskill\_Are-
we-living-at-the-hinge-of-history.pdf](https://www.google.com/url?q=https://globalprioritiesinstitute.org/wp-content/uploads/William-MacAskill_Are-we-living-at-the-hinge-of-history.pdf&sa=D&source=editors&ust=1692801435207033&usg=AOvVaw0aKVuYf4aLrnX1XVlL9X55).

[[47]](#ftnt_ref47) Yudkowsky, Eliezer. ‘Intelligence Explosion Microeconomics’. Machine Intelligence Research Institute, 2013. [http://intelligence.org/files/IEM.pdf](https://www.google.com/url?q=http://intelligence.org/files/IEM.pdf&sa=D&source=editors&ust=1692801435207401&usg=AOvVaw13JspxmZ_RErWnOrg02mB4)

[[48]](#ftnt_ref48) Davidson, Tom. ‘Could Advanced AI Drive Explosive Economic Growth?’ Open Philanthropy Project, 8 April 2021.[https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-
economic-growth](https://www.google.com/url?q=https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth&sa=D&source=editors&ust=1692801435208048&usg=AOvVaw0W7unK6zfGJHIKQVXc9eNb).

[[49]](#ftnt_ref49) Roodman, David. ‘Modeling the Human Trajectory’. Open Philanthropy, 15 June 2020.[https://www.openphilanthropy.org/blog/modeling-human-trajectory](https://www.google.com/url?q=https://www.openphilanthropy.org/blog/modeling-human-trajectory&sa=D&source=editors&ust=1692801435208803&usg=AOvVaw0b3OklX8oKRrYR4N9ty56m). See also previously: AI Impacts. ‘Precedents for Economic N-Year Doubling before 4n-Year Doubling’. AI Impacts (blog), 14 April 2020.[https://aiimpacts.org/precedents-for-economic-n-year-doubling-before-4n-year-
doubling/](https://www.google.com/url?q=https://aiimpacts.org/precedents-for-economic-n-year-doubling-before-4n-year-doubling/&sa=D&source=editors&ust=1692801435209388&usg=AOvVaw1lxDGHGd8SreVsW7gs5y9V). This approach has also grounded some skeptical accounts: see for instance Thorstad, David. ‘Against the Singularity Hypothesis’. GPI Working Paper. Global Priorities Institute, University of Oxford, 1 November 2022.[https://globalprioritiesinstitute.org/against-the-
singularity-hypothesis-david-thorstad/](https://www.google.com/url?q=https://globalprioritiesinstitute.org/against-the-singularity-hypothesis-david-thorstad/&sa=D&source=editors&ust=1692801435209901&usg=AOvVaw2auiEesLpBnyGYr73ZtKbj). (arguing that the scenario rests on ‘implausible growth assumptions’).

[[50]](#ftnt_ref50) Roser, Max. ‘Technology over the Long Run: Zoom out to See How Dramatically the World Can Change within a Lifetime’. Our World in Data, 6 December 2022.[https://ourworldindata.org/
technology-long-run](https://www.google.com/url?q=https://ourworldindata.org/technology-long-run&sa=D&source=editors&ust=1692801435210422&usg=AOvVaw3qLYjMcGLcDzsGLNh5tSWb).

[[51]](#ftnt_ref51) Maas, Matthijs. ‘Paths Untaken: The History, Epistemology and Strategy of Technological Restraint, and Lessons for AI’. Verfassungsblog (blog), 9 August 2022.[https://verfassungsblog.de/
paths-untaken/](https://www.google.com/url?q=https://verfassungsblog.de/paths-untaken/&sa=D&source=editors&ust=1692801435210978&usg=AOvVaw07qB2zmORar8vW0ULGydMo).

[[52]](#ftnt_ref52) Barnett. ‘Three Reasons to Expect Long AI Timelines’. LessWrong, 22 April 2021.[https://www.lesswrong.com/posts/Z5gPrKTR2oDmm6fqJ/three-reasons-to-expect-long-ai-timelines](https://www.google.com/url?q=https://www.lesswrong.com/posts/Z5gPrKTR2oDmm6fqJ/three-reasons-to-expect-long-ai-timelines&sa=D&source=editors&ust=1692801435211582&usg=AOvVaw2QB6w3-vIwBuBxZy-ULEkz).; see also Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9
Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435212017&usg=AOvVaw2S0T4S9pYzNCuDEFSsVyOW). (pg. 76-80);

[[53]](#ftnt_ref53) Davidson, Tom. ‘Semi-Informative Priors over AI Timelines’. Open Philanthropy Project, 25 March 2021.[https://www.openphilanthropy.org/research/semi-informative-priors-over-ai-
timelines/](https://www.google.com/url?q=https://www.openphilanthropy.org/research/semi-informative-priors-over-ai-timelines/&sa=D&source=editors&ust=1692801435212600&usg=AOvVaw3WMag_GbzB4gbEZVxA0wY7).

[[54]](#ftnt_ref54) Roser, Max. ‘Artificial Intelligence Has Advanced despite Having Few Resources Dedicated to Its Development – Now Investments Have Increased Substantially’. Our World in Data, 6 December 2022.[https://ourworldindata.org/ai-investments](https://www.google.com/url?q=https://ourworldindata.org/ai-investments&sa=D&source=editors&ust=1692801435213130&usg=AOvVaw3Y0mgGzFXHqvuAdlNnnoyb).

[[55]](#ftnt_ref55) On the track record of past futurist predictions: see previously Lukeprog. ‘Futurism’s Track Record’. LessWrong, 2014.[https://www.lesswrong.com/posts/6ycPKhdmDgWsoyvKd/futurism-s-
track-record](https://www.google.com/url?q=https://www.lesswrong.com/posts/6ycPKhdmDgWsoyvKd/futurism-s-track-record&sa=D&source=editors&ust=1692801435213734&usg=AOvVaw3sndYElxNJSKAz8Ste48O8).; for an optimistic recent analysis, see: Karnofsky, Holden. ‘The Track Record of Futurists Seems ... Fine’. Cold Takes, 30 June 2022.[https://www.cold-takes.com/the-track-record-
of-futurists-seems-fine/](https://www.google.com/url?q=https://www.cold-takes.com/the-track-record-of-futurists-seems-fine/&sa=D&source=editors&ust=1692801435214230&usg=AOvVaw1_tqyl2wsX8Hk4O8f-fufF)., based off an analysis of predictions made by the ‘Big Three’ SF authors (Arthur C Clarke, Robert Heinlein, and Isaac Asimov); though for a critical response, see: Luu, Dan. ‘Futurist Prediction Methods and Accuracy’, 14 September 2022.[https://danluu.com/
futurist-predictions/](https://www.google.com/url?q=https://danluu.com/futurist-predictions/&sa=D&source=editors&ust=1692801435214609&usg=AOvVaw0EFwrnj2IqQHvVFyLFaYDh). (reviewing the track record of many other influential futurists, to argue that the predictive track record of many is quite bad, and that more recent longtermist analyses such as Karnofsky’s “fundamentally use the same techniques as the futurists analyses we looked at here and then add a few things on top that are also things that people who make accurate predictions do.”. For commentary see also Sempere, Nuño. ‘Forecasting Newsletter: September 2022.’ Substack newsletter. Forecasting (blog), 12 October 2022.[https://forecasting.substack.com/
p/forecasting-newsletter-september-57b](https://www.google.com/url?q=https://forecasting.substack.com/p/forecasting-newsletter-september-57b&sa=D&source=editors&ust=1692801435215118&usg=AOvVaw3NVysSAbZunVwbKzcxOtHv). For a specific discussion of the forecasting track record of some prominent contributors, with implications for estimates of AI risk, see also: Garfinkel, Benjamin. ‘On Deference and Yudkowsky’s AI Risk Estimates’. EA Forum, 19 June 2022.[https://forum.effectivealtruism.org/posts/NBgpPaz5vYe3tH4ga/on-deference-and-yudkowsky-s-ai-risk-estimates](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/NBgpPaz5vYe3tH4ga/on-deference-and-yudkowsky-s-ai-risk-estimates&sa=D&source=editors&ust=1692801435215616&usg=AOvVaw0pWvdLCcGQcMoIs3PTGCFr).

[[56]](#ftnt_ref56) For more discussion of epistemic pitfalls that may steer technology forecasting towards excessive conservatism, see also: Branwen, Gwern. ‘Complexity No Bar to AI’, 1 June 2014.[https://www.gwern.net/Complexity-vs-AI](https://www.google.com/url?q=https://www.gwern.net/Complexity-vs-AI&sa=D&source=editors&ust=1692801435216167&usg=AOvVaw1kXSSxouMrimaYkJeLPPDa). (Appendix: Technology Forecasting Errors: Functional Fixedness in Assuming Dependencies). See also Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435216623&usg=AOvVaw3_a-pwnwLzYCwLVjrMrUVz). (pg. 59-64); (reviewing cases of excessive optimism (Human Brain Project, ITER, DARPA’s 1983-1993 Strategic Computing Initiative…), as well as cases of sudden, discontinuous progress (Wright flyer, nuclear fission, nuclear bombs, penicillin), or progress that was previously held to be impossible (eg. Dreyfus’ rejection of the very possibility of web search, three years before Google’s 2004 IPO), and suggesting:

“it can be useful to consider the epistemic situation of society’s relation to various failures of technological prediction. Prima facie, in cases where predictions of a certain new technology repeatedly fail or are postponed, we would expect to see more high-profile and protracted scientific and public debates held over a longer period of time, than in cases where a predicted technology arrives more or less on schedule, or where an unexpected breakthrough occurred (where there may have been little public anticipation in the preceding years). If that is so, we would expect frustrated technological predictions to generally produce a bigger cultural footprint over a longer period of time, than do successful predictions or unexpected breakthroughs. This outsized footprint in turn may shape or skew our idea of technological prediction as being categorically over-optimistic”).

(pg. 63-64).

[[57]](#ftnt_ref57) On the track record of past AI predictions: Armstrong, Stuart, Kaj Sotala, and Seán S. Ó hÉigeartaigh. ‘The Errors, Insights and Lessons of Famous AI Predictions – and What They Mean for the Future’. Journal of Experimental & Theoretical Artificial Intelligence 26, no. 3 (3 July 2014): 317–42.[https://doi.org/10.1080/0952813X.2014.895105](https://www.google.com/url?q=https://doi.org/10.1080/0952813X.2014.895105&sa=D&source=editors&ust=1692801435217594&usg=AOvVaw0kGEQD2UpBmMxFK1I05NvO).; AI Impacts. ‘Accuracy of AI Predictions’. AI Impacts (blog), 4 June 2015.[https://aiimpacts.org/accuracy-of-ai-predictions/](https://www.google.com/url?q=https://aiimpacts.org/accuracy-of-ai-predictions/&sa=D&source=editors&ust=1692801435218049&usg=AOvVaw2XQCsLXFwNK1bHGdYlj00z).; AI Impacts. ‘Similar Predictions’. Accessed 16 August 2022.[http://www.aiimpacts.org/ai-timelines/
predictions-of-human-level-ai-dates/similar-predictions](https://www.google.com/url?q=http://www.aiimpacts.org/ai-timelines/predictions-of-human-level-ai-dates/similar-predictions&sa=D&source=editors&ust=1692801435218500&usg=AOvVaw2S0vVlmps7Rf-UQu_jkdqb).

[[58]](#ftnt_ref58) Muelhauser, Luke. ‘What Should We Learn from Past AI Forecasts?’ Open Philanthropy (blog), 1 May 2016.[https://www.openphilanthropy.org/research/what-should-we-learn-from-past-ai-
forecasts/](https://www.google.com/url?q=https://www.openphilanthropy.org/research/what-should-we-learn-from-past-ai-forecasts/&sa=D&source=editors&ust=1692801435219203&usg=AOvVaw0ipgUAGz847NR-XO2em-gQ).

[[59]](#ftnt_ref59) For influential surveys of AI experts, see: Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. ‘When Will AI Exceed Human Performance? Evidence from AI Experts’. Journal of Artificial Intelligence Research 62 (2018): 729–54. [http://arxiv.org/abs/1705.
08807](https://www.google.com/url?q=http://arxiv.org/abs/1705.08807&sa=D&source=editors&ust=1692801435219568&usg=AOvVaw1Zt9T8zJ36xWw8RKTrE3cU) See also: Michael, Julian, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, et al. ‘What Do NLP Researchers Believe? Results of the NLP Community Metasurvey’, 2022, 31. [https://nlpsurvey.net/nlp-metasurvey-results.pdf](https://www.google.com/url?q=https://nlpsurvey.net/nlp-metasurvey-results.pdf&sa=D&source=editors&ust=1692801435219795&usg=AOvVaw31Am6Hg-tYPh0OtvyKJdAt) ; Zhang, Baobao, Noemi Dreksler, Markus Anderljung, Lauren Kahn, Charlie Giattino, Allan Dafoe, and Michael C. Horowitz. ‘Forecasting AI Progress: Evidence from a Survey of Machine Learning Researchers’. arXiv, 8 June 2022.[https://doi.org/10.48550/arXiv.2206.04132](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2206.04132&sa=D&source=editors&ust=1692801435220173&usg=AOvVaw0vBvLimGUBEbSPkcqproJk).; Stein-Perlman, Zach, Benjamin Weinstein-Raun, and Katja Grace. ‘2022 Expert Survey on Progress in AI’. AI Impacts, 4 August 2022.[https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/](https://www.google.com/url?q=https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/&sa=D&source=editors&ust=1692801435220560&usg=AOvVaw3UIepdrp50Tx79JgwQ8mRx). See also: Gruetzemacher, Ross, David Paradice, and Kang Bok Lee. ‘Forecasting Extreme Labor Displacement: A Survey of AI Practitioners’. Technological Forecasting and Social Change, Forthcoming 2020, 43.; Gruetzemacher, Ross, David Paradice, and Kang Bok Lee. ‘Forecasting Transformative AI: An Expert Survey’, 16 July 2019.[http://arxiv.org/abs/1901.08579](https://www.google.com/url?q=http://arxiv.org/abs/1901.08579&sa=D&source=editors&ust=1692801435221004&usg=AOvVaw3LYqxKH1Upt5LrEUYae-TT).

For older surveys, see also Müller, Vincent C., and Nick Bostrom. ‘Future Progress in Artificial Intelligence: A Survey of Expert Opinion’. In Fundamental Issues of Artificial Intelligence, 555–72. Springer, 2016. [http://www.nickbostrom.com/papers/survey.pdf](https://www.google.com/url?q=http://www.nickbostrom.com/papers/survey.pdf&sa=D&source=editors&ust=1692801435221358&usg=AOvVaw1JHJ687fUCkrSCIku0mA4h); Baum, Seth D., Ben Goertzel, and Ted G. Goertzel. ‘How Long until Human-Level AI? Results from an Expert Assessment’. Technological Forecasting and Social Change 78, no. 1 (January 2011): 185–95.[https://doi.org/10.1016/j.techfore.2010.09.006](https://www.google.com/url?q=https://doi.org/10.1016/j.techfore.2010.09.006&sa=D&source=editors&ust=1692801435221835&usg=AOvVaw3mHTIN1Ak7O6xRNMpe84eK).

[[60]](#ftnt_ref60) Gruetzemacher, Ross, Florian E. Dorner, Niko Bernaola-Alvarez, Charlie Giattino, and David Manheim. ‘Forecasting AI Progress: A Research Agenda’. Technological Forecasting and Social Change 170 (1 September 2021): 120909.[https://doi.org/10.1016/j.techfore.2021.120909](https://www.google.com/url?q=https://doi.org/10.1016/j.techfore.2021.120909&sa=D&source=editors&ust=1692801435222346&usg=AOvVaw1rJSkv9ZeAJGfsNUjZoubG).

[[61]](#ftnt_ref61) For a general evaluation of when to expect generalist forecasters in prediction markets to beat domain experts, see: Leech, Gavin, and Mischa Yagudin. ‘Comparing Top Forecasters and Domain Experts’. Effective Altruism Forum, 6 March 2022.[https://forum.effectivealtruism.org/
posts/qZqvBLvR5hX9sEkjR/comparing-top-forecasters-and-domain-experts](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/qZqvBLvR5hX9sEkjR/comparing-top-forecasters-and-domain-experts&sa=D&source=editors&ust=1692801435222919&usg=AOvVaw0nthuPmX0xSi6MPGcO1nrK).

[[62]](#ftnt_ref62) See for instance the questions on the forecasting platform Metaculus: Aguirre, Anthony. ‘Will There Be Human-Machine Intelligence Parity Before 2040?’ Metaculus, 1 December 2016.[https://www.metaculus.com/questions/384/humanmachine-intelligence-parity-by-2040/](https://www.google.com/url?q=https://www.metaculus.com/questions/384/humanmachine-intelligence-parity-by-2040/&sa=D&source=editors&ust=1692801435223481&usg=AOvVaw3wxs_LSQ7nbuSSGFBfab8a).; Aguirre, Anthony. ‘When Will the First Weakly General AI System Be Devised, Tested, and Publicly Announced?’ Metaculus, 18 January 2020.[https://www.metaculus.com/questions/3479/date-
weakly-general-ai-is-publicly-known/](https://www.google.com/url?q=https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-publicly-known/&sa=D&source=editors&ust=1692801435223905&usg=AOvVaw1Qjz0NMl2VNSmgPVuM5prm).; Barnett, Matthew. ‘When Will the First General AI System Be Devised, Tested, and Publicly Announced?’ Metaculus, 23 August 2020.[https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/](https://www.google.com/url?q=https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/&sa=D&source=editors&ust=1692801435224411&usg=AOvVaw3D7Iy11REnLZg6JIa_Z7Fk). Though see a discussion of the Metaculus approach: Trazzi, Michaël. ‘Alex Lawsen On Forecasting AI Progress’. LessWrong, 6 September 2022.[https://www.lesswrong.com/posts/
pT86qTHDALskxCXsC/alex-lawsen-on-forecasting-ai-progress](https://www.google.com/url?q=https://www.lesswrong.com/posts/pT86qTHDALskxCXsC/alex-lawsen-on-forecasting-ai-progress&sa=D&source=editors&ust=1692801435224900&usg=AOvVaw2G8CcNKRInvEz5dYGia2Vp).

[[63]](#ftnt_ref63) Steinhardt, Jacob. ‘Updates and Lessons from AI Forecasting’. Bounded Regret, 18 August 2021.[https://bounded-regret.ghost.io/ai-forecasting/](https://www.google.com/url?q=https://bounded-regret.ghost.io/ai-forecasting/&sa=D&source=editors&ust=1692801435225417&usg=AOvVaw0ki1g-41OrOOpQf8RMUyDr). But for a critique of this approach, and how it scores, see: nostalgebraist. ‘On “Ai Forecasting: One Year In”’. Nostalgebraist (blog), 15 September 2022. [https://nostalgebraist.tumblr.com/post/695521414035406848/on-ai-forecasting-
one-year-in](https://www.google.com/url?q=https://nostalgebraist.tumblr.com/post/695521414035406848/on-ai-forecasting-one-year-in&sa=D&source=editors&ust=1692801435225716&usg=AOvVaw00vAG0NJonU-GrgYwjacks)

[[64]](#ftnt_ref64) Chow, Trevor, Basil Halperin, and J. Zachary Mazlish. ‘AGI and the EMH: Markets Are Not Expecting Aligned or Unaligned AI in the next 30 Years’. Effective Altruism Forum, 10 January 2023.[https://forum.effectivealtruism.org/posts/8c7LycgtkypkgYjZx/agi-and-the-emh-markets-are-
not-expecting-aligned-or](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/8c7LycgtkypkgYjZx/agi-and-the-emh-markets-are-not-expecting-aligned-or&sa=D&source=editors&ust=1692801435226280&usg=AOvVaw1_DaOnc4URz9fNMU4PIbbY).

[[65]](#ftnt_ref65) On the ‘Biological anchors’ approach to forecasting, see Cotra, Ajeya. ‘Forecasting TAI with Biological Anchors (Draft)’. Open Philanthropy Project, July 2020.[https://drive.google.com/drive/
folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP](https://www.google.com/url?q=https://drive.google.com/drive/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP&sa=D&source=editors&ust=1692801435226880&usg=AOvVaw29nJiqAjy5ubuKwozwRoMc). For summaries of the report, see: Alexander, Scott. ‘Biological Anchors: A Trick That Might Or Might Not Work’. Substack newsletter. Astral Codex Ten (blog), 23 February 2022.[https://astralcodexten.substack.com/p/biological-anchors-a-
trick-that-might](https://www.google.com/url?q=https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might&sa=D&source=editors&ust=1692801435227387&usg=AOvVaw2R3CCpbRxAqSKXd6aL_rau). For an overview of other summaries, critiques and responses, see also: Aldred, Will. ‘AI Timelines via Bioanchors: The Debate in One Place’. EA Forum, 31 July 2022.[https://forum.effectivealtruism.org/posts/NnygBgntvoGSuvsRH/ai-timelines-via-bioanchors-the-debate-in-one-place-1](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/NnygBgntvoGSuvsRH/ai-timelines-via-bioanchors-the-debate-in-one-place-1&sa=D&source=editors&ust=1692801435227866&usg=AOvVaw2Xfe1dAnpkxg8K1cUHauvL). See also reviews, including: Lin, Jennifer. ‘Biological Anchors External Review’. Google Docs, 2022.[https://docs.google.com/document/d/1\_GqOrCo29qKly1z48-mR86IV7
TUDfzaEXxD3lGFQ8Wk/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1_GqOrCo29qKly1z48-mR86IV7TUDfzaEXxD3lGFQ8Wk/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435228313&usg=AOvVaw0uLE_3oKIig-CibINZDWFY); Hobbhahn, Marius. ‘Disagreement with Bio Anchors That Lead to Shorter Timelines’. Effective Altruism Forum, 16 November 2022.[https://forum.effectivealtruism.org/posts/gWSa7e2CS7KCu78D8/disagreement-with-bio-anchors-that-lead-to-shorter-timelines](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/gWSa7e2CS7KCu78D8/disagreement-with-bio-anchors-that-lead-to-shorter-timelines&sa=D&source=editors&ust=1692801435228801&usg=AOvVaw30hd6enLzP1EfW-ZqrVguE).

[[66]](#ftnt_ref66) Cotra’s report draws, in part, on Carlsmith, Joseph. ‘How Much Computational Power Does It Take to Match the Human Brain?’ Open Philanthropy Project, 11 September 2020.[https://www.openphilanthropy.org/research/how-much-computational-power-does-it-take-to-match-the-human-brain/](https://www.google.com/url?q=https://www.openphilanthropy.org/research/how-much-computational-power-does-it-take-to-match-the-human-brain/&sa=D&source=editors&ust=1692801435229451&usg=AOvVaw0yNRgk7qjz9J5S_-Ehf7Jh). For an older account of human-level hardware, see Grace, Katja. ‘Human-Level Hardware Timeline’. AI Impacts (blog), 22 December 2017.[https://aiimpacts.org/human-level-hardware-timeline/](https://www.google.com/url?q=https://aiimpacts.org/human-level-hardware-timeline/&sa=D&source=editors&ust=1692801435229953&usg=AOvVaw2BRnkAj9gL5Sg878wiY-Yc).

[[67]](#ftnt_ref67) Bowman, Samuel R. ‘Eight Things to Know about Large Language Models’, 2023.[https://cims.nyu.edu/~sbowman/eightthings.pdf](https://www.google.com/url?q=https://cims.nyu.edu/~sbowman/eightthings.pdf&sa=D&source=editors&ust=1692801435437293&usg=AOvVaw32PfpyhgvafSi1lxn3grvd).

[[68]](#ftnt_ref68) Villalobos, Pablo, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. ‘Will We Run out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning’. arXiv, 25 October 2022.[https://doi.org/10.48550/arXiv.2211.04325](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2211.04325&sa=D&source=editors&ust=1692801435415279&usg=AOvVaw1O5ifszaQ2dkVutxwUc8_3).; see also dynomight. ‘First Principles on AI Progress’. Substack newsletter. DYNOMIGHT INTERNET NEWSLETTER, 6 March 2023.[https://dynomight.substack.com/p/scaling?publication\_id=327510](https://www.google.com/url?q=https://dynomight.substack.com/p/scaling?publication_id%3D327510&sa=D&source=editors&ust=1692801435415707&usg=AOvVaw1_ZujOghI8Oo5e0CHYnyI9).

[[69]](#ftnt_ref69) There are too many to discuss in detail; but see informally: Cotra, Ajeya. ‘Two-Year Update on My Personal AI Timelines’. AI Alignment Forum, 3 August 2022.[https://www.alignmentforum.org/posts/AfH2oPHCApdKicM4m/two-year-update-on-my-personal-ai-timelines](https://www.google.com/url?q=https://www.alignmentforum.org/posts/AfH2oPHCApdKicM4m/two-year-update-on-my-personal-ai-timelines&sa=D&source=editors&ust=1692801435230542&usg=AOvVaw3-ruKtY9mduDVxZqarOEBn).; Alexander, Scott. ‘Somewhat Contra Marcus On AI Scaling’. Substack newsletter. Astral Codex Ten (blog), 10 June 2022.[https://astralcodexten.substack.com/p/somewhat-contra-
marcus-on-ai-scaling](https://www.google.com/url?q=https://astralcodexten.substack.com/p/somewhat-contra-marcus-on-ai-scaling&sa=D&source=editors&ust=1692801435231017&usg=AOvVaw0pHxxa7csTBkFoxx69l-Lz). Marcus, Gary. ‘Does AI Really Need a Paradigm Shift?’ Substack newsletter. The Road to AI We Can Trust (blog), 11 June 2022.[https://garymarcus.substack.com/
p/does-ai-really-need-a-paradigm-shift](https://www.google.com/url?q=https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift&sa=D&source=editors&ust=1692801435231498&usg=AOvVaw1tAfpnCq7qc6JuW8pEyJiy).

[[70]](#ftnt_ref70) Burden, John, and Jose Hernandez-Orallo. ‘Exploring AI Safety in Degrees: Generality, Capability and Control’, In Proceedings of the Workshop on Artificial Intelligence Safety (SafeAI 2020), 2020, 5. [http://ceur-ws.org/Vol-2560/paper21.pdf](https://www.google.com/url?q=http://ceur-ws.org/Vol-2560/paper21.pdf&sa=D&source=editors&ust=1692801435231904&usg=AOvVaw0EyqflbX51HSiQAFf-Uckv) ; Casares, Pablo Antonio Moreno, Bao Sheng Loe, John Burden, Sean hEigeartaigh, and José Hernández-Orallo. ‘How General-Purpose Is a Language Model? Usefulness and Safety with Human Prompters in the Wild’. Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 5 (28 June 2022): 5295–5303.[https://doi.org/10.1609/aaai.v36i5.20466](https://www.google.com/url?q=https://doi.org/10.1609/aaai.v36i5.20466&sa=D&source=editors&ust=1692801435232447&usg=AOvVaw31uuWfydobpwzkWE9CFniV).; and see: Hernández-Orallo, José, Bao Sheng Loe, Lucy Cheke, Fernando Martínez-Plumed, and Seán Ó hÉigeartaigh. ‘General Intelligence Disentangled via a Generality Metric for Natural and Artificial Intelligence’. Scientific Reports 11, no. 1 (24 November 2021): 22822.[https://doi.org/10.1038/s41598-021-01997-7](https://www.google.com/url?q=https://doi.org/10.1038/s41598-021-01997-7&sa=D&source=editors&ust=1692801435232982&usg=AOvVaw1Y4z-gI9H8lqOcDZ7Kzxeh).

[[71]](#ftnt_ref71) On the problem of eliciting expert judgment in conditions where there may not be a relevant reference class of experts to draw from, see generally Morgan, M. Granger. ‘Use (and Abuse) of Expert Elicitation in Support of Decision Making for Public Policy’. Proceedings of the National Academy of Sciences 111, no. 20 (20 May 2014): 7176–84.[https://doi.org/10.1073/pnas.1319946111](https://www.google.com/url?q=https://doi.org/10.1073/pnas.1319946111&sa=D&source=editors&ust=1692801435233532&usg=AOvVaw02U0VBifBEfz-zCOr9IrtY).

[[72]](#ftnt_ref72) Landau-Taylor, Ben. ‘Against AGI Timelines’. Ben Landau-Taylor (blog), 12 March 2023.[https://benlandautaylor.com/2023/03/12/against-agi-timelines/](https://www.google.com/url?q=https://benlandautaylor.com/2023/03/12/against-agi-timelines/&sa=D&source=editors&ust=1692801435417077&usg=AOvVaw2nX1_nxTcZr7w1z4PryCUL).

[[73]](#ftnt_ref73) nostalgebraist. ‘On “AI Forecasting: One Year In”’. Nostalgebraist (blog), 15 September 2022.[https://nostalgebraist.tumblr.com/post/695521414035406848/on-ai-forecasting-one-year-in](https://www.google.com/url?q=https://nostalgebraist.tumblr.com/post/695521414035406848/on-ai-forecasting-one-year-in&sa=D&source=editors&ust=1692801435234165&usg=AOvVaw00y_FZjx-id3Y4laF2kME5).

[[74]](#ftnt_ref74) Clarke, Sam. ‘When Reporting AI Timelines, Be Clear Who You’re (Not) Deferring To’. EA Forum, 10 October 2022.[https://forum.effectivealtruism.org/posts/FtggfJ2oxNSN8Niix/when-reporting-ai-timelines-be-clear-who-you-re-not](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/FtggfJ2oxNSN8Niix/when-reporting-ai-timelines-be-clear-who-you-re-not&sa=D&source=editors&ust=1692801435234724&usg=AOvVaw3qYnSO57gj7UQhpiSgmzZ-).

[[75]](#ftnt_ref75) Clarke, Sam, and mccaffary. ‘Deference on AI Timelines: Survey Results’. EA Forum, 31 March 2023.[https://forum.effectivealtruism.org/posts/BGFbwca4nfagvB9Xb/deference-on-ai-timelines-
survey-results](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/BGFbwca4nfagvB9Xb/deference-on-ai-timelines-survey-results&sa=D&source=editors&ust=1692801435431357&usg=AOvVaw3gs7etFgPc02LBW1ZhP5HN).

[[76]](#ftnt_ref76) Brundage, Miles. ‘Why AGI Timeline Research/Discourse Might Be Overrated’. EA Forum, 2022.[https://forum.effectivealtruism.org/posts/SEqJoRL5Y8cypFasr/why-agi-timeline-research-discourse-might-be-overrated](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/SEqJoRL5Y8cypFasr/why-agi-timeline-research-discourse-might-be-overrated&sa=D&source=editors&ust=1692801435235259&usg=AOvVaw2pD4HHE2h6VjITIDUwAbV2).

[[77]](#ftnt_ref77) Campos, Simon. ‘AGI Timelines in Governance: Different Strategies for Different Timeframes’. EA Forum, 19 December 2022.[https://forum.effectivealtruism.org/posts/Pt7MxstXxXHak4wkt/
agi-timelines-in-governance-different-strategies-for](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Pt7MxstXxXHak4wkt/agi-timelines-in-governance-different-strategies-for&sa=D&source=editors&ust=1692801435235798&usg=AOvVaw2OkC0yLysnNFzNo6gj4OL_).

[[78]](#ftnt_ref78) For a discussion of the term, see: Barnett, Matthew. ‘Distinguishing Definitions of Takeoff’. AI Alignment Forum, 14 February 2020.[https://www.alignmentforum.org/posts/
YgNYA6pj2hPSDQiTE/distinguishing-definitions-of-takeoff](https://www.google.com/url?q=https://www.alignmentforum.org/posts/YgNYA6pj2hPSDQiTE/distinguishing-definitions-of-takeoff&sa=D&source=editors&ust=1692801435236436&usg=AOvVaw3Nd4D__0bYjNZBT7swqNkR).

See also Alexander, Scott. ‘Yudkowsky Contra Christiano On AI Takeoff Speeds’. Substack newsletter. Astral Codex Ten (blog), 4 April 2022.[https://astralcodexten.substack.com/p/
yudkowsky-contra-christiano-on-ai](https://www.google.com/url?q=https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai&sa=D&source=editors&ust=1692801435236968&usg=AOvVaw0TX8Q_MlGNUwjUr3kXwVlC). For an older account, see: Hanson, Robin, and Eliezer Yudkowsky. ‘The Hanson-Yudkowsky AI-Foom Debate’, 2008, 741. [https://intelligence.org/files/
AIFoomDebate.pdf](https://www.google.com/url?q=https://intelligence.org/files/AIFoomDebate.pdf&sa=D&source=editors&ust=1692801435237166&usg=AOvVaw2ZPaaQnpkkcS6XdJzQTgMw)

[[79]](#ftnt_ref79) Muelhauser, Luke. ‘Intelligence Explosion FAQ’. Machine Intelligence Research Institute, 2013.[https://intelligence.org/ie-faq/](https://www.google.com/url?q=https://intelligence.org/ie-faq/&sa=D&source=editors&ust=1692801435237642&usg=AOvVaw2oxFjpqzIo3TgCNNTsnKjG). and many others. For a recent argument based on a compute-centric framework, see: Davidson, Tom. ‘What a Compute-Centric Framework Says about AI Takeoff Speeds - Draft Report’. EA Forum, 23 January 2023.[https://forum.effectivealtruism.org/posts/3vDarp6adLPBTux5g/what-a-compute-centric-framework-says-about-ai-takeoff](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/3vDarp6adLPBTux5g/what-a-compute-centric-framework-says-about-ai-takeoff&sa=D&source=editors&ust=1692801435238142&usg=AOvVaw3AdBcWZixeDZUS5m1CtB48).

[[80]](#ftnt_ref80) Christiano, Paul. ‘Takeoff Speeds’. The Sideways View (blog), 24 February 2018.[https://sideways-view.com/2018/02/24/takeoff-speeds/](https://www.google.com/url?q=https://sideways-view.com/2018/02/24/takeoff-speeds/&sa=D&source=editors&ust=1692801435238695&usg=AOvVaw1awgn2uM1K4Dws1AsI8ZvB). Grace, Katja. ‘Likelihood of Discontinuous Progress around the Development of AGI’. AI Impacts (blog), 23 February 2018.[https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/](https://www.google.com/url?q=https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/&sa=D&source=editors&ust=1692801435239155&usg=AOvVaw0L1sFCzoiVgdgEY6urD-lW).

[[81]](#ftnt_ref81) AI Impacts. ‘Cases of Discontinuous Technological Progress’. AI Impacts (blog), 31 December 2014.[https://aiimpacts.org/cases-of-discontinuous-technological-progress/](https://www.google.com/url?q=https://aiimpacts.org/cases-of-discontinuous-technological-progress/&sa=D&source=editors&ust=1692801435239816&usg=AOvVaw14B6ADAe7iLP6kMQn6s6XL). Grace, Katja. ‘Discontinuous Progress in History: An Update’. AI Impacts, 13 April 2020.[https://aiimpacts.org/discontinuous-progress-in-history-an-update/](https://www.google.com/url?q=https://aiimpacts.org/discontinuous-progress-in-history-an-update/&sa=D&source=editors&ust=1692801435240209&usg=AOvVaw3NhurWc3d2973ViFnHawh0). See also Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435240746&usg=AOvVaw0raOi78K0MXbS8w3ZH9trM). (pg. 59-64).

[[82]](#ftnt_ref82) Cremer, Carla Zoe, and Jess Whittlestone. ‘Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI’. International Journal of Interactive Multimedia and Artificial Intelligence 6, no. 5 (2021): 100–109. [https://www.ijimai.org/journal/sites/default/files/2021-02/ijimai\_6\_5\_10.pdf](https://www.google.com/url?q=https://www.ijimai.org/journal/sites/default/files/2021-02/ijimai_6_5_10.pdf&sa=D&source=editors&ust=1692801435241313&usg=AOvVaw16yRy2zc2p2FAWMaZI91Sx)

[[83]](#ftnt_ref83) Yudkowsky, Eliezer. ‘There’s No Fire Alarm for Artificial General Intelligence’. Machine Intelligence Research Institute (blog), 14 October 2017.[https://intelligence.org/2017/10/13/fire-alarm/](https://www.google.com/url?q=https://intelligence.org/2017/10/13/fire-alarm/&sa=D&source=editors&ust=1692801435241957&usg=AOvVaw2nfh0-K760iN8JObYd6X_J). But see also: Grace, Katja. ‘Beyond Fire Alarms: Freeing the Groupstruck’. AI Impacts, 26 September 2021.[https://aiimpacts.org/beyond-fire-alarms-freeing-the-groupstruck/](https://www.google.com/url?q=https://aiimpacts.org/beyond-fire-alarms-freeing-the-groupstruck/&sa=D&source=editors&ust=1692801435242364&usg=AOvVaw0ZfUIcIXLFPM1dHbDJc4Ha).

[[84]](#ftnt_ref84) ‘Threat model’ can be defined as: “[a] [c]ombination of a development model that says how we get AGI and a risk model that says how AGI leads to existential catastrophe.”; Shah, Rohin. ‘The Importance of Threat Models for AI Alignment’. 16 February 2021.[https://www.youtube.com/watch?v=VC\_J\_skJNMs](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DVC_J_skJNMs&sa=D&source=editors&ust=1692801435361900&usg=AOvVaw0MdaAjEbbl08mBIRoyBJ-y).

[[85]](#ftnt_ref85) ‘Takeover’ has been defined by Holden Karnofsky as “AI systems disempowering humans entirely, leading to a future that has little to do with anything humans value”; Karnofsky, Holden. ‘AI Could Defeat All Of Us Combined’. Cold Takes, 9 June 2022.[https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/](https://www.google.com/url?q=https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/&sa=D&source=editors&ust=1692801435362504&usg=AOvVaw0Ul-dhiy5x7ZvCqtp46h-7).; and by Ajeya Cotra as “a possibly violent uprising or coup by AI systems’; Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801435362958&usg=AOvVaw0QQlp1VRskM7ERFRwNDCsi).

[[86]](#ftnt_ref86) Ngo, Richard. ‘AGI Safety From First Principles’, 2020.[https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ](https://www.google.com/url?q=https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ&sa=D&source=editors&ust=1692801435079837&usg=AOvVaw0PuCohv1uj4kaJ8YzxSEzw).

[[87]](#ftnt_ref87) Leech, Gavin. ‘Why Worry about Future AI?’ Argmin gravitas, 21 March 2021.[https://www.gleech.org/ai-risk](https://www.google.com/url?q=https://www.gleech.org/ai-risk&sa=D&source=editors&ust=1692801435080266&usg=AOvVaw2LUpCXWS4ieD-elWGqWrot).

[[88]](#ftnt_ref88) Kokotajlo, Daniel. ‘Cortés, Pizarro, and Afonso as Precedents for Takeover’. AI Alignment Forum, 1 March 2020.[https://www.alignmentforum.org/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover](https://www.google.com/url?q=https://www.alignmentforum.org/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover&sa=D&source=editors&ust=1692801435080815&usg=AOvVaw1tWnJOpxJwTk_vG1ZE-82Z).

[[89]](#ftnt_ref89) Clarke, Sam. ‘Clarifying “What Failure Looks like” (Part 1)’. AI Alignment Forum, 20 September 2020.[https://www.alignmentforum.org/posts/v6Q7T335KCMxujhZu/clarifying-what-failure-looks-like-part-1](https://www.google.com/url?q=https://www.alignmentforum.org/posts/v6Q7T335KCMxujhZu/clarifying-what-failure-looks-like-part-1&sa=D&source=editors&ust=1692801435081354&usg=AOvVaw2gcWkflMH1CNpQjyN5mR9C).

[[90]](#ftnt_ref90) Carlier, Alexis. ‘What Can the Principal-Agent Literature Tell Us about AI Risk?’ AI Alignment Forum, 8 February 2020.[https://www.alignmentforum.org/posts/Z5ZBPEgufmDsm7LAv/what-can-the-principal-agent-literature-tell-us-about-ai](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Z5ZBPEgufmDsm7LAv/what-can-the-principal-agent-literature-tell-us-about-ai&sa=D&source=editors&ust=1692801435081922&usg=AOvVaw2S33ZhqC1MutR-PGLm1HHM).

[[91]](#ftnt_ref91) Hadfield-Menell, Dylan, and Gillian Hadfield. ‘Incomplete Contracting and AI Alignment’. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 2019.[http://arxiv.org/abs/1804.04268](https://www.google.com/url?q=http://arxiv.org/abs/1804.04268&sa=D&source=editors&ust=1692801435082395&usg=AOvVaw2A5P0YSo5JyxXJHBOWDg7X).

[[92]](#ftnt_ref92) Hadshar, Rose, and particlemania. ‘The Economy as an Analogy for Advanced AI Systems’. AI Alignment Forum, 15 November 2022.[https://www.alignmentforum.org/posts/oH3XmScSFnZt6x2eN/the-economy-as-an-analogy-for-advanced-ai-systems-2](https://www.google.com/url?q=https://www.alignmentforum.org/posts/oH3XmScSFnZt6x2eN/the-economy-as-an-analogy-for-advanced-ai-systems-2&sa=D&source=editors&ust=1692801435099338&usg=AOvVaw13F9JFfoG09_mAvjWt1vF8).

[[93]](#ftnt_ref93) Danzig, Richard. ‘Machines, Bureaucracies, and Markets as Artificial Intelligences’. Center for Security and Emerging Technology, January 2022.[https://cset.georgetown.edu/publication/machines-bureaucracies-and-markets-as-artificial-intelligences/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/machines-bureaucracies-and-markets-as-artificial-intelligences/&sa=D&source=editors&ust=1692801435149598&usg=AOvVaw0XE9cHuOP1t5l9iGapOkFj).

[[94]](#ftnt_ref94) Manheim, David, and Scott Garrabrant. ‘Categorizing Variants of Goodhart’s Law’. ArXiv:1803.04585 [Cs, q-Fin, Stat], 12 March 2018.[http://arxiv.org/abs/1803.04585](https://www.google.com/url?q=http://arxiv.org/abs/1803.04585&sa=D&source=editors&ust=1692801435082863&usg=AOvVaw39Vd1o8L_ywrZfhCPdlTpE).

[[95]](#ftnt_ref95) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022. [https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435090693&usg=AOvVaw3h063DEUqxlZ2ThzgpPgNT)

[[96]](#ftnt_ref96) Tim G. J. Rduner, and Helen Toner. ‘Key Concepts in AI Safety: Specification in Machine Learning’. Center for Security and Emerging Technology, December 2021.[https://doi.org/10.51593/20210031](https://www.google.com/url?q=https://doi.org/10.51593/20210031&sa=D&source=editors&ust=1692801435201609&usg=AOvVaw2Lf8a0P8oJrf4W0seD_d_G). Pg. 3 (“for example, a captive dolphin in Mississippi, upon learning it would be rewarded for bringing trash to its handler, was observed stowing trash in a corner of its habitat and tearing off small pieces to maximize the number of fish it could “earn.” Like humans and animals, machines respond to the incentives presented to them”).

[[97]](#ftnt_ref97) Yudkowsky, Eliezer. ‘AI Alignment: Why It’s Hard, and Where to Start’. Machine Intelligence Research Institute, 28 December 2016.[https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/](https://www.google.com/url?q=https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/&sa=D&source=editors&ust=1692801435083372&usg=AOvVaw2_QY2r5eONBDBqA74ujZq4).

[[98]](#ftnt_ref98) Conn, Ariel. ‘Artificial Intelligence and the King Midas Problem’. Future of Life Institute (blog), 12 December 2016.[https://futureoflife.org/ai/artificial-intelligence-king-midas-problem/](https://www.google.com/url?q=https://futureoflife.org/ai/artificial-intelligence-king-midas-problem/&sa=D&source=editors&ust=1692801435131179&usg=AOvVaw2r4DbzyBx1WNZlBvEiFMwT).; Russell, Stuart. ‘Of Myths and Moonshine’. Reality Club Conversation: The Myth of AI, 2014.[https://www.edge.org/conversation/the-myth-of-ai#26015](https://www.google.com/url?q=https://www.edge.org/conversation/the-myth-of-ai%2326015&sa=D&source=editors&ust=1692801435131647&usg=AOvVaw0p-SRKx9VsS42yehL2mGM3).

[[99]](#ftnt_ref99) Bostrom, Nick. ‘Ethical Issues in Advanced Artificial Intelligence’. In Machine Ethics and Robot Ethics, by Wendell Wallach and Peter Asaro, 69–75. edited by Wendell Wallach and Peter Asaro, 1st ed. Routledge, 2003.[https://doi.org/10.4324/9781003074991-7](https://www.google.com/url?q=https://doi.org/10.4324/9781003074991-7&sa=D&source=editors&ust=1692801435130502&usg=AOvVaw1pEVKJaP0kx57N6GuP3rJz).

[[100]](#ftnt_ref100) Ngo, Richard. ‘The Alignment Problem from a Deep Learning Perspective’. arXiv, 29 August 2022.[https://doi.org/10.48550/arXiv.2209.00626](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2209.00626&sa=D&source=editors&ust=1692801435083827&usg=AOvVaw1qQd7__KeeD094RthXLV6O). Cotra, Ajeya. ‘Why AI Alignment Could Be Hard with Modern Deep Learning’. Cold Takes, 21 September 2021.[https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/](https://www.google.com/url?q=https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/&sa=D&source=editors&ust=1692801435084297&usg=AOvVaw1D8qiuOvhGSb-hlTXJUGYJ).; see also the ongoing project: Dewey, Daniel. ‘Global Risk from Deep Learning’. Accessed 1 December 2022.[https://www.danieldewey.net/risk/index.html](https://www.google.com/url?q=https://www.danieldewey.net/risk/index.html&sa=D&source=editors&ust=1692801435084719&usg=AOvVaw0lYlH9Xw3xwf-qWrhX15RF).

[[101]](#ftnt_ref101) Amodei, Dario, and Jack Clark. ‘Faulty Reward Functions in the Wild’. OpenAI (blog), 2016.[https://openai.com/blog/faulty-reward-functions/](https://www.google.com/url?q=https://openai.com/blog/faulty-reward-functions/&sa=D&source=editors&ust=1692801435085363&usg=AOvVaw1oW2dq8XLaRb9Tumy6uI04).

[[102]](#ftnt_ref102) Krakovna, Victoria, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zachary Kenton, Jan Leike, and Shane Legg. ‘Specification gaming: the flip side of AI ingenuity’. Deepmind (blog), 21 April 2020.[https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity](https://www.google.com/url?q=https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity&sa=D&source=editors&ust=1692801435086167&usg=AOvVaw1sjXf9Y1t-eFbKyEvngUlp). See also Krakovna, Victoria. ‘Specification Gaming Examples in AI’. Deep Safety (blog), 1 April 2018.[https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/](https://www.google.com/url?q=https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/&sa=D&source=editors&ust=1692801435086718&usg=AOvVaw2IfjjOgJwBrPDrNjkrBazx).

[[103]](#ftnt_ref103) Gao, Leo, John Schulman, and Jacob Hilton. ‘Scaling Laws for Reward Model Overoptimization’. arXiv, 19 October 2022.[https://doi.org/10.48550/arXiv.2210.10760](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2210.10760&sa=D&source=editors&ust=1692801435087264&usg=AOvVaw2Yn4LSIEp_4drCXlffaUVN).

[[104]](#ftnt_ref104) Turner, Alexander Matt, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. ‘Optimal Policies Tend to Seek Power’. ArXiv:1912.01683 [Cs], 3 December 2021.[http://arxiv.org/abs/1912.01683](https://www.google.com/url?q=http://arxiv.org/abs/1912.01683&sa=D&source=editors&ust=1692801435087939&usg=AOvVaw23KxQpDUj7Zw56n5bF93Dm). See also: Turner, Alexander Matt, and Prasad Tadepalli. ‘Parametrically Retargetable Decision-Makers Tend To Seek Power’. arXiv, 11 October 2022.[https://doi.org/10.48550/arXiv.2206.13477](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2206.13477&sa=D&source=editors&ust=1692801435088343&usg=AOvVaw3SxAAOROv0l2WVvamK4AyQ).

[[105]](#ftnt_ref105) Langosco, Lauro, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger. ‘Goal Misgeneralization in Deep Reinforcement Learning’. arXiv, 7 September 2022.[https://doi.org/10.48550/arXiv.2105.14111](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2105.14111&sa=D&source=editors&ust=1692801435088819&usg=AOvVaw2Oho4xNT9M3azspcugeLYG).; Shah, Rohin, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. ‘Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals’. arXiv, 2 November 2022.[https://doi.org/10.48550/arXiv.2210.01790](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2210.01790&sa=D&source=editors&ust=1692801435089214&usg=AOvVaw19fwmVyqUlKddldX-AhQAL).

[[106]](#ftnt_ref106) Kenton, Zachary, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. ‘Alignment of Language Agents’. ArXiv:2103.14659 [Cs], 26 March 2021.[http://arxiv.org/abs/2103.14659](https://www.google.com/url?q=http://arxiv.org/abs/2103.14659&sa=D&source=editors&ust=1692801435089793&usg=AOvVaw1jkYs2iqv7Hc4UcC_v5QWi).

[[107]](#ftnt_ref107) Hendrycks, Dan, Nicholas Carlini, John Schulman, and Jacob Steinhardt. ‘Unsolved Problems in ML Safety’. ArXiv:2109.13916 [Cs], 28 September 2021.[http://arxiv.org/abs/2109.13916](https://www.google.com/url?q=http://arxiv.org/abs/2109.13916&sa=D&source=editors&ust=1692801435090274&usg=AOvVaw2gzFTLjnh2sdjckzY5K8tZ).

[[108]](#ftnt_ref108) Chan, Alan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, et al. ‘Harms from Increasingly Agentic Algorithmic Systems’. arXiv, 20 February 2023.[https://doi.org/10.48550/arXiv.2302.10329](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2302.10329&sa=D&source=editors&ust=1692801435414749&usg=AOvVaw07PTIfsER9Ybguf1dKn-dR).

[[109]](#ftnt_ref109) Hendrycks, Dan. ‘Natural Selection Favors AIs over Humans’. arXiv, 28 March 2023.[https://doi.org/10.48550/arXiv.2303.16200](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2303.16200&sa=D&source=editors&ust=1692801435432365&usg=AOvVaw0K5PLEY7Yt9s69IKLuOlup).

[[110]](#ftnt_ref110) This taxonomy draws loosely on: Clarke, Sam. ‘Classifying Sources of AI X-Risk’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/e55QpEExmtkRjw9CD/classifying-sources-of-ai-x-risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/e55QpEExmtkRjw9CD/classifying-sources-of-ai-x-risk&sa=D&source=editors&ust=1692801434957858&usg=AOvVaw3zmGWsTlSg2HMtTDtxynC5).

[[111]](#ftnt_ref111) Barrett, Anthony M., and Seth D. Baum. ‘A Model of Pathways to Artificial Superintelligence Catastrophe for Risk and Decision Analysis’. Journal of Experimental & Theoretical Artificial Intelligence 29, no. 2 (4 March 2017): 397–414.[https://doi.org/10.1080/0952813X.2016.1186228](https://www.google.com/url?q=https://doi.org/10.1080/0952813X.2016.1186228&sa=D&source=editors&ust=1692801435135166&usg=AOvVaw0ew33bcsPDxblfBoVjLbSg).

[[112]](#ftnt_ref112) Clarke, Sam, Ben Cottier, Aryeh Englander, Daniel Eth, David Manheim, Samuel Dylan Martin, and Issa Rice. ‘Modeling Transformative AI Risks (MTAIR) Project -- Summary Report’. arXiv, 19 June 2022.[https://doi.org/10.48550/arXiv.2206.09360](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2206.09360&sa=D&source=editors&ust=1692801434957145&usg=AOvVaw1bgZQdD3opz2Jar9184wE-).

[[113]](#ftnt_ref113) Clarke, Sam, and Samuel Dylan Martin. ‘Distinguishing AI Takeover Scenarios’. AI Alignment Forum, 8 September 2021.[https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios](https://www.google.com/url?q=https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios&sa=D&source=editors&ust=1692801434964212&usg=AOvVaw1-oxCClm9woyPdh1h5-VVJ).

[[114]](#ftnt_ref114) Martin, Samuel Dylan. ‘Investigating AI Takeover Scenarios’. AI Alignment Forum, 17 September 2021.[https://www.alignmentforum.org/posts/zkF9PNSyDKusoyLkP/investigating-ai-takeover-scenarios](https://www.google.com/url?q=https://www.alignmentforum.org/posts/zkF9PNSyDKusoyLkP/investigating-ai-takeover-scenarios&sa=D&source=editors&ust=1692801435387095&usg=AOvVaw2fkjDsCiaK73CLn7Ch9qft).

[[115]](#ftnt_ref115) Clarke, Sam. ‘Classifying Sources of AI X-Risk’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/e55QpEExmtkRjw9CD/classifying-sources-of-ai-x-risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/e55QpEExmtkRjw9CD/classifying-sources-of-ai-x-risk&sa=D&source=editors&ust=1692801434964902&usg=AOvVaw2ERgTOVeWxXwpLzPvY2N_C).

[[116]](#ftnt_ref116) Vold, Karina, and Daniel R. Harris. ‘How Does Artificial Intelligence Pose an Existential Risk?’ In The Oxford Handbook of Digital Ethics, 2021.[https://doi.org/10.1093/oxfordhb/9780198857815.013.36](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780198857815.013.36&sa=D&source=editors&ust=1692801434965564&usg=AOvVaw305L3EFeeLhlf1GZHLso6O).

[[117]](#ftnt_ref117) Ngo, Richard. ‘Disentangling Arguments for the Importance of AI Safety’. AI Alignment Forum, 2019.[https://www.alignmentforum.org/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety](https://www.google.com/url?q=https://www.alignmentforum.org/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety&sa=D&source=editors&ust=1692801434962693&usg=AOvVaw0LohetECedZMvul8qYONhQ).

[[118]](#ftnt_ref118) Grace, Katja. ‘List of Sources Arguing for Existential Risk from AI’. AI Impacts, 6 August 2022.[https://aiimpacts.org/list-of-sources-arguing-for-existential-risk-from-ai/](https://www.google.com/url?q=https://aiimpacts.org/list-of-sources-arguing-for-existential-risk-from-ai/&sa=D&source=editors&ust=1692801434963333&usg=AOvVaw21lcHoPVKQxqfqx-VfOt9A).

[[119]](#ftnt_ref119) Nanda, Neel. ‘My Overview of the AI Alignment Landscape: Threat Models’. Alignment Forum, 26 December 2021.[https://www.alignmentforum.org/posts/3DFBbPFZyscrAiTKS/my-overview-of-the-ai-alignment-landscape-threat-models](https://www.google.com/url?q=https://www.alignmentforum.org/posts/3DFBbPFZyscrAiTKS/my-overview-of-the-ai-alignment-landscape-threat-models&sa=D&source=editors&ust=1692801435070859&usg=AOvVaw35Jr2_clnM0H41b3c6vTTR).

[[120]](#ftnt_ref120) Kenton, Zachary, Rohin Shah, David Lindner, Vikrant Varma, Victoria Krakovna, Mary Phuong, Ramana Kumar, and Elliot Catt. ‘Clarifying AI X-Risk’. Alignment Forum, 1 November 2022.[https://www.alignmentforum.org/posts/GctJD5oCDRxCspEaZ/clarifying-ai-x-risk](https://www.google.com/url?q=https://www.alignmentforum.org/posts/GctJD5oCDRxCspEaZ/clarifying-ai-x-risk&sa=D&source=editors&ust=1692801435069826&usg=AOvVaw3Ej6Y6mWxqJf0uqJoGqYMW). Summarizing: Kenton, Zachary, Rohin Shah, David Lindner, Vikrant Varma, Victoria Krakovna, Mary Phuong, Ramana Kumar, and Elliot Catt. ‘Threat Model Literature Review’. Alignment Forum, 1 November 2022.[https://www.alignmentforum.org/posts/wnnkD6P2k2TfHnNmt/threat-model-literature-review](https://www.google.com/url?q=https://www.alignmentforum.org/posts/wnnkD6P2k2TfHnNmt/threat-model-literature-review&sa=D&source=editors&ust=1692801435070308&usg=AOvVaw2L7zTYxEWFtWrODemjsBH5).

[[121]](#ftnt_ref121) Yudkowsky, Eliezer. ‘Artificial Intelligence as a Positive and Negative Factor in Global Risk.’ In Global Catastrophic Risks, by Eliezer Yudkowsky, 308–45. New York: Oxford University Press, 2008.[https://oxford.universitypressscholarship.com/view/10.1093/oso/9780198570509.001.0001/isbn-9780198570509-book-part-21](https://www.google.com/url?q=https://oxford.universitypressscholarship.com/view/10.1093/oso/9780198570509.001.0001/isbn-9780198570509-book-part-21&sa=D&source=editors&ust=1692801434967062&usg=AOvVaw3XhYshX8VS5Sprh-GuO9R2).

[[122]](#ftnt_ref122) Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014.

[[123]](#ftnt_ref123) Sotala, Kaj. ‘Disjunctive Scenarios of Catastrophic AI Risk’. In Artificial Intelligence Safety and Security, edited by Roman V. Yampolskiy, 1st ed., 315–37. First edition. | Boca Raton, FL : CRC Press/Taylor & Francis Group, 2018.: Chapman and Hall/CRC, 2018.[https://doi.org/10.1201/9781351251389-22](https://www.google.com/url?q=https://doi.org/10.1201/9781351251389-22&sa=D&source=editors&ust=1692801434966255&usg=AOvVaw0ekHzRUFV8-PcBGhHsGTwF).

[[124]](#ftnt_ref124) Sotala, Kaj, and Roman Yampolskiy. ‘Risks of the Journey to the Singularity’. In The Technological Singularity: Managing the Journey, edited by Victor Callaghan, James Miller, Roman Yampolskiy, and Stuart Armstrong, 11–23. The Frontiers Collection. Berlin, Heidelberg: Springer, 2017.[https://doi.org/10.1007/978-3-662-54033-6\_2](https://www.google.com/url?q=https://doi.org/10.1007/978-3-662-54033-6_2&sa=D&source=editors&ust=1692801435025468&usg=AOvVaw0l89j-UoxDJMAI6q4V2hHR).

[[125]](#ftnt_ref125) Alexander, Scott. ‘Superintelligence FAQ’, 2016.[https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq](https://www.google.com/url?q=https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq&sa=D&source=editors&ust=1692801434958568&usg=AOvVaw3KL9CQ51jeF8ArPg9ehY7u).

[[126]](#ftnt_ref126) Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801434967969&usg=AOvVaw2xl5lTKEFkPSgCR2u6IOLx).; Karnofsky, Holden. ‘AI Could Defeat All Of Us Combined’. Cold Takes, 9 June 2022.[https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/](https://www.google.com/url?q=https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/&sa=D&source=editors&ust=1692801434968533&usg=AOvVaw3LymFY6t4qk1XS0iw049wv).

[[127]](#ftnt_ref127) Christiano, Paul. ‘What Failure Looks Like’. AI Alignment Forum (blog), 2019.[https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like](https://www.google.com/url?q=https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like&sa=D&source=editors&ust=1692801435405122&usg=AOvVaw2pejF17NrB5BgHtTIATEiu).

[[128]](#ftnt_ref128) Carlsmith, Joseph. ‘Is Power-Seeking AI an Existential Risk?’ arXiv, April 2021.[http://arxiv.org/abs/2206.13353](https://www.google.com/url?q=http://arxiv.org/abs/2206.13353&sa=D&source=editors&ust=1692801435077011&usg=AOvVaw2NCHGr2FfruTlivQ_nm14o).

[[129]](#ftnt_ref129) Ngo, Richard. ‘AGI Safety From First Principles’, 2020.[https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ](https://www.google.com/url?q=https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ&sa=D&source=editors&ust=1692801435077470&usg=AOvVaw0LJ-1kkmAJOgXNPROoTG7_).

[[130]](#ftnt_ref130) Yudkowsky, Eliezer. ‘AGI Ruin: A List of Lethalities’. Machine Intelligence Research Institute, 11 June 2022.[https://intelligence.org/2022/06/10/agi-ruin/](https://www.google.com/url?q=https://intelligence.org/2022/06/10/agi-ruin/&sa=D&source=editors&ust=1692801435145023&usg=AOvVaw0ab24EO5Go2ys_7fU7YoGR).; see also: Soares, Nate. ‘AGI Ruin Scenarios Are Likely (and Disjunctive)’. AI Alignment Forum, 27 July 2022.[https://www.alignmentforum.org/posts/ervaGwJ2ZcwqfCcLx/agi-ruin-scenarios-are-likely-and-disjunctive](https://www.google.com/url?q=https://www.alignmentforum.org/posts/ervaGwJ2ZcwqfCcLx/agi-ruin-scenarios-are-likely-and-disjunctive&sa=D&source=editors&ust=1692801435145558&usg=AOvVaw0k-4hNfbZ9udwbZ8IaCZga).; and Lintz, Alex. ‘Three Pillars for Avoiding AGI Catastrophe: Technical Alignment, Deployment Decisions, and Coordination’. Effective Altruism Forum. Accessed 4 August 2022.[https://forum.effectivealtruism.org/posts/eggdG27y75ot8dNn7/three-pillars-for-avoiding-agi-catastrophe-technical](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/eggdG27y75ot8dNn7/three-pillars-for-avoiding-agi-catastrophe-technical&sa=D&source=editors&ust=1692801435146136&usg=AOvVaw3pLAEgqGvaQDkdQhX5hxRa).

[[131]](#ftnt_ref131) Karnofsky, Holden. ‘AI Could Defeat All Of Us Combined’. Cold Takes, 9 June 2022.[https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/](https://www.google.com/url?q=https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/&sa=D&source=editors&ust=1692801435146769&usg=AOvVaw16gKKt47hkHIS_kns4VHqK).; see also: Ricon, Jose Luis. ‘Set Sail For Fail? On AI Risk’. Nintil, 4 August 2022.[https://nintil.com/ai-safety](https://www.google.com/url?q=https://nintil.com/ai-safety&sa=D&source=editors&ust=1692801435147204&usg=AOvVaw0BEwjEgrBuNjgDuRC0xA38). And see: Clarke, Sam, and Samuel Dylan Martin. ‘Distinguishing AI Takeover Scenarios’. AI Alignment Forum, 8 September 2021.[https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios](https://www.google.com/url?q=https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios&sa=D&source=editors&ust=1692801435147763&usg=AOvVaw3oKu3qHS-gNlrXytAsxiZj).

[[132]](#ftnt_ref132) Barak, Boaz, and Ben Edelman. ‘AI Will Change the World, but Won’t Take It over by Playing “3-Dimensional Chess”.’ Windows On Theory (blog), 22 November 2022.[https://windowsontheory.org/2022/11/22/ai-will-change-the-world-but-wont-take-it-over-by-playing-3-dimensional-chess/](https://www.google.com/url?q=https://windowsontheory.org/2022/11/22/ai-will-change-the-world-but-wont-take-it-over-by-playing-3-dimensional-chess/&sa=D&source=editors&ust=1692801435172027&usg=AOvVaw32URqwYfjjHITckrIFXn5P).; Fodor, James. ‘A Critique of AI Takeover Scenarios’. Effective Altruism Forum, 2022.[https://forum.effectivealtruism.org/posts/j7X8nQ7YvvA7Pi4BX/a-critique-of-ai-takeover-scenarios](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/j7X8nQ7YvvA7Pi4BX/a-critique-of-ai-takeover-scenarios&sa=D&source=editors&ust=1692801435172478&usg=AOvVaw0QE7rvzzOVrTGupcTcBKlp).

[[133]](#ftnt_ref133) Christiano, Paul. ‘What Failure Looks Like’. AI Alignment Forum (blog), 2019.[https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like](https://www.google.com/url?q=https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like&sa=D&source=editors&ust=1692801435445272&usg=AOvVaw2Zh2ybVYOomw8Uj8CN_PbK).; discussed in Clarke, Sam. ‘Clarifying “What Failure Looks like” (Part 1)’. AI Alignment Forum (blog), 2020.[https://www.alignmentforum.org/posts/v6Q7T335KCMxujhZu/clarifying-what-failure-looks-like-part-1](https://www.google.com/url?q=https://www.alignmentforum.org/posts/v6Q7T335KCMxujhZu/clarifying-what-failure-looks-like-part-1&sa=D&source=editors&ust=1692801435445854&usg=AOvVaw2MeqT1QGmAbQtNdWhqe6mI).

[[134]](#ftnt_ref134) Sotala, Kaj, and Lukas Gloor. ‘Superintelligence As a Cause or Cure For Risks of Astronomical Suffering’. Informatica 41, no. 4 (27 December 2017).[http://www.informatica.si/index.php/informatica/article/view/1877](https://www.google.com/url?q=http://www.informatica.si/index.php/informatica/article/view/1877&sa=D&source=editors&ust=1692801435335095&usg=AOvVaw1VLp73Lgv-5gha_2Lzo3eF).; Baumann, Tobias. ‘An Introduction to Worst-Case AI Safety’. Reducing Risks of Future Suffering (blog), 5 July 2018.[https://s-risks.org/an-introduction-to-worst-case-ai-safety/](https://www.google.com/url?q=https://s-risks.org/an-introduction-to-worst-case-ai-safety/&sa=D&source=editors&ust=1692801435335616&usg=AOvVaw0kX-_CdYpHoKB2YHlHX8jg).; Baumann, Tobias. ‘Focus Areas of Worst-Case AI Safety’. Reducing Risks of Future Suffering, 16 September 2017.[https://s-risks.org/focus-areas-of-worst-case-ai-safety/](https://www.google.com/url?q=https://s-risks.org/focus-areas-of-worst-case-ai-safety/&sa=D&source=editors&ust=1692801435336135&usg=AOvVaw0T7S0LOww5ubquRYwdI24a); Tomasik, Brian. ‘Astronomical Suffering from Slightly Misaligned Artificial Intelligence’, 2018.[https://reducing-suffering.org/near-miss/](https://www.google.com/url?q=https://reducing-suffering.org/near-miss/&sa=D&source=editors&ust=1692801435336524&usg=AOvVaw1tNYdmn2KaT6YENTkv9-lY).

[[135]](#ftnt_ref135)  Metzinger, Thomas. ‘Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology’. Journal of Artificial Intelligence and Consciousness, 19 February 2021, 1–24.[https://doi.org/10.1142/S270507852150003X](https://www.google.com/url?q=https://doi.org/10.1142/S270507852150003X&sa=D&source=editors&ust=1692801435416324&usg=AOvVaw1zs7ruNFXLsNjAcEaZMVXY). Pg. 3.

[[136]](#ftnt_ref136) Clifton, Jesse. ‘Cooperation, Conflict, and Transformative Artificial Intelligence - A Research Agenda’. Center on Long-Term Risk, March 2020.[https://longtermrisk.org/files/Cooperation-Conflict-and-Transformative-Artificial-Intelligence-A-Research-Agenda.pdf](https://www.google.com/url?q=https://longtermrisk.org/files/Cooperation-Conflict-and-Transformative-Artificial-Intelligence-A-Research-Agenda.pdf&sa=D&source=editors&ust=1692801434971870&usg=AOvVaw0Ow2ITMG8hzsza-Ah9i-AH).

[[137]](#ftnt_ref137) Critch, Andrew, and Thomas Krendl Gilbert. ‘What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)’. LessWrong, 1 April 2021.[https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic](https://www.google.com/url?q=https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic&sa=D&source=editors&ust=1692801434972565&usg=AOvVaw0TqSJkWd7bYdSe3T0naQIL).

[[138]](#ftnt_ref138) Dafoe, Allan. ‘AI Governance: A Research Agenda’. Oxford: Center for the Governance of AI, Future of Humanity Institute, 2018.[https://www.fhi.ox.ac.uk/govaiagenda/](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/govaiagenda/&sa=D&source=editors&ust=1692801435148343&usg=AOvVaw0t4BvpieygBcFsI8adVGcY). Pg 7; Hilton, Benjamin. ‘Risks of Stable Totalitarianism’. 80,000 Hours, 23 September 2022.[https://80000hours.org/problem-profiles/risks-of-stable-totalitarianism/](https://www.google.com/url?q=https://80000hours.org/problem-profiles/risks-of-stable-totalitarianism/&sa=D&source=editors&ust=1692801435148849&usg=AOvVaw2iez10uAJkQLLqPVbJFSX9).

[[139]](#ftnt_ref139) Finnveden, Lukas, C. Jess Riedel, and Carl Shulman. ‘Artificial General Intelligence and Lock-In’, 2022.[https://docs.google.com/document/d/1mkLFhxixWdT5peJHq4rfFzq4QbHyfZtANH1nou68q88/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1mkLFhxixWdT5peJHq4rfFzq4QbHyfZtANH1nou68q88/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435060007&usg=AOvVaw1cu_MvHVyV05YiP6bNToUy).

[[140]](#ftnt_ref140) Critch, Andrew, and David Krueger. ‘AI Research Considerations for Human Existential Safety (ARCHES)’, 29 May 2020.[http://acritch.com/arches/](https://www.google.com/url?q=http://acritch.com/arches/&sa=D&source=editors&ust=1692801435405696&usg=AOvVaw1wYSmLGUhs0rIZ1QJEqvvb). Pg. 12-13

(“We say that an AI system or technology is prepotent [...] (relative to humanity) if its deployment would transform the state of humanity’s habitat—currently the Earth—in a manner that is at least as impactful as humanity and unstoppable to humanity, as follows:

* at least as impactful as humanity: By this we mean that if the AI system or technology is deployed, then its resulting transformative effects on the world would be at least as significant as humanity’s transformation of the Earth thus far, including past events like the agricultural and industrial revolutions.
* unstoppable to humanity: By this we mean that if the AI system or technology is deployed, then no concurrently existing collective of humans would have the ability to reverse or stop the transformative impact of the technology (even if every human in the collective were suddenly in unanimous agreement that the transformation should be reversed or stopped). Merely altering the nature of the transformative impact does not count as stopping it.”)
[[141]](#ftnt_ref141) Aguirre, Anthony. ‘Why Those Who Care about Catastrophic and Existential Risk Should Care about Autonomous Weapons’. EA Forum, 11 November 2020.[https://forum.effectivealtruism.org/posts/oR9tLNRSAep293rr5/why-those-who-care-about-catastrophic-and-existential-risk-2](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/oR9tLNRSAep293rr5/why-those-who-care-about-catastrophic-and-existential-risk-2&sa=D&source=editors&ust=1692801434941747&usg=AOvVaw2mSCaFhSedZF7K_c7ZreeF).; but for critical responses, see: Ruhl, Christian. ‘Risks from Autonomous Weapon Systems and Military AI’. Founders Pledge, 19 May 2022.[https://forum.effectivealtruism.org/posts/RKMNZn7r6cT2Yaorf/risks-from-autonomous-weapon-systems-and-military-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/RKMNZn7r6cT2Yaorf/risks-from-autonomous-weapon-systems-and-military-ai&sa=D&source=editors&ust=1692801434942342&usg=AOvVaw20snLMEPnGl5y7pLCTMSCo)

[[142]](#ftnt_ref142) Maas, Matthijs M, Kayla Matteucci, and Di Cooke. ‘Military Artificial Intelligence as Contributor to Global Catastrophic Risk’, 2023, in The Era of Global Risk (2023). (eds. SJ Beard, Martin Rees, Catherine Richards & Clarissa Rios-Rojas). Open Book Publishers. 36. [https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=4115010](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D4115010&sa=D&source=editors&ust=1692801434942765&usg=AOvVaw26QMmkbuzRCsR70yDps4Iz) ; Avin, Shahar, and S. M. Amadae. ‘Autonomy and Machine Learning at the Interface of Nuclear Weapons, Computers and People’. In The Impact of Artificial Intelligence on Strategic Stability and Nuclear Risk, edited by V. Boulanin. Stockholm International Peace Research Institute, 2019.[https://doi.org/10.17863/CAM.44758](https://www.google.com/url?q=https://doi.org/10.17863/CAM.44758&sa=D&source=editors&ust=1692801434943281&usg=AOvVaw20-5SI_YKvY77-Q4kwcv76).; Rautenbach, Peter. ‘Machine Learning & NC3: The Risk of Integration’. Cambridge Existential Risk Initiative, 9 November 2022.[https://docs.google.com/document/d/1E2e2gn1LadgwREPb9SfruXq48tPFnd\_JNePaXWYSgmo/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1E2e2gn1LadgwREPb9SfruXq48tPFnd_JNePaXWYSgmo/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801434943853&usg=AOvVaw0qxzF5zzNsmJMQuhmGJIEu).

[[143]](#ftnt_ref143) Horowitz, Michael C. ‘When Speed Kills: Lethal Autonomous Weapon Systems, Deterrence and Stability’. Journal of Strategic Studies 42, no. 6 (19 September 2019): 764–88.[https://doi.org/10.1080/01402390.2019.1621174](https://www.google.com/url?q=https://doi.org/10.1080/01402390.2019.1621174&sa=D&source=editors&ust=1692801435387833&usg=AOvVaw0rrT4pvqCYRTQJmC_-QTap). I thank Christian Ruhl for this suggestion.

[[144]](#ftnt_ref144) MichaelA. ‘8 Possible High-Level Goals for Work on Nuclear Risk’. EA Forum, 29 March 2022.[https://forum.effectivealtruism.org/posts/dASEFCurRpNot4Gpc/8-possible-high-level-goals-for-work-on-nuclear-risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/dASEFCurRpNot4Gpc/8-possible-high-level-goals-for-work-on-nuclear-risk&sa=D&source=editors&ust=1692801434944572&usg=AOvVaw0SrTQGGKz_rgMjy0Hmi1yD).

[[145]](#ftnt_ref145) Clarke, Sam, and Jess Whittlestone. ‘A Survey of the Potential Long-Term Impacts of AI: How AI Could Lead to Long-Term Changes in Science, Cooperation, Power, Epistemics and Values’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 192–202. AIES ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3514094.3534131](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534131&sa=D&source=editors&ust=1692801434975401&usg=AOvVaw0IXPTA8ryhZYiLSLD14g7a).

[[146]](#ftnt_ref146) AI Impacts. ‘AI Vignettes Project’. AI Impacts, 12 October 2021.[https://aiimpacts.org/ai-vignettes-project/](https://www.google.com/url?q=https://aiimpacts.org/ai-vignettes-project/&sa=D&source=editors&ust=1692801435300220&usg=AOvVaw2nNIsNC812VeYQSQ5lEBkd).

[[147]](#ftnt_ref147) ggilgallon. ‘FLI Launches Worldbuilding Contest with $100,000 in Prizes’. EA Forum, 17 January 2022.[https://forum.effectivealtruism.org/posts/LjExZCPCHnNNTFDfq/fli-launches-worldbuilding-contest-with-usd100-000-in-prizes](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/LjExZCPCHnNNTFDfq/fli-launches-worldbuilding-contest-with-usd100-000-in-prizes&sa=D&source=editors&ust=1692801435300932&usg=AOvVaw3y3-ehUJmqHl-2BcVgJrSz).; Future of Life Institute. ‘About’. FLI Worldbuilding Contest (blog), 2022.[https://worldbuild.ai/about/](https://www.google.com/url?q=https://worldbuild.ai/about/&sa=D&source=editors&ust=1692801435301407&usg=AOvVaw18ANdt319q-mQuSzUAI46A).

[[148]](#ftnt_ref148) Avin, Shahar, Ross Gruetzemacher, and James Fox. ‘Exploring AI Futures Through Role Play’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8–14. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375817](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375817&sa=D&source=editors&ust=1692801435301968&usg=AOvVaw3hghS3WXpx6MLthdSezxV7).

[[149]](#ftnt_ref149) Karnofsky, Holden. ‘AI Could Defeat All Of Us Combined’. Cold Takes, 9 June 2022.[https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/](https://www.google.com/url?q=https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/&sa=D&source=editors&ust=1692801435302463&usg=AOvVaw2wH7kLiy5kzWf1g3D_DasO).; see also: Ricon, Jose Luis. ‘Set Sail For Fail? On AI Risk’. Nintil, 4 August 2022.[https://nintil.com/ai-safety](https://www.google.com/url?q=https://nintil.com/ai-safety&sa=D&source=editors&ust=1692801435302878&usg=AOvVaw2UtL0aWco9NH6l3kN1blRY). And see: Clarke, Sam, and Samuel Dylan Martin. ‘Distinguishing AI Takeover Scenarios’. AI Alignment Forum, 8 September 2021.[https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios](https://www.google.com/url?q=https://www.alignmentforum.org/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios&sa=D&source=editors&ust=1692801435303308&usg=AOvVaw0hAKag0DHyYbW7DGb22Hox).

[[150]](#ftnt_ref150) Hilton, Benjamin. ‘What Could an AI-Caused Existential Catastrophe Actually Look Like?’ 80,000 Hours, 15 August 2022.[https://80000hours.org/articles/what-could-an-ai-caused-existential-catastrophe-actually-look-like/](https://www.google.com/url?q=https://80000hours.org/articles/what-could-an-ai-caused-existential-catastrophe-actually-look-like/&sa=D&source=editors&ust=1692801435303918&usg=AOvVaw2PaaD3SBAOZtpGKbZNZEjg).

[[151]](#ftnt_ref151) Branwen, Gwern. ‘It Looks Like You’re Trying To Take Over The World’, 6 March 2022.[https://www.gwern.net/fiction/Clippy](https://www.google.com/url?q=https://www.gwern.net/fiction/Clippy&sa=D&source=editors&ust=1692801435403177&usg=AOvVaw1hzUHIzFSdMjMA28zRsb8-).

[[152]](#ftnt_ref152) Carlier, Alexis, Sam Clarke, and Jonas Schuett. ‘Existential Risks from AI: A Survey of Expert Opinion’, 2021, 16.; Clarke, Sam, Alexis Carlier, and Jonas Schuett. ‘Survey on AI Existential Risk Scenarios’. Effective Altruism Forum, 8 June 2021.[https://forum.effectivealtruism.org/posts/2tumunFmjBuXdfF2F/survey-on-ai-existential-risk-scenarios-1](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/2tumunFmjBuXdfF2F/survey-on-ai-existential-risk-scenarios-1&sa=D&source=editors&ust=1692801434970006&usg=AOvVaw1tWoFS-fZ6PdvFGgi41LJF).

[[153]](#ftnt_ref153) Bensinger, Rob. ‘“Existential Risk from AI” Survey Results’. AI Alignment Forum, 1 June 2021.[https://www.alignmentforum.org/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results](https://www.google.com/url?q=https://www.alignmentforum.org/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results&sa=D&source=editors&ust=1692801434969237&usg=AOvVaw0kWBtzEZeA2w98sGKQZ1aP).

[[154]](#ftnt_ref154) Graham, Ross. ‘Discourse Analysis of Academic Debate of Ethics for AGI’. AI & SOCIETY, 2 June 2021.[https://doi.org/10.1007/s00146-021-01228-7](https://www.google.com/url?q=https://doi.org/10.1007/s00146-021-01228-7&sa=D&source=editors&ust=1692801434960661&usg=AOvVaw22Fl2lA6YNKnzSSTA5Q7jq).

[[155]](#ftnt_ref155) Burden, John, Sam Clarke, and Jess Whittlestone. ‘From Turing’s Speculations to an Academic Discipline: A History of AI Existential Safety’. In Cambridge Conference on Catastrophic Risk 2020, 2022; see also: lukeprog. ‘AI Risk and Opportunity: Humanity’s Efforts So Far’. Accessed 2 February 2023.[https://www.lesswrong.com/posts/i4susk4W3ieR5K92u/ai-risk-and-opportunity-humanity-s-efforts-so-far](https://www.google.com/url?q=https://www.lesswrong.com/posts/i4susk4W3ieR5K92u/ai-risk-and-opportunity-humanity-s-efforts-so-far&sa=D&source=editors&ust=1692801435378733&usg=AOvVaw1gqfax0sLo_BXbKW0zTiin).; lukeprog. ‘AI Risk & Opportunity: A Timeline of Early Ideas and Arguments’. Accessed 2 February 2023.[https://www.lesswrong.com/posts/Qdq2SKyMi8vf7Snxq/ai-risk-and-opportunity-a-timeline-of-early-ideas-and](https://www.google.com/url?q=https://www.lesswrong.com/posts/Qdq2SKyMi8vf7Snxq/ai-risk-and-opportunity-a-timeline-of-early-ideas-and&sa=D&source=editors&ust=1692801435379359&usg=AOvVaw13AAkoV70AGkKPj4iiBhJ9).

[[156]](#ftnt_ref156) Chivers, Tom. The AI Does Not Hate You: Superintelligence, Rationality and the Race to Save the World. London: Weidenfeld & Nicolson, 2019.

[[157]](#ftnt_ref157) Adamczewski, Tom. ‘A Shift in Arguments for AI Risk’. Fragile Credences, 25 May 2019.[https://fragile-credences.github.io/prioritising-ai/](https://www.google.com/url?q=https://fragile-credences.github.io/prioritising-ai/&sa=D&source=editors&ust=1692801434961891&usg=AOvVaw2zveBMuTc-ehgFvPhmOStQ).

[[158]](#ftnt_ref158) Garfinkel, Ben. ‘How Sure Are We about This AI Stuff?’ EA Forum, 9 February 2019.[https://forum.effectivealtruism.org/posts/9sBAW3qKppnoG3QPq/ben-garfinkel-how-sure-are-we-about-this-ai-stuff](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/9sBAW3qKppnoG3QPq/ben-garfinkel-how-sure-are-we-about-this-ai-stuff&sa=D&source=editors&ust=1692801434989853&usg=AOvVaw0RdICjb4NUILRV-WQ3EZXv).

[[159]](#ftnt_ref159) Grace, Katja. ‘Counterarguments to the Basic AI X-Risk Case’. AI Impacts, 14 October 2022.[https://aiimpacts.org/counterarguments-to-the-basic-ai-x-risk-case/](https://www.google.com/url?q=https://aiimpacts.org/counterarguments-to-the-basic-ai-x-risk-case/&sa=D&source=editors&ust=1692801434990452&usg=AOvVaw305l0gj3vZMAMj5yj1gOHZ).; and response: Jenner, Erik, and Johannes Treutlein. ‘Response to Katja Grace’s AI x-Risk Counterarguments’. Alignment Forum, 19 October 2022.[https://www.alignmentforum.org/posts/GQat3Nrd9CStHyGaq/response-to-katja-grace-s-ai-x-risk-counterarguments](https://www.google.com/url?q=https://www.alignmentforum.org/posts/GQat3Nrd9CStHyGaq/response-to-katja-grace-s-ai-x-risk-counterarguments&sa=D&source=editors&ust=1692801434991078&usg=AOvVaw1kVdRDn4xqzXSPZs0HbPMF).

[[160]](#ftnt_ref160) Barak, Boaz, and Ben Edelman. ‘AI Will Change the World, but Won’t Take It over by Playing “3-Dimensional Chess”.’ Windows On Theory (blog), 22 November 2022.[https://windowsontheory.org/2022/11/22/ai-will-change-the-world-but-wont-take-it-over-by-playing-3-dimensional-chess/](https://www.google.com/url?q=https://windowsontheory.org/2022/11/22/ai-will-change-the-world-but-wont-take-it-over-by-playing-3-dimensional-chess/&sa=D&source=editors&ust=1692801435157601&usg=AOvVaw1YCsNTjOjY5K5INvMNfk5e).

[[161]](#ftnt_ref161) E.g. Heninger, Jeffrey. ‘Against a General Factor of Doom’. AI Impacts, 23 November 2022.[https://aiimpacts.org/against-a-general-factor-of-doom/](https://www.google.com/url?q=https://aiimpacts.org/against-a-general-factor-of-doom/&sa=D&source=editors&ust=1692801435298594&usg=AOvVaw0bl583QqR_mLUBf0I2R9F8).

[[162]](#ftnt_ref162) NunoSempere. ‘My Highly Personal Skepticism Braindump on Existential Risk from Artificial Intelligence.’ EA Forum, 23 January 2023.[https://forum.effectivealtruism.org/posts/L6ZmggEJw8ri4KB8X/my-highly-personal-skepticism-braindump-on-existential-risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/L6ZmggEJw8ri4KB8X/my-highly-personal-skepticism-braindump-on-existential-risk&sa=D&source=editors&ust=1692801435299215&usg=AOvVaw0zewpNRbyBob9xpyS42K_4).

[[163]](#ftnt_ref163) Trammell, Philip. ‘But Have They Engaged with the Arguments?’ Philip Trammell, 29 December 2019.[https://philiptrammell.com/blog/46](https://www.google.com/url?q=https://philiptrammell.com/blog/46&sa=D&source=editors&ust=1692801435299719&usg=AOvVaw0x-HhqUcAqWHxiioSSvOu_).

[[164]](#ftnt_ref164) Bostrom, Nick, Allan Dafoe, and Carrick Flynn. ‘Public Policy and Superintelligent AI: A Vector Field Approach’. In Ethics of Artificial Intelligence, edited by S.M. Liao. Oxford University Press, 2019.[http://www.nickbostrom.com/papers/aipolicy.pdf](https://www.google.com/url?q=http://www.nickbostrom.com/papers/aipolicy.pdf&sa=D&source=editors&ust=1692801434945233&usg=AOvVaw300nT4PJ9n6QvsZjBJtb-l).

[[165]](#ftnt_ref165) Ord, Toby. The Precipice: Existential Risk and the Future of Humanity. Illustrated Edition. New York: Hachette Books, 2020. pg. 175–180.

[[166]](#ftnt_ref166) Clarke, Sam, and Jess Whittlestone. ‘A Survey of the Potential Long-Term Impacts of AI: How AI Could Lead to Long-Term Changes in Science, Cooperation, Power, Epistemics and Values’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 192–202. AIES ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3514094.3534131](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534131&sa=D&source=editors&ust=1692801435172998&usg=AOvVaw1xG9kR9XBB_MnVnQTviJ-k).

[[167]](#ftnt_ref167) For an overview of some of these themes, see also: Dafoe, Allan. ‘AI Governance: Opportunity and Theory of Impact’, 17 September 2020.[https://www.allandafoe.com/opportunity](https://www.google.com/url?q=https://www.allandafoe.com/opportunity&sa=D&source=editors&ust=1692801434946014&usg=AOvVaw3BlQmqdQi8R_aQy0DW-F9O). ‘AI Strategy, Policy, and Governance’ by Allan Dafoe, 2019.[https://www.youtube.com/watch?v=2IpJ8TIKKtI](https://www.google.com/url?q=https://www.youtube.com/watch?v%3D2IpJ8TIKKtI&sa=D&source=editors&ust=1692801434946508&usg=AOvVaw3rSpWXhooPqcKKS8JKdvYi).

[[168]](#ftnt_ref168) Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, et al. ‘The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation’, 20 February 2018.[http://arxiv.org/abs/1802.07228](https://www.google.com/url?q=http://arxiv.org/abs/1802.07228&sa=D&source=editors&ust=1692801434947075&usg=AOvVaw2Z9j52z49kOVk4GJ1oBW2B).

[[169]](#ftnt_ref169) Seger, Elizabeth, Shahar Avin, Gavin Pearson, Mark Briers, Seán Ó hÉigeartaigh, and Helena Bacon. ‘Tackling Threats to Informed Decisionmaking in Democratic Societies: Promoting Epistemic Security in a Technologically-Advanced World’. The Alan Turing Institute, October 2020.[https://www.turing.ac.uk/research/publications/tackling-threats-informed-decision-making-democratic-societies](https://www.google.com/url?q=https://www.turing.ac.uk/research/publications/tackling-threats-informed-decision-making-democratic-societies&sa=D&source=editors&ust=1692801434947819&usg=AOvVaw3PuXo-rlf3HVEDzTLubGky).

[[170]](#ftnt_ref170) Maas, Matthijs M. ‘International Law Does Not Compute: Artificial Intelligence and The Development, Displacement or Destruction of the Global Legal Order’. Melbourne Journal of International Law 20, no. 1 (2019): 29–56. [https://law.unimelb.edu.au/\_\_data/assets/pdf\_file/0005/3144308/Maas.pdf](https://www.google.com/url?q=https://law.unimelb.edu.au/__data/assets/pdf_file/0005/3144308/Maas.pdf&sa=D&source=editors&ust=1692801434948360&usg=AOvVaw03-jf1knRime3-fnqxAVKo)

[[171]](#ftnt_ref171) Kolt, Noam. ‘Algorithmic Black Swans’. Washington University Law Review 101 (25 February 2023).[https://papers.ssrn.com/abstract=4370566](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4370566&sa=D&source=editors&ust=1692801435414271&usg=AOvVaw3qE21zoCwfer7n7rgul9fX).

[[172]](#ftnt_ref172) Hilton, Benjamin. ‘Preventing an AI-Related Catastrophe - Problem Profile’. 80,000 Hours, 25 August 2022.[https://80000hours.org/problem-profiles/artificial-intelligence/](https://www.google.com/url?q=https://80000hours.org/problem-profiles/artificial-intelligence/&sa=D&source=editors&ust=1692801435374086&usg=AOvVaw2lezZdxkjwefw9u4fIb2e9).

[[173]](#ftnt_ref173) Aird, Michael. ‘Database of Orgs Relevant to Longtermist/x-Risk Work’. EA Forum, 19 November 2021.[https://forum.effectivealtruism.org/posts/twMs8xsgwnYvaowWX/database-of-orgs-relevant-to-longtermist-x-risk-work](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/twMs8xsgwnYvaowWX/database-of-orgs-relevant-to-longtermist-x-risk-work&sa=D&source=editors&ust=1692801435374886&usg=AOvVaw2VnkwhZb5NoeV9VF-KnQu6). (see [link](https://www.google.com/url?q=https://airtable.com/shrl64Yz5Q8lidMAD/tblWmQnJoaxgYMiZG/viwjHEUBUGEvJVXUc?blocks%3Dhide&sa=D&source=editors&ust=1692801435375185&usg=AOvVaw1MNHhDL0cvwaV-ot1D4ebs) to database; and [view](https://www.google.com/url?q=https://airtable.com/shrl64Yz5Q8lidMAD/tblWmQnJoaxgYMiZG/viw80CkuBogp953rd?blocks%3Dhide&sa=D&source=editors&ust=1692801435375463&usg=AOvVaw0nlyVvjH4_9FLrgUN3NnYT) filtered by organizations focused on AI strategy / alignment).

[[174]](#ftnt_ref174) Riedel, Jess, and Angelica Deibel. ‘TAI Safety Bibliographic Database’. AI Alignment Forum, 22 December 2020.[https://www.alignmentforum.org/posts/4DegbDJJiMX2b3EKm/tai-safety-bibliographic-database](https://www.google.com/url?q=https://www.alignmentforum.org/posts/4DegbDJJiMX2b3EKm/tai-safety-bibliographic-database&sa=D&source=editors&ust=1692801435376212&usg=AOvVaw0Thkenvrz9hJCZlmZhqeiz).

[[175]](#ftnt_ref175) Everitt, Tom, Gary Lea, and Marcus Hutter. ‘AGI Safety Literature Review’. ArXiv:1805.01109 [Cs], 3 May 2018.[http://arxiv.org/abs/1805.01109](https://www.google.com/url?q=http://arxiv.org/abs/1805.01109&sa=D&source=editors&ust=1692801434971155&usg=AOvVaw1OgBVd2q8BAkNNmIcXunVV).

[[176]](#ftnt_ref176) Christiano, Paul. ‘Current Work in AI Alignment’. Effective Altruism, 3 April 2020.[https://www.effectivealtruism.org/articles/paul-christiano-current-work-in-ai-alignment](https://www.google.com/url?q=https://www.effectivealtruism.org/articles/paul-christiano-current-work-in-ai-alignment&sa=D&source=editors&ust=1692801434973244&usg=AOvVaw0HgawLQKRyVvJUV3QEMX_j).

[[177]](#ftnt_ref177) Nanda, Neel. ‘My Overview of the AI Alignment Landscape: Threat Models’. Alignment Forum, 26 December 2021.[https://www.alignmentforum.org/posts/3DFBbPFZyscrAiTKS/my-overview-of-the-ai-alignment-landscape-threat-models](https://www.google.com/url?q=https://www.alignmentforum.org/posts/3DFBbPFZyscrAiTKS/my-overview-of-the-ai-alignment-landscape-threat-models&sa=D&source=editors&ust=1692801435073367&usg=AOvVaw0b-3Ak_XTAj-o_Ot332AMn).

[[178]](#ftnt_ref178) Krakovna, Victoria. ‘Paradigms of AI Alignment: Components and Enablers’. Victoria Krakovna (blog), 2 June 2022.[https://vkrakovna.wordpress.com/2022/06/02/paradigms-of-ai-alignment-components-and-enablers/](https://www.google.com/url?q=https://vkrakovna.wordpress.com/2022/06/02/paradigms-of-ai-alignment-components-and-enablers/&sa=D&source=editors&ust=1692801435073979&usg=AOvVaw1lN-cGjlxgeixerK9_yYJb).

[[179]](#ftnt_ref179) Kirchner, Jan Hendrik, Logan Riggs Smith, Jacques Thibodeau, and janus. ‘A Descriptive, Not Prescriptive, Overview of Current AI Alignment Research’. AI Alignment Forum, 6 June 2022.[https://www.alignmentforum.org/posts/FgjcHiWvADgsocE34/a-descriptive-not-prescriptive-overview-of-current-ai](https://www.google.com/url?q=https://www.alignmentforum.org/posts/FgjcHiWvADgsocE34/a-descriptive-not-prescriptive-overview-of-current-ai&sa=D&source=editors&ust=1692801435074546&usg=AOvVaw0kWHrUQYtEsLAKhdAzHU6F).

[[180]](#ftnt_ref180) Shah, Rohin. ‘AI Alignment 2018-19 Review’. AI Alignment Forum (blog), 2020.[https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review](https://www.google.com/url?q=https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review&sa=D&source=editors&ust=1692801435075096&usg=AOvVaw0CyM1mZet21DuRq1ToHGIu).

[[181]](#ftnt_ref181) Murphy, Evan R. ‘Interpretability Research for the Most Important Century’. AI Alignment Forum, 26 April 2022.[https://www.alignmentforum.org/s/jef8ntrWuJ7SvZjCM](https://www.google.com/url?q=https://www.alignmentforum.org/s/jef8ntrWuJ7SvZjCM&sa=D&source=editors&ust=1692801435075615&usg=AOvVaw1ANvtU4u2Qy6Ch1JsT28B8).

[[182]](#ftnt_ref182) Hubinger, Evan. ‘An Overview of 11 Proposals for Building Safe Advanced AI’. arXiv, 4 December 2020.[https://doi.org/10.48550/arXiv.2012.07532](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2012.07532&sa=D&source=editors&ust=1692801434976044&usg=AOvVaw2q0I1GL1TT30xMfF4U-spj).

[[183]](#ftnt_ref183) As in many fields at an early stage of development, there may be significant challenges to meaningfully evaluating or comparing the relative promises of different paradigms of alignment research. As such, while some assessment of past work can compare the evaluations of different approaches, any larger comparisons of these agendas will be quite precarious. I thank Richard Ngo for this point.

[[184]](#ftnt_ref184) Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801435406803&usg=AOvVaw3mgOlMZ4uYtejWcxatpej-). (see subsection ‘Simple “baseline” behavioral safety interventions’).

[[185]](#ftnt_ref185) Soares, Nate. ‘On How Various Plans Miss the Hard Bits of the Alignment Challenge’. Effective Altruism Forum, 12 July 2022.[https://forum.effectivealtruism.org/posts/jydymb23NWF3Q4oDt/on-how-various-plans-miss-the-hard-bits-of-the-alignment](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/jydymb23NWF3Q4oDt/on-how-various-plans-miss-the-hard-bits-of-the-alignment&sa=D&source=editors&ust=1692801435407373&usg=AOvVaw1a3aIRPZdjasgEdcSTNvIP).

[[186]](#ftnt_ref186) Leike, Jan. ‘Why I’m Optimistic about Our Alignment Approach’. Musings on the Alignment Problem (blog), 5 December 2022.[https://aligned.substack.com/p/alignment-optimism?publication\_id=328633&isFreemail=true](https://www.google.com/url?q=https://aligned.substack.com/p/alignment-optimism?publication_id%3D328633%26isFreemail%3Dtrue&sa=D&source=editors&ust=1692801435407926&usg=AOvVaw2rRl_-91aSRpM4vlhN3PRB).

[[187]](#ftnt_ref187) Leech, Gavin. ‘The Academic Contribution to AI Safety Seems Large’. Effective Altruism Forum, 2020.[https://forum.effectivealtruism.org/posts/8ErtxW7FRPGMtDqJy/the-academic-contribution-to-ai-safety-seems-large](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/8ErtxW7FRPGMtDqJy/the-academic-contribution-to-ai-safety-seems-large&sa=D&source=editors&ust=1692801435377003&usg=AOvVaw1HsD3GHPiYd04fJGNu_uJr).

[[188]](#ftnt_ref188) Shah, Rohin. ‘[AN #80]: Why AI Risk Might Be Solved without Additional Intervention from Longtermists’. AI Alignment Forum, 2 January 2020.[https://www.alignmentforum.org/posts/QknPz9JQTQpGdaWDp/an-80-why-ai-risk-might-be-solved-without-additional](https://www.google.com/url?q=https://www.alignmentforum.org/posts/QknPz9JQTQpGdaWDp/an-80-why-ai-risk-might-be-solved-without-additional&sa=D&source=editors&ust=1692801435377796&usg=AOvVaw3EH5H-dYlBnX1SKmmT_GfF).

[[189]](#ftnt_ref189) For an older (2014) estimate, see: Muehlhauser, Luke. ‘How Big Is the Field of Artificial Intelligence? (Initial Findings)’. Machine Intelligence Research Institute, 28 January 2014.[https://intelligence.org/2014/01/28/how-big-is-ai/](https://www.google.com/url?q=https://intelligence.org/2014/01/28/how-big-is-ai/&sa=D&source=editors&ust=1692801435337179&usg=AOvVaw0fUI3As688k_ggrHn4u8pz).

[[190]](#ftnt_ref190) Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum. ‘2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy’. Global Catastrophic Risk Institute Technical Report. Global Catastrophic Risk Institute, 2020.[https://gcrinstitute.org/papers/055\_agi-2020.pdf](https://www.google.com/url?q=https://gcrinstitute.org/papers/055_agi-2020.pdf&sa=D&source=editors&ust=1692801435337726&usg=AOvVaw0wzOhqajCOA3bc5S6j0y1I). And for a previous 2017 version: Baum, Seth. ‘A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy’. Global Catastrophic Risk Institute Technical Report. Global Catastrophic Risk Institute, 2017.[https://papers.ssrn.com/abstract=3070741](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3070741&sa=D&source=editors&ust=1692801435338130&usg=AOvVaw3c71W6IdOUTmyGr2t6crPY).

[[191]](#ftnt_ref191) See for instance: Hannas, William, Huey-Meei Chang, Daniel Chou, and Brian Fleeger. ‘China’s Advanced AI Research: Monitoring China’s Paths to “General” Artificial Intelligence’. Center for Security and Emerging Technology, July 2022.[https://cset.georgetown.edu/publication/chinas-advanced-ai-research/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/chinas-advanced-ai-research/&sa=D&source=editors&ust=1692801435338726&usg=AOvVaw3S_eYgzvnsB47PyHd_JNSI). And also: Hannas, William, Huey-Meei Chang, Catherine Aiken, and Daniel Chou. ‘China AI-Brain Research: Brain-Inspired AI, Connectomics, Brain-Computer Interfaces’. Center for Security and Emerging Technology, September 2020.[https://cset.georgetown.edu/publication/china-ai-brain-research/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/china-ai-brain-research/&sa=D&source=editors&ust=1692801435339214&usg=AOvVaw1wleEk8UlxJJxHzvGpuCqk). (focusing on connectionist approaches).

[[192]](#ftnt_ref192) For an older sketch, see: Muehlhauser, Luke. ‘The World’s Distribution of Computation (Initial Findings)’. Machine Intelligence Research Institute, 1 March 2014.[https://intelligence.org/2014/02/28/the-worlds-distribution-of-computation-initial-findings/](https://www.google.com/url?q=https://intelligence.org/2014/02/28/the-worlds-distribution-of-computation-initial-findings/&sa=D&source=editors&ust=1692801435339864&usg=AOvVaw3GsLli_hafxUpQwkZqyc3A). For a project guide to a continued project, see: Grace, Katja, and Luke Muelhauser. ‘Project Guide: Map the Computing Landscape’. Google Docs. Accessed 28 October 2022.[https://docs.google.com/document/d/19K37J6VzN7aigZC4IwydEWDAYMSVBTWFcrxN6YFMxig/edit?usp=sharing&](https://www.google.com/url?q=https://docs.google.com/document/d/19K37J6VzN7aigZC4IwydEWDAYMSVBTWFcrxN6YFMxig/edit?usp%3Dsharing%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435340379&usg=AOvVaw3c3EuVDkN0pN8ugQIJFGsj).

[[193]](#ftnt_ref193) Khan, Saif. ‘The Semiconductor Supply Chain: Assessing National Competitiveness’. Center for Security and Emerging Technology, January 2021.[https://cset.georgetown.edu/publication/the-semiconductor-supply-chain/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/the-semiconductor-supply-chain/&sa=D&source=editors&ust=1692801435340968&usg=AOvVaw2UrkKQysDo66J86awY-Adu).

[[194]](#ftnt_ref194) Huang, Tina, and Zachary Arnold. ‘Immigration Policy and the Global Competition for AI Talent’. Center for Security and Emerging Technology, June 2020.[https://cset.georgetown.edu/research/immigration-policy-and-the-global-competition-for-ai-talent/](https://www.google.com/url?q=https://cset.georgetown.edu/research/immigration-policy-and-the-global-competition-for-ai-talent/&sa=D&source=editors&ust=1692801435341575&usg=AOvVaw21OHt2lM1v00Nu_eQjqeDF).

[[195]](#ftnt_ref195) Langenkamp, Max, and Daniel N. Yue. ‘How Open Source Machine Learning Software Shapes AI’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 385–95. Oxford United Kingdom: ACM, 2022.[https://doi.org/10.1145/3514094.3534167](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534167&sa=D&source=editors&ust=1692801435342156&usg=AOvVaw1Jn-_D7OqDxo4iP18MYX-h).

[[196]](#ftnt_ref196) Salisbury, Adam. ‘How Will the AI Supply Chain Evolve?’, 2022.[https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/edit?usp=sharing&](https://www.google.com/url?q=https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/edit?usp%3Dsharing%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435342785&usg=AOvVaw3PjbFy55mftf9nps5nO6PW). (arguing the AI supply chain is currently mostly vertically integrated, with the main users of AI technology also producing the majority of their AI capabilities in-house).

[[197]](#ftnt_ref197) Garfinkel, Benjamin. ‘A Tour of Emerging Cryptographic Technologies: What They Are and How They Could Matter’. Centre for the Governance of AI, Future of Humanity Institute, University of Oxford, May 2021.[https://assets.website-files.com/614b70a71b9f71c9c240c7a7/617938781d1308004d007e2d\_Garfinkel\_Tour\_Of\_Emerging\_Cryptographic\_Technologies.pdf](https://www.google.com/url?q=https://assets.website-files.com/614b70a71b9f71c9c240c7a7/617938781d1308004d007e2d_Garfinkel_Tour_Of_Emerging_Cryptographic_Technologies.pdf&sa=D&source=editors&ust=1692801435343506&usg=AOvVaw39coqnFG67o-j55HYI_SNo).

[[198]](#ftnt_ref198) Snodin, Ben. ‘My Thoughts on Nanotechnology Strategy Research as an EA Cause Area’. EA Forum, 2 May 2022.[https://forum.effectivealtruism.org/posts/oqBJk2Ae3RBegtFfn/my-thoughts-on-nanotechnology-strategy-research-as-an-ea](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/oqBJk2Ae3RBegtFfn/my-thoughts-on-nanotechnology-strategy-research-as-an-ea&sa=D&source=editors&ust=1692801435344179&usg=AOvVaw39_m-BDqunsoE--cZaRhKQ).

[[199]](#ftnt_ref199) Epoch. ‘Announcing Epoch: A Research Initiative Investigating the Road to Transformative AI’, 27 June 2022.[https://epochai.org/blog/announcing-epoch](https://www.google.com/url?q=https://epochai.org/blog/announcing-epoch&sa=D&source=editors&ust=1692801435344753&usg=AOvVaw1IY19Us4b3r4OMMf7QTAsF).

[[200]](#ftnt_ref200) Sevilla, Jaime, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. ‘Compute Trends Across Three Eras of Machine Learning’. ArXiv:2202.05924 [Cs], 11 February 2022.[http://arxiv.org/abs/2202.05924](https://www.google.com/url?q=http://arxiv.org/abs/2202.05924&sa=D&source=editors&ust=1692801435345350&usg=AOvVaw3vXTqZGSWl9LWs97VyEpsY). See also Lohn, Andrew, and Micah Musser. ‘AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?’ Center for Security and Emerging Technology, January 2022.[https://cset.georgetown.edu/publication/ai-and-compute/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/ai-and-compute/&sa=D&source=editors&ust=1692801435345820&usg=AOvVaw2hwomtuU_IZHrU73Hwa2YB).

[[201]](#ftnt_ref201) Cottier, Ben. ‘Trends in the Dollar Training Cost of Machine Learning Systems’. Epoch, 31 January 2023.[https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems](https://www.google.com/url?q=https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems&sa=D&source=editors&ust=1692801435359444&usg=AOvVaw1cbES3fIK18BSeSXwzYZ7P).

[[202]](#ftnt_ref202) Villalobos, Pablo, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. ‘Will We Run out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning’. arXiv, 25 October 2022.[https://doi.org/10.48550/arXiv.2211.04325](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2211.04325&sa=D&source=editors&ust=1692801435346377&usg=AOvVaw2coSp9LcNnglc7A3v1Njpj).

[[203]](#ftnt_ref203) Kirchner, Jan Hendrik. ‘Compute Governance: The Role of Commodity Hardware’. On Brains, Minds, And Their Possible Uses (blog), 26 March 2022.[https://universalprior.substack.com/p/compute-governance-the-role-of-commodity](https://www.google.com/url?q=https://universalprior.substack.com/p/compute-governance-the-role-of-commodity&sa=D&source=editors&ust=1692801435351723&usg=AOvVaw3-NZFJIPKkjDernNGASP-a).

[[204]](#ftnt_ref204) Tucker, Aaron D., Markus Anderljung, and Allan Dafoe. ‘Social and Governance Implications of Improved Data Efficiency’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 378–84. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375863](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375863&sa=D&source=editors&ust=1692801435346988&usg=AOvVaw0nlOoHndk5GnJBV2VuAZ_L).

[[205]](#ftnt_ref205) Nostalgebraist. ‘Chinchilla’s Wild Implications’. LessWrong, 31 July 2022.[https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications](https://www.google.com/url?q=https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications&sa=D&source=editors&ust=1692801435347587&usg=AOvVaw0KKt7Ba2QfytJh-78IY2Ah). (discussing: Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, et al. ‘Training Compute-Optimal Large Language Models’. arXiv, 29 March 2022.[http://arxiv.org/abs/2203.15556](https://www.google.com/url?q=http://arxiv.org/abs/2203.15556&sa=D&source=editors&ust=1692801435348036&usg=AOvVaw2dFNGLtVKI4skcOT1Wa2T5).).

[[206]](#ftnt_ref206) See: Salisbury, Adam. ‘How Will the AI Supply Chain Evolve?’, 2022.[https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/](https://www.google.com/url?q=https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/edit?usp%3Dsharing%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435348779&usg=AOvVaw2qTn6O4ElfvHrYDmGuMTan). (reviewing several trends, to anticipate “the emergence of a hybrid industry structure in which i) AI firms sell access to some of their technology ii) some non-AI firms develop their own AI capabilities and iii) AI firms retain a major downstream presence themselves.”). See also: Mindermann, Sören. ‘Summary: Will AI Companies Sell or Use Their Technology? V2’. Google Docs, 2021.[https://docs.google.com/document/d/1ltCvbI8xYQ49izUUjbHLlQI6MqS6wCJFsIb6zCjSKMA/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1ltCvbI8xYQ49izUUjbHLlQI6MqS6wCJFsIb6zCjSKMA/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435349354&usg=AOvVaw22TvPqhHWWr7voklRMUHZE). (drawing on established economic theory and historical evidence from general-purpose technologies (GPTs), to argue that the AI industry will likely become less vertically integrated); see also: Uuk, Risto. ‘Emerging Non-European Monopolies in the Global AI Market’. Future of Life Institute, November 2022.[https://futureoflife.org/wp-content/uploads/2022/11/Emerging\_Non-European\_Monopolies\_in\_the\_Global\_AI\_Market.pdf](https://www.google.com/url?q=https://futureoflife.org/wp-content/uploads/2022/11/Emerging_Non-European_Monopolies_in_the_Global_AI_Market.pdf&sa=D&source=editors&ust=1692801435349872&usg=AOvVaw2DbGYO5KHCBIde3Wfuj5ZX).

[[207]](#ftnt_ref207) Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801435350533&usg=AOvVaw38UDBIdZLfiY1BQxYBZTfD). See also: Karnofsky, Holden. ‘How Might We Align Transformative AI If It’s Developed Very Soon?’ EA Forum, 29 August 2022.[https://forum.effectivealtruism.org/posts/sW6RggfddDrcmM6Aw/how-might-we-align-transformative-ai-if-it-s-developed-very](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/sW6RggfddDrcmM6Aw/how-might-we-align-transformative-ai-if-it-s-developed-very&sa=D&source=editors&ust=1692801435351047&usg=AOvVaw1vFOIKQh2Ct--crJcaJWPa).

[[208]](#ftnt_ref208) Zhang, Baobao, and Allan Dafoe. ‘Artificial Intelligence: American Attitudes and Trends’. Center for the Governance of AI and Future of Humanity Institute, January 2019.[https://www.ssrn.com/abstract=3312874](https://www.google.com/url?q=https://www.ssrn.com/abstract%3D3312874&sa=D&source=editors&ust=1692801435305693&usg=AOvVaw1LZv0s9To6zhfkWcIhW_oa).

[[209]](#ftnt_ref209) Zhang, Baobao. ‘No Rage Against the Machines: Threat of Automation Does Not Change Policy Preferences’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 856–66. AIES ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3514094.3534179](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534179&sa=D&source=editors&ust=1692801435306200&usg=AOvVaw15rOyL5uTY-tQEfQb6inod).; see also Cave, Stephen, Kate Coughlan, and Kanta Dihal. ‘“Scary Robots”: Examining Public Responses to AI’. In Proceedings of AAAI / ACM Conference on Artificial Intelligence, Ethics and Society 2019, 8, 2019. [https://dl.acm.org/doi/abs/10.1145/3306618.3314232](https://www.google.com/url?q=https://dl.acm.org/doi/abs/10.1145/3306618.3314232&sa=D&source=editors&ust=1692801435306489&usg=AOvVaw3mHfnzk7XfLMUPVr-3x3QR) ; O’Shaughnessy, Matthew, Daniel S. Schiff, Lav R. Varshney, Christopher Rozell, and Mark Davenport. ‘What Governs Attitudes toward Artificial Intelligence Adoption and Governance?’ OSF Preprints, 14 December 2021.[https://doi.org/10.31219/osf.io/pkeb8](https://www.google.com/url?q=https://doi.org/10.31219/osf.io/pkeb8&sa=D&source=editors&ust=1692801435306878&usg=AOvVaw3gDErohY9G0Zdfhv8g6sck).

[[210]](#ftnt_ref210) O’Shaughnessy, Matthew, Daniel S. Schiff, Lav R. Varshney, Christopher Rozell, and Mark Davenport. ‘What Governs Attitudes toward Artificial Intelligence Adoption and Governance?’ OSF Preprints, 14 December 2021.[https://doi.org/10.31219/osf.io/pkeb8](https://www.google.com/url?q=https://doi.org/10.31219/osf.io/pkeb8&sa=D&source=editors&ust=1692801435307358&usg=AOvVaw0grUBDQMuefWIX-9LUWyOR).; Stein-Perlman, Zach. ‘The Public Supports Regulating AI for Safety’. AI Impacts, 16 February 2023.[https://aiimpacts.org/the-public-supports-regulating-ai-for-safety/](https://www.google.com/url?q=https://aiimpacts.org/the-public-supports-regulating-ai-for-safety/&sa=D&source=editors&ust=1692801435307764&usg=AOvVaw31VsRDvSyGomTJTRXimkAQ). Citing: Monmouth University Poll. ‘National: Artificial Intelligence Use Prompts Concerns’. Monmouth University, 15 February 2023.[https://www.monmouth.edu/polling-institute/documents/monmouthpoll\_us\_021523.pdf/](https://www.google.com/url?q=https://www.monmouth.edu/polling-institute/documents/monmouthpoll_us_021523.pdf/&sa=D&source=editors&ust=1692801435308206&usg=AOvVaw0Hg4xksmWpuHrOtw0HkLKx).

[[211]](#ftnt_ref211) Martínez, Eric, and Christoph Winter. ‘Protecting Sentient Artificial Intelligence: A Survey of Lay Intuitions on Standing, Personhood, and General Legal Protection’. Frontiers in Robotics and AI 8 (2021).[https://www.frontiersin.org/articles/10.3389/frobt.2021.788355](https://www.google.com/url?q=https://www.frontiersin.org/articles/10.3389/frobt.2021.788355&sa=D&source=editors&ust=1692801435308785&usg=AOvVaw0wqt6i_R7N8gR0Z8c8MF3H).; see also generally: De Graaf, Maartje M. A., Frank A. Hindriks, and Koen V. Hindriks. ‘Who Wants to Grant Robots Rights?’ Frontiers in Robotics and AI 8 (2022).[https://www.frontiersin.org/articles/10.3389/frobt.2021.781985](https://www.google.com/url?q=https://www.frontiersin.org/articles/10.3389/frobt.2021.781985&sa=D&source=editors&ust=1692801435309238&usg=AOvVaw1RptPzqB1k9U2OSupdlQBU).

[[212]](#ftnt_ref212) Zhang, Baobao, and Allan Dafoe. ‘U.S. Public Opinion on the Governance of Artificial Intelligence’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 187–93. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375827](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375827&sa=D&source=editors&ust=1692801435309750&usg=AOvVaw0SqVyjdPrUjVQeDxQmMd4f).

[[213]](#ftnt_ref213) Zhang, Baobao, Markus Anderljung, Lauren Kahn, Noemi Dreksler, Michael C. Horowitz, and Allan Dafoe. ‘Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers’. Journal of Artificial Intelligence Research 71 (2 August 2021): 591-666-591–666.[https://doi.org/10.1613/jair.1.12895](https://www.google.com/url?q=https://doi.org/10.1613/jair.1.12895&sa=D&source=editors&ust=1692801435310277&usg=AOvVaw2bTpw6D865Xcr2I_FhvYbm).

[[214]](#ftnt_ref214) Krafft, P. M., Meg Young, Michael Katell, Karen Huang, and Ghislain Bugingo. ‘Defining AI in Policy versus Practice’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 72–78. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375835](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375835&sa=D&source=editors&ust=1692801435310810&usg=AOvVaw3-z_HGY-nrw0L5TBHi85tb).

[[215]](#ftnt_ref215) Imbrie, Andrew, James Dunham, Rebecca Gelles, and Catherine Aiken. ‘Mainframes: A Provisional Analysis of Rhetorical Frames in AI’. Center for Security and Emerging Technology, August 2020.[https://cset.georgetown.edu/research/mainframes-a-provisional-analysis-of-rhetorical-frames-in-ai/](https://www.google.com/url?q=https://cset.georgetown.edu/research/mainframes-a-provisional-analysis-of-rhetorical-frames-in-ai/&sa=D&source=editors&ust=1692801435311420&usg=AOvVaw1a0oem-ZcAuUpVoG3HkrY-). ; Imbrie, Andrew, Rebecca Gelles, James Dunham, and Catherine Aiken. ‘Contending Frames: Evaluating Rhetorical Dynamics in AI’. Center for Security and Emerging Technology, May 2021.[https://cset.georgetown.edu/publication/contending-frames/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/contending-frames/&sa=D&source=editors&ust=1692801435311884&usg=AOvVaw0HQ5U4V4cflAibiE_vnZYI).; on the history of AI narratives see also generally: Cave, Stephen, Kanta Dihal, and Sarah Dillon, eds. AI Narratives: A History of Imaginative Thinking about Intelligent Machines. New York: Oxford University Press, 2020.

[[216]](#ftnt_ref216) This overlaps with the evaluation of actors’ [Levers of control](#h.5romymfdade3), as well as pathways of influence. See especially the section on [Existing and Applicable Principles, Norms and Laws](#h.xtzugu6j7dxo).

[[217]](#ftnt_ref217) Solaiman, Irene. ‘The Gradient of Generative AI Release: Methods and Considerations’. arXiv, 5 February 2023.[https://doi.org/10.48550/arXiv.2302.04844](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2302.04844&sa=D&source=editors&ust=1692801435404413&usg=AOvVaw3AdytAjbUD2lG5DCWmDyHM).

[[218]](#ftnt_ref218) Hua, Shin-Shin, and Haydn Belfield. ‘AI & Antitrust: Reconciling Tensions Between Competition Law and Cooperative AI Development’. Yale Journal of Law and Technology 23 (Spring 2021): 127. [https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development](https://www.google.com/url?q=https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development&sa=D&source=editors&ust=1692801435312833&usg=AOvVaw3lBBMCQDyBvX4N44DCW1f3)

[[219]](#ftnt_ref219) Cihon, Peter. ‘Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development’. Technical Report. Oxford: Center for the Governance of AI, Future of Humanity Institute, University of Oxford, April 2019.[https://www.fhi.ox.ac.uk/wp-content/uploads/Standards\_-FHI-Technical-Report.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Standards_-FHI-Technical-Report.pdf&sa=D&source=editors&ust=1692801435313413&usg=AOvVaw3LrhTOQ7FByK62lvAZxkan).; O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801435313890&usg=AOvVaw0kkJST06vdo17lraZA9I5V).

[[220]](#ftnt_ref220) Kunz, Martina, and Seán Ó hÉigeartaigh. ‘Artificial Intelligence and Robotization’. In Oxford Handbook on the International Law of Global Security, edited by Robin Geiss and Nils Melzer. Oxford University Press, 2021.[https://papers.ssrn.com/abstract=3310421](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3310421&sa=D&source=editors&ust=1692801435314466&usg=AOvVaw2LQlr1ifzgZamqVnSO1Yfv).; Kemp, Luke, and Catherine Rhodes. ‘The Cartography of Global Catastrophic Governance’. Global Challenges Foundation, 2020.[https://globalchallenges.org/the-cartography-of-global-catastrophic-governance/](https://www.google.com/url?q=https://globalchallenges.org/the-cartography-of-global-catastrophic-governance/&sa=D&source=editors&ust=1692801435314903&usg=AOvVaw2sni32Bc9Pbv_jTB7qWiiZ). (pg. 4-6); Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435315335&usg=AOvVaw0UWp-C-tJo_tCueXCRSUQw) (pg. 94-118); Garcia, Eugenio V. ‘Multilateralism and Artificial Intelligence: What Role for the United Nations?’ In The Global Politics of Artificial Intelligence, edited by Maurizio Tinnirello, 18. Boca Raton: CRC Press, 2020.[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3779866](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3779866&sa=D&source=editors&ust=1692801435315771&usg=AOvVaw1hCP2DAO9AUF9-YTYQ5miU). See also the overview at: Kunz, Martina. ‘Global AI Governance’. Accessed 26 August 2022.[https://globalaigov.org/#](https://www.google.com/url?q=https://globalaigov.org/%23&sa=D&source=editors&ust=1692801435316157&usg=AOvVaw3JIZNEfIDYelLyXWigPYIJ). For an older account, see also Castel, J.G., and Mathew E. Castel. ‘The Road to Artificial Superintelligence - Has International Law a Role to Play?’ Canadian Journal of Law & Technology 14 (2016).[https://ojs.library.dal.ca/CJLT/article/download/7211/6256](https://www.google.com/url?q=https://ojs.library.dal.ca/CJLT/article/download/7211/6256&sa=D&source=editors&ust=1692801435316680&usg=AOvVaw23FDH5tMF7FASepRchuHRX).

[[221]](#ftnt_ref221) Schmitt, Lewin. ‘Mapping Global AI Governance: A Nascent Regime in a Fragmented Landscape’. AI and Ethics, 17 August 2021.[https://doi.org/10.1007/s43681-021-00083-y](https://www.google.com/url?q=https://doi.org/10.1007/s43681-021-00083-y&sa=D&source=editors&ust=1692801435317275&usg=AOvVaw0VWgqVNOR7M1uwDofEbolr).; see also Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435317704&usg=AOvVaw2hmq8Fppeed8xOHzwCxYJy). (pg. 94-121).

[[222]](#ftnt_ref222) Schiff, Daniel S. ‘Looking through a Policy Window with Tinted Glasses: Setting the Agenda for U.S. AI Policy’. Review of Policy Research n/a, no. n/a. Accessed 1 February 2023.[https://doi.org/10.1111/ropr.12535](https://www.google.com/url?q=https://doi.org/10.1111/ropr.12535&sa=D&source=editors&ust=1692801435357515&usg=AOvVaw26lD5usrUBR2QTzF7uBL8h).

[[223]](#ftnt_ref223) Scherer, Matthew U. ‘Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies’. Harvard Journal of Law & Technology, no. 2 (Spring 2016).[http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf](https://www.google.com/url?q=http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf&sa=D&source=editors&ust=1692801435428897&usg=AOvVaw3-5YoDO4hcqlICIsa5EdmI).; Schuett, Jonas. ‘Defining the Scope of AI Regulations’. Law, Innovation and Technology 0, no. 0 (3 March 2023): 1–23.[https://doi.org/10.1080/17579961.2023.2184135](https://www.google.com/url?q=https://doi.org/10.1080/17579961.2023.2184135&sa=D&source=editors&ust=1692801435429300&usg=AOvVaw3OR_l0_19wZV5UYH3KRJ0x).

[[224]](#ftnt_ref224) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435429927&usg=AOvVaw11nfVyTyqztXAjONWTBKti).

[[225]](#ftnt_ref225) Trager, Robert F. ‘The Security Governance Challenge of Emerging Technologies’. Orbis 66, no. 4 (2022): 536–50.[https://doi.org/10.1016/j.orbis.2022.08.008](https://www.google.com/url?q=https://doi.org/10.1016/j.orbis.2022.08.008&sa=D&source=editors&ust=1692801435430589&usg=AOvVaw3HtHCflrk8e_yDlW95K05B).

[[226]](#ftnt_ref226) Clare, Stephen. ‘The US Expands Restrictions on AI Exports to China. What Are the x-Risk Effects?’ EA Forum, 14 October 2022.[https://forum.effectivealtruism.org/posts/c6RnqjBd3BAkqsknB/the-us-expands-restrictions-on-ai-exports-to-china-what-are](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/c6RnqjBd3BAkqsknB/the-us-expands-restrictions-on-ai-exports-to-china-what-are&sa=D&source=editors&ust=1692801435318319&usg=AOvVaw18TDbnZumPzQBoXzsnKVD7).

[[227]](#ftnt_ref227) Casper, Stephen, Phillip Christoffersen, and Rui-Jie Yew. ‘The Slippery Slope from DALLE-2 to Deepfake Anarchy’, 5 November 2022.[https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarch](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy&sa=D&source=editors&ust=1692801435318926&usg=AOvVaw0IrIt0VpEuMXjLpQmFs6MM)[y](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy&sa=D&source=editors&ust=1692801435319144&usg=AOvVaw2RrpxDEgR2G8HQpDYs2d8Q).

[[228]](#ftnt_ref228) Huang, Michael. ‘NYT: Google Will “Recalibrate” the Risk of Releasing AI Due to Competition with OpenAI’. EA Forum, 22 January 2023.[https://forum.effectivealtruism.org/posts/Nm9ahJzKsDGFfF66b/nyt-google-will-recalibrate-the-risk-of-releasing-ai-due-to](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Nm9ahJzKsDGFfF66b/nyt-google-will-recalibrate-the-risk-of-releasing-ai-due-to&sa=D&source=editors&ust=1692801435319810&usg=AOvVaw1clki3ArsTwnPfMF-hGRRa).

[[229]](#ftnt_ref229) Krueger, David Scott. ‘An Update on Academia vs. Industry (One Year into My Faculty Job)’. AI Alignment Forum, 3 September 2022.[https://www.alignmentforum.org/posts/HXxHcRCxR4oHrAsEr/an-update-on-academia-vs-industry-one-year-into-my-faculty](https://www.google.com/url?q=https://www.alignmentforum.org/posts/HXxHcRCxR4oHrAsEr/an-update-on-academia-vs-industry-one-year-into-my-faculty&sa=D&source=editors&ust=1692801435320450&usg=AOvVaw2vC1TUCzKI47VsT6ckQwxk). But see also: Bensinger, Rob. ‘The Inordinately Slow Spread of Good AGI Conversations in ML’. LessWrong, 21 June 2022.[https://www.lesswrong.com/posts/Rkxj7TFxhbm59AKJh/the-inordinately-slow-spread-of-good-agi-conversations-in-ml](https://www.google.com/url?q=https://www.lesswrong.com/posts/Rkxj7TFxhbm59AKJh/the-inordinately-slow-spread-of-good-agi-conversations-in-ml&sa=D&source=editors&ust=1692801435320909&usg=AOvVaw18UKvxmTJrgmuCgN1fuyLM).

[[230]](#ftnt_ref230) Gabs, Nick. ‘Lessons from Three Mile Island for AI Warning Shots’. EA Forum, 26 September 2022.[https://forum.effectivealtruism.org/posts/NyCHoZGGw5YssvDJB/lessons-from-three-mile-island-for-ai-warning-shots](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/NyCHoZGGw5YssvDJB/lessons-from-three-mile-island-for-ai-warning-shots&sa=D&source=editors&ust=1692801435158241&usg=AOvVaw1vJkbzkFHyvnnr9MTELA6A).

[[231]](#ftnt_ref231) Baum, Seth D. ‘Countering Superintelligence Misinformation’. Information 9, no. 10 (30 September 2018): 244.[https://doi.org/10.3390/info9100244](https://www.google.com/url?q=https://doi.org/10.3390/info9100244&sa=D&source=editors&ust=1692801435051810&usg=AOvVaw32HzLR9cIuMXvxH6eU4MWJ). ; Baum, Seth D. ‘Superintelligence Skepticism as a Political Tool’. Information 9, no. 9 (22 August 2018): 209.[https://doi.org/10.3390/info9090209](https://www.google.com/url?q=https://doi.org/10.3390/info9090209&sa=D&source=editors&ust=1692801435052274&usg=AOvVaw0boS5yng8ztiuGB6yAXeUk).

[[232]](#ftnt_ref232) Dafoe, Allan, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. ‘Cooperative AI: Machines Must Learn to Find Common Ground’. Nature 593, no. 7857 (May 2021): 33–36.[https://doi.org/10.1038/d41586-021-01170-0](https://www.google.com/url?q=https://doi.org/10.1038/d41586-021-01170-0&sa=D&source=editors&ust=1692801435035738&usg=AOvVaw0laeRsn7LLvJGntr1xWh44).

[[233]](#ftnt_ref233) Deeks, Ashley. ‘High-Tech International Law’. George Washington Law Review 88 (2020): 575–653. [https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3531976](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3531976&sa=D&source=editors&ust=1692801434987929&usg=AOvVaw0iAHllExxr9V1rRCAnAW28) ; Maas, Matthijs M. ‘International Law Does Not Compute: Artificial Intelligence and The Development, Displacement or Destruction of the Global Legal Order’. Melbourne Journal of International Law 20, no. 1 (2019): 29–56. [https://law.unimelb.edu.au/\_\_data/assets/pdf\_file/0005/3144308/Maas.pdf](https://www.google.com/url?q=https://law.unimelb.edu.au/__data/assets/pdf_file/0005/3144308/Maas.pdf&sa=D&source=editors&ust=1692801434988360&usg=AOvVaw3mbJcQkAAD-U0sCxDO6YqM) ; Maas, Matthijs M. ‘AI, Governance Displacement, and the (De)Fragmentation of International Law’. In ISA Annual Convention, 2021.[https://www.cser.ac.uk/resources/ai-governance-displacement-and-defragmentation-international-law/](https://www.google.com/url?q=https://www.cser.ac.uk/resources/ai-governance-displacement-and-defragmentation-international-law/&sa=D&source=editors&ust=1692801434989050&usg=AOvVaw2Cs1c5aM0z2XOH8DJk09YZ).

[[234]](#ftnt_ref234) Mittelsteadt, Matthew. ‘AI Verification: Mechanisms to Ensure AI Arms Control Compliance’. Center for Security and Emerging Technology, February 2021.[https://live-cset-georgetown.pantheonsite.io/research/ai-verification/](https://www.google.com/url?q=https://live-cset-georgetown.pantheonsite.io/research/ai-verification/&sa=D&source=editors&ust=1692801434984387&usg=AOvVaw0T1e1_0m-VLOrZY0Ud7wAe).; see more generally work beyond the community, such as: Vaynman, Jane. ‘Better Monitoring and Better Spying: The Implications of Emerging Technology for Arms Control’. Texas National Security Review 4, no. 4 (23 September 2021).[https://tnsr.org/2021/09/better-monitoring-and-better-spying-the-implications-of-emerging-technology-for-arms-control/](https://www.google.com/url?q=https://tnsr.org/2021/09/better-monitoring-and-better-spying-the-implications-of-emerging-technology-for-arms-control/&sa=D&source=editors&ust=1692801434985197&usg=AOvVaw27U-OxCmS5f62_5CbbdN7q).; Reinhold, Thomas, and Niklas Schörnig. Armament, Arms Control and Artificial Intelligence: The Janus-Faced Nature of Machine Learning in the Military Realm. Springer Nature, 2022. [https://link.springer.com/book/10.1007/978-3-031-11043-6](https://www.google.com/url?q=https://link.springer.com/book/10.1007/978-3-031-11043-6&sa=D&source=editors&ust=1692801434985559&usg=AOvVaw33VragDvIY-bUChV6xOWMr); Lück, Nico. Machine Learning-Powered Artificial Intelligence in Arms Control. PRIF Report 2019, 8. Frankfurt am Main: Peace Research Institute Frankfurt, 2019.[https://www.hsfk.de/publikationen/publikationssuche/publikation/machine-learning-powered-artificial-intelligence-in-arms-control](https://www.google.com/url?q=https://www.hsfk.de/publikationen/publikationssuche/publikation/machine-learning-powered-artificial-intelligence-in-arms-control&sa=D&source=editors&ust=1692801434986354&usg=AOvVaw16Ux6amwK6nbRjqpUoKgYH).; Schörnig, Niklas. ‘AI for Arms Control : How Artificial Intelligence Can Foster Verification and Support Arms Control’. Peace Research Institute Frankfurt, 2022.[https://doi.org/10.48809/PRIFSPOT2201](https://www.google.com/url?q=https://doi.org/10.48809/PRIFSPOT2201&sa=D&source=editors&ust=1692801434986852&usg=AOvVaw28ic1n0jNOIWxOZroyarxW). See also Cox, Jessica, and Heather Williams. ‘The Unavoidable Technology: How Artificial Intelligence Can Strengthen Nuclear Stability’. The Washington Quarterly 44, no. 1 (2 January 2021): 69–85.[https://doi.org/10.1080/0163660X.2021.1893019](https://www.google.com/url?q=https://doi.org/10.1080/0163660X.2021.1893019&sa=D&source=editors&ust=1692801434987381&usg=AOvVaw0JMKGxDSs9ARl5hnyeXojO). Pg. 77-79 (on AI applications in arms control).

[[235]](#ftnt_ref235) See also ‘[Key Strategic Parameters of Transformative AI Governance](https://www.google.com/url?q=https://docs.google.com/document/u/0/d/1Mlt3rHcxJCBCGjSqrNJool0xB33GwmyH0bHjcveI7oc/edit&sa=D&source=editors&ust=1692801435242731&usg=AOvVaw3HklOvkmEEVK-0g03tW5Ro)’.

[[236]](#ftnt_ref236) Zwetsloot, Remco, and Allan Dafoe. ‘Thinking About Risks From AI: Accidents, Misuse and Structure’. Lawfare, 11 February 2019.[https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure](https://www.google.com/url?q=https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure&sa=D&source=editors&ust=1692801435245614&usg=AOvVaw087DM2_YiqyYmIpjOqzkyh).

[[237]](#ftnt_ref237) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435246295&usg=AOvVaw1Ybc43GbncrFp3Ec6Tp5_L). (Chapter 4).; Maas, Matthijs M. ‘Aligning AI Regulation to Sociotechnical Change’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.22](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.22&sa=D&source=editors&ust=1692801435246853&usg=AOvVaw37yKDBRTsP4PKT9w5dXioa).

[[238]](#ftnt_ref238) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022. [https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435247414&usg=AOvVaw0hgpWikUt_XDP5eBnfeh_l)

[[239]](#ftnt_ref239) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435248239&usg=AOvVaw1K2OQWz9oSc5hQ0H6QtUO0). For an argument that current large language models may have reached the level of performance to be GPTs, see: Eloundou, Tyna, Sam Manning, Pamela Mishkin, and Daniel Rock. ‘GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models’. arXiv, 19 March 2023.[https://doi.org/10.48550/arXiv.2303.10130](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2303.10130&sa=D&source=editors&ust=1692801435248673&usg=AOvVaw1yY_Py518dAaUB_GqXyWIS).

[[240]](#ftnt_ref240) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435249189&usg=AOvVaw1jlQm9H0QXLWibrGQZDmCS). pg 77-78; drawing on: Drezner, Daniel W. ‘Technological Change and International Relations’. International Relations 33, no. 2 (1 June 2019): 286–303.[https://doi.org/10.1177/0047117819834629](https://www.google.com/url?q=https://doi.org/10.1177/0047117819834629&sa=D&source=editors&ust=1692801435249634&usg=AOvVaw3aNAB93Ml_E5H3GcSLWB87).

[[241]](#ftnt_ref241) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435250333&usg=AOvVaw2LG_816Qc2YtD9WSPrbdi1).

[[242]](#ftnt_ref242) Ibid.

[[243]](#ftnt_ref243) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435250976&usg=AOvVaw1hhz9vKj_FzjSfM3vqhphA). Pg. 263-264; drawing on frameworks presented in: Crootof, Rebecca. ‘Jurisprudential Space Junk: Treaties and New Technologies’. In Resolving Conflicts in the Law, edited by Chiara Giorgetti and Natalie Klein, 106–29, 2019.[https://brill.com/view/book/edcoll/9789004316539/BP000015.xml](https://www.google.com/url?q=https://brill.com/view/book/edcoll/9789004316539/BP000015.xml&sa=D&source=editors&ust=1692801435251433&usg=AOvVaw3TiRjjT1Qpnsmmph1msI8Q).; Watts, Sean. ‘Regulation-Tolerant Weapons, Regulation-Resistant Weapons and the Law of War’. International Law Studies 91 (2015): 83. [https://digital-commons.usnwc.edu/cgi/viewcontent.cgi?article=1411&context=ils](https://www.google.com/url?q=https://digital-commons.usnwc.edu/cgi/viewcontent.cgi?article%3D1411%26context%3Dils&sa=D&source=editors&ust=1692801435251725&usg=AOvVaw1d2vwdR03GlxApPg15T4Yn) ; Watts, Sean. ‘Autonomous Weapons: Regulation Tolerant or Regulation Resistant?’ SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, 9 October 2015.[https://doi.org/10.2139/ssrn.2681283](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.2681283&sa=D&source=editors&ust=1692801435252090&usg=AOvVaw2unrt_0N1VkiOgp6x4sYUG).

[[244]](#ftnt_ref244) AI Impacts. ‘Friendly AI as a Global Public Good’. AI Impacts, 8 August 2016.[https://aiimpacts.org/friendly-ai-as-a-global-public-good/](https://www.google.com/url?q=https://aiimpacts.org/friendly-ai-as-a-global-public-good/&sa=D&source=editors&ust=1692801435252584&usg=AOvVaw1zBJAUgbYLahMloxDOIvUm). See also Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. (Chapter 2.2).

[[245]](#ftnt_ref245) Neufville, Robert de, and Seth D. Baum. ‘Collective Action on Artificial Intelligence: A Primer and Review’. Technology in Society 66 (1 August 2021): 101649.[https://doi.org/10.1016/j.techsoc.2021.101649](https://www.google.com/url?q=https://doi.org/10.1016/j.techsoc.2021.101649&sa=D&source=editors&ust=1692801435253117&usg=AOvVaw1x2W0thci_npxaMFm15CFV); Askell, Amanda, Miles Brundage, and Gillian Hadﬁeld. ‘The Role of Cooperation in Responsible AI Development’, 10 July 2019, 23. [https://arxiv.org/abs/1907.04534](https://www.google.com/url?q=https://arxiv.org/abs/1907.04534&sa=D&source=editors&ust=1692801435253308&usg=AOvVaw1V2SlgCPVNT1KHk4Lpe_o0)

[[246]](#ftnt_ref246) Karnofsky, Holden. ‘Racing through a Minefield: The AI Deployment Problem’. Cold Takes, 22 December 2022.[https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/](https://www.google.com/url?q=https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/&sa=D&source=editors&ust=1692801435253858&usg=AOvVaw358Pa3JPVA0XP871HSW538).

[[247]](#ftnt_ref247) Shulman, Carl. ‘Arms Control and Intelligence Explosions’, 6. Bellaterra, Spain, 2009.[https://intelligence.org/files/ArmsControl.pdf](https://www.google.com/url?q=https://intelligence.org/files/ArmsControl.pdf&sa=D&source=editors&ust=1692801435254378&usg=AOvVaw14C0K9r5tQ3CG5YjRRJvz1).; Armstrong, Stuart, Nick Bostrom, and Carl Shulman. ‘Racing to the Precipice: A Model of Artificial Intelligence Development’. AI & Society 31, no. 2 (2016): 201–6. [https://link.springer.com/article/10.1007/s00146-015-0590-y](https://www.google.com/url?q=https://link.springer.com/article/10.1007/s00146-015-0590-y&sa=D&source=editors&ust=1692801435254645&usg=AOvVaw1yEwjnN2hHIapCA6XdbsaG)

[[248]](#ftnt_ref248) Scharre, Paul. ‘Debunking the AI Arms Race Theory’. Texas National Security Review 4, no. 3 (28 June 2021).[https://tnsr.org/2021/06/debunking-the-ai-arms-race-theory/](https://www.google.com/url?q=https://tnsr.org/2021/06/debunking-the-ai-arms-race-theory/&sa=D&source=editors&ust=1692801435255180&usg=AOvVaw1qck_KYlHbePawKEHIq7rH).; Roff, Heather M. ‘The Frame Problem: The AI “Arms Race” Isn’t One’. Bulletin of the Atomic Scientists 0, no. 0 (26 April 2019): 1–4.[https://doi.org/10.1080/00963402.2019.1604836](https://www.google.com/url?q=https://doi.org/10.1080/00963402.2019.1604836&sa=D&source=editors&ust=1692801435255632&usg=AOvVaw2RJ8r8_nm0ISdePWY4f1-R).

[[249]](#ftnt_ref249) Kania, Elsa B. ‘Technological Entanglement: Cooperation, Competition and the Dual-Use Dilemma in Artificial Intelligence’. Policy Brief. Australian Strategic Policy Institute, 2018.[https://s3-ap-southeast-2.amazonaws.com/ad-aspi/2018-07/Tech-Entanglemen\_PolicyBrief\_20180702-v2.pdf?7BahbUgNHCY1umz4PCrLOEdBJUrjULCg](https://www.google.com/url?q=https://s3-ap-southeast-2.amazonaws.com/ad-aspi/2018-07/Tech-Entanglemen_PolicyBrief_20180702-v2.pdf?7BahbUgNHCY1umz4PCrLOEdBJUrjULCg&sa=D&source=editors&ust=1692801435256299&usg=AOvVaw3KsiabTGmZt9AmPgNtw41x). ; Bryson, Joanna J., and Helena Malikova. ‘Is There an AI Cold War?’ Global Perspectives 2, no. 1 (28 June 2021): 24803.[https://doi.org/10.1525/gp.2021.24803](https://www.google.com/url?q=https://doi.org/10.1525/gp.2021.24803&sa=D&source=editors&ust=1692801435256751&usg=AOvVaw0dDAG8D2BgVzzhcAMXIq6M).; see also Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435257298&usg=AOvVaw3Vp4ulReRlbsGBUwaoKCzS). (“At present the “arms” modifier is largely literally off-point,since most of the geopolitical activity in AI is not about weapons per se, but is insteadabout supply chains, infrastructure, industrial base, strategic industries, scientific capability, and prestige achievements”).

[[250]](#ftnt_ref250) Cave, Stephen, and Seán S. ÓhÉigeartaigh. ‘An AI Race for Strategic Advantage: Rhetoric and Risks’. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 36–40. New Orleans LA USA: ACM, 2018.[https://doi.org/10.1145/3278721.3278780](https://www.google.com/url?q=https://doi.org/10.1145/3278721.3278780&sa=D&source=editors&ust=1692801435257848&usg=AOvVaw0cGLuyrmafVSolYz02c1KJ).; Belfield, Haydn. ‘Are You Really in a Race? The Cautionary Tales of Szilárd and Ellsberg’. EA Forum, 2022.[https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and&sa=D&source=editors&ust=1692801435258365&usg=AOvVaw398Dhq3bGnNovQ3B8grD31).

[[251]](#ftnt_ref251) Han, The Anh, Luis Moniz Pereira, and Tom Lenaerts. ‘Modelling and Influencing the AI Bidding War: A Research Agenda’, 2019.[http://www.aies-conference.com/wp-content/papers/main/AIES-19\_paper\_28.pdf](https://www.google.com/url?q=http://www.aies-conference.com/wp-content/papers/main/AIES-19_paper_28.pdf&sa=D&source=editors&ust=1692801435258958&usg=AOvVaw0vZmH0idxpj8hx_aKNipZW).; Stafford, Eoghan, Robert Trager, and Allan Dafoe. ‘International Strategic Dynamics of Risky Technology Races’, June 2021.[https://www.academia.edu/49586612/International\_Strategic\_Dynamics\_of\_Risky\_Technology\_Races](https://www.google.com/url?q=https://www.academia.edu/49586612/International_Strategic_Dynamics_of_Risky_Technology_Races&sa=D&source=editors&ust=1692801435259381&usg=AOvVaw2fRYfot18ZJjQZMQTrEptc)

[[252]](#ftnt_ref252) Barnhart, Joslyn. ‘Emerging Technologies, Prestige Motivations and the Dynamics of International Competition’, January 2022, 56.

[https://www.governance.ai/research-paper/emerging-technologies-prestige-motivations-and-the-dynamics-of-international-competition](https://www.google.com/url?q=https://www.governance.ai/research-paper/emerging-technologies-prestige-motivations-and-the-dynamics-of-international-competition&sa=D&source=editors&ust=1692801435259841&usg=AOvVaw0rVdSsMvjr9vSrHbrHXMbS)

[[253]](#ftnt_ref253) Gruetzemacher, Ross. ‘Rethinking AI Strategy and Policy as Entangled Super Wicked Problems’, 6. AIES 2018; New Orleans, 2018.[http://www.rossgritz.com/wp-content/uploads/2018/11/aies\_gruetzemacher\_revisions.pdf](https://www.google.com/url?q=http://www.rossgritz.com/wp-content/uploads/2018/11/aies_gruetzemacher_revisions.pdf&sa=D&source=editors&ust=1692801435260420&usg=AOvVaw0of6eKr9svdee7JOmdWRQL); Liu, Hin-Yan, and Matthijs M. Maas. ‘“Solving for X?” Towards a Problem-Finding Framework to Ground Long-Term Governance Strategies for Artificial Intelligence’. Futures 126 (1 February 2021): 22.[https://doi.org/10.1016/j.futures.2020.102672](https://www.google.com/url?q=https://doi.org/10.1016/j.futures.2020.102672&sa=D&source=editors&ust=1692801435261151&usg=AOvVaw2PYxiuD3J2ADfmvQE5P2in).

[[254]](#ftnt_ref254) Kemp, Luke. ‘Agents of Doom: Who Is Creating the Apocalypse and Why’. BBC Future, 26 October 2021.[https://www.bbc.com/future/article/20211014-agents-of-doom-who-is-hastening-the-apocalypse-and-why](https://www.google.com/url?q=https://www.bbc.com/future/article/20211014-agents-of-doom-who-is-hastening-the-apocalypse-and-why&sa=D&source=editors&ust=1692801435261736&usg=AOvVaw0eHq7B2X4UW3IkwSL3-kn1).

[[255]](#ftnt_ref255) See generally: Koremenos, Barbara, Charles Lipson, and Duncan Snidal. ‘The Rational Design of International Institutions’. International Organization 55, no. 4 (ed 2001): 761–99.[https://doi.org/10.1162/002081801317193592](https://www.google.com/url?q=https://doi.org/10.1162/002081801317193592&sa=D&source=editors&ust=1692801435321439&usg=AOvVaw2qIfa9A-Z5dDnF_WqFxspS).

[[256]](#ftnt_ref256) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Fragmentation and the Future: Investigating Architectures for International AI Governance’. Global Policy 11, no. 5 (November 2020): 545–56.[https://doi.org/10.1111/1758-5899.12890](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12890&sa=D&source=editors&ust=1692801435322056&usg=AOvVaw3k7Uce_GDOZOoXhKfvgihi).; Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Should Artificial Intelligence Governance Be Centralised?: Design Lessons from History’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 228–34. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375857](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375857&sa=D&source=editors&ust=1692801435322458&usg=AOvVaw0fZ9TYThsTN3xOwrtP1Y6-). See also generally: Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435322886&usg=AOvVaw3OaCzWIWTIkQ2RbIvMCH0I). (Chapter 7).

[[257]](#ftnt_ref257) Staﬀord, Eoghan, Robert F Trager, and Allan Dafoe. ‘Safety Not Guaranteed: International Strategic Dynamics of Risky Technology Races’, June 2022, 31.

[[258]](#ftnt_ref258) Emery-Xu, Nicholas, Andrew Park, and Robert Trager. ‘Uncertainty, Information, and Risk in International Technology Races’, June 2022.[https://drive.google.com/file/d/18j\_wnA4HDMA3ofclLcfpgyV-0INMn1ZW/view?usp=sharing&](https://www.google.com/url?q=https://drive.google.com/file/d/18j_wnA4HDMA3ofclLcfpgyV-0INMn1ZW/view?usp%3Dsharing%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435001005&usg=AOvVaw0SODYbbWVFJMFUFHUphm41).

[[259]](#ftnt_ref259) Han, The Anh, Luis Moniz Pereira, and Tom Lenaerts. ‘Modelling and Influencing the AI Bidding War: A Research Agenda’. In AAAI/ACM Conference on AI, Ethics, and Society, 5–11, 27 January 2019. [https://doi.org/10.1145/3306618.3314265](https://www.google.com/url?q=https://doi.org/10.1145/3306618.3314265&sa=D&source=editors&ust=1692801435106500&usg=AOvVaw3i9T8J-Xk-e0qW48oj3bp5), also available at[http://www.aies-conference.com/wp-content/papers/main/AIES-19\_paper\_28.pdf](https://www.google.com/url?q=http://www.aies-conference.com/wp-content/papers/main/AIES-19_paper_28.pdf&sa=D&source=editors&ust=1692801435106966&usg=AOvVaw3W3T6u7y8I2wj7OMyw8Pj2). Han, The Anh, Luis Moniz Pereira, Tom Lenaerts, and Francisco C. Santos. ‘Mediating Artificial Intelligence Developments through Negative and Positive Incentives’. ArXiv:2010.00403 [Nlin, q-Bio], 1 October 2020.[http://arxiv.org/abs/2010.00403](https://www.google.com/url?q=http://arxiv.org/abs/2010.00403&sa=D&source=editors&ust=1692801435107422&usg=AOvVaw1ybs4GfXWtueHCAfmA0dVl).

[[260]](#ftnt_ref260) Bova, Paolo, Jonas Emanuel Müller, Tanja Rüegg, and Robert Trager. ‘Announcing the SPT Model Web App for AI Governance’. EA Forum, 4 August 2022.[https://forum.effectivealtruism.org/posts/c73nsggC2GQE5wBjq/announcing-the-spt-model-web-app-for-ai-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/c73nsggC2GQE5wBjq/announcing-the-spt-model-web-app-for-ai-governance&sa=D&source=editors&ust=1692801435108619&usg=AOvVaw3InXK7lrjeIDoeWrd89nD5).; See model at [https://spt.modelingcooperation.com/](https://www.google.com/url?q=https://spt.modelingcooperation.com/&sa=D&source=editors&ust=1692801435108838&usg=AOvVaw0ACmYme3OzVX8b9Zyz-j2O) ; See also Robert Trager, Paolo Bova, Nicholas Emery-Xu, Eoghan Stafford, and Allan Dafoe, "Welfare Implications of Safety-Performance Tradeoffs in AI Safety Research", Working paper, August 2022.

[[261]](#ftnt_ref261) Han, The Anh, Francisco C. Santos, Luís Moniz Pereira, and Tom Lenaerts. ‘A Regulation Dilemma in Artificial Intelligence Development’. MIT Press, 2021.[https://doi.org/10.1162/isal\_a\_00385](https://www.google.com/url?q=https://doi.org/10.1162/isal_a_00385&sa=D&source=editors&ust=1692801435109324&usg=AOvVaw1hP5CPjhDfd7qtGS-91pPX).

[[262]](#ftnt_ref262) Garfinkel, Ben, and Allan Dafoe. ‘How Does the Offense-Defense Balance Scale?’ Journal of Strategic Studies 42, no. 6 (19 September 2019): 736–63.[https://doi.org/10.1080/01402390.2019.1631810](https://www.google.com/url?q=https://doi.org/10.1080/01402390.2019.1631810&sa=D&source=editors&ust=1692801435001742&usg=AOvVaw0SXUhuDYLNKYCsaPJjccQ2).

[[263]](#ftnt_ref263) Shevlane, Toby, and Allan Dafoe. ‘The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?’ In AIES ’20: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2020.[http://arxiv.org/abs/2001.00463](https://www.google.com/url?q=http://arxiv.org/abs/2001.00463&sa=D&source=editors&ust=1692801435002453&usg=AOvVaw0olvZJHG1uQs0kiz2Y6dEb).

[[264]](#ftnt_ref264) Pulver, Tobias. ‘Shaping Policy as Experts: An Epistemic Community for (Transformative) AI Governance?’ 2019. [https://docs.google.com/document/d/1h7YHlp44kANhXPo8zJdr5ea1mttF8E5l-pJ\_w8v\_quE/edit?usp=drive\_web&ouid=107201564093427841585&](https://www.google.com/url?q=https://docs.google.com/document/d/1h7YHlp44kANhXPo8zJdr5ea1mttF8E5l-pJ_w8v_quE/edit?usp%3Ddrive_web%26ouid%3D107201564093427841585%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435123782&usg=AOvVaw1xnbAw7XJLIkJ2UrwcA5Dl)

[[265]](#ftnt_ref265) Avin, Shahar, Ross Gruetzemacher, and James Fox. ‘Exploring AI Futures Through Role Play’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8–14. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375817](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375817&sa=D&source=editors&ust=1692801435063587&usg=AOvVaw1vwjYG6iQ09sYsFY3QJipX).; see more broadly the review of methodologies in Avin, Shahar. ‘Exploring Artificial Intelligence Futures’. AIHumanities, 17 January 2019.[http://aihumanities.org/en/journal/past-issues/?board\_name=Enjournal&search\_field=fn\_title&search\_text=Exploring%20&vid=15](https://www.google.com/url?q=http://aihumanities.org/en/journal/past-issues/?board_name%3DEnjournal%26search_field%3Dfn_title%26search_text%3DExploring%2520%26vid%3D15&sa=D&source=editors&ust=1692801435064126&usg=AOvVaw1VmaqHTyPHgEOApqk_WF3W).

[[266]](#ftnt_ref266) This list is obviously not exhaustive. It lists cases that have been identified, flagged, or studied by researchers in the field, however there are many additional possible cases. In the discussion of ‘historical analogies’ of each perspective, I will suggest a number of additional plausible historical cases that could yield valuable lessons, insights, or support to a given perspective; as well as counter-examples that highlight potential failure modes or barriers to be overcome.

[[267]](#ftnt_ref267) Ord, Toby. ‘Lessons from the Development of the Atomic Bomb’. Center for the Governance of AI, November 2022.[https://www.governance.ai/research-paper/lessons-atomic-bomb-ord](https://www.google.com/url?q=https://www.governance.ai/research-paper/lessons-atomic-bomb-ord&sa=D&source=editors&ust=1692801435104970&usg=AOvVaw2CalShOF98PBx8YP67jQJN). (exploring insights in terms of the scientific, engineering and political prerequisites for making the atomic bomb; the labour and money invested; the role and efficacy of secrecy in various national programmes; the role of spying; the ability of scientists to provide decision makers with useful estimates of the cost and effects of an atomic bomb; US decision-making about whether and how to use the bomb; the effects of the atomic bombings of Japan; the subsequent efforts of atomic scientists to control the development and use of the technology; the impact of individual actors on the development of atomic weapons, and how scientists managed the potential existential risk of nuclear weapons igniting the atmosphere).

[[268]](#ftnt_ref268) Levin, John-Clark, and Matthijs M. Maas. ‘Roadmap to a Roadmap: How Could We Tell When AGI Is a “Manhattan Project” Away?’, 7. Santiago de Compostela, Spain, 2020.[http://dmip.webs.upv.es/EPAI2020/papers/EPAI\_2020\_paper\_11.pdf](https://www.google.com/url?q=http://dmip.webs.upv.es/EPAI2020/papers/EPAI_2020_paper_11.pdf&sa=D&source=editors&ust=1692801435105561&usg=AOvVaw2Jt7vlTfQ349x3r5C4EjPn).

[[269]](#ftnt_ref269) Barnhart, Joslyn. ‘Emerging Technologies, Prestige Motivations and the Dynamics of International Competition’, January 2022, 56. [https://www.governance.ai/research-paper/emerging-technologies-prestige-motivations-and-the-dynamics-of-international-competition](https://www.google.com/url?q=https://www.governance.ai/research-paper/emerging-technologies-prestige-motivations-and-the-dynamics-of-international-competition&sa=D&source=editors&ust=1692801435002984&usg=AOvVaw2YyQYYuSdNC09hKVEcIi8D)

[[270]](#ftnt_ref270) Sagan, Scott D. ‘Why Do States Build Nuclear Weapons?: Three Models in Search of a Bomb’. International Security 21, no. 3 (1996): 54–86. [http://www.jstor.org/stable/2539273](https://www.google.com/url?q=http://www.jstor.org/stable/2539273&sa=D&source=editors&ust=1692801435380161&usg=AOvVaw3_QB9M3APKkzApEcO_fhe6)

[[271]](#ftnt_ref271) Belfield, Haydn. ‘Are You Really in a Race? The Cautionary Tales of Szilárd and Ellsberg’. EA Forum, 2022.[https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and&sa=D&source=editors&ust=1692801435056066&usg=AOvVaw1dEuJWmREhajOYW-Y14XUl).

[[272]](#ftnt_ref272) Narang, Vipin. Seeking the Bomb: Strategies of Nuclear Proliferation. Princeton Studies in International History and Politics. Princeton University Press, 2022.[https://press.princeton.edu/books/paperback/9780691172620/seeking-the-bomb](https://www.google.com/url?q=https://press.princeton.edu/books/paperback/9780691172620/seeking-the-bomb&sa=D&source=editors&ust=1692801435437856&usg=AOvVaw1S8rssmk6YdpMdt-44d0uC).

[[273]](#ftnt_ref273) Robinson, Mark. ‘Big Science Collaborations; Lessons for Global Governance and Leadership’. Global Policy n/a, no. n/a (2020).[https://doi.org/10.1111/1758-5899.12861](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12861&sa=D&source=editors&ust=1692801435358009&usg=AOvVaw2FQurRzz6Lh4yPzifmmPUT). Robinson, Mark. ‘The CERN Community; A Mechanism for Effective Global Collaboration?’ Global Policy, 18 November 2018.[https://doi.org/10.1111/1758-5899.12608](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12608&sa=D&source=editors&ust=1692801435358404&usg=AOvVaw09BxbsB-X--WnClYVzCnK9). (discussing CERN, ITER, and ISS); see also: Kerry, Cameron F, Joshua P Meltzer, and Andrea Renda. ‘AI Cooperation on the Ground: AI Research and Development on a Global Scale’. Brookings Institute & Forum for Cooperation on Artificial Intelligence (FCAI), October 2022.[https://www.brookings.edu/wp-content/uploads/2022/11/FCAI-October-2022.pdf](https://www.google.com/url?q=https://www.brookings.edu/wp-content/uploads/2022/11/FCAI-October-2022.pdf&sa=D&source=editors&ust=1692801435358847&usg=AOvVaw0lgt0CBFq-r-d0Jy_sczSe). Appendix. (discussing the governance and finance arrangements of the HGP, ISS, and CERN, to derive lessons for AI).

[[274]](#ftnt_ref274) Krige, John, Angelina Long Callahan, and Ashok Maharaj. ‘Sustaining Soviet-American Collaboration, 1957–1989’. In NASA in the World: Fifty Years of International Collaboration in Space, edited by John Krige, Angelina Long Callahan, and Ashok Maharaj, 127–51. Palgrave Studies in the History of Science and Technology. New York: Palgrave Macmillan US, 2013.[https://doi.org/10.1057/9781137340931\_7](https://www.google.com/url?q=https://doi.org/10.1057/9781137340931_7&sa=D&source=editors&ust=1692801435363605&usg=AOvVaw2Qoxpg_dbQu2sOmxaRiKgE). I thank Christian Ruhl for this suggestion.

[[275]](#ftnt_ref275) See generally: Fleming, James Rodger. ‘The Pathological History of Weather and Climate Modification: Three Cycles of Promise and Hype’. Historical Studies in the Physical and Biological Sciences 37, no. 1 (1 September 2006): 3–25.[https://doi.org/10.1525/hsps.2006.37.1.3](https://www.google.com/url?q=https://doi.org/10.1525/hsps.2006.37.1.3&sa=D&source=editors&ust=1692801435109825&usg=AOvVaw0D0GDXii-YobIX6n3H7KSz).

[[276]](#ftnt_ref276) Picker, Colin B. ‘A View from 40,000 Feet: International Law and the Invisible Hand of Technology’. Cardozo Law Review 23 (2001): 151–219. [https://papers.ssrn.com/abstract=987524](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D987524&sa=D&source=editors&ust=1692801435102977&usg=AOvVaw3zm3_fuqrdIspMfFHlGwxA)

[[277]](#ftnt_ref277) See for example Chafkin, Max. ‘Even After $100 Billion, Self-Driving Cars Are Going Nowhere’. Bloomberg.Com, 6 October 2022.[https://www.bloomberg.com/news/features/2022-10-06/even-after-100-billion-self-driving-cars-are-going-nowhere](https://www.google.com/url?q=https://www.bloomberg.com/news/features/2022-10-06/even-after-100-billion-self-driving-cars-are-going-nowhere&sa=D&source=editors&ust=1692801435110475&usg=AOvVaw3cqvx5NVd51wRq8cxfXaUX).; but see also: Templeton, Brad. ‘Reports Of The Death Of Self-Driving Cars Are Greatly Exaggerated’. Forbes, 15 November 2022, sec. Transportation.[https://www.forbes.com/sites/bradtempleton/2022/11/15/reports-of-the-death-of-self-driving-cars-are-greatly-exaggerated/](https://www.google.com/url?q=https://www.forbes.com/sites/bradtempleton/2022/11/15/reports-of-the-death-of-self-driving-cars-are-greatly-exaggerated/&sa=D&source=editors&ust=1692801435111034&usg=AOvVaw0QSrxxI7O8d6aaKVUOc7mA).

[[278]](#ftnt_ref278) Liao, Tony, and Andrew Iliadis. ‘A Future so Close: Mapping 10 Years of Promises and Futures across the Augmented Reality Development Cycle’. New Media & Society 23, no. 2 (1 February 2021): 258–83.[https://doi.org/10.1177/1461444820924623](https://www.google.com/url?q=https://doi.org/10.1177/1461444820924623&sa=D&source=editors&ust=1692801435352796&usg=AOvVaw3ocpMVsqY42rlbriUHIQLu).

[[279]](#ftnt_ref279) E.g. see: Gartner. ‘What’s New in the 2022 Gartner Hype Cycle for Emerging Technologies’. Gartner, 2023.[https://www.gartner.com/en/articles/what-s-new-in-the-2022-gartner-hype-cycle-for-emerging-technologies](https://www.google.com/url?q=https://www.gartner.com/en/articles/what-s-new-in-the-2022-gartner-hype-cycle-for-emerging-technologies&sa=D&source=editors&ust=1692801435353463&usg=AOvVaw1aMx4Nwg1B_aJmwtKSEQBd).; though for a critique, see also: Steinert, Martin, and Larry Leifer. ‘Scrutinizing Gartner’s Hype Cycle Approach’. In PICMET 2010 Proceedings. Phuket, Thailand, 2010.[https://www.researchgate.net/profile/Martin\_Steinert/publication/224182916\_Scrutinizing\_Gartner%27s\_hype\_cycle\_approach/links/543005400cf29bbc1273c7e1/Scrutinizing-Gartners-hype-cycle-approach.pdf](https://www.google.com/url?q=https://www.researchgate.net/profile/Martin_Steinert/publication/224182916_Scrutinizing_Gartner%2527s_hype_cycle_approach/links/543005400cf29bbc1273c7e1/Scrutinizing-Gartners-hype-cycle-approach.pdf&sa=D&source=editors&ust=1692801435354122&usg=AOvVaw2UAfr5Xdx9hagaxxnvmMUs).

[[280]](#ftnt_ref280) Grace, Katja. ‘Discontinuous Progress in History: An Update’. AI Impacts, 13 April 2020.[https://aiimpacts.org/discontinuous-progress-in-history-an-update/](https://www.google.com/url?q=https://aiimpacts.org/discontinuous-progress-in-history-an-update/&sa=D&source=editors&ust=1692801435412455&usg=AOvVaw3gg-6JWGp3za6L6PxtTygW). (defining such ‘large robust discontinuities’ as events which “abruptly and clearly contributed more to progress on some technological metric than another century would have seen on the previous trend”).

[[281]](#ftnt_ref281) Ibid. (In addition, they also identify 5 ‘moderate robust discontinuities’ (events that suddenly contribute around 10-100 years of progress of previous trends).

[[282]](#ftnt_ref282) Yudkowsky, Eliezer. ‘There’s No Fire Alarm for Artificial General Intelligence’. Machine Intelligence Research Institute (blog), 14 October 2017.[https://intelligence.org/2017/10/13/fire-alarm/](https://www.google.com/url?q=https://intelligence.org/2017/10/13/fire-alarm/&sa=D&source=editors&ust=1692801435103512&usg=AOvVaw0ExlzkKzAcD1mdKqSt-Mq2). See also Schwartz, Baron. ‘Heavier-Than-Air Flight Is Impossible’, 4 June 2017.[https://www.xaprb.com/blog/flight-is-impossible/](https://www.google.com/url?q=https://www.xaprb.com/blog/flight-is-impossible/&sa=D&source=editors&ust=1692801435103879&usg=AOvVaw0ABQD8z9XSJ4BBXriaIzuV).

[[283]](#ftnt_ref283) See examples discussed in: Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435104437&usg=AOvVaw3oENlBEqrMriit-RMJvHUE). (pg. 62-63, citing sources).

[[284]](#ftnt_ref284) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435262411&usg=AOvVaw0x58AkJSC92dgxGxGRtFm4). Pg. 62, ftn 169 (“in the first edition of his 2001 book On the Internet, Hubert Dreyfus built on his previous critique of AI to argue against the very possibility of searching the internet, claiming that without embodied knowledge, online search would hit an intractable wall. [...] These sections were quietly dropped from the book’s Second Edition, published after Google’s 2004 IPO.”). Drawing on: Dreyfus, Hubert. On the Internet. 1st ed. Routledge, 2001.[https://www.abebooks.com/first-edition/Internet-Thinking-Action-HUBERT-DREYFUS-Routledge/30612233740/bd](https://www.google.com/url?q=https://www.abebooks.com/first-edition/Internet-Thinking-Action-HUBERT-DREYFUS-Routledge/30612233740/bd&sa=D&source=editors&ust=1692801435262981&usg=AOvVaw3FoF2hZUzt8R-5E8oe5tzY).

[[285]](#ftnt_ref285) For work on the broader relevance of technological diffusion patterns, rather than just innovation capabilities, in determining national competitiveness, see: Ding, Jeffrey. ‘The Diffusion Deficit in Scientific and Technological Power: Re-Assessing China’s Rise’. Review of International Political Economy 0, no. 0 (13 March 2023): 1–26.[https://doi.org/10.1080/09692290.2023.2173633](https://www.google.com/url?q=https://doi.org/10.1080/09692290.2023.2173633&sa=D&source=editors&ust=1692801435438452&usg=AOvVaw1BQP8Xli8P5rZGCg4Lg4zk).

[[286]](#ftnt_ref286) Korzekwa, Rick. ‘Observed Patterns around Major Technological Advancements’. AI Impacts, 2 February 2022.[https://aiimpacts.org/observed-patterns-around-major-technological-advancements/](https://www.google.com/url?q=https://aiimpacts.org/observed-patterns-around-major-technological-advancements/&sa=D&source=editors&ust=1692801435006185&usg=AOvVaw2nfDjnLNNN5j69p35My8WP). For an overview of the underlying 53 case studies, see: AI Impacts. ‘Discontinuous Progress Investigation’. AI Impacts (blog), 2 February 2015.[https://aiimpacts.org/discontinuous-progress-investigation/](https://www.google.com/url?q=https://aiimpacts.org/discontinuous-progress-investigation/&sa=D&source=editors&ust=1692801435006814&usg=AOvVaw3unvpq-zxxz13bRStzZLto).

[[287]](#ftnt_ref287) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435007566&usg=AOvVaw07946wgeuK01Kc_rtL2AQN). Pg. 79. Drawing on: Gill, Indermit. ‘Whoever Leads in Artificial Intelligence in 2030 Will Rule the World until 2100’. Brookings (blog), 17 January 2020.[https://www.brookings.edu/blog/future-development/2020/01/17/whoever-leads-in-artificial-intelligence-in-2030-will-rule-the-world-until-2100/](https://www.google.com/url?q=https://www.brookings.edu/blog/future-development/2020/01/17/whoever-leads-in-artificial-intelligence-in-2030-will-rule-the-world-until-2100/&sa=D&source=editors&ust=1692801435008361&usg=AOvVaw2WdX9PT8mHgXxIbIMWQrvA).; Comin, Diego, and Martí Mestieri. ‘If Technology Has Arrived Everywhere, Why Has Income Diverged?’ American Economic Journal: Macroeconomics 10, no. 3 (July 2018): 137–78.[https://doi.org/10.1257/mac.20150175](https://www.google.com/url?q=https://doi.org/10.1257/mac.20150175&sa=D&source=editors&ust=1692801435008933&usg=AOvVaw3K-kCYYAYdJscpeoJzTOvS); Comin, Diego, and Martı Mestieri. ‘Technology Adoption and Growth Dynamics’, 2014, 38. [https://pdfs.semanticscholar.org/1f76/728473ee4fb154fe1655a4645c4b43b29358.pdf](https://www.google.com/url?q=https://pdfs.semanticscholar.org/1f76/728473ee4fb154fe1655a4645c4b43b29358.pdf&sa=D&source=editors&ust=1692801435009278&usg=AOvVaw3Lq9xqKakpQbpvWbU7rWGH).

See more generally the discussion of distinct types of technologies and their proliferation profiles, in Drezner, Daniel W. ‘Technological Change and International Relations’. International Relations 33, no. 2 (1 June 2019): 286–303.[https://doi.org/10.1177/0047117819834629](https://www.google.com/url?q=https://doi.org/10.1177/0047117819834629&sa=D&source=editors&ust=1692801435009988&usg=AOvVaw1lw44y2bZZpZYGRD6pQbSY).

[[288]](#ftnt_ref288) Ord, Toby. ‘Lessons from the Development of the Atomic Bomb’. Center for the Governance of AI, November 2022.[https://www.governance.ai/research-paper/lessons-atomic-bomb-ord](https://www.google.com/url?q=https://www.governance.ai/research-paper/lessons-atomic-bomb-ord&sa=D&source=editors&ust=1692801435017196&usg=AOvVaw22mwKSMGu5TZk1oEj1DEIb).; see also: GAA. ‘Nuclear Espionage and AI Governance’. EA Forum, 2021.[https://forum.effectivealtruism.org/posts/CKfHDw5Lmoo6jahZD/nuclear-espionage-and-ai-governance-1](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/CKfHDw5Lmoo6jahZD/nuclear-espionage-and-ai-governance-1&sa=D&source=editors&ust=1692801435017742&usg=AOvVaw2o38N4MBuAD4jTDlyQQm93).

[[289]](#ftnt_ref289) Muelhauser, Luke. ‘Example High-Stakes Information Security Breaches [Public]’, June 2020.[https://docs.google.com/document/d/1\_smEDPWDVIaLuZ14Cm7KLHcWx4LkJ0DCTk8bcHjYy\_Y/edit#heading=h.hqf76e8phc7g](https://www.google.com/url?q=https://docs.google.com/document/d/1_smEDPWDVIaLuZ14Cm7KLHcWx4LkJ0DCTk8bcHjYy_Y/edit%23heading%3Dh.hqf76e8phc7g&sa=D&source=editors&ust=1692801435058935&usg=AOvVaw2FWOf679CCy26WvuND25bR).

[[290]](#ftnt_ref290) Salisbury, Adam. ‘How Will the AI Supply Chain Evolve?’, 2022.[https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/edit?usp=sharing&](https://www.google.com/url?q=https://docs.google.com/document/d/1s3QGFJ8Ochosksl4JgQCWekJrsY3YFAfGgEiEt6zFpA/edit?usp%3Dsharing%26usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435173583&usg=AOvVaw0ZDAW7QxiPkxYi3Z7CWuox). (pg. 21-25).

[[291]](#ftnt_ref291) The distinction between ‘narrowly transformative’, ‘transformative’, and ‘radically transformative’ is found in Gruetzemacher, Ross, and Jess Whittlestone. ‘The Transformative Potential of Artificial Intelligence’. Futures 135 (2022): 102884.[https://doi.org/10.1016/j.futures.2021.102884](https://www.google.com/url?q=https://doi.org/10.1016/j.futures.2021.102884&sa=D&source=editors&ust=1692801435019084&usg=AOvVaw1pCo6T6F031LCdPCHLIFMb).

[[292]](#ftnt_ref292) Ding, Jeffrey, and Allan Dafoe. ‘Engines of Power: Electricity, AI, and General-Purpose, Military Transformations’. European Journal of International Security, 7 February 2023, 1–18.[https://doi.org/10.1017/eis.2023.1](https://www.google.com/url?q=https://doi.org/10.1017/eis.2023.1&sa=D&source=editors&ust=1692801435010737&usg=AOvVaw01dY12AYmvOOMAo2rk69KE).

[[293]](#ftnt_ref293) Garfinkel, Ben. ‘The Impact of Artificial Intelligence: A Historical Perspective’. In The Oxford Handbook of AI Governance, edited by Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang, 0. Oxford University Press, 2023.[https://doi.org/10.1093/oxfordhb/9780197579329.013.5](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.5&sa=D&source=editors&ust=1692801435011414&usg=AOvVaw2tHMPAtYwi-aMydH9pItd2).

[[294]](#ftnt_ref294) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435092873&usg=AOvVaw3YWSEXdVCl0ExNuYnrw0Kw).

[[295]](#ftnt_ref295) The former term is by Garfinkel; the latter by Whittlestone & Gruetzemacher.

[[296]](#ftnt_ref296) Garfinkel, Benjamin. ‘The Impact of Artificial Intelligence: A Historical Perspective’, 2022.[https://docs.google.com/document/d/1I13\_0o3kUe1AVQNfevOF9sHpc4mCQkuFDxOXFj\_4g-I/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1I13_0o3kUe1AVQNfevOF9sHpc4mCQkuFDxOXFj_4g-I/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435018453&usg=AOvVaw0xdLrOoIXrnuSFni0Amg69).

[[297]](#ftnt_ref297) Mentioned in: Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435093651&usg=AOvVaw3DxQJaUQ0pPFML5uLh2ijx).

[[298]](#ftnt_ref298) Ibid.

[[299]](#ftnt_ref299) dynomight. ‘Historical Analogies for Large Language Models’. DYNOMIGHT INTERNET NEWSLETTER (blog), 8 December 2022.[https://dynomight.substack.com/p/llms](https://www.google.com/url?q=https://dynomight.substack.com/p/llms&sa=D&source=editors&ust=1692801435177000&usg=AOvVaw34FLMF08DsRt5N_p0Rxe9O).

[[300]](#ftnt_ref300) Maas, Matthijs M. ‘Regulating for “Normal AI Accidents”: Operational Lessons for the Responsible Governance of Artificial Intelligence Deployment’. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 223–28. AIES ’18. New York, NY, USA: Association for Computing Machinery, 2018.[https://doi.org/10.1145/3278721.3278766](https://www.google.com/url?q=https://doi.org/10.1145/3278721.3278766&sa=D&source=editors&ust=1692801435019860&usg=AOvVaw3gcCyPFgsYfXQ4PNZQ62kp).; Williams, Robert, and Roman Yampolskiy. ‘Understanding and Avoiding AI Failures: A Practical Guide’. Philosophies 6, no. 3 (September 2021): 53.[https://doi.org/10.3390/philosophies6030053](https://www.google.com/url?q=https://doi.org/10.3390/philosophies6030053&sa=D&source=editors&ust=1692801435020324&usg=AOvVaw0JkxzpnNOn0oaY8BHAa30I).; Dietterich, Thomas G. ‘Robust Artificial Intelligence and Robust Human Organizations’. ArXiv:1811.10840 [Cs], 27 November 2018.[http://arxiv.org/abs/1811.10840](https://www.google.com/url?q=http://arxiv.org/abs/1811.10840&sa=D&source=editors&ust=1692801435020809&usg=AOvVaw3csrvHCcAeI4EoS-gXXeF2).

[[301]](#ftnt_ref301) Hayward, Keith J, and Matthijs M Maas. ‘Artificial Intelligence and Crime: A Primer for Criminologists’. Crime, Media, Culture 17, no. 2 (30 June 2020): 209–33.[https://doi.org/10.1177/1741659020917434](https://www.google.com/url?q=https://doi.org/10.1177/1741659020917434&sa=D&source=editors&ust=1692801435174106&usg=AOvVaw08hF1meswi4052zPNbAuw6). Pg. 10.

[[302]](#ftnt_ref302) Branwen, Gwern. ‘Slowing Moore’s Law: How It Could Happen’, 16 March 2012.[https://www.gwern.net/Slowing-Moores-Law](https://www.google.com/url?q=https://www.gwern.net/Slowing-Moores-Law&sa=D&source=editors&ust=1692801435058400&usg=AOvVaw1tDOAbKmYZc4JGnmE7Q7eJ). (see subsection ‘Case-study: Suppressing Nuclear Weapons’).

[[303]](#ftnt_ref303) Zwetsloot, Remco, and Allan Dafoe. ‘Thinking About Risks From AI: Accidents, Misuse and Structure’. Lawfare, 11 February 2019.[https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure](https://www.google.com/url?q=https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure&sa=D&source=editors&ust=1692801435078021&usg=AOvVaw2CB7UTvwDX3pjhsDD-jVMk).; drawing on the famous argument in: Van Evera, Stephen. ‘The Cult of the Offensive and the Origins of the First World War’. International Security 9, no. 1 (1984): 58–107.[https://doi.org/10.2307/2538636](https://www.google.com/url?q=https://doi.org/10.2307/2538636&sa=D&source=editors&ust=1692801435078412&usg=AOvVaw0GuuAPS5N2llECZtyNfmwy).         However, this interpretation remains contested. See also: Lieber, Keir A. War and the Engineers: The Primacy of Politics over Technology. 1 edition. Ithaca; London: Cornell University Press, 2008; Snyder, Jack, and Keir A. Lieber. ‘Defensive Realism and the “New” History of World War I’. International Security 33, no. 1 (26 June 2008): 174–94.[https://doi.org/10.1162/isec.2008.33.1.174](https://www.google.com/url?q=https://doi.org/10.1162/isec.2008.33.1.174&sa=D&source=editors&ust=1692801435078872&usg=AOvVaw3LIOh2O2ttpDTsXOAs3oiE).

[[304]](#ftnt_ref304) Dafoe, Allan. ‘AI Governance: Overview and Theoretical Lenses’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press, 2022.[https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu\_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k\_jCIsCX1/pub](https://www.google.com/url?q=https://docs.google.com/document/d/e/2PACX-1vQOQ0EBIaEu_LaJqWvdPKu8xlmrOCM6h6gq7eFHnN0Y2GPYoodQjLeilxQ8SUwnbVThXc0k_jCIsCX1/pub&sa=D&source=editors&ust=1692801435091325&usg=AOvVaw2Ul-RWxwTqNsQlu3B4-3Ue).

[[305]](#ftnt_ref305) MacInnes, Morgan, Ben Garfinkel, and Allan Dafoe. ‘Anarchy as Architect: Competitive Pressure, Technology, and the Internal Structure of States’, (under review 2023).(“it is possible that progress in artificial intelligence (AI) will bias the future evolution of states towards inegalitarian and autocratic forms. The distinguishing feature of AI is that it can perform cognitive tasks which have hitherto been humanity’s exclusive domain. Sufficiently advanced AI could mentally outcompete humans in terms of both cost and performance, replacing them both on the battlefield and the home front. Assuming states continue to be subject to competitive pressure in the future, selection dynamics may once again shape them into forms detrimental to human welfare.” pg 21).

[[306]](#ftnt_ref306) Ibid.

[[307]](#ftnt_ref307) Danaher, John. ‘Axiological Futurism: The Systematic Study of the Future of Values’. Futures 132 (1 September 2021): 102780.[https://doi.org/10.1016/j.futures.2021.102780](https://www.google.com/url?q=https://doi.org/10.1016/j.futures.2021.102780&sa=D&source=editors&ust=1692801435174792&usg=AOvVaw1xVBUdprZRof-CpwfEETz_). (drawing on: Morris, Ian, Foragers, Farmers, and Fossil Fuels: How Human Values Evolve. Edited by Stephen Macedo. Updated ed. edition. Princeton: Princeton University Press, 2015.]

[[308]](#ftnt_ref308) See generally: Hopster, J. K. G., C. Arora, C. Blunden, C. Eriksen, L. E. Frank, J. S. Hermann, M. B. O. T. Klenk, E. R. H. O’Neill, and S. Steinert. ‘Pistols, Pills, Pork and Ploughs: The Structure of Technomoral Revolutions’. Inquiry 0, no. 0 (8 July 2022): 1–33.[https://doi.org/10.1080/0020174X.2022.2090434](https://www.google.com/url?q=https://doi.org/10.1080/0020174X.2022.2090434&sa=D&source=editors&ust=1692801435132310&usg=AOvVaw0PHtF9v2fan1-cL2tTxg2w).

[[309]](#ftnt_ref309) Leung, Jade. ‘Who Will Govern Artificial Intelligence? Learning from the History of Strategic Politics in Emerging Technologies’. University of Oxford, 2019.[https://ora.ox.ac.uk/objects/uuid:ea3c7cb8-2464-45f1-a47c-c7b568f27665](https://www.google.com/url?q=https://ora.ox.ac.uk/objects/uuid:ea3c7cb8-2464-45f1-a47c-c7b568f27665&sa=D&source=editors&ust=1692801435022963&usg=AOvVaw3mzZbAXnNtswqwv51Tqxb0).

[[310]](#ftnt_ref310) Ding, Jeffrey, and Allan Dafoe. ‘The Logic of Strategic Assets: From Oil to AI’. Security Studies, 3 June 2021, 1–31.[https://doi.org/10.1080/09636412.2021.1915583](https://www.google.com/url?q=https://doi.org/10.1080/09636412.2021.1915583&sa=D&source=editors&ust=1692801435003731&usg=AOvVaw2lkzG1XNZXIWWU3TnV62C-).

[[311]](#ftnt_ref311) Walker, J. Samuel, and Thomas R. Wellock. ‘A Short History of Nuclear Regulation, 1946–2009’. U.S. Nuclear Regulatory Commission, October 2010.[https://www.nrc.gov/docs/ML1029/ML102980443.pdf](https://www.google.com/url?q=https://www.nrc.gov/docs/ML1029/ML102980443.pdf&sa=D&source=editors&ust=1692801435438928&usg=AOvVaw1gWFuW_a46mLhvhy6q1fTT). I thank Aishwarya Saxena for discussions and suggestions on this topic.

[[312]](#ftnt_ref312) O’Keefe, Cullen. ‘How Will National Security Considerations Affect Antitrust Decisions in AI? An Examination of Historical Precedents’. Future of Humanity Institute, 7 July 2020.[https://www.fhi.ox.ac.uk/wp-content/uploads/How-Will-National-Security-Considerations-Affect-Antitrust-Decisions-in-AI-Cullen-OKeefe.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/How-Will-National-Security-Considerations-Affect-Antitrust-Decisions-in-AI-Cullen-OKeefe.pdf&sa=D&source=editors&ust=1692801435004575&usg=AOvVaw3CGi0MYUdzxJLudFLTsMOK).

[[313]](#ftnt_ref313) Di Cooke, and Sam Clarke. ‘The “Old AI”: Lessons for AI Governance from the Early Days of Electricity Regulation’. Effective Altruism Forum. 2022. [https://forum.effectivealtruism.org/posts/k73qrirnxcKtKZ4ng/the-old-ai-lessons-for-ai-governance-from-early-electricity-1](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/k73qrirnxcKtKZ4ng/the-old-ai-lessons-for-ai-governance-from-early-electricity-1&sa=D&source=editors&ust=1692801435052772&usg=AOvVaw09vFbtMB2Aw-PtqV6LATJn) Drawing among others on: Isser, Steve. Electricity Restructuring in the United States: Markets and Policy from the 1978 Energy Act to the Present. Cambridge University Press, 2015.

[[314]](#ftnt_ref314) See influentially Boyer, Paul. By the Bomb’s Early Light. The University of North Carolina Press, 1985.[https://uncpress.org/book/9780807844809/by-the-bombs-early-light/](https://www.google.com/url?q=https://uncpress.org/book/9780807844809/by-the-bombs-early-light/&sa=D&source=editors&ust=1692801435417619&usg=AOvVaw2UsJDQCRS2z1kbc5DteMXn). Lente, Dick van, ed. The Nuclear Age in Popular Media. New York: Palgrave Macmillan US, 2012.[https://doi.org/10.1057/9781137086181](https://www.google.com/url?q=https://doi.org/10.1057/9781137086181&sa=D&source=editors&ust=1692801435418049&usg=AOvVaw3amXaeVrTiKSGrhzU7nG2V). I thank Lara Thurnherr for these suggestions.

[[315]](#ftnt_ref315) Gabs, Nick. ‘Lessons from Three Mile Island for AI Warning Shots’. EA Forum, 26 September 2022.[https://forum.effectivealtruism.org/posts/NyCHoZGGw5YssvDJB/lessons-from-three-mile-island-for-ai-warning-shots](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/NyCHoZGGw5YssvDJB/lessons-from-three-mile-island-for-ai-warning-shots&sa=D&source=editors&ust=1692801435099928&usg=AOvVaw3lVYmffOXo8x67jS4dsUP-).

[[316]](#ftnt_ref316) Krakovna, Victoria. ‘Possible Takeaways from the Coronavirus Pandemic for Slow AI Takeoff’. AI Alignment Forum (blog), 31 May 2020.[https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai](https://www.google.com/url?q=https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai&sa=D&source=editors&ust=1692801435100630&usg=AOvVaw09295N-h5TsBkUHIPA8ZMv).; Soares, Nate. ‘Warning Shots Probably Wouldn’t Change The Picture Much’. Alignment Forum, 6 October 2022.[https://www.alignmentforum.org/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much](https://www.google.com/url?q=https://www.alignmentforum.org/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much&sa=D&source=editors&ust=1692801435101098&usg=AOvVaw27PqiIm9wM0yx0kBel2rTU). See generally: Liu, Hin-Yan, Kristian Lauta, and Matthijs Maas. ‘Apocalypse Now?: Initial Lessons from the Covid-19 Pandemic for the Governance of Existential and Global Catastrophic Risks’. Journal of International Humanitarian Legal Studies 11, no. 2 (13 August 2020): 295–310.[https://doi.org/10.1163/18781527-01102004](https://www.google.com/url?q=https://doi.org/10.1163/18781527-01102004&sa=D&source=editors&ust=1692801435101534&usg=AOvVaw1dp-0o0eqf74MA3y3k6x8_).

[[317]](#ftnt_ref317) Grotto, Andrew. ‘Genetically Modified Organisms: A Precautionary Tale For AI Governance’. AI Pulse, 24 January 2019.[https://aipulse.org/genetically-modified-organisms-a-precautionary-tale-for-ai-governance-2/](https://www.google.com/url?q=https://aipulse.org/genetically-modified-organisms-a-precautionary-tale-for-ai-governance-2/&sa=D&source=editors&ust=1692801435005414&usg=AOvVaw0XRVpup2zc__KNVK16SYCB).

[[318]](#ftnt_ref318) See also the section on ‘containing’. As well as: Maas, Matthijs. ‘Paths Untaken: The History, Epistemology and Strategy of Technological Restraint, and Lessons for AI’. Verfassungsblog (blog), 9 August 2022.[https://verfassungsblog.de/paths-untaken/](https://www.google.com/url?q=https://verfassungsblog.de/paths-untaken/&sa=D&source=editors&ust=1692801435016617&usg=AOvVaw1SzE2lMSSO_gNvVRJDtmL6).

[[319]](#ftnt_ref319) Kemp, Luke. ‘Agents of Doom: Who Is Creating the Apocalypse and Why’. BBC Future, 26 October 2021.[https://www.bbc.com/future/article/20211014-agents-of-doom-who-is-hastening-the-apocalypse-and-why](https://www.google.com/url?q=https://www.bbc.com/future/article/20211014-agents-of-doom-who-is-hastening-the-apocalypse-and-why&sa=D&source=editors&ust=1692801435023596&usg=AOvVaw233TYACET4u4WPWxWeMLJm).;

[[320]](#ftnt_ref320) See generally; Deshman, Abigail C. ‘Horizontal Review between International Organizations: Why, How, and Who Cares about Corporate Regulatory Capture’. European Journal of International Law 22, no. 4 (1 November 2011): 1089–1113.[https://doi.org/10.1093/ejil/chr093](https://www.google.com/url?q=https://doi.org/10.1093/ejil/chr093&sa=D&source=editors&ust=1692801435388530&usg=AOvVaw30vUs8_EY8t4BcmnwacQSD). As discussed in: Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Fragmentation and the Future: Investigating Architectures for International AI Governance’. Global Policy 11, no. 5 (November 2020): 545–56.[https://doi.org/10.1111/1758-5899.12890](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12890&sa=D&source=editors&ust=1692801435389096&usg=AOvVaw3zs5-Fvm46ryQQSz9jEccg).

[[321]](#ftnt_ref321) See generally: Bower, Adam, and Jeffrey S. Lantis. ‘Contesting the Heavens: US Antipreneurship and the Regulation of Space Weapons’. European Journal of International Security, 8 February 2023, 1–22.[https://doi.org/10.1017/eis.2023.2](https://www.google.com/url?q=https://doi.org/10.1017/eis.2023.2&sa=D&source=editors&ust=1692801435389800&usg=AOvVaw0SDzGAjUn69znECSjbw3Mm).

[[322]](#ftnt_ref322) Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020.[https://drive.google.com/file/d/1vIJUAp\_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp=sharing](https://www.google.com/url?q=https://drive.google.com/file/d/1vIJUAp_i41A5gc9Tb9EvO9aSuLn15ixq/view?usp%3Dsharing&sa=D&source=editors&ust=1692801435053438&usg=AOvVaw3tLsm0dS_ebhf9ztY_4zo4). Pg. 193-195. Drawing on: Crootof, Rebecca. ‘Jurisprudential Space Junk: Treaties and New Technologies’. In Resolving Conflicts in the Law, edited by Chiara Giorgetti and Natalie Klein, 106–29, 2019.[https://brill.com/view/book/edcoll/9789004316539/BP000015.xml](https://www.google.com/url?q=https://brill.com/view/book/edcoll/9789004316539/BP000015.xml&sa=D&source=editors&ust=1692801435053856&usg=AOvVaw2a9sA9KsKflkA-6aJBC5US).

[[323]](#ftnt_ref323) Maas (ibid. Pg. 224-227). See also: Picker, Colin B. ‘A View from 40,000 Feet: International Law and the Invisible Hand of Technology’. Cardozo Law Review 23 (2001): 151–219. [https://papers.ssrn.com/abstract=987524](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D987524&sa=D&source=editors&ust=1692801435054250&usg=AOvVaw2Wu69lU0iC-acxnPuRttmT)

[[324]](#ftnt_ref324) Grace, Katja. ‘Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation’. Technical Report. Berkeley, CA: Machine Intelligence Research Institute, October 2015. [https://intelligence.org/files/SzilardNuclearWeapons.pdf](https://www.google.com/url?q=https://intelligence.org/files/SzilardNuclearWeapons.pdf&sa=D&source=editors&ust=1692801434973639&usg=AOvVaw0bnHRfY5rnCwUsGbYvmBFI).

[[325]](#ftnt_ref325) Napper, Brian. ‘Early Computers (1946-51)’. Computer50, 20 August 1999.[http://curation.cs.manchester.ac.uk/computer50/www.computer50.org/mark1/contemporary.html](https://www.google.com/url?q=http://curation.cs.manchester.ac.uk/computer50/www.computer50.org/mark1/contemporary.html&sa=D&source=editors&ust=1692801435418629&usg=AOvVaw2ck431n67lKzKHV6R6fJoh). I thank Lara Thurnherr for this suggestion.

[[326]](#ftnt_ref326) Critch, Andrew, and David Krueger. ‘AI Research Considerations for Human Existential Safety (ARCHES)’, 29 May 2020.[http://acritch.com/arches/](https://www.google.com/url?q=http://acritch.com/arches/&sa=D&source=editors&ust=1692801435072414&usg=AOvVaw2K6gNwylm75gGiVHbt4o5z). (pg. 83-84).

[[327]](#ftnt_ref327) See generally: Beaumier, Guillaume, Kevin Kalomeni, Malcolm Campbell‐Verduyn, Marc Lenglet, Serena Natile, Marielle Papin, Daivi Rodima‐Taylor, Arthur Silve, and Falin Zhang. ‘Global Regulations for a Digital Economy: Between New and Old Challenges’. Global Policy 11, no. 4 (September 2020): 515–22.[https://doi.org/10.1111/1758-5899.12823](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12823&sa=D&source=editors&ust=1692801435091828&usg=AOvVaw1tTankXmxhdEjcmPI5bT38).; see also the analysis of the regulatability of hardware in Brundage, Miles, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, et al. ‘Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims’. ArXiv:2004.07213 [Cs], 15 April 2020.[http://arxiv.org/abs/2004.07213](https://www.google.com/url?q=http://arxiv.org/abs/2004.07213&sa=D&source=editors&ust=1692801435092209&usg=AOvVaw3wlZda7Rg4VuVUQyrcBgc_).

[[328]](#ftnt_ref328) Ruhl, Christian. ‘Autonomous Weapon Systems & Military AI: Cause Area Report’. Founders Pledge, May 2022.[https://founderspledge.com/stories/autonomous-weapon-systems-and-military-artificial-intelligence-ai](https://www.google.com/url?q=https://founderspledge.com/stories/autonomous-weapon-systems-and-military-artificial-intelligence-ai&sa=D&source=editors&ust=1692801435014509&usg=AOvVaw0EzjPA7QOjFUiRMzdotc7g).

[[329]](#ftnt_ref329) Horowitz, Michael C, and Paul Scharre. ‘AI and International Stability: Risks and Confidence-Building Measures’. Center for a New American Security, 12 January 2021.[https://www.cnas.org/publications/reports/ai-and-international-stability-risks-and-confidence-building-measures](https://www.google.com/url?q=https://www.cnas.org/publications/reports/ai-and-international-stability-risks-and-confidence-building-measures&sa=D&source=editors&ust=1692801435124591&usg=AOvVaw2QprsCd2fp0r0etVVth9LJ).

[[330]](#ftnt_ref330) Stafford, Eoghan, and Robert F Trager. ‘The IAEA Solution: Knowledge Sharing to Prevent Dangerous Technology Races’, 2022, 91.

[[331]](#ftnt_ref331) Baker, Mauricio. ‘Nuclear Arms Control Verification and Lessons for AI Treaties’. arXiv, 8 April 2023.[http://arxiv.org/abs/2304.04123](https://www.google.com/url?q=http://arxiv.org/abs/2304.04123&sa=D&source=editors&ust=1692801435439376&usg=AOvVaw3G-9FtuO7GaKNq6kIYIHvN).

[[332]](#ftnt_ref332) Grace, Katja. ‘The Asilomar Conference: A Case Study in Risk Mitigation’. Technical Report. Berkeley, CA: Machine Intelligence Research Institute, 15 July 2015.[https://intelligence.org/files/TheAsilomarConference.pdf](https://www.google.com/url?q=https://intelligence.org/files/TheAsilomarConference.pdf&sa=D&source=editors&ust=1692801435054848&usg=AOvVaw3cP1Zge-LcYT_C30pVqYpJ).

[[333]](#ftnt_ref333) Wang, Jasmine. ‘What the AI Community Can Learn From Sneezing Ferrets and a Mutant Virus Debate’. Partnership on AI (blog), 9 December 2020.[https://medium.com/partnership-on-ai/lessons-for-the-ai-community-from-the-h5n1-controversy-32432438a82e](https://www.google.com/url?q=https://medium.com/partnership-on-ai/lessons-for-the-ai-community-from-the-h5n1-controversy-32432438a82e&sa=D&source=editors&ust=1692801435021592&usg=AOvVaw3TcPguCbuQyCXoeMOU_rPb)

[[334]](#ftnt_ref334) jia. ‘Case Studies of Self-Governance to Reduce Technology Risk’. EA Forum, 6 April 2021.[https://forum.effectivealtruism.org/posts/Xf6QE6txgvfCGvZpk/case-studies-of-self-governance-to-reduce-technology-risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Xf6QE6txgvfCGvZpk/case-studies-of-self-governance-to-reduce-technology-risk&sa=D&source=editors&ust=1692801435056654&usg=AOvVaw20zql5_gaIyDaC4Ut7SGYr).

[[335]](#ftnt_ref335) Maas, Matthijs. ‘Paths Untaken: The History, Epistemology and Strategy of Technological Restraint, and Lessons for AI’. Verfassungsblog (blog), 9 August 2022.[https://verfassungsblog.de/paths-untaken/](https://www.google.com/url?q=https://verfassungsblog.de/paths-untaken/&sa=D&source=editors&ust=1692801435015848&usg=AOvVaw2SSey-DnbQ1NiawmuP7rQA).

[[336]](#ftnt_ref336) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Should Artificial Intelligence Governance Be Centralised?: Design Lessons from History’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 228–34. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375857](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375857&sa=D&source=editors&ust=1692801435055475&usg=AOvVaw3NzXGec-wmV1H6VbkRVcgf).

[[337]](#ftnt_ref337) Gutierrez, Carlos Ignacio, Gary E. Marchant, and Lucille Tournas. ‘Lessons for Artificial Intelligence from Historical Uses of Soft Law Governance’. JURIMETRICS 61, no. 1 (29 December 2020).[https://doi.org/10.2139/ssrn.3775271](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.3775271&sa=D&source=editors&ust=1692801435244770&usg=AOvVaw3d_wu-EfbqCRVK_1pL5rol).

[[338]](#ftnt_ref338) Hollis, Helena, and Jess Whittlestone. ‘Participatory AI Futures: Lessons from Research in Climate Change’. Medium (blog), 24 August 2021.[https://medium.com/@helena.hollis.14/participatory-ai-futures-lessons-from-research-in-climate-change-34e3580553f8](https://www.google.com/url?q=https://medium.com/@helena.hollis.14/participatory-ai-futures-lessons-from-research-in-climate-change-34e3580553f8&sa=D&source=editors&ust=1692801435022357&usg=AOvVaw3aRG0HN2a-Hp47zyDlgtoG).

[[339]](#ftnt_ref339) Grace, Katja. ‘Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation’. Technical Report. Berkeley, CA: Machine Intelligence Research Institute, October 2015.[https://intelligence.org/files/SzilardNuclearWeapons.pdf](https://www.google.com/url?q=https://intelligence.org/files/SzilardNuclearWeapons.pdf&sa=D&source=editors&ust=1692801435115250&usg=AOvVaw3HPnoN4dD5YyZopcPivsBn).

[[340]](#ftnt_ref340) Korzekwa, Rick. ‘Preliminary Survey of Prescient Actions’. AI Impacts, 3 April 2020.[https://aiimpacts.org/survey-of-prescient-actions/](https://www.google.com/url?q=https://aiimpacts.org/survey-of-prescient-actions/&sa=D&source=editors&ust=1692801435012014&usg=AOvVaw2uzxRIWMRqgAgkuc33QmuB). (briefly surveying 20 possible cases).

[[341]](#ftnt_ref341) Eigner, Peter, and Thomas S. Umlauft. ‘The Great Depression(s) of 1929-1933 and 2007-2009? Parallels, Differences and Policy Lessons’. Hungarian Academy of Science MTA-ELTE Crisis History Working Paper No. 2. Rochester, NY, 1 July 2015.[https://doi.org/10.2139/ssrn.2612243](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.2612243&sa=D&source=editors&ust=1692801435433877&usg=AOvVaw09hNVj6a_uRbqA0N5JYt9v). I thank Lara Thurnherr for these and other suggestions.

[[342]](#ftnt_ref342) Muehlhauser, Luke. ‘How Well Will Policy-Makers Handle AGI? (Initial Findings)’. Machine Intelligence Research Institute, 12 September 2013.[https://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/](https://www.google.com/url?q=https://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/&sa=D&source=editors&ust=1692801435059465&usg=AOvVaw3BvBnSY839b-U12Qj4OJFN). (also discussing a set of other examples or analogies, which are however dismissed for not being sufficiently similar to AGI risk on various grounds).

[[343]](#ftnt_ref343) Krakovna, Victoria. ‘Possible Takeaways from the Coronavirus Pandemic for Slow AI Takeoff’. AI Alignment Forum (blog), 31 May 2020.[https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai](https://www.google.com/url?q=https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai&sa=D&source=editors&ust=1692801435057379&usg=AOvVaw27k5-JI24dNH4AsKrLFP3i).; Soares, Nate. ‘Warning Shots Probably Wouldn’t Change The Picture Much’. Alignment Forum, 6 October 2022.[https://www.alignmentforum.org/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much](https://www.google.com/url?q=https://www.alignmentforum.org/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much&sa=D&source=editors&ust=1692801435057908&usg=AOvVaw38nojZQmHg13PH9mVkRbDl).

[[344]](#ftnt_ref344) Baum, Seth. ‘Lessons for Artificial Intelligence from Other Global Risks’. In The Global Politics of Artificial Intelligence, edited by Maurizio Tinnirello, 20, 2019.

[[345]](#ftnt_ref345) I thank Oliver Guest for suggesting this category.

[[346]](#ftnt_ref346) Gaulkin, Thomas. ‘Facing Nuclear Reality, 35 Years after The Day After’. Bulletin of the Atomic Scientists (blog), 13 December 2018.[https://thebulletin.org/2018/12/facing-nuclear-reality-35-years-after-the-day-after/](https://www.google.com/url?q=https://thebulletin.org/2018/12/facing-nuclear-reality-35-years-after-the-day-after/&sa=D&source=editors&ust=1692801435111923&usg=AOvVaw2N2stcHYImzMqj2Z5NpYAO).; Knoblauch, William M. Nuclear Freeze in a Cold War: The Reagan Administration, Cultural Activism, and the End of the Arms Race. University of Massachusetts Press, 2017.[https://doi.org/10.2307/j.ctv346v1z](https://www.google.com/url?q=https://doi.org/10.2307/j.ctv346v1z&sa=D&source=editors&ust=1692801435112317&usg=AOvVaw2dD7kPa-jRoYoidqkwOWOw).; see also Feldman, Stanley, and Lee Sigelman. ‘The Political Impact of Prime-Time Television: “The Day After”’. The Journal of Politics 47, no. 2 (1985): 556–78.[https://doi.org/10.2307/2130896](https://www.google.com/url?q=https://doi.org/10.2307/2130896&sa=D&source=editors&ust=1692801435112728&usg=AOvVaw0hTWBdPWGcAAA_znLbDga6).

[[347]](#ftnt_ref347) Wiblin, Robert, and Keiran Harris. ‘Carl Shulman on the Common-Sense Case for Existential Risk Work and Its Practical Implications’. 80,000 Hours Podcast. Accessed 11 October 2021.[https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/](https://www.google.com/url?q=https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/&sa=D&source=editors&ust=1692801435114141&usg=AOvVaw1s8Wy127mIkg5a38b23Q3t).

[[348]](#ftnt_ref348) Luce, Dan De. ‘A Novel About War With China Strikes a Chord at the Pentagon’. Foreign Policy (blog), 15 May 2016.[https://foreignpolicy.com/2016/05/15/a-novel-about-war-with-china-strikes-a-chord-at-the-pentagon/](https://www.google.com/url?q=https://foreignpolicy.com/2016/05/15/a-novel-about-war-with-china-strikes-a-chord-at-the-pentagon/&sa=D&source=editors&ust=1692801435113371&usg=AOvVaw28hAUwa5-E-5GsECQUIdHP).; Harper, Jon. ‘Pentagon Betting on New Technologies to Foil Future Adversaries’. National Defense 101, no. 756 (2016): 26–29. [https://www.jstor.org/stable/27021585](https://www.google.com/url?q=https://www.jstor.org/stable/27021585&sa=D&source=editors&ust=1692801435113607&usg=AOvVaw0j2g5LV1Aen_uKRbcGQhe2)

[[349]](#ftnt_ref349) Forrest, Adam. ‘Matt Hancock Admits Hollywood Film Contagion Shaped Vaccine Response’. The Independent, 4 February 2021.[https://www.independent.co.uk/news/uk/politics/covid-vaccine-strategy-hancock-contagion-movie-b1796923.html](https://www.google.com/url?q=https://www.independent.co.uk/news/uk/politics/covid-vaccine-strategy-hancock-contagion-movie-b1796923.html&sa=D&source=editors&ust=1692801435114714&usg=AOvVaw3-4JH92tbAu_kmgb34ZVq2).

[[350]](#ftnt_ref350) Maas, Matthijs M. ‘How Viable Is International Arms Control for Military Artificial Intelligence? Three Lessons from Nuclear Weapons’. Contemporary Security Policy 40, no. 3 (6 February 2019): 285–311.[https://doi.org/10.1080/13523260.2019.1576464](https://www.google.com/url?q=https://doi.org/10.1080/13523260.2019.1576464&sa=D&source=editors&ust=1692801435106086&usg=AOvVaw3DQVecg-VtMb3LUzTxqcHM).

[[351]](#ftnt_ref351) Zaidi, Waqar, and Allan Dafoe. ‘International Control of Powerful Technology: Lessons from the Baruch Plan’. Center for the Governance of AI, Future of Humanity Institute, March 2021.[https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/International-Control-of-Powerful-Technology-Lessons-from-the-Baruch-Plan-Zaidi-Dafoe-2021.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/International-Control-of-Powerful-Technology-Lessons-from-the-Baruch-Plan-Zaidi-Dafoe-2021.pdf&sa=D&source=editors&ust=1692801435013046&usg=AOvVaw2t2NyyBHPZnQd9QoHnwW9r).;

[[352]](#ftnt_ref352) Scharre, Paul, and Megan Lamberth. ‘Artificial Intelligence and Arms Control’. Center for a New American Security, 12 October 2022.[https://www.cnas.org/publications/reports/artificial-intelligence-and-arms-control](https://www.google.com/url?q=https://www.cnas.org/publications/reports/artificial-intelligence-and-arms-control&sa=D&source=editors&ust=1692801435013713&usg=AOvVaw2h2OYohrvmoCPLEVekBaiP).

[[353]](#ftnt_ref353) Zaidi, Waqar, and Allan Dafoe. ‘International Control of Powerful Technology: Lessons from the Baruch Plan’. Center for the Governance of AI, Future of Humanity Institute, March 2021.[https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/International-Control-of-Powerful-Technology-Lessons-from-the-Baruch-Plan-Zaidi-Dafoe-2021.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/International-Control-of-Powerful-Technology-Lessons-from-the-Baruch-Plan-Zaidi-Dafoe-2021.pdf&sa=D&source=editors&ust=1692801435356808&usg=AOvVaw3M-qy53kCXtZLSKNe7tStV).

[[354]](#ftnt_ref354) Zaidi, Waqar H. Technological Internationalism and World Order: Aviation, Atomic Energy, and the Search for International Peace, 1920–1950. Cambridge ; New York, NY: Cambridge University Press, 2021.

[[355]](#ftnt_ref355) Harding, Verity. ‘Lessons from History: What Can Past Technological Breakthroughs Teach the AI Community Today’, 2020.[https://www.bennettinstitute.cam.ac.uk/blog/lessons-history-what-can-past-technological-breakt/](https://www.google.com/url?q=https://www.bennettinstitute.cam.ac.uk/blog/lessons-history-what-can-past-technological-breakt/&sa=D&source=editors&ust=1692801435150305&usg=AOvVaw3iZh08Lexbze9RxNshz4X1).

[[356]](#ftnt_ref356) Cooke, Di. ‘The Role of Activism in Nuclear Arms Control’, 2020. (private draft).

[[357]](#ftnt_ref357) Martin, Rani. ‘The Pugwash Conferences and the Anti-Ballistic Missile Treaty as a Case Study of Track II Diplomacy’. EA Forum, 16 September 2022.[https://forum.effectivealtruism.org/posts/ggiCDnYcSKLxwFbBv/the-pugwash-conferences-and-the-anti-ballistic-missile](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ggiCDnYcSKLxwFbBv/the-pugwash-conferences-and-the-anti-ballistic-missile&sa=D&source=editors&ust=1692801435015209&usg=AOvVaw2xMZFr7ExOuQ7_moMySVu4).

[[358]](#ftnt_ref358) The term is from: Dafoe, Allan. ‘AI Governance: A Research Agenda’. Oxford: Center for the Governance of AI, Future of Humanity Institute, 2018.[https://www.fhi.ox.ac.uk/govaiagenda/](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/govaiagenda/&sa=D&source=editors&ust=1692801435079346&usg=AOvVaw3pkL5_fU3JAUatNpFZIOaL).

[[359]](#ftnt_ref359) Bostrom, Nick, Allan Dafoe, and Carrick Flynn. ‘Public Policy and Superintelligent AI: A Vector Field Approach’. In Ethics of Artificial Intelligence, edited by S.M. Liao. Oxford University Press, 2019.[http://www.nickbostrom.com/papers/aipolicy.pdf](https://www.google.com/url?q=http://www.nickbostrom.com/papers/aipolicy.pdf&sa=D&source=editors&ust=1692801434949025&usg=AOvVaw3zUGQjT3aQfm8-I3Ryj7TN). See also: RoryG. ‘What If AI Development Goes Well?’. EA Forum, 3 August 2022.[https://forum.effectivealtruism.org/posts/9EjMoD8BRhXEsfzMh/what-if-ai-development-goes-well-3](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/9EjMoD8BRhXEsfzMh/what-if-ai-development-goes-well-3&sa=D&source=editors&ust=1692801434949551&usg=AOvVaw3mRJjcQJoOI--aT2vtf3Gc).

[[360]](#ftnt_ref360) Karnofsky, Holden. ‘Important, Actionable Research Questions for the Most Important Century’. EA Forum, 24 February 2022.[https://forum.effectivealtruism.org/posts/zGiD94SHwQ9MwPyfW/important-actionable-research-questions-for-the-most](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zGiD94SHwQ9MwPyfW/important-actionable-research-questions-for-the-most&sa=D&source=editors&ust=1692801434950232&usg=AOvVaw12wRKTfeJqQlfRFWi_L0Xf).

[[361]](#ftnt_ref361) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Fragmentation and the Future: Investigating Architectures for International AI Governance’. Global Policy 11, no. 5 (November 2020): 545–56.[https://doi.org/10.1111/1758-5899.12890](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12890&sa=D&source=editors&ust=1692801435394017&usg=AOvVaw0gPyW3xOOiR4xFGLz8dcqO).

[[362]](#ftnt_ref362) The terms ‘mid-transition’ and ‘pre-emergence’ are drawn from: Manheim, David. ‘AI Governance across Slow/Fast Takeoff and Easy/Hard Alignment Spectra’. AI Alignment Forum, 3 April 2022.[https://www.alignmentforum.org/posts/xxMYFKLqiBJZRNoPj/ai-governance-across-slow-fast-takeoff-and-easy-hard](https://www.google.com/url?q=https://www.alignmentforum.org/posts/xxMYFKLqiBJZRNoPj/ai-governance-across-slow-fast-takeoff-and-easy-hard&sa=D&source=editors&ust=1692801434950949&usg=AOvVaw3bd6lcDI5BMffnovQzXpcV).

[[363]](#ftnt_ref363) Clarke, Sam. ‘The Longtermist AI Governance Landscape: A Basic Overview’. EA Forum, 18 January 2022.[https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview&sa=D&source=editors&ust=1692801435151052&usg=AOvVaw2wb1LntlPf5JeHVi9gR-AV).

[[364]](#ftnt_ref364) For definitions of these terms, see the start of each subsection; and also: Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023.

[[365]](#ftnt_ref365) In the current model, different pathways are mostly treated as being actor-specific (that is, bucketed by which actor they are meant to influence). In some cases however, we might consider that some pathways might be lever-specific (e.g. some types of advocacy are more suited to prompting the use of some types of government action, than others). I thank Suzanne van Arsdale for pointing out this distinction.

[[366]](#ftnt_ref366) Note: in this section, I bucket research horizontally, along the categories of ‘key actors’, ‘levers’ (of these actors), and ‘pathways to influence’ (on these actors). However, in practice, many specific analysis of interventions would integrate these three–discussing each actor both in terms of the levers available to it, as well as pathways by which their decisions might be informed.

[[367]](#ftnt_ref367) For discussion of these terms, see also Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023.

[[368]](#ftnt_ref368) Ibid.

[[369]](#ftnt_ref369) Leung, Jade. ‘Who Will Govern Artificial Intelligence? Learning from the History of Strategic Politics in Emerging Technologies’. University of Oxford, 2019.[https://ora.ox.ac.uk/objects/uuid:ea3c7cb8-2464-45f1-a47c-c7b568f27665](https://www.google.com/url?q=https://ora.ox.ac.uk/objects/uuid:ea3c7cb8-2464-45f1-a47c-c7b568f27665&sa=D&source=editors&ust=1692801435125259&usg=AOvVaw0Cqpl6ypp8tfrhQRHiCFdE).

[[370]](#ftnt_ref370) Maas, Matthijs M. ‘How Viable Is International Arms Control for Military Artificial Intelligence? Three Lessons from Nuclear Weapons’. Contemporary Security Policy 40, no. 3 (6 February 2019): 285–311.[https://doi.org/10.1080/13523260.2019.1576464](https://www.google.com/url?q=https://doi.org/10.1080/13523260.2019.1576464&sa=D&source=editors&ust=1692801435151733&usg=AOvVaw3lpacc0jtZb2Om20jre0hx).

[[371]](#ftnt_ref371) Shevlane, Toby. ‘The Artefacts of Intelligence: Governing Scientists’ Contribution to AI Proliferation’. University of Oxford, 2022.[https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation](https://www.google.com/url?q=https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation&sa=D&source=editors&ust=1692801435152381&usg=AOvVaw1F6NmmPDNOWmRBZD7qrfhM).

[[372]](#ftnt_ref372) Deshpande, Advait, and Helen Sharp. ‘Responsible AI Systems: Who Are the Stakeholders?’ In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 227–36. AIES ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3514094.3534187](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534187&sa=D&source=editors&ust=1692801435390504&usg=AOvVaw3Z4RqzUA_PI9ROWjUg9SSD).

[[373]](#ftnt_ref373) Schiff, Daniel S. ‘Setting the Agenda for AI: Actors, Issues, and Influence in United States Artificial Intelligence Policy’. Georgia Institute of Technology, 2022.[https://osf.io/kw8xd/](https://www.google.com/url?q=https://osf.io/kw8xd/&sa=D&source=editors&ust=1692801435125698&usg=AOvVaw3Kxkr0LgpsjMP_wobMLPvE). (in US context).

[[374]](#ftnt_ref374) Cihon, Peter, Jonas Schuett, and Seth D. Baum. ‘Corporate Governance of Artificial Intelligence in the Public Interest’. Information 12, no. 7 (July 2021): 275.[https://doi.org/10.3390/info12070275](https://www.google.com/url?q=https://doi.org/10.3390/info12070275&sa=D&source=editors&ust=1692801435126211&usg=AOvVaw0YqSAf2fdt5KPEiNt0PFj4). (discussing actors both “inside the corporation—managers, workers, and investors—and outside the corporation—corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and governments”); Baum, Seth, and Jonas Schuett. ‘The Case for Long-Term Corporate Governance of AI’. Effective Altruism Forum, 3 November 2021.[https://forum.effectivealtruism.org/posts/5MZpxbJJ5pkEBpAAR/the-case-for-long-term-corporate-governance-of-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/5MZpxbJJ5pkEBpAAR/the-case-for-long-term-corporate-governance-of-ai&sa=D&source=editors&ust=1692801435126832&usg=AOvVaw0hLggCpTF78K39s4j0RlAd).; Leung, Jade. ‘Why Companies Should Be Leading on AI Governance’. 16 May 2019.[https://forum.effectivealtruism.org/posts/fniRhiPYw8b6FETsn/jade-leung-why-companies-should-be-leading-on-ai-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/fniRhiPYw8b6FETsn/jade-leung-why-companies-should-be-leading-on-ai-governance&sa=D&source=editors&ust=1692801435127409&usg=AOvVaw0kI13wkRZUSe5Ko2yNm3f6).

[[375]](#ftnt_ref375) Belfield, Haydn. ‘Activism by the AI Community: Analysing Recent Achievements and Future Prospects’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 15–21. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375814](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375814&sa=D&source=editors&ust=1692801435128111&usg=AOvVaw03_jMhxAQf-hu3GUs-Tuv8). And see generally: Nedzhvetskaya, Nataliya, and J. S. Tan. ‘The Role of Workers in AI Ethics and Governance’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press. Accessed 21 October 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.68](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.68&sa=D&source=editors&ust=1692801435128664&usg=AOvVaw1v4UqKckuN3Z9U-G_195xe).

[[376]](#ftnt_ref376) Cihon, Peter, Jonas Schuett, and Seth D. Baum. ‘Corporate Governance of Artificial Intelligence in the Public Interest’. Information 12, no. 7 (July 2021): 275.[https://doi.org/10.3390/info12070275](https://www.google.com/url?q=https://doi.org/10.3390/info12070275&sa=D&source=editors&ust=1692801435175370&usg=AOvVaw2jzM05WkmSTYuPY8ArfiSx).

[[377]](#ftnt_ref377) Listed in: Hogarth, Ian. ‘We Must Slow down the Race to God-like AI’. Financial Times, 13 April 2023.[https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2](https://www.google.com/url?q=https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2&sa=D&source=editors&ust=1692801435443013&usg=AOvVaw3L1O-LD-nVOOMye5jFPvzs). For older overviews of the landscape of labs pursuing AGI, see: Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum. ‘2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy’. Global Catastrophic Risk Institute Technical Report. Global Catastrophic Risk Institute, 2020.[https://gcrinstitute.org/papers/055\_agi-2020.pdf](https://www.google.com/url?q=https://gcrinstitute.org/papers/055_agi-2020.pdf&sa=D&source=editors&ust=1692801435443479&usg=AOvVaw0gmU98vYIh2GR6HO8_rwFb). And previously: Baum, Seth. ‘A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy’. Global Catastrophic Risk Institute Technical Report. Global Catastrophic Risk Institute, 12 November 2017.[https://papers.ssrn.com/abstract=3070741](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3070741&sa=D&source=editors&ust=1692801435443929&usg=AOvVaw1muNMz0pMwPxLqBMCxpO8r).

[[378]](#ftnt_ref378) Hannas, William, Huey-Meei Chang, Daniel Chou, and Brian Fleeger. ‘China’s Advanced AI Research: Monitoring China’s Paths to “General” Artificial Intelligence’. Center for Security and Emerging Technology, July 2022.[https://cset.georgetown.edu/publication/chinas-advanced-ai-research/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/chinas-advanced-ai-research/&sa=D&source=editors&ust=1692801435444571&usg=AOvVaw2Hg9Urz_0Akja8vnFIy3AG).

[[379]](#ftnt_ref379) See also: Leung, Jade. ‘Why Companies Should Be Leading on AI Governance’. 16 May 2019.[https://forum.effectivealtruism.org/posts/fniRhiPYw8b6FETsn/jade-leung-why-companies-should-be-leading-on-ai-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/fniRhiPYw8b6FETsn/jade-leung-why-companies-should-be-leading-on-ai-governance&sa=D&source=editors&ust=1692801435026044&usg=AOvVaw0emJ2VFSicMC9CqL_BkcXt).

[[380]](#ftnt_ref380) The particular name comes from: Cotra, Ajeya. ‘Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover’. AI Alignment Forum, 18 July 2022.[https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to](https://www.google.com/url?q=https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to&sa=D&source=editors&ust=1692801435244119&usg=AOvVaw33I-eslENEibpxAlf5m8zM).

[[381]](#ftnt_ref381) MLCommons. ‘MLCommons - Philosophy’. Accessed 2 December 2022.[https://mlcommons.org/](https://www.google.com/url?q=https://mlcommons.org/&sa=D&source=editors&ust=1692801435278347&usg=AOvVaw30jM23gGS486e0cVpdC4wK).

[[382]](#ftnt_ref382) O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801435323582&usg=AOvVaw2fbaTsq07IYsMohjJ_gHdd).

[[383]](#ftnt_ref383) For an interactive supply chain exploration tool, see: Emerging Technology Observatory. ‘Supply Chain Explorer: Advanced Chips’, 16 October 2022.[https://chipexplorer.eto.tech/](https://www.google.com/url?q=https://chipexplorer.eto.tech/&sa=D&source=editors&ust=1692801435278951&usg=AOvVaw3UrlTaxjrd6y1kw9ewpW8F).; for overviews, see: See Khan, Saif. ‘The Semiconductor Supply Chain: Assessing National Competitiveness’. Center for Security and Emerging Technology, January 2021.[https://cset.georgetown.edu/publication/the-semiconductor-supply-chain/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/the-semiconductor-supply-chain/&sa=D&source=editors&ust=1692801435279511&usg=AOvVaw3CvkDWGqfwNQiUZK1NPYB8).; Belfield, Haydn, and Shin-Shin Hua. ‘Compute and Antitrust: Regulatory implications of the AI hardware supply chain, from chip design to cloud APIs’. Verfassungsblog (blog), 19 August 2022.[https://verfassungsblog.de/compute-and-antitrust/](https://www.google.com/url?q=https://verfassungsblog.de/compute-and-antitrust/&sa=D&source=editors&ust=1692801435280121&usg=AOvVaw2w21unbOpxzAbE0IOVNWwv).; Elmgren, Karson. ‘China’s Silicon Future’. Asterisk, November 2022.[https://asteriskmag.com/issues/1/china-s-silicon-future](https://www.google.com/url?q=https://asteriskmag.com/issues/1/china-s-silicon-future&sa=D&source=editors&ust=1692801435280609&usg=AOvVaw2XE6oZEBjQrT0KnIAyfHyX).

[[384]](#ftnt_ref384) ASML. ‘ZEISS and ASML Strengthen Partnership for Next Generation of EUV Lithography’, 2016.[https://www.asml.com/en/news/press-releases/2016/zeiss-and-asml-strengthen-partnership-for-next-generation-of-euv-lithography](https://www.google.com/url?q=https://www.asml.com/en/news/press-releases/2016/zeiss-and-asml-strengthen-partnership-for-next-generation-of-euv-lithography&sa=D&source=editors&ust=1692801435409354&usg=AOvVaw2DeeU6OtbUY24tqxe_fnAU). Hoyng Rokh Monegier. ‘ASML and Carl Zeiss SMT v. Nikon – Immersion Lithography’, 28 July 2020.[https://www.hoyngrokhmonegier.com/news-insights/case-studies/asml-and-carl-zeiss-smt-v-nikon-immersion-lithography/](https://www.google.com/url?q=https://www.hoyngrokhmonegier.com/news-insights/case-studies/asml-and-carl-zeiss-smt-v-nikon-immersion-lithography/&sa=D&source=editors&ust=1692801435409874&usg=AOvVaw3xGghSlQgsHOGlYkIHskuN).

[[385]](#ftnt_ref385) Belfield, Haydn, and Shin-Shin Hua. ‘Compute and Antitrust: Regulatory implications of the AI hardware supply chain, from chip design to cloud APIs’. Verfassungsblog (blog), 19 August 2022.[https://verfassungsblog.de/compute-and-antitrust/](https://www.google.com/url?q=https://verfassungsblog.de/compute-and-antitrust/&sa=D&source=editors&ust=1692801435281308&usg=AOvVaw1X7NVqx8rBTv1_q0qGhRgI).; See also: Yglesias, Matthew. ‘At Last, an AI Existential Risk Policy Idea’. Slow Boring, 28 September 2022.[https://www.slowboring.com/p/at-last-an-ai-existential-risk-policy](https://www.google.com/url?q=https://www.slowboring.com/p/at-last-an-ai-existential-risk-policy&sa=D&source=editors&ust=1692801435281840&usg=AOvVaw0JA2obyt514MekvWt5sxq_).

[[386]](#ftnt_ref386) Elmgren, Karson. ‘China’s Silicon Future’. Asterisk, November 2022.[https://asteriskmag.com/issues/1/china-s-silicon-future](https://www.google.com/url?q=https://asteriskmag.com/issues/1/china-s-silicon-future&sa=D&source=editors&ust=1692801435282449&usg=AOvVaw0i9mcpyU6sJp5dVjBKr3vO).

[[387]](#ftnt_ref387) Belfield, Haydn, and Shin-Shin Hua. ‘Compute and Antitrust: Regulatory implications of the AI hardware supply chain, from chip design to cloud APIs’. Verfassungsblog (blog), 19 August 2022.[https://verfassungsblog.de/compute-and-antitrust/](https://www.google.com/url?q=https://verfassungsblog.de/compute-and-antitrust/&sa=D&source=editors&ust=1692801435283176&usg=AOvVaw2B0x3c2u1hMUKIaqAGurSO).

[[388]](#ftnt_ref388) Shevlane, Toby. ‘The Artefacts of Intelligence: Governing Scientists’ Contribution to AI Proliferation’. University of Oxford, 2022.[https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation](https://www.google.com/url?q=https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation&sa=D&source=editors&ust=1692801435284019&usg=AOvVaw258WRjewZ6HO7wyqsdNuGG).; Prunkl, Carina E. A., Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe. ‘Institutionalizing Ethics in AI through Broader Impact Requirements’. Nature Machine Intelligence 3, no. 2 (February 2021): 104–10.[https://doi.org/10.1038/s42256-021-00298-y](https://www.google.com/url?q=https://doi.org/10.1038/s42256-021-00298-y&sa=D&source=editors&ust=1692801435284605&usg=AOvVaw1_I3J_p8_E0R9jD1KC5pUG).

[[389]](#ftnt_ref389) Langenkamp, Max, and Daniel N. Yue. ‘How Open Source Machine Learning Software Shapes AI’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 385–95. Oxford United Kingdom: ACM, 2022.[https://doi.org/10.1145/3514094.3534167](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3534167&sa=D&source=editors&ust=1692801435364258&usg=AOvVaw03kRtdaA_zoasDzIWzXCiz).; Engler, Alex. ‘How Open-Source Software Shapes AI Policy’. Brookings (blog), 10 August 2021.[https://www.brookings.edu/research/how-open-source-software-shapes-ai-policy/](https://www.google.com/url?q=https://www.brookings.edu/research/how-open-source-software-shapes-ai-policy/&sa=D&source=editors&ust=1692801435364748&usg=AOvVaw043Cqx02I-W3DnR6xdE6gz).

[[390]](#ftnt_ref390) I thank Di Cooke for suggesting this term.

[[391]](#ftnt_ref391) Phang, Jason, Herbie Bradley, Leo Gao, Louis Castricato, and Stella Biderman. ‘EleutherAI: Going Beyond “Open Science” to “Science in the Open”’. arXiv, 12 October 2022.[https://doi.org/10.48550/arXiv.2210.06413](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2210.06413&sa=D&source=editors&ust=1692801435365385&usg=AOvVaw2QEWCAezI23r36rT2zc0nl).; though for critical views of actors in this community, see: Seger, Elizabeth. ‘What Do We Mean When We Talk About “AI Democratisation”?’ GovAI Blog (blog), 7 February 2023.[https://www.governance.ai/post/what-do-we-mean-when-we-talk-about-ai-democratisation](https://www.google.com/url?q=https://www.governance.ai/post/what-do-we-mean-when-we-talk-about-ai-democratisation&sa=D&source=editors&ust=1692801435365908&usg=AOvVaw3nOnZyRiX0v__iSoS7q5NA).; Casper, Stephen, Phillip Christoffersen, and Rui-Jie Yew. ‘The Slippery Slope from DALLE-2 to Deepfake Anarchy’, 5 November 2022.[https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy&sa=D&source=editors&ust=1692801435366447&usg=AOvVaw1zryZt4RIw0i1-RtungNJI).

[[392]](#ftnt_ref392) Ladish, Jeffrey. ‘Information Security Considerations for AI and the Long Term Future’. EA Forum, 2 May 2022.[https://forum.effectivealtruism.org/posts/WqQDCCLWbYfFRwubf/information-security-considerations-for-ai-and-the-long-term](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/WqQDCCLWbYfFRwubf/information-security-considerations-for-ai-and-the-long-term&sa=D&source=editors&ust=1692801435367204&usg=AOvVaw36g68cFIEgClrsjfLT0lMP).; Zabel, Claire, and Luke Muelhauser. ‘Information Security Careers for GCR Reduction’. EA Forum, 21 June 2019.[https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction&sa=D&source=editors&ust=1692801435367891&usg=AOvVaw1RC5VMglwyd8wgwo-9esbo).

[[393]](#ftnt_ref393) Bowerman, Niel. ‘The Case for Building Expertise to Work on US AI Policy’. 80,000 Hours, December 2020.[https://80000hours.org/articles/us-ai-policy/](https://www.google.com/url?q=https://80000hours.org/articles/us-ai-policy/&sa=D&source=editors&ust=1692801435026535&usg=AOvVaw3qVSE0TY4a4bojMXwot53z).

[[394]](#ftnt_ref394) I especially thank Di Cooke and Carlos Ignacio Gutierrez for some suggestions with regards to US actors.

[[395]](#ftnt_ref395) LegiStorm. ‘Congressional Artificial Intelligence Caucus’. Accessed 20 February 2023.[https://www.legistorm.com/organization/summary/129204/Congressional\_Artificial\_Intelligence\_Caucus.html](https://www.google.com/url?q=https://www.legistorm.com/organization/summary/129204/Congressional_Artificial_Intelligence_Caucus.html&sa=D&source=editors&ust=1692801435394709&usg=AOvVaw0EykH9D-LozXZCjfxgotVM).

[[396]](#ftnt_ref396) National Science Foundation. ‘Artificial Intelligence (AI) at NSF’. Accessed 20 February 2023.[https://www.nsf.gov/cise/ai.jsp](https://www.google.com/url?q=https://www.nsf.gov/cise/ai.jsp&sa=D&source=editors&ust=1692801435395276&usg=AOvVaw1ry4JQqWtWcGzcRRPNMaQw).

[[397]](#ftnt_ref397) Barrett, Anthony M., Dan Hendrycks, Jessica Newman, and Brandie Nonnecke. ‘Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks’. arXiv, 17 June 2022.[https://doi.org/10.48550/arXiv.2206.08966](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2206.08966&sa=D&source=editors&ust=1692801435275654&usg=AOvVaw1sT2EKiNAkMbjIl5vis6XD).

[[398]](#ftnt_ref398) National Security Commission on Artificial Intelligence. ‘Final Report’. National Security Commission on Artificial Intelligence, March 2021.[https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf](https://www.google.com/url?q=https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf&sa=D&source=editors&ust=1692801435116361&usg=AOvVaw0KUolag9OtgPaeOXXKqzhz).

[[399]](#ftnt_ref399) ‘U.S. Department of Homeland Security Artificial Intelligence Strategy’. U.S. Department of Homeland Security, 3 December 2020.[https://www.dhs.gov/publication/us-department-homeland-security-artificial-intelligence-strategy](https://www.google.com/url?q=https://www.dhs.gov/publication/us-department-homeland-security-artificial-intelligence-strategy&sa=D&source=editors&ust=1692801435243268&usg=AOvVaw0hTl9NntEmsbUGZSOn5Xc4). I thank Christoph Winter for this suggestion.

[[400]](#ftnt_ref400) FTC. ‘FTC Launches New Office of Technology to Bolster Agency’s Work’. Federal Trade Commission, 16 February 2023.[https://www.ftc.gov/news-events/news/press-releases/2023/02/ftc-launches-new-office-technology-bolster-agencys-work](https://www.google.com/url?q=https://www.ftc.gov/news-events/news/press-releases/2023/02/ftc-launches-new-office-technology-bolster-agencys-work&sa=D&source=editors&ust=1692801435440545&usg=AOvVaw0u0ZGYan-LuDfFInDsy2ka).

[[401]](#ftnt_ref401) Selbst, Andrew D., and Solon Barocas. ‘Unfair Artificial Intelligence: How FTC Intervention Can Overcome the Limitations of Discrimination Law’. SSRN Scholarly Paper. Rochester, NY, 8 August 2022.[https://papers.ssrn.com/abstract=4185227](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4185227&sa=D&source=editors&ust=1692801435396521&usg=AOvVaw2MT16Wvle_JkbvJ-_eizD-).; and Okerlund, Johanna, Evan Klasky, Aditya Middha, Sujin Kim, Hannah Rosenfeld, Molly Kleinman, and Shobita Parthasarathy. ‘What’s in the Chatterbox? Large Language Models, Why They Matter, and What We Should Do About Them’. Ford School of Public Policy, University of Michigan, 2022.[https://stpp.fordschool.umich.edu/research/research-report/whats-in-the-chatterbox](https://www.google.com/url?q=https://stpp.fordschool.umich.edu/research/research-report/whats-in-the-chatterbox&sa=D&source=editors&ust=1692801435397042&usg=AOvVaw1N5sRGNxXuDvHxD0YDyqqb).

[[402]](#ftnt_ref402) Casper, Stephen, Phillip Christoffersen, and Rui-Jie Yew. ‘The Slippery Slope from DALLE-2 to Deepfake Anarchy’. Effective Altruism Forum, 5 November 2022.[https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy&sa=D&source=editors&ust=1692801435397724&usg=AOvVaw24tDNt0X-cj-ZwmF-pPvdV).

[[403]](#ftnt_ref403) See e.g. Josephson, Henry. ‘A California Effect for Artificial Intelligence’, 2022.[https://www.henryjos.com/p/a-california-effect-for-artificial.html](https://www.google.com/url?q=https://www.henryjos.com/p/a-california-effect-for-artificial.html&sa=D&source=editors&ust=1692801435068587&usg=AOvVaw3QQaOGkrsxva7IFlkC8Iaa).

[[404]](#ftnt_ref404) 80,000 Hours. ‘China-Related AI Safety and Governance Paths’. 80,000 Hours, February 2022.[https://80000hours.org/career-reviews/china-related-ai-safety-and-governance-paths/](https://www.google.com/url?q=https://80000hours.org/career-reviews/china-related-ai-safety-and-governance-paths/&sa=D&source=editors&ust=1692801435027048&usg=AOvVaw2Yx8qw7fYoZpe3RV-zoRLM).; Cheng, Jing, and Jinghan Zeng. ‘Shaping AI’s Future? China in Global AI Governance’. Journal of Contemporary China 0, no. 0 (8 August 2022): 1–17.[https://doi.org/10.1080/10670564.2022.2107391](https://www.google.com/url?q=https://doi.org/10.1080/10670564.2022.2107391&sa=D&source=editors&ust=1692801435027457&usg=AOvVaw0WIao3YzDXL-grEpHnIMNI).; Ding, Jeffrey. ‘Deciphering China’s AI Dream: The Context, Components, Capabilities, and Consequences of China’s Strategy to Lead the World in AI’. Future of Humanity Institute, Governance of AI Program, March 2018.[https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering\_Chinas\_AI-Dream.pdf?platform=hootsuite](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering_Chinas_AI-Dream.pdf?platform%3Dhootsuite&sa=D&source=editors&ust=1692801435027893&usg=AOvVaw0tv0_i-Fo2jTg7fcHSOrvX).; Roberts, Huw, Josh Cowls, Jessica Morley, Mariarosaria Taddeo, Vincent Wang, and Luciano Floridi. ‘The Chinese Approach to Artificial Intelligence: An Analysis of Policy, Ethics, and Regulation’. AI & SOCIETY 36, no. 1 (1 March 2021): 59–77.[https://doi.org/10.1007/s00146-020-00992-2](https://www.google.com/url?q=https://doi.org/10.1007/s00146-020-00992-2&sa=D&source=editors&ust=1692801435028330&usg=AOvVaw05d0pvGMVb5P84IDyOrLFm).

[[405]](#ftnt_ref405) Sheehan, Matt. ‘China’s New AI Governance Initiatives Shouldn’t Be Ignored’. Carnegie Endowment for International Peace, 4 January 2022.[https://carnegieendowment.org/2022/01/04/china-s-new-ai-governance-initiatives-shouldn-t-be-ignored-pub-86127](https://www.google.com/url?q=https://carnegieendowment.org/2022/01/04/china-s-new-ai-governance-initiatives-shouldn-t-be-ignored-pub-86127&sa=D&source=editors&ust=1692801435042977&usg=AOvVaw2wkNm4RrfFrfJfiyfUDvKZ).

[[406]](#ftnt_ref406) Zhou, Frank, and Summer Sun. ‘How China Regulates Ethical Issues in “AI+life Science”’. International Bar Association, 21 October 2022.[https://www.ibanet.org/china-regulates-ai-life-science](https://www.google.com/url?q=https://www.ibanet.org/china-regulates-ai-life-science&sa=D&source=editors&ust=1692801435324151&usg=AOvVaw23ytjWrYSN8hT7uFN0WuVY).

[[407]](#ftnt_ref407) Hannas, William, Huey-Meei Chang, Daniel Chou, and Brian Fleeger. ‘China’s Advanced AI Research: Monitoring China’s Paths to “General” Artificial Intelligence’. Center for Security and Emerging Technology, July 2022.[https://cset.georgetown.edu/publication/chinas-advanced-ai-research/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/chinas-advanced-ai-research/&sa=D&source=editors&ust=1692801435115786&usg=AOvVaw3yTraOOppPBaxyd6KuI3Ju).

[[408]](#ftnt_ref408) Ying, Fu, and John Allen. ‘Together, The U.S. And China Can Reduce The Risks From AI’. Noema, 17 December 2020.[https://www.noemamag.com/together-the-u-s-and-china-can-reduce-the-risks-from-ai](https://www.google.com/url?q=https://www.noemamag.com/together-the-u-s-and-china-can-reduce-the-risks-from-ai&sa=D&source=editors&ust=1692801435410490&usg=AOvVaw3ngCVS6XefATyOWBLWzpJC).; U.S.-China Perception Monitor. ‘Who Is Fu Ying?’ Accessed 22 February 2023.[https://uscnpm.org/who-is-fu-ying/](https://www.google.com/url?q=https://uscnpm.org/who-is-fu-ying/&sa=D&source=editors&ust=1692801435410833&usg=AOvVaw0XSDlduN9jj0w-mRr6Sb6V).

[[409]](#ftnt_ref409) Stix, Charlotte. ‘The European Artificial Intelligence Landscape’. Workshop Report. European Commission, 2018.[https://ec.europa.eu/digital-single-market/en/news/european-artificial-intelligence-landscape](https://www.google.com/url?q=https://ec.europa.eu/digital-single-market/en/news/european-artificial-intelligence-landscape&sa=D&source=editors&ust=1692801435028881&usg=AOvVaw0ZQLdz-LUg-z31bGZz-Ky2).; Siegmann, Charlotte, and Markus Anderljung. ‘The Brussels Effect and Artificial Intelligence: How EU Regulation Will Impact the Global AI Market’. Centre for the Governance of AI, August 2022.[https://www.governance.ai/research-paper/brussels-effect-ai](https://www.google.com/url?q=https://www.governance.ai/research-paper/brussels-effect-ai&sa=D&source=editors&ust=1692801435029296&usg=AOvVaw2MMpBiRSRLXbap1PfEoWTk).; see also; Langosco, Lauro. ‘AI Policy Careers in the EU’. EA Forum, 11 November 2019.[https://forum.effectivealtruism.org/posts/XGPW25NZHq2WHbK9w/ai-policy-careers-in-the-eu](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/XGPW25NZHq2WHbK9w/ai-policy-careers-in-the-eu&sa=D&source=editors&ust=1692801435029728&usg=AOvVaw1eNQ0hDu614yzyUUrVNYJc).; Nicolas. ‘Should You Work in the European Union to Do AGI Governance?’ EA Forum, 31 January 2022.[https://forum.effectivealtruism.org/posts/fbG6wWZhJ3jt3xHxS/should-you-work-in-the-european-union-to-do-agi-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/fbG6wWZhJ3jt3xHxS/should-you-work-in-the-european-union-to-do-agi-governance&sa=D&source=editors&ust=1692801435030207&usg=AOvVaw0CI73TTiMdt58N63i9LNMV).

[[410]](#ftnt_ref410) I thank Haydn Belfield, Jacob Arbeid, and Moritz Kleinalterkamp for suggestions and input.

[[411]](#ftnt_ref411) European Commission. ‘International Outreach for Human-Centric Artificial Intelligence Initiative’, 2022.[https://digital-strategy.ec.europa.eu/en/policies/international-outreach-ai](https://www.google.com/url?q=https://digital-strategy.ec.europa.eu/en/policies/international-outreach-ai&sa=D&source=editors&ust=1692801435324740&usg=AOvVaw2jmcFzqQx_Um2gen8CMTDB).

[[412]](#ftnt_ref412) Stix, Charlotte. ‘The Ghost of AI Governance Past, Present, and Future: AI Governance in the European Union’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press. Accessed 21 October 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.56](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.56&sa=D&source=editors&ust=1692801435325287&usg=AOvVaw3Q7jyzlJrEypYF95BIf5_6).; and generally Stahl, Bernd Carsten, Rowena Rodrigues, Nicole Santiago, and Kevin Macnish. ‘A European Agency for Artificial Intelligence: Protecting Fundamental Rights and Ethical Values’. Computer Law & Security Review 45 (1 July 2022): 105661.[https://doi.org/10.1016/j.clsr.2022.105661](https://www.google.com/url?q=https://doi.org/10.1016/j.clsr.2022.105661&sa=D&source=editors&ust=1692801435325693&usg=AOvVaw1hOhAZRJH5U39KREQjxMKG).

[[413]](#ftnt_ref413) See Hadshar, Rose. ‘Current UK Government Levers on AI Development’. EA Forum, 10 April 2023.[https://forum.effectivealtruism.org/posts/BFBf5yPLoJMGozygE/current-uk-government-levers-on-ai-development](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/BFBf5yPLoJMGozygE/current-uk-government-levers-on-ai-development&sa=D&source=editors&ust=1692801435152965&usg=AOvVaw3_MtjR9_1iwj3p0doBRjxU). And also: Roberts, Huw, Alexander Babuta, Jessica Morley, Christopher Thomas, Mariarosaria Taddeo, and Luciano Floridi. ‘Artificial Intelligence Regulation in the United Kingdom: A Path to Global Leadership?’ SSRN Scholarly Paper. Rochester, NY, 1 September 2022.[https://doi.org/10.2139/ssrn.4209504](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4209504&sa=D&source=editors&ust=1692801435153370&usg=AOvVaw2-GTYGq3_g_JrPjzDoW0tT). For input on this section, I also thank Jess Whittlestone, Haydn Belfield, and Di Cooke.

[[414]](#ftnt_ref414) Maas, Matthijs M., Diane Cooke, Tom Hobson, Lalitha Sundaram, Haydn Belfield, Lara Mani, Jess Whittlestone, and Seán Ó HÉigeartaigh. ‘Reconfiguring Resilience for Existential Risk: Submission of Evidence to the Cabinet Office on the New UK National Resilience Strategy’. Centre for the Study of Existential Risk, 27 September 2021.[https://www.cser.ac.uk/resources/reconfiguring-resilience-existential-risk-submission-evidence-cabinet-office-new-uk-national-resilience-strategy/](https://www.google.com/url?q=https://www.cser.ac.uk/resources/reconfiguring-resilience-existential-risk-submission-evidence-cabinet-office-new-uk-national-resilience-strategy/&sa=D&source=editors&ust=1692801435065702&usg=AOvVaw03uf8oPmlIr9hsbEG3Z-XW).

[[415]](#ftnt_ref415) See also the section on [‘hardware supply chain’](#h.qtapto9yloz0) actors.

[[416]](#ftnt_ref416) Chua, Yi-Yang. ‘Singapore AI Policy Career Guide’. EA Forum, 21 January 2021.[https://forum.effectivealtruism.org/posts/umeMcbD4jDseLjsgT/singapore-ai-policy-career-guide](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/umeMcbD4jDseLjsgT/singapore-ai-policy-career-guide&sa=D&source=editors&ust=1692801435060533&usg=AOvVaw1Dk6WdhEem7K8sUoOCrM1z).

[[417]](#ftnt_ref417) Png, Marie-Therese. ‘At the Tensions of South and North: Critical Roles of Global South Stakeholders in AI Governance’. In 2022 ACM Conference on Fairness, Accountability, and Transparency, 1434–45. FAccT ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3531146.3533200](https://www.google.com/url?q=https://doi.org/10.1145/3531146.3533200&sa=D&source=editors&ust=1692801435154126&usg=AOvVaw01IkAUvYdzC3DQQGAf2b27).; Garcia, Eugenio V. ‘The Militarization of Artificial Intelligence: A Wake-up Call for the Global South’, September 2019.[https://www.researchgate.net/publication/335787908\_The\_militarization\_of\_artificial\_intelligence\_a\_wake-up\_call\_for\_the\_Global\_South/references](https://www.google.com/url?q=https://www.researchgate.net/publication/335787908_The_militarization_of_artificial_intelligence_a_wake-up_call_for_the_Global_South/references&sa=D&source=editors&ust=1692801435154622&usg=AOvVaw29-NJHweqz-PRJg_a64_Ax).

[[418]](#ftnt_ref418) See generally: Andersen, Philip Hall, Henrik Øberg Myhre, Andreas Massey, Jakob Graabak, Sanna Baug Warholm, and Erik Aunvåg Matsen. ‘Why Scale Is Overrated: The Case for Increasing EA Policy Efforts in Smaller Countries’. EA Forum, 15 August 2021.[https://forum.effectivealtruism.org/posts/7SjtFYo6sCe3588Tx/why-scale-is-overrated-the-case-for-increasing-ea-policy](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/7SjtFYo6sCe3588Tx/why-scale-is-overrated-the-case-for-increasing-ea-policy&sa=D&source=editors&ust=1692801435276499&usg=AOvVaw0WZvWNoUhv6Q7Ap0EP8FQY).

[[419]](#ftnt_ref419) Cihon, Peter. ‘Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development’. Technical Report. Oxford: Center for the Governance of AI, Future of Humanity Institute, University of Oxford, April 2019.[https://www.fhi.ox.ac.uk/wp-content/uploads/Standards\_-FHI-Technical-Report.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Standards_-FHI-Technical-Report.pdf&sa=D&source=editors&ust=1692801435030755&usg=AOvVaw1ew8iAcJuufaqzalQrHSv6).; Lorenz, Philippe. ‘AI Governance through Political Fora and Standards Developing Organizations’. Stiftung Neue Verantwortung, September 2020.[https://www.stiftung-nv.de/de/publikation/ai-governance-through-political-fora-and-standards-developing-organizations](https://www.google.com/url?q=https://www.stiftung-nv.de/de/publikation/ai-governance-through-political-fora-and-standards-developing-organizations&sa=D&source=editors&ust=1692801435031196&usg=AOvVaw0RuFA2coEJHJR_eZ61B2aQ).

[[420]](#ftnt_ref420) ‘Artificial Intelligence Put to Test: State of Hesse and VDE Present First AI Quality & Testing Hub Nationwide’, 13 February 2023.[https://www.vde.com/en/press/press-releases/2023-02-13-pk-aiq-hub](https://www.google.com/url?q=https://www.vde.com/en/press/press-releases/2023-02-13-pk-aiq-hub&sa=D&source=editors&ust=1692801435395891&usg=AOvVaw0peuPBnPSR3qyUxgz-N4gf).

[[421]](#ftnt_ref421) Garcia, Eugenio V. ‘Multilateralism and Artificial Intelligence: What Role for the United Nations?’ In The Global Politics of Artificial Intelligence, edited by Maurizio Tinnirello, 18. Boca Raton: CRC Press, 2020.[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3779866](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3779866&sa=D&source=editors&ust=1692801435031750&usg=AOvVaw3wqryaf0__dF-qwv4vqiqd).; see also: Nindler, Reinmar. ‘The United Nation’s Capability to Manage Existential Risks with a Focus on Artificial Intelligence’. International Community Law Review 21, no. 1 (11 March 2019): 5–34.[https://doi.org/10.1163/18719732-12341388](https://www.google.com/url?q=https://doi.org/10.1163/18719732-12341388&sa=D&source=editors&ust=1692801435032193&usg=AOvVaw0dgrICiQ3YEI01G3wWYQ_G).; and see the overview at: Kunz, Martina. ‘AI and International Organizations’. Accessed 31 October 2022.[https://globalaigov.org/participants/igos.html](https://www.google.com/url?q=https://globalaigov.org/participants/igos.html&sa=D&source=editors&ust=1692801435032570&usg=AOvVaw26tJeIP8fQ4KC1ufdVuSfC). I thank Eugenio Vargas Garcia for additional suggestions.

[[422]](#ftnt_ref422) ITU. ‘United Nations Activities on Artificial Intelligence (AI) 2019’. ITU, 2019.[https://www.itu.int/dms\_pub/itu-s/opb/gen/S-GEN-UNACT-2019-1-PDF-E.pdf](https://www.google.com/url?q=https://www.itu.int/dms_pub/itu-s/opb/gen/S-GEN-UNACT-2019-1-PDF-E.pdf&sa=D&source=editors&ust=1692801435326206&usg=AOvVaw3uCoh5TyXvx_LsXcQKfg-j).

[[423]](#ftnt_ref423) Garcia, Eugenio V. ‘Multilateralism and Artificial Intelligence: What Role for the United Nations?’ In The Global Politics of Artificial Intelligence, edited by Maurizio Tinnirello, 18. Boca Raton: CRC Press, 2020.[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3779866](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3779866&sa=D&source=editors&ust=1692801435050694&usg=AOvVaw167_gCzt8IY5KYbdyiCzNZ).; see also: UNESCO. ‘UNESCO Member States Adopt the First Ever Global Agreement on the Ethics of Artificial Intelligence’. UNESCO, 25 November 2021.[https://en.unesco.org/news/unesco-member-states-adopt-first-ever-global-agreement-ethics-artificial-intelligence](https://www.google.com/url?q=https://en.unesco.org/news/unesco-member-states-adopt-first-ever-global-agreement-ethics-artificial-intelligence&sa=D&source=editors&ust=1692801435051245&usg=AOvVaw21rHQbGLv948m8ofpyNIqo).

[[424]](#ftnt_ref424) European Center for Not-for-Profit Law. ‘UN HRC 51: New and Emerging Technologies and Human Rights at the Heart of New Resolutions Adopted’, 13 October 2022.[https://ecnl.org/news/un-hrc-51-new-and-emerging-technologies-and-human-rights-heart-new-resolutions-adopted](https://www.google.com/url?q=https://ecnl.org/news/un-hrc-51-new-and-emerging-technologies-and-human-rights-heart-new-resolutions-adopted&sa=D&source=editors&ust=1692801435269610&usg=AOvVaw0SJZBq0Z-k-KbyQzgdHk6m).

[[425]](#ftnt_ref425) UN OHCHR. ‘New and Emerging Digital Technologies and Human Rights’. OHCHR. Accessed 30 January 2023.[https://www.ohchr.org/en/hr-bodies/hrc/advisory-committee/digital-technologiesand-hr](https://www.google.com/url?q=https://www.ohchr.org/en/hr-bodies/hrc/advisory-committee/digital-technologiesand-hr&sa=D&source=editors&ust=1692801435270297&usg=AOvVaw2qO6bt8Kqq_u4ve6OObK-O).

[[426]](#ftnt_ref426) UN - CEB. ‘Artificial Intelligence’, 2020.[https://unsceb.org/topics/artificial-intelligence](https://www.google.com/url?q=https://unsceb.org/topics/artificial-intelligence&sa=D&source=editors&ust=1692801435270893&usg=AOvVaw23rtmvaU9QQHHUQmnA2Ct0).

[[427]](#ftnt_ref427) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Fragmentation and the Future: Investigating Architectures for International AI Governance’. Global Policy 11, no. 5 (November 2020): 545–56.[https://doi.org/10.1111/1758-5899.12890](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12890&sa=D&source=editors&ust=1692801435033119&usg=AOvVaw0w8GtqonMK19SKK4nA50aA).; Schmitt, Lewin. ‘Mapping Global AI Governance: A Nascent Regime in a Fragmented Landscape’. AI and Ethics, 17 August 2021.[https://doi.org/10.1007/s43681-021-00083-y](https://www.google.com/url?q=https://doi.org/10.1007/s43681-021-00083-y&sa=D&source=editors&ust=1692801435033521&usg=AOvVaw0jAtQYl1-MbpNZDb6qp2ZW).

[[428]](#ftnt_ref428) See generally: Morin, Jean‐Frédéric, Hugo Dobson, Claire Peacock, Miriam Prys‐Hansen, Abdoulaye Anne, Louis Bélanger, Peter Dietsch, et al. ‘How Informality Can Address Emerging Issues: Making the Most of the G7’. Global Policy 10, no. 2 (May 2019): 267–73.[https://doi.org/10.1111/1758-5899.12668](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12668&sa=D&source=editors&ust=1692801435047763&usg=AOvVaw24Q2cu9P5cToHAGfDPUxf4). (briefly discussing AI issues as an area where informal, like-minded club governance could excel).

[[429]](#ftnt_ref429) Jelinek, Thorsten, Wendell Wallach, and Danil Kerimi. ‘Policy Brief: The Creation of a G20 Coordinating Committee for the Governance of Artificial Intelligence’. AI and Ethics, 6 October 2020.[https://doi.org/10.1007/s43681-020-00019-y](https://www.google.com/url?q=https://doi.org/10.1007/s43681-020-00019-y&sa=D&source=editors&ust=1692801435047204&usg=AOvVaw35nQELNzwrwaMXrJtorFT6).

[[430]](#ftnt_ref430) Breuer, Marten. ‘The Council of Europe as an AI Standard Setter’. Verfassungsblog (blog), 4 April 2022.[https://verfassungsblog.de/the-council-of-europe-as-an-ai-standard-setter/](https://www.google.com/url?q=https://verfassungsblog.de/the-council-of-europe-as-an-ai-standard-setter/&sa=D&source=editors&ust=1692801435048438&usg=AOvVaw2V42rGZW2Qp9R-_jGsqp5Q).

[[431]](#ftnt_ref431) NATO. ‘NATO’s Data and Artificial Intelligence Review Board’. NATO, 13 October 2022.[https://www.nato.int/cps/en/natohq/official\_texts\_208374.htm](https://www.google.com/url?q=https://www.nato.int/cps/en/natohq/official_texts_208374.htm&sa=D&source=editors&ust=1692801435049017&usg=AOvVaw2x7Y3sDyPXGIQii2RuDZ7k).; see also: Stanley-Lockman, Zoe, and Lena Trabucco. ‘NATO’s Role in Responsible AI Governance in Military Affairs’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press. Accessed 21 October 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.69](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.69&sa=D&source=editors&ust=1692801435049460&usg=AOvVaw3PvII57boFqU9sdUpoyYgI).

[[432]](#ftnt_ref432) Trabucco, Lena. ‘AI Partnership for Defense Is a Step in the Right Direction – But Will Face Challenges’. Opinio Juris (blog), 5 October 2020.[http://opiniojuris.org/2020/10/05/ai-partnership-for-defense-is-a-step-in-the-right-direction-but-will-face-challenges/](https://www.google.com/url?q=http://opiniojuris.org/2020/10/05/ai-partnership-for-defense-is-a-step-in-the-right-direction-but-will-face-challenges/&sa=D&source=editors&ust=1692801435050146&usg=AOvVaw1dBxAyHd2BnAgCSEwq1vYm).

[[433]](#ftnt_ref433) Smith, Bryant Walker. ‘New Technologies and Old Treaties’. AJIL Unbound 114 (ed 2020): 152–57.[https://doi.org/10.1017/aju.2020.28](https://www.google.com/url?q=https://doi.org/10.1017/aju.2020.28&sa=D&source=editors&ust=1692801435069113&usg=AOvVaw0GmrDaeBGtGKzfoPb9EP2M).

[[434]](#ftnt_ref434) European Commission. ‘EU-US Trade and Technology Council’. Accessed 30 January 2023.[https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/stronger-europe-world/eu-us-trade-and-technology-council\_en](https://www.google.com/url?q=https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/stronger-europe-world/eu-us-trade-and-technology-council_en&sa=D&source=editors&ust=1692801435061119&usg=AOvVaw0WIaR_plyi3M3poQE89oBV). See also: O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801435061604&usg=AOvVaw1cV0zTgn0HL4Rv2lTu-f9X).

[[435]](#ftnt_ref435) Kunz, Martina. ‘AI and Multi-Stakeholder Fora’. Accessed 31 October 2022.[https://globalaigov.org/participants/fora.html](https://www.google.com/url?q=https://globalaigov.org/participants/fora.html&sa=D&source=editors&ust=1692801435064582&usg=AOvVaw2kKDM7sjbfeg7W0Q2BMcSG).

[[436]](#ftnt_ref436) See also the section on some [institutional proposals](#h.13k9jc79o5ll).

[[437]](#ftnt_ref437) On the relatively slow 2000s response to the threat of LAWS, see Carpenter, Charli. ‘Lost’ Causes, Agenda Vetting in Global Issue Networks and the Shaping of Human Security. Ithaca: Cornell University Press, 2014.[https://doi.org/10.7591/9780801470363](https://www.google.com/url?q=https://doi.org/10.7591/9780801470363&sa=D&source=editors&ust=1692801435156149&usg=AOvVaw3q1uolS9jxdbITnLBbbyvE).

[[438]](#ftnt_ref438) Campaign to Stop Killer Robots. ‘About Us’. Accessed 5 September 2020.[https://www.stopkillerrobots.org/about/](https://www.google.com/url?q=https://www.stopkillerrobots.org/about/&sa=D&source=editors&ust=1692801435155159&usg=AOvVaw3QrFH-5O18QTafrwlIDoWK). But see: Rosert, Elvira, and Frank Sauer. ‘How (Not) to Stop the Killer Robots: A Comparative Analysis of Humanitarian Disarmament Campaign Strategies’. Contemporary Security Policy 0, no. 0 (30 May 2020): 1–26.[https://doi.org/10.1080/13523260.2020.1771508](https://www.google.com/url?q=https://doi.org/10.1080/13523260.2020.1771508&sa=D&source=editors&ust=1692801435155633&usg=AOvVaw1IC3We4VIvnoafNgxL-oUF).

[[439]](#ftnt_ref439) See e.g. AlgorithmWatch. ‘AI Ethics Guidelines Global Inventory’. AlgorithmWatch (blog), 2019.[https://algorithmwatch.org/en/project/ai-ethics-guidelines-global-inventory/](https://www.google.com/url?q=https://algorithmwatch.org/en/project/ai-ethics-guidelines-global-inventory/&sa=D&source=editors&ust=1692801435391274&usg=AOvVaw0JHPurN47ngcgKwV6LweSP).

[[440]](#ftnt_ref440) I thank Di Cooke for suggesting many of these.

[[441]](#ftnt_ref441) Geneva Centre for Security Policy. ‘The Geneva Process on AI Principles’. Accessed 28 January 2023.[https://www.gcsp.ch/the-geneva-process-on-AI-Principles](https://www.google.com/url?q=https://www.gcsp.ch/the-geneva-process-on-AI-Principles&sa=D&source=editors&ust=1692801435277261&usg=AOvVaw098kGReCuoASZ-tjbky8yu).; Centre for Humanitarian Dialogue. ‘Code of Conduct on Artificial Intelligence in Military Systems’. Centre for Humanitarian Dialogue, 2021.[https://www.hdcentre.org/wp-content/uploads/2021/08/AI-Code-of-Conduct.pdf](https://www.google.com/url?q=https://www.hdcentre.org/wp-content/uploads/2021/08/AI-Code-of-Conduct.pdf&sa=D&source=editors&ust=1692801435277797&usg=AOvVaw1NT0XxYR1PZL4F6PkdaakO).

[[442]](#ftnt_ref442) See broadly: Bettle, Rosie. ‘Mass Media Interventions: Shallow Investigation’. Founders Pledge, 17 November 2022.[https://docs.google.com/document/d/1SZ590oQkLYFQtU82kdtP9WBajpqcxVTVu302lYg\_Eeo/edit?](https://www.google.com/url?q=https://docs.google.com/document/d/1SZ590oQkLYFQtU82kdtP9WBajpqcxVTVu302lYg_Eeo/edit?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435102119&usg=AOvVaw1rdqcMwaWYb_iSBDM1tP_q).

[[443]](#ftnt_ref443) For a comparative analysis of the role of para-scientific media in shaping public perceptions and policy courses in a different technological domain, that of nanotechnology, see: Kaplan, Sarah, and Joanna Radin. ‘Bounding an Emerging Technology: Para-Scientific Media and the Drexler-Smalley Debate about Nanotechnology’. Social Studies of Science 41, no. 4 (2011): 457–85. [https://www.jstor.org/stable/41301944](https://www.google.com/url?q=https://www.jstor.org/stable/41301944&sa=D&source=editors&ust=1692801435069484&usg=AOvVaw2Wwj8cygPYWWLfd_KRj5br)

[[444]](#ftnt_ref444) Eg. Neal Stephenson, Cixin Liu, or many others, depending on the intended audiences.

[[445]](#ftnt_ref445) For discussion of these terms, see also Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023

[[446]](#ftnt_ref446) Another mapping of tools (called ‘catalysts’) is provided in: Hobbhahn, Marius, Max Räuker, Yannick Mühlhäuser, Jasper Götting, and Simon Grimm. ‘What Success Looks Like’. Effective Altruism Forum, 28 June 2022.[https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like&sa=D&source=editors&ust=1692801435434864&usg=AOvVaw1KBOgu5kgmk9Qcivb9Dgrh). (distinguishing between:

* “Governance: domestic laws, international treaties, safety regulations, whistleblower protection, auditing firms, compute governance and contingency plans;
* Technical: Red teaming, benchmarks, fire alarms, forecasting and information security;
* Societal: Norms in AI, widespread publicity, expert publicity and field-building”).
[[447]](#ftnt_ref447) This draws inspiration from ongoing work by Jonas Schuett. For another overview, see also: AI Impacts. ‘Affordances for AI Labs’. AI Impacts Wiki, 25 January 2023.[https://wiki.aiimpacts.org/doku.php?id=responses\_to\_ai:affordances:lab\_affordances](https://www.google.com/url?q=https://wiki.aiimpacts.org/doku.php?id%3Dresponses_to_ai:affordances:lab_affordances&sa=D&source=editors&ust=1692801435065076&usg=AOvVaw08C3xKUO83iiN3HWcmpmt1).

[[448]](#ftnt_ref448) Karnofsky, Holden. ‘What AI Companies Can Do Today to Help with the Most Important Century’. Cold Takes, 20 February 2023.[https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/](https://www.google.com/url?q=https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/&sa=D&source=editors&ust=1692801435424559&usg=AOvVaw04rn7gTtuKXd8I8gyyAHwg)

[[449]](#ftnt_ref449) Karnofsky, Holden. ‘How Might We Align Transformative AI If It’s Developed Very Soon?’ EA Forum, 29 August 2022.[https://forum.effectivealtruism.org/posts/sW6RggfddDrcmM6Aw/how-might-we-align-transformative-ai-if-it-s-developed-very](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/sW6RggfddDrcmM6Aw/how-might-we-align-transformative-ai-if-it-s-developed-very&sa=D&source=editors&ust=1692801435420805&usg=AOvVaw0PFH4vGCYsTZ1UrNkiDykO).

[[450]](#ftnt_ref450) Ashmore, Rob, Radu Calinescu, and Colin Paterson. ‘Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges’. arXiv, 10 May 2019.[https://doi.org/10.48550/arXiv.1905.04223](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.1905.04223&sa=D&source=editors&ust=1692801435421272&usg=AOvVaw0iMEDseY0gDvKnFk2LesRl).

[[451]](#ftnt_ref451) ARC Evals. ‘Update on ARC’s Recent Eval Efforts’, 17 March 2023.[https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/](https://www.google.com/url?q=https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/&sa=D&source=editors&ust=1692801435422298&usg=AOvVaw1ySqIbV7Ygw4aW1Ms7Ykfu).

[[452]](#ftnt_ref452) Schuett, Jonas. ‘Three Lines of Defense against Risks from AI’. arXiv, 16 December 2022.[https://doi.org/10.48550/arXiv.2212.08364](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2212.08364&sa=D&source=editors&ust=1692801435202752&usg=AOvVaw2YhGRKB_UBdgqCuXuF_Hbz).

[[453]](#ftnt_ref453) Yudkowsky, Eliezer. ‘Six Dimensions of Operational Adequacy in AGI Projects’. Machine Intelligence Research Institute, 8 June 2022.[https://intelligence.org/2022/06/07/six-dimensions-of-operational-adequacy-in-agi-projects/](https://www.google.com/url?q=https://intelligence.org/2022/06/07/six-dimensions-of-operational-adequacy-in-agi-projects/&sa=D&source=editors&ust=1692801435413021&usg=AOvVaw0Fr-BCD2OHkxpdsfMAI_c6).

[[454]](#ftnt_ref454) Muelhauser, Luke. ‘A Personal Take on Longtermist AI Governance’. EA Forum, 16 July 2021.[https://forum.effectivealtruism.org/posts/M2SBwctwC6vBqAmZW/a-personal-take-on-longtermist-ai-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/M2SBwctwC6vBqAmZW/a-personal-take-on-longtermist-ai-governance&sa=D&source=editors&ust=1692801435421831&usg=AOvVaw0E6EsWfoUcieNpXJLAQWxV). (footnote 19).

[[455]](#ftnt_ref455) Korinek, Anton. ‘Why We Need a New Agency to Regulate Advanced Artificial Intelligence: Lessons on AI Control from the Facebook Files’. Brookings (blog), 8 December 2021.[https://www.brookings.edu/research/why-we-need-a-new-agency-to-regulate-advanced-artificial-intelligence-lessons-on-ai-control-from-the-facebook-files/](https://www.google.com/url?q=https://www.brookings.edu/research/why-we-need-a-new-agency-to-regulate-advanced-artificial-intelligence-lessons-on-ai-control-from-the-facebook-files/&sa=D&source=editors&ust=1692801435422963&usg=AOvVaw1SBkPir4-HwhN1-iLYjjGW).; for broader work on oversight boards at Meta, see: Wong, David, and Luciano Floridi. ‘Meta’s Oversight Board: A Review and Critical Assessment’. SSRN Scholarly Paper. Rochester, NY, 22 October 2022.[https://papers.ssrn.com/abstract=4255817](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4255817&sa=D&source=editors&ust=1692801435423314&usg=AOvVaw2CvKDVjb4K-n5DARtX5htg).; Helfer, Laurence R., and Molly K. Land. ‘The Facebook Oversight Board’s Human Rights Future’. SSRN Scholarly Paper. Rochester, NY, 22 August 2022.[https://doi.org/10.2139/ssrn.4197107](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4197107&sa=D&source=editors&ust=1692801435423636&usg=AOvVaw1vR21C2uRCwnGAWFqnA1dD).; Kulick, Andreas. ‘Corporations as Interpreters and Adjudicators of International Human Rights Norms – Meta‘s Oversight Board and Beyond’. SSRN Scholarly Paper. Rochester, NY, 22 September 2022.[https://papers.ssrn.com/abstract=4226521](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4226521&sa=D&source=editors&ust=1692801435424009&usg=AOvVaw3oGDYtQ9L7AMdaavFIEXe0).

[[456]](#ftnt_ref456) Cihon, Peter, Jonas Schuett, and Seth D. Baum. ‘Corporate Governance of Artificial Intelligence in the Public Interest’. Information 12, no. 7 (July 2021): 275.[https://doi.org/10.3390/info12070275](https://www.google.com/url?q=https://doi.org/10.3390/info12070275&sa=D&source=editors&ust=1692801434993093&usg=AOvVaw3Mj59XGfgFBt1k9BLLqv28).

[[457]](#ftnt_ref457) Zabel, Claire, and Luke Muehlhauser. ‘Information Security Careers for GCR Reduction’. Effective Altruism Forum (blog), 2019.[https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction&sa=D&source=editors&ust=1692801435425158&usg=AOvVaw3K-HAidjM1pGONdN-mCgaK).; Ladish, Jeffrey. ‘Information Security Considerations for AI and the Long Term Future’. EA Forum, 2 May 2022.[https://forum.effectivealtruism.org/posts/WqQDCCLWbYfFRwubf/information-security-considerations-for-ai-and-the-long-term](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/WqQDCCLWbYfFRwubf/information-security-considerations-for-ai-and-the-long-term&sa=D&source=editors&ust=1692801435425592&usg=AOvVaw0Gvq6ZXFK0Q7Rkzto2Csy2).

[[458]](#ftnt_ref458) Shevlane, Toby. ‘Structured Access: An Emerging Paradigm for Safe AI Deployment’. In The Oxford Handbook of AI Governance, by Toby Shevlane, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.39](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.39&sa=D&source=editors&ust=1692801435426123&usg=AOvVaw0xlkIE764HQ7uO8nKQxHIs).

[[459]](#ftnt_ref459) See generally: Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. ‘Model Cards for Model Reporting’. In Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–29, 2019.[https://doi.org/10.1145/3287560.3287596](https://www.google.com/url?q=https://doi.org/10.1145/3287560.3287596&sa=D&source=editors&ust=1692801435426649&usg=AOvVaw3hIaT2z9sLE2tGri_sUHtn). And for an application, see: OpenAI. ‘GPT-4 System Card’. OpenAI, 14 March 2023.[https://cdn.openai.com/papers/gpt-4-system-card.pdf](https://www.google.com/url?q=https://cdn.openai.com/papers/gpt-4-system-card.pdf&sa=D&source=editors&ust=1692801435427005&usg=AOvVaw3eyiP9yMb7EpqW7Mj1_geA).

[[460]](#ftnt_ref460) I thank Zach Stein-Perlman for suggesting this. For a discussion of some of the challenges involved with maintaining such policies, see: Raemon. ‘“Carefully Bootstrapped Alignment” Is Organizationally Hard’. LessWrong, 17 March 2023.[https://www.lesswrong.com/posts/thkAtqoQwN6DtaiGT/carefully-bootstrapped-alignment-is-organizationally-hard](https://www.google.com/url?q=https://www.lesswrong.com/posts/thkAtqoQwN6DtaiGT/carefully-bootstrapped-alignment-is-organizationally-hard&sa=D&source=editors&ust=1692801435432918&usg=AOvVaw0VVqqLT5wdMrzlcaeGyHYU).

[[461]](#ftnt_ref461) The Promise of AI with Demis Hassabis - DeepMind: The Podcast (S2, Ep9), 2022.[https://www.youtube.com/watch?v=GdeY-MrXD74](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DGdeY-MrXD74&sa=D&source=editors&ust=1692801435433397&usg=AOvVaw0xeRee_4VJ5kEbap1Zhtm6).

[[462]](#ftnt_ref462) Han, The Anh, Tom Lenaerts, Francisco C. Santos, and Luis Moniz Pereira. ‘Voluntary Safety Commitments Provide an Escape from Over-Regulation in AI Development’. ArXiv:2104.03741 [Nlin], 8 April 2021.[http://arxiv.org/abs/2104.03741](https://www.google.com/url?q=http://arxiv.org/abs/2104.03741&sa=D&source=editors&ust=1692801435108039&usg=AOvVaw3M1XDdDAaopmvFxGG8Iykm).

[[463]](#ftnt_ref463) The term derives from: Sunstein, Cass R. ‘Social Norms and Social Roles’. Columbia Law Review 96, no. 4 (May 1996): 903.[https://doi.org/10.2307/1123430](https://www.google.com/url?q=https://doi.org/10.2307/1123430&sa=D&source=editors&ust=1692801435368526&usg=AOvVaw2l1qfMjgFntJ2hPqJgCA1y). pg. 1996 (defining ‘norm entrepreneurs’ as:

“Political actors [who] might be able to exploit private dissatisfaction with existing norms in order to bring about large-scale social change [...] norm entrepreneurs can alert people to the existence of a shared complaint and can suggest a collective solution. [...] Thus political actors, whether public or private, can exploit widespread dissatisfaction with existing norms by (a) signalling their own commitment to change, (b) creating coalitions, (c) making defiance of the norms seem or be less costly, and (d) making compliance with new norms seem or be more beneficial.”).

The norm entrepreneurship framework has been applied to many other domains, such as internet governance: Hurel, Louise Marie, and Luisa Cruz Lobato. ‘Unpacking Cyber Norms: Private Companies as Norm Entrepreneurs’. Journal of Cyber Policy 3, no. 1 (2 January 2018): 61–76.[https://doi.org/10.1080/23738871.2018.1467942](https://www.google.com/url?q=https://doi.org/10.1080/23738871.2018.1467942&sa=D&source=editors&ust=1692801435369353&usg=AOvVaw14F4IS2nPPEAldFuYZCmiT).; Radu, Roxana, Matthias C. Kettemann, Trisha Meyer, and Jamal Shahin. ‘Normfare: Norm Entrepreneurship in Internet Governance’. Telecommunications Policy, Norm entrepreneurship in Internet Governance, 45, no. 6 (1 July 2021): 102148.[https://doi.org/10.1016/j.telpol.2021.102148](https://www.google.com/url?q=https://doi.org/10.1016/j.telpol.2021.102148&sa=D&source=editors&ust=1692801435369873&usg=AOvVaw1bTVCqoCLdHWjklnyeYLg1).; multilateral arms control: Müller, Harald, and Carmen Wunderlich. Norm Dynamics in Multilateral Arms Control: Interests, Conflicts, and Justice. University of Georgia Press, 2013.; and others.

[[464]](#ftnt_ref464) O’Keefe, Cullen. ‘Antitrust-Compliant AI Industry Self-Regulation’. LPP WORKING PAPER SERIES. Rochester, NY, 30 September 2021.[https://doi.org/10.2139/ssrn.3933677](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.3933677&sa=D&source=editors&ust=1692801435285219&usg=AOvVaw0uBKFXmyyOWrArEf58YvKV).

[[465]](#ftnt_ref465) Schiff, D., J. Borenstein, J. Biddle, and K. Laas. ‘AI Ethics in the Public, Private, and NGO Sectors: A Review of a Global Document Collection’. IEEE Transactions on Technology and Society, 2021, 1–1.[https://doi.org/10.1109/TTS.2021.3052127](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3052127&sa=D&source=editors&ust=1692801435326773&usg=AOvVaw0zbA0h03He8kDXVv4GEPW_).

[[466]](#ftnt_ref466) Gasser, Urs, and Carolyn Schmitt. ‘The Role of Professional Norms in the Governance of Artificial Intelligence’. In The Oxford Handbook of AI Ethics, edited by M Dubber and F. Pasquale, 34. Oxford University Press, 2019.

[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3378267](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3378267&sa=D&source=editors&ust=1692801435327237&usg=AOvVaw2b6gY7-etAzr3OvpW7hFGH)

[[467]](#ftnt_ref467) Newman, Jessica Cussins. ‘Decision Points in AI Governance’. Berkeley, CA: Center for Long-Term Cybersecurity, 5 May 2020.[https://cltc.berkeley.edu/wp-content/uploads/2020/05/Decision\_Points\_AI\_Governance.pdf](https://www.google.com/url?q=https://cltc.berkeley.edu/wp-content/uploads/2020/05/Decision_Points_AI_Governance.pdf&sa=D&source=editors&ust=1692801435327816&usg=AOvVaw1VOpFWUrwKZkLOlFqX1Cr3).

[[468]](#ftnt_ref468) McGregor, Sean. ‘Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database’. arXiv, 17 November 2020.[https://doi.org/10.48550/arXiv.2011.08512](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2011.08512&sa=D&source=editors&ust=1692801435328334&usg=AOvVaw3kNX30j94nbkuDN4r1-PJK).

[[469]](#ftnt_ref469) Brundage, Miles, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, et al. ‘Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims’. ArXiv:2004.07213 [Cs], 15 April 2020.[http://arxiv.org/abs/2004.07213](https://www.google.com/url?q=http://arxiv.org/abs/2004.07213&sa=D&source=editors&ust=1692801435328903&usg=AOvVaw04Lol_k5p7hbgda0aiDbLc).

[[470]](#ftnt_ref470) See generally; Kenway, Josh, Camille François, Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. ‘Bug Bounties for Algorithmic Harms: Lessons from Cybersecurity Vulnerability Disclosure for Algorithmic Harms Discovery, Disclosure, and Redress’. Algorithmic Justice League, January 2022.[https://drive.google.com/file/d/1f4hVwQNiwp13zy62wUhwIg84lOq0ciG\_/view?](https://www.google.com/url?q=https://drive.google.com/file/d/1f4hVwQNiwp13zy62wUhwIg84lOq0ciG_/view?usp%3Dembed_facebook&sa=D&source=editors&ust=1692801435329509&usg=AOvVaw0CjZ7-4iCUNIurzYFgjlE3).

[[471]](#ftnt_ref471) Askell, Amanda, Miles Brundage, and Gillian Hadfield. ‘The Role of Cooperation in Responsible AI Development’. arXiv, 10 July 2019.[http://arxiv.org/abs/1907.04534](https://www.google.com/url?q=http://arxiv.org/abs/1907.04534&sa=D&source=editors&ust=1692801435285827&usg=AOvVaw36Oa0ocs7AV_DaDVgUrG95).

[[472]](#ftnt_ref472) See notably: OpenAI. ‘OpenAI Charter’. OpenAI Blog, 9 April 2018. [https://openai.com/charter](https://www.google.com/url?q=https://openai.com/charter&sa=D&source=editors&ust=1692801435370308&usg=AOvVaw3GFqB1Lmx7cFhS5-77YiUG). (“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”).

[[473]](#ftnt_ref473) O’Keefe, Cullen, Peter Cihon, Ben Garfinkel, Carrick Flynn, Jade Leung, and Allan Dafoe. ‘The Windfall Clause: Distributing the Benefits of AI for the Common Good’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 327–31. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375842](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375842&sa=D&source=editors&ust=1692801435370951&usg=AOvVaw0VY_i5bHSGMlq4HzetTlrd); Bova, Paolo, Jonas Emanuel Müller, and Benjamin Harack. ‘Safe Transformative AI via a Windfall Clause’. ArXiv:2108.09404 [Cs], 28 August 2021.[http://arxiv.org/abs/2108.09404](https://www.google.com/url?q=http://arxiv.org/abs/2108.09404&sa=D&source=editors&ust=1692801435371391&usg=AOvVaw3gRQNz7N2Z1j7lKsMldaT4); for legal analysis, see also: Bridge, John. ‘Towards a Worldwide, Watertight Windfall Clause’. EA Forum, 7 April 2022.[https://forum.effectivealtruism.org/s/68dCXfuvykT3RmYy4](https://www.google.com/url?q=https://forum.effectivealtruism.org/s/68dCXfuvykT3RmYy4&sa=D&source=editors&ust=1692801435371782&usg=AOvVaw0H4ww4tTZkzUPxh5u6D4jI).

[[474]](#ftnt_ref474) See list of policies enumerated in: Hua, Shin-Shin, and Haydn Belfield. ‘AI & Antitrust: Reconciling Tensions Between Competition Law and Cooperative AI Development’. Yale Journal of Law and Technology 23 (Spring 2021): 127. (Appendix A).

[[475]](#ftnt_ref475) Wang, Jasmine. ‘What the AI Community Can Learn From Sneezing Ferrets and a Mutant Virus Debate’. Partnership on AI (blog), 9 December 2020.[https://medium.com/partnership-on-ai/lessons-for-the-ai-community-from-the-h5n1-controversy-32432438a82e](https://www.google.com/url?q=https://medium.com/partnership-on-ai/lessons-for-the-ai-community-from-the-h5n1-controversy-32432438a82e&sa=D&source=editors&ust=1692801435286834&usg=AOvVaw39AQ3RmxnzfWuY4lmaPxLt).

[[476]](#ftnt_ref476) Cihon, Peter, Moritz J. Kleinaltenkamp, Jonas Schuett, and Seth D. Baum. ‘AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries’. IEEE Transactions on Technology and Society 2, no. 4 (2021): 200–209.[https://doi.org/10.1109/TTS.2021.3077595](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3077595&sa=D&source=editors&ust=1692801435287575&usg=AOvVaw3jehWG8J874Z2lOjtHFGAG).; Winter, Philip Matthias, Sebastian Eder, Johannes Weissenböck, Christoph Schwald, Thomas Doms, Tom Vogt, Sepp Hochreiter, and Bernhard Nessler. ‘Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications’. ArXiv:2103.16910 [Cs, Stat], 31 March 2021.[http://arxiv.org/abs/2103.16910](https://www.google.com/url?q=http://arxiv.org/abs/2103.16910&sa=D&source=editors&ust=1692801435288130&usg=AOvVaw39PiE9gt4JuYRjUoMqaoVL).

[[477]](#ftnt_ref477) Avin, Shahar, Haydn Belfield, Miles Brundage, Gretchen Krueger, Jasmine Wang, Adrian Weller, Markus Anderljung, et al. ‘Filling Gaps in Trustworthy Development of AI’. Science (New York, N.Y.) 374, no. 6573 (10 December 2021): 1327–29.[https://doi.org/10.1126/science.abi7176](https://www.google.com/url?q=https://doi.org/10.1126/science.abi7176&sa=D&source=editors&ust=1692801435288907&usg=AOvVaw1mXZNKzdAsM16Ws82RPAYV).; Mökander, Jakob, Maria Axente, Federico Casolari, and Luciano Floridi. ‘Conformity Assessments and Post-Market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation’. Minds and Machines 32, no. 2 (1 June 2022): 241–68.[https://doi.org/10.1007/s11023-021-09577-4](https://www.google.com/url?q=https://doi.org/10.1007/s11023-021-09577-4&sa=D&source=editors&ust=1692801435289475&usg=AOvVaw3Gaat3NVEoXztucxCSEiKh).; Raji, Inioluwa Deborah. ‘From Algorithmic Audits to Actual Accountability: Overcoming Practical Roadblocks on the Path to Meaningful Audit Interventions for AI Governance’. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 5. AIES ’22. New York, NY, USA: Association for Computing Machinery, 2022.[https://doi.org/10.1145/3514094.3539566](https://www.google.com/url?q=https://doi.org/10.1145/3514094.3539566&sa=D&source=editors&ust=1692801435289970&usg=AOvVaw1Mn7DETPPsLtVH06gT2m-v).

[[478]](#ftnt_ref478) See Blackman, Reid. ‘If Your Company Uses AI, It Needs an Institutional Review Board’. Harvard Business Review, 1 April 2021.[https://hbr.org/2021/04/if-your-company-uses-ai-it-needs-an-institutional-review-board](https://www.google.com/url?q=https://hbr.org/2021/04/if-your-company-uses-ai-it-needs-an-institutional-review-board&sa=D&source=editors&ust=1692801435290554&usg=AOvVaw2E3lKY4lkLXIyKvoF7Cy0p).; on the history of IRBs generally, see: Stark, Laura. Behind Closed Doors: IRBs and the Making of Ethical Research. Morality and Society Series. Chicago, IL: University of Chicago Press, 2012.[https://press.uchicago.edu/ucp/books/book/chicago/B/bo12182576.html](https://www.google.com/url?q=https://press.uchicago.edu/ucp/books/book/chicago/B/bo12182576.html&sa=D&source=editors&ust=1692801435291010&usg=AOvVaw1R-NF1da3CdEQnflTcLBbf).

[[479]](#ftnt_ref479) Prunkl, Carina E. A., Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe. ‘Institutionalizing Ethics in AI through Broader Impact Requirements’. Nature Machine Intelligence 3, no. 2 (February 2021): 104–10.[https://doi.org/10.1038/s42256-021-00298-y](https://www.google.com/url?q=https://doi.org/10.1038/s42256-021-00298-y&sa=D&source=editors&ust=1692801435291578&usg=AOvVaw1O_eNgR5xyG3hzkqqGH-lW).

[[480]](#ftnt_ref480) CIFAR. ‘A Culture of Ethical AI: Report’. CIFAR, Partnership on AI, July 2022.[https://partnershiponai.org//wp-content/uploads/dlm\_uploads/2022/08/CIFAR-AI-Insights-EN-AM-220803-1.pdf](https://www.google.com/url?q=https://partnershiponai.org//wp-content/uploads/dlm_uploads/2022/08/CIFAR-AI-Insights-EN-AM-220803-1.pdf&sa=D&source=editors&ust=1692801435292137&usg=AOvVaw1mRltHOGFr1Qi49x_EAbA2).

[[481]](#ftnt_ref481) Solaiman, Irene, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, et al. ‘Release Strategies and the Social Impacts of Language Models’. ArXiv:1908.09203 [Cs], 12 November 2019.[http://arxiv.org/abs/1908.09203](https://www.google.com/url?q=http://arxiv.org/abs/1908.09203&sa=D&source=editors&ust=1692801435292653&usg=AOvVaw0aAbAZSOvWUAnWiY-LPnP9).; Ovadya, Aviv, and Jess Whittlestone. ‘Reducing Malicious Use of Synthetic Media Research: Considerations and Potential Release Practices for Machine Learning’. arXiv, 28 July 2019.[https://doi.org/10.48550/arXiv.1907.11274](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.1907.11274&sa=D&source=editors&ust=1692801435293053&usg=AOvVaw1Rx3xnUIXbhnxbntTcGgtQ). See also Partnership on AI. ‘Managing the Risks of AI Research: Six Recommendations for Responsible Publication’. Accessed 14 October 2022.[https://partnershiponai.org/paper/responsible-publication-recommendations/](https://www.google.com/url?q=https://partnershiponai.org/paper/responsible-publication-recommendations/&sa=D&source=editors&ust=1692801435293483&usg=AOvVaw3hIRicXFIiY0ceUdk1c7jz).; Shevlane, Toby. ‘The Artefacts of Intelligence: Governing Scientists’ Contribution to AI Proliferation’. University of Oxford, 2022.[https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation](https://www.google.com/url?q=https://www.governance.ai/research-paper/the-artefacts-of-intelligence-governing-scientists-contribution-to-ai-proliferation&sa=D&source=editors&ust=1692801435294049&usg=AOvVaw0T9mkr1XRCAIc19Lx1IF8R). And for a recent review: Wasil, Akash R, Charlotte Siegmann, Carson Ezell, and Aris Richardson. ‘Publication Policies and Model-Sharing Decisions: A Literature Review and Recommendations for AI Labs’, 2023.[https://static1.squarespace.com/static/6276a63ecf564172c125f58e/t/641cbc1d84814a4d0f3e1788/1679604766050/WasilEzellRichardsonSiegmann+%2810%29.pdf](https://www.google.com/url?q=https://static1.squarespace.com/static/6276a63ecf564172c125f58e/t/641cbc1d84814a4d0f3e1788/1679604766050/WasilEzellRichardsonSiegmann%2B%252810%2529.pdf&sa=D&source=editors&ust=1692801435294616&usg=AOvVaw068skPKl7AEgnFBkeMCbCN).

[[482]](#ftnt_ref482) Duan, Isabella. ‘Race to the Top: Rethink Benchmark-Making for Safe AI Development’, 3 December 2022.[https://isaduan.github.io/isabelladuan.github.io/posts/first/](https://www.google.com/url?q=https://isaduan.github.io/isabelladuan.github.io/posts/first/&sa=D&source=editors&ust=1692801435295140&usg=AOvVaw3YMC2WANPphstuIesBXr6a).; on the role of benchmarks in steering AI development, see also: Dehghani, Mostafa, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. ‘The Benchmark Lottery’. arXiv, 14 July 2021.[https://doi.org/10.48550/arXiv.2107.07002](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2107.07002&sa=D&source=editors&ust=1692801435295502&usg=AOvVaw2vJRjwCmFJVJGAilyfvHB9).

[[483]](#ftnt_ref483) Sandbrink, Jonas, Hamish Hobbs, Jacob Swett, Allan Dafoe, and Anders Sandberg. ‘Differential Technology Development: A Responsible Innovation Principle for Navigating Technology Risks’. SSRN Scholarly Paper. Rochester, NY, 8 September 2022.[https://papers.ssrn.com/abstract=4213670](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4213670&sa=D&source=editors&ust=1692801435295988&usg=AOvVaw1orI2q5IjhRAsN314mmyk3). (on the general principle, not specifically focused on AI); However, for a skeptical take on the efficacy of innovation prizes, see: Howes, Anton. ‘Why Innovation Prizes Fail’. Works in Progress (blog), 21 April 2022.[https://www.worksinprogress.co/issue/why-innovation-prizes-fail/](https://www.google.com/url?q=https://www.worksinprogress.co/issue/why-innovation-prizes-fail/&sa=D&source=editors&ust=1692801435296433&usg=AOvVaw1a38JW_vMdNjvMzRIudGFh).

[[484]](#ftnt_ref484) Vöneky, Silja. ‘Human Rights and Legitimate Governance of Existential and Global Catastrophic Risks’. In Human Rights, Democracy, and Legitimacy in a World of Disorder, edited by Silja Vöneky and Gerald Neuman, 139–62. Cambridge University Press, 2018.[https://papers.ssrn.com/abstract=3363552](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3363552&sa=D&source=editors&ust=1692801435297006&usg=AOvVaw2h0r0cKPvSqwqQOsWuXnnh). (discussing scientific moratoria generally);

[[485]](#ftnt_ref485) Heim, Lennart. ‘Transformative AI and Compute’. AI Alignment Forum, 2021.[https://www.alignmentforum.org/s/bJi3hd8E8qjBeHz9Z](https://www.google.com/url?q=https://www.alignmentforum.org/s/bJi3hd8E8qjBeHz9Z&sa=D&source=editors&ust=1692801435297487&usg=AOvVaw2i3jsF54FgLcp7EDjtyKBH).

[[486]](#ftnt_ref486) Barbe, Andre, and Will Hunt. ‘Preserving the Chokepoints: Reducing the Risks of Offshoring Among U.S. Semiconductor Manufacturing Equipment Firms’. Center for Security and Emerging Technology, May 2022.[https://cset.georgetown.edu/publication/preserving-the-chokepoints/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/preserving-the-chokepoints/&sa=D&source=editors&ust=1692801435298041&usg=AOvVaw28Lf001T4vTZ2lQL2hAuuW).

[[487]](#ftnt_ref487) See also: Scherer, Matthew U. ‘Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies’. Harvard Journal of Law & Technology, no. 2 (Spring 2016).[http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf](https://www.google.com/url?q=http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf&sa=D&source=editors&ust=1692801435177537&usg=AOvVaw2LOs2ZdvGr5NQ-SfYpxOsT). (reviewing legislatures, expert agencies & common law tort system); see also Winter, Christoph, Jonas Schuett, Eric Martínez, Suzanne Van Arsdale, Renan Araújo, Nick Hollman, Jeff Sebo, Andrew Stawasz, Cullen O’Keefe, and Giuliana Rotola. ‘Legal Priorities Research: A Research Agenda’. Legal Priorities Project, January 2021.[https://www.legalpriorities.org/research\_agenda.pdf](https://www.google.com/url?q=https://www.legalpriorities.org/research_agenda.pdf&sa=D&source=editors&ust=1692801435177915&usg=AOvVaw3XvcB88EgcH8LM7tD9G8xy). See also: AI Impacts. ‘Affordances for States’. AI Impacts Wiki, 25 January 2023.[https://wiki.aiimpacts.org/doku.php?id=responses\_to\_ai:affordances:state\_affordances](https://www.google.com/url?q=https://wiki.aiimpacts.org/doku.php?id%3Dresponses_to_ai:affordances:state_affordances&sa=D&source=editors&ust=1692801435178322&usg=AOvVaw2SoNLSpqUxUS-0VpcFepUD). And broadly: Karnofsky, Holden. ‘How Major Governments Can Help with the Most Important Century’, 24 February 2023.[https://forum.effectivealtruism.org/posts/ruJnXtdDS7XiiwzSP/how-major-governments-can-help-with-the-most-important](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ruJnXtdDS7XiiwzSP/how-major-governments-can-help-with-the-most-important&sa=D&source=editors&ust=1692801435178779&usg=AOvVaw1qyPZKibiSfoF9YSSmjfUL).

[[488]](#ftnt_ref488) There are various collections that discuss the regulation of AI on the basis of extant bodies of law, though these focus primarily on the regulation of algorithms that exist today, rather than more capable or transformative AI systems. Barfield, Woodrow, and Ugo Pagallo, eds. Research Handbook on the Law of Artificial Intelligence. Cheltenham, UK: Edward Elgar Publishing, 2018.[https://www.elgaronline.com/view/edcoll/9781786439048/9781786439048.xml](https://www.google.com/url?q=https://www.elgaronline.com/view/edcoll/9781786439048/9781786439048.xml&sa=D&source=editors&ust=1692801435179367&usg=AOvVaw13W2NoWhrrnLf2cbZ7Bbgp).; Wischmeyer, Thomas, and Timo Rademacher, eds. Regulating Artificial Intelligence. Springer International Publishing, 2020.[https://doi.org/10.1007/978-3-030-32361-5](https://www.google.com/url?q=https://doi.org/10.1007/978-3-030-32361-5&sa=D&source=editors&ust=1692801435180528&usg=AOvVaw3YyJSQHWrE1WH7Xvxfzj-j).; DiMatteo, Larry A., Cristina Poncibò, and Michel Cannarsa, eds. The Cambridge Handbook of Artificial Intelligence: Global Perspectives on Law and Ethics. Cambridge Law Handbooks. Cambridge: Cambridge University Press, 2022.[https://doi.org/10.1017/9781009072168](https://www.google.com/url?q=https://doi.org/10.1017/9781009072168&sa=D&source=editors&ust=1692801435180937&usg=AOvVaw2S0mUNyiqGGPwo7D3Hgt0t).

[[489]](#ftnt_ref489) Petit, Nicolas, and Jerome De Cooman. ‘Models of Law and Regulation for AI’. EUI Working Paper RSCAS 2020/63. Social Science Research Network, 1 October 2020.[https://doi.org/10.2139/ssrn.3706771](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.3706771&sa=D&source=editors&ust=1692801435398258&usg=AOvVaw3TdzzC7lVShXNqvy1TWBGB).; see also Maas, Matthijs M. ‘Aligning AI Regulation to Sociotechnical Change’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.22](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.22&sa=D&source=editors&ust=1692801435398855&usg=AOvVaw13wMv1ub998GghCNeqYbDH).

[[490]](#ftnt_ref490) Higgins, Brian Wm. ‘Legal Elements of an AI Regulatory Permit Program’. In The Oxford Handbook of AI Governance, by Brian Wm. Higgins, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.18](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.18&sa=D&source=editors&ust=1692801435399527&usg=AOvVaw0rHoywPyAxC3lIiibYBV9H).

[[491]](#ftnt_ref491) See also the later section on ‘[existing and applicable law](#h.xtzugu6j7dxo)’. For a distinction of regulatory responses between ‘drawing analogies’, ‘extending existing law’, ‘creating new law’, and ‘reassessing the regulatory regime’, see: Crootof, Rebecca, and B. J. Ard. ‘Structuring Techlaw’. Harvard Journal of Law & Technology 34, no. 2 (2021): 347–417. [https://jolt.law.harvard.edu/assets/articlePDFs/v34/1.-Crootof-Ard-Structuring-Techlaw.pdf](https://www.google.com/url?q=https://jolt.law.harvard.edu/assets/articlePDFs/v34/1.-Crootof-Ard-Structuring-Techlaw.pdf&sa=D&source=editors&ust=1692801435181471&usg=AOvVaw0Asn4x5Gpo5H22L2jFjg_j)

[[492]](#ftnt_ref492) Hua, Shin-Shin, and Haydn Belfield. ‘AI & Antitrust: Reconciling Tensions Between Competition Law and Cooperative AI Development’. Yale Journal of Law and Technology 23 (Spring 2021): 127. [https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development](https://www.google.com/url?q=https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development&sa=D&source=editors&ust=1692801435181942&usg=AOvVaw2QVfCa5T6PSAr1LuKrVaC6)

[[493]](#ftnt_ref493) White, Trevor N., and Seth D. Baum. ‘Liability For Present And Future Robotics Technology’. Robot Ethics 2.0: From Autonomous Cars to Artificial Intelligence, 2017, 5.; Erdélyi, Olivia J., and Gábor Erdélyi. ‘The AI Liability Puzzle and a Fund-Based Work-Around’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 50–56. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375806](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375806&sa=D&source=editors&ust=1692801435182580&usg=AOvVaw0-VK0OXcXuyfrCdCvMbfGB).

[[494]](#ftnt_ref494) Lior, Anat. ‘Insuring AI: The Role of Insurance in Artificial Intelligence Regulation’. Harvard Journal of Law & Technology 35, no. 2 (2022): 64. [https://jolt.law.harvard.edu/assets/articlePDFs/v35/2.-Lior-Insuring-AI.pdf](https://www.google.com/url?q=https://jolt.law.harvard.edu/assets/articlePDFs/v35/2.-Lior-Insuring-AI.pdf&sa=D&source=editors&ust=1692801435183012&usg=AOvVaw1UK60Yksi6rxDFpo6cIagv)

[[495]](#ftnt_ref495) See also: Linarelli, John. ‘Artificial General Intelligence and Contract’. Uniform Law Review 24, no. 2 (1 June 2019): 330–47.[https://doi.org/10.1093/ulr/unz015](https://www.google.com/url?q=https://doi.org/10.1093/ulr/unz015&sa=D&source=editors&ust=1692801435183658&usg=AOvVaw2YbLFM19X7BF8SS-2vYPJI).

[[496]](#ftnt_ref496) Calvin, Nathan, and Jade Leung. ‘Who Owns Artificial Intelligence? A Preliminary Analysis of Corporate Intellectual Property Strategies and Why They Matter’. Oxford: Center for the Governance of AI, Future of Humanity Institute, February 2020.[https://www.fhi.ox.ac.uk/wp-content/uploads/Patents\_-FHI-Working-Paper-Final-.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Patents_-FHI-Working-Paper-Final-.pdf&sa=D&source=editors&ust=1692801435184287&usg=AOvVaw36g58fi5m7-FkjgVQcPWQ7).; and previously Koepsell, David. ‘Can the Singularity Be Patented? (And Other IP Conundrums for Converging Technologies)’. In The Technological Singularity: Managing the Journey, edited by Victor Callaghan, James Miller, Roman Yampolskiy, and Stuart Armstrong, 181–91. The Frontiers Collection. Berlin, Heidelberg: Springer, 2017.[https://doi.org/10.1007/978-3-662-54033-6\_10](https://www.google.com/url?q=https://doi.org/10.1007/978-3-662-54033-6_10&sa=D&source=editors&ust=1692801435184801&usg=AOvVaw1DjkNswt2sg7J5YU9dkK88).

[[497]](#ftnt_ref497) See also Vincent, James. ‘The Lawsuit That Could Rewrite the Rules of AI Copyright’. The Verge, 8 November 2022.[https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data](https://www.google.com/url?q=https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data&sa=D&source=editors&ust=1692801435185384&usg=AOvVaw3_IL1gPcwg3r_55bszgjzx).

[[498]](#ftnt_ref498) See generally King, Thomas C., Nikita Aggarwal, Mariarosaria Taddeo, and Luciano Floridi. ‘Artificial Intelligence Crime: An Interdisciplinary Analysis of Foreseeable Threats and Solutions’. Science and Engineering Ethics, 14 February 2019.[https://doi.org/10.1007/s11948-018-00081-0](https://www.google.com/url?q=https://doi.org/10.1007/s11948-018-00081-0&sa=D&source=editors&ust=1692801435185898&usg=AOvVaw2cjN-ui-ScGbBohP0I_xK6).

[[499]](#ftnt_ref499) Belfield, Haydn, Amritha Jayanti, and Shahar Avin. ‘Written Evidence - Defence Industrial Policy: Procurement and Prosperity’, 2020.[https://committees.parliament.uk/writtenevidence/4785/default/](https://www.google.com/url?q=https://committees.parliament.uk/writtenevidence/4785/default/&sa=D&source=editors&ust=1692801435186470&usg=AOvVaw2aUHMbzZbElVR-NPnEtlhJ).; See generally: Dor, Lavi M. Ben, and Cary Coglianese. ‘Procurement as AI Governance’. IEEE Transactions on Technology and Society, 2021, 1–1.[https://doi.org/10.1109/TTS.2021.3111764](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3111764&sa=D&source=editors&ust=1692801435186910&usg=AOvVaw1MUCopbRXTjfBsjK4MYgJ5).

[[500]](#ftnt_ref500) Dawson, Gregory, Kevin Desouza, and James Denford. ‘Understanding Artificial Intelligence Spending by the U.S. Federal Government’. Brookings (blog), 22 September 2022.[https://www.brookings.edu/blog/techtank/2022/09/22/understanding-artificial-intelligence-spending-by-the-u-s-federal-government/](https://www.google.com/url?q=https://www.brookings.edu/blog/techtank/2022/09/22/understanding-artificial-intelligence-spending-by-the-u-s-federal-government/&sa=D&source=editors&ust=1692801435187558&usg=AOvVaw3b1SPwgT1hl8FDLWqsMDyN).

[[501]](#ftnt_ref501) Sandbrink, Jonas, Hamish Hobbs, Jacob Swett, Allan Dafoe, and Anders Sandberg. ‘Differential Technology Development: A Responsible Innovation Principle for Navigating Technology Risks’. SSRN Scholarly Paper. Rochester, NY, 8 September 2022.[https://papers.ssrn.com/abstract=4213670](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4213670&sa=D&source=editors&ust=1692801435188082&usg=AOvVaw2sLZ56hMOc3_gkRh3X8nW_).

[[502]](#ftnt_ref502) Ibid.

[[503]](#ftnt_ref503) See also the section on ‘[existing and applicable law](#h.xtzugu6j7dxo)’.

[[504]](#ftnt_ref504) Perault, Matt. ‘Section 230 Won’t Protect ChatGPT’. Lawfare, 23 February 2023.[https://www.lawfareblog.com/section-230-wont-protect-chatgpt](https://www.google.com/url?q=https://www.lawfareblog.com/section-230-wont-protect-chatgpt&sa=D&source=editors&ust=1692801435400391&usg=AOvVaw1Lk2ce7SR2BmY8f3S_iJ21).; Robertson, Adi. ‘The Supreme Court Could Be About to Decide the Legal Fate of AI Search’. The Verge, 16 February 2023.[https://www.theverge.com/2023/2/16/23591290/supreme-court-section-230-gonzalez-google-bard-bing-ai-search-algorithms](https://www.google.com/url?q=https://www.theverge.com/2023/2/16/23591290/supreme-court-section-230-gonzalez-google-bard-bing-ai-search-algorithms&sa=D&source=editors&ust=1692801435400984&usg=AOvVaw0Ylit6H-29xOV8SLoc8EqC). And see generally: Kosseff, Jeff. ‘A User’s Guide to Section 230, and a Legislator’s Guide to Amending It (or Not)’. Berkeley Technology Law Journal 37, no. 2 (2022).[https://papers.ssrn.com/abstract=3905347](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3905347&sa=D&source=editors&ust=1692801435401481&usg=AOvVaw18828Svq3m_2KIsVyKNnLK).

[[505]](#ftnt_ref505) There has been little direct work on applying this to AI; however, for discussions of this in the context of COVID responses, see: Ginsburg, Tom, and Mila Versteeg. ‘The Bound Executive: Emergency Powers during the Pandemic’. International Journal of Constitutional Law 19, no. 5 (1 December 2021): 1498–1535.[https://doi.org/10.1093/icon/moab059](https://www.google.com/url?q=https://doi.org/10.1093/icon/moab059&sa=D&source=editors&ust=1692801435188748&usg=AOvVaw3XbDFZYFL6Iv1hl2Zj9iDe). I thank Christoph Winter for this suggestion.

[[506]](#ftnt_ref506) For the use of courts in other domains, see: Martinsen, Dorte Sindbjerg. ‘Judicial Policy-Making and Europeanization: The Proportionality of National Control and Administrative Discretion’. Journal of European Public Policy 18, no. 7 (1 October 2011): 944–61.[https://doi.org/10.1080/13501763.2011.599962](https://www.google.com/url?q=https://doi.org/10.1080/13501763.2011.599962&sa=D&source=editors&ust=1692801435189286&usg=AOvVaw1mcsZNi7LipkFhiXt7ocXQ).; I thank Christoph Winter for this suggestion.

[[507]](#ftnt_ref507) Scherer, Matthew U. ‘Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies’. Harvard Journal of Law & Technology, no. 2 (Spring 2016).[http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf](https://www.google.com/url?q=http://jolt.law.harvard.edu/articles/pdf/v29/29HarvJLTech353.pdf&sa=D&source=editors&ust=1692801435402630&usg=AOvVaw14rfDm-r8vIHLzw0mOlekm). Pg. 382.

[[508]](#ftnt_ref508) Villasenor, John. ‘Soft Law as a Complement to AI Regulation’. Brookings (blog), 31 July 2020.[https://www.brookings.edu/research/soft-law-as-a-complement-to-ai-regulation/](https://www.google.com/url?q=https://www.brookings.edu/research/soft-law-as-a-complement-to-ai-regulation/&sa=D&source=editors&ust=1692801435189920&usg=AOvVaw0pIG2dYhQRYWDxWiGJjM7H). See also Marchant, Gary E., and Carlos Ignacio Gutierrez. ‘Indirect Enforcement of Artificial Intelligence “Soft Law”’. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, 15 December 2020.[https://doi.org/10.2139/ssrn.3749776](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.3749776&sa=D&source=editors&ust=1692801435190341&usg=AOvVaw3pOCVrBkMUU7L_nOdBecHE).; Gutierrez, Carlos Ignacio, Gary E. Marchant, and Katina Michael. ‘Effective and Trustworthy Implementation of AI Soft Law Governance’. IEEE Transactions on Technology and Society 2, no. 4 (December 2021): 168–70.[https://doi.org/10.1109/TTS.2021.3121959](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3121959&sa=D&source=editors&ust=1692801435190780&usg=AOvVaw34GbFhbGWzodUg9-AQYNaF).

[[509]](#ftnt_ref509) Whittlestone, Jess, and Jack Clark. ‘Why and How Governments Should Monitor AI Development’. ArXiv:2108.12427 [Cs], 31 August 2021.[http://arxiv.org/abs/2108.12427](https://www.google.com/url?q=http://arxiv.org/abs/2108.12427&sa=D&source=editors&ust=1692801435191356&usg=AOvVaw3G7X5hBwIwwDVWbb6DYmFc).; also Clark, Jack. ‘Technical Observatories for Better AI Governance’. In The Oxford Handbook of AI Governance, edited by Valerie Hudson and Justin Bullock. Oxford Univ. Press, [forthcoming].

[[510]](#ftnt_ref510) Hudson, Valerie M. ‘Standing Up a Regulatory Ecosystem for Governing AI Decision-Making: Principles and Components’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press. Accessed 21 October 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.17](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.17&sa=D&source=editors&ust=1692801435192071&usg=AOvVaw1yGmaCeaN1_XEgwwOsiAUv).

[[511]](#ftnt_ref511) See generally Morse, Julia C., and Robert O. Keohane. ‘Contested Multilateralism’. The Review of International Organizations 9, no. 4 (1 December 2014): 385–412.[https://doi.org/10.1007/s11558-014-9188-2](https://www.google.com/url?q=https://doi.org/10.1007/s11558-014-9188-2&sa=D&source=editors&ust=1692801435372880&usg=AOvVaw2-Me_E7FM85HzbNJbqjVi3).

[[512]](#ftnt_ref512) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Should Artificial Intelligence Governance Be Centralised?: Design Lessons from History’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 228–34. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375857](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375857&sa=D&source=editors&ust=1692801435372323&usg=AOvVaw0TwTj-CpyqbfDqNSnBC_dF).; and see generally: Fehl, Caroline. ‘Forum Shopping from above and below: Power Shifts and Institutional Choice in a Stratified International Society’, 36. Munich, 2016.

[[513]](#ftnt_ref513) Deeks, Ashley. ‘High-Tech International Law’. George Washington Law Review 88 (2020): 575–653. See pg. 590-591. [https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3531976](https://www.google.com/url?q=https://papers.ssrn.com/sol3/papers.cfm?abstract_id%3D3531976&sa=D&source=editors&ust=1692801435401956&usg=AOvVaw3oznvK4xYFYVUQcZu2MiUh)

[[514]](#ftnt_ref514) For in-depth discussion of the role of CIL, see: Hakimi, Monica. ‘Making Sense of Customary International Law’. Michigan Law Review 118 (16 June 2020).[https://papers.ssrn.com/abstract=3627905](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3627905&sa=D&source=editors&ust=1692801435411385&usg=AOvVaw3oVdG-cWG3_21zhoso3ysW). ; for an analysis of the merits and roles of CIL, see: Helfer, Laurence R, and Ingrid B Wuerth. ‘Customary International Law: An Instrument Choice Perspective’. Michigan Journal of International Law 37 (2016): 563. [https://repository.law.umich.edu/mjil/vol37/iss4/1/](https://www.google.com/url?q=https://repository.law.umich.edu/mjil/vol37/iss4/1/&sa=D&source=editors&ust=1692801435411669&usg=AOvVaw2Cnq23CM79NkAlJ3hLXibI) ; Crootof has argued that changing state practice may even modify established treaty law; Crootof, Rebecca. ‘Change Without Consent: How Customary International Law Modifies Treaties’. Yale Journal of International Law 41, no. 2 (2016): 65. [https://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article=1670&context=yjil](https://www.google.com/url?q=https://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article%3D1670%26context%3Dyjil&sa=D&source=editors&ust=1692801435411967&usg=AOvVaw0n8Y_X-Xip-u9h_zFvtzp2)

[[515]](#ftnt_ref515) The White House. ‘Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People’, October 2022, 73. [https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf](https://www.google.com/url?q=https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf&sa=D&source=editors&ust=1692801435352139&usg=AOvVaw16g4aG3EYl1GhdPidG0rTD)

[[516]](#ftnt_ref516) Fischer, Sophie-Charlotte, Jade Leung, Markus Anderljung, Cullen O’Keefe, Stefan Torges, Saif M. Khan, Ben Garfinkel, and Allan Dafoe. ‘AI Policy Levers: A Review of the U.S. Government’s Tools to Shape AI Research, Development, and Deployment’. Centre for the Governance of AI, Future of Humanity Institute, University of Oxford, March 2021.[https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/AI-Policy-Levers-A-Review-of-the-U.S.-Governments-tools-to-shape-AI-research-development-and-deployment-%E2%80%93-Fischer-et-al.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/AI-Policy-Levers-A-Review-of-the-U.S.-Governments-tools-to-shape-AI-research-development-and-deployment-%25E2%2580%2593-Fischer-et-al.pdf&sa=D&source=editors&ust=1692801435192889&usg=AOvVaw3S2UaA44Mzz6YRbEF0ytsj).

[[517]](#ftnt_ref517) Flynn, Carrick. ‘Recommendations on Export Controls for Artificial Intelligence’. Center for Security and Emerging Technology, February 2020.[https://cset.georgetown.edu/research/recommendations-on-export-controls-for-artificial-intelligence/](https://www.google.com/url?q=https://cset.georgetown.edu/research/recommendations-on-export-controls-for-artificial-intelligence/&sa=D&source=editors&ust=1692801435193531&usg=AOvVaw3PmQ3nvydl-_f0KOIw-UAL). Leung, Jade, Sophie-Charlotte Fischer, and Allan Dafoe. ‘Export Controls in the Age of AI’. War on the Rocks, 28 August 2019.[https://warontherocks.com/2019/08/export-controls-in-the-age-of-ai/](https://www.google.com/url?q=https://warontherocks.com/2019/08/export-controls-in-the-age-of-ai/&sa=D&source=editors&ust=1692801435193940&usg=AOvVaw104M6RRW9lHC_pZ6kIYW4z). For more recent work, see: Brockmann, Kolja. ‘Applying Export Controls to AI: Current Coverage and Potential Future Controls’. In Armament, Arms Control and Artificial Intelligence: The Janus-Faced Nature of Machine Learning in the Military Realm, edited by Thomas Reinhold and Niklas Schörnig, 193–209. Studies in Peace and Security. Cham: Springer International Publishing, 2022.[https://doi.org/10.1007/978-3-031-11043-6\_14](https://www.google.com/url?q=https://doi.org/10.1007/978-3-031-11043-6_14&sa=D&source=editors&ust=1692801435194444&usg=AOvVaw0vR7fdz_zqZAHNUoVcsLKB).

[[518]](#ftnt_ref518) See generally Baker, James E. ‘A DPA for the 21st Century’. Center for Security and Emerging Technology, April 2021.[https://cset.georgetown.edu/publication/a-dpa-for-the-21st-century/](https://www.google.com/url?q=https://cset.georgetown.edu/publication/a-dpa-for-the-21st-century/&sa=D&source=editors&ust=1692801435195021&usg=AOvVaw2Wt0a-q-dJnSqZx_iRU2WV).

[[519]](#ftnt_ref519) NIST. ‘AI Risk Management Framework: AI RMF (1.0)’. Gaithersburg, MD: National Institute of Standards and Technology, 2023.[https://doi.org/10.6028/NIST.AI.100-1](https://www.google.com/url?q=https://doi.org/10.6028/NIST.AI.100-1&sa=D&source=editors&ust=1692801435195526&usg=AOvVaw1rUUGrMJZ-dOOkgFce8VNb); See also: Barrett, Anthony M., Dan Hendrycks, Jessica Newman, and Brandie Nonnecke. ‘Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks’. arXiv, 17 June 2022.[https://doi.org/10.48550/arXiv.2206.08966](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2206.08966&sa=D&source=editors&ust=1692801435195906&usg=AOvVaw1ZbQwYNg99L7q5OogwDrhB).

[[520]](#ftnt_ref520) Belfield, Haydn. ‘Competition Law Is One of the Most Important Levers for AI Governance’. EAG: SF 22, 24 August 2022.[https://www.youtube.com/watch?v=2-GbRe6oewE](https://www.google.com/url?q=https://www.youtube.com/watch?v%3D2-GbRe6oewE&sa=D&source=editors&ust=1692801435196380&usg=AOvVaw3iXdyE4ZK2db0Ug9WzmOpF).

[[521]](#ftnt_ref521) Casper, Stephen, Phillip Christoffersen, and Rui-Jie Yew. ‘The Slippery Slope from DALLE-2 to Deepfake Anarchy’. Effective Altruism Forum, 5 November 2022.[https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Bnp9YDqErNXHmTvvE/the-slippery-slope-from-dalle-2-to-deepfake-anarchy&sa=D&source=editors&ust=1692801435196958&usg=AOvVaw2wZ2dOO9hGN6SHjQSar5qA).

[[522]](#ftnt_ref522) Stix, Charlotte. ‘The Ghost of AI Governance Past, Present, and Future: AI Governance in the European Union’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang, 0. Oxford University Press. Accessed 21 October 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.56](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.56&sa=D&source=editors&ust=1692801435197539&usg=AOvVaw0nDtsl00Eq3gx8X-QWNCSr).

[[523]](#ftnt_ref523) Siegmann, Charlotte, and Markus Anderljung. ‘The Brussels Effect and Artificial Intelligence’: Centre for the Governance of AI, August 2022.[https://www.governance.ai/research-paper/brussels-effect-ai](https://www.google.com/url?q=https://www.governance.ai/research-paper/brussels-effect-ai&sa=D&source=editors&ust=1692801435198106&usg=AOvVaw22pAROe8I6lYVXzbR4UERv). See also broadly: Dempsey, Mark, Keegan McBride, Meeri Haataja, and Joanna J. Bryson. ‘Transnational Digital Governance and Its Impact on Artificial Intelligence’. In The Oxford Handbook of AI Governance, by Mark Dempsey, Keegan McBride, Meeri Haataja, and Joanna J. Bryson, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.16](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.16&sa=D&source=editors&ust=1692801435198645&usg=AOvVaw25G2f04OSZ9u-RlFWZRcHV).

[[524]](#ftnt_ref524) O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801435199332&usg=AOvVaw1bQtnjaWocTWu1fiOAYz4o).

[[525]](#ftnt_ref525) European Commission. ‘Proposal for a Directive of the European Parliament and of the Council on Adapting Non-Contractual Civil Liability Rules to Artificial Intelligence (AI Liability Directive)’. European Commission, 28 September 2022.[https://ec.europa.eu/info/sites/default/files/1\_1\_197605\_prop\_dir\_ai\_en.pdf](https://www.google.com/url?q=https://ec.europa.eu/info/sites/default/files/1_1_197605_prop_dir_ai_en.pdf&sa=D&source=editors&ust=1692801435199883&usg=AOvVaw0ERfgKQwHYiRmJzzIx7hOt).

[[526]](#ftnt_ref526) Sheehan, Matt. ‘China’s New AI Governance Initiatives Shouldn’t Be Ignored’. Carnegie Endowment for International Peace, 4 January 2022.[https://carnegieendowment.org/2022/01/04/china-s-new-ai-governance-initiatives-shouldn-t-be-ignored-pub-86127](https://www.google.com/url?q=https://carnegieendowment.org/2022/01/04/china-s-new-ai-governance-initiatives-shouldn-t-be-ignored-pub-86127&sa=D&source=editors&ust=1692801435200524&usg=AOvVaw0m9IzhmZLLyZLJvJu9mfPx).

[[527]](#ftnt_ref527) Cheng, Jing, and Jinghan Zeng. ‘Shaping AI’s Future? China in Global AI Governance’. Journal of Contemporary China 0, no. 0 (8 August 2022): 1–17.[https://doi.org/10.1080/10670564.2022.2107391](https://www.google.com/url?q=https://doi.org/10.1080/10670564.2022.2107391&sa=D&source=editors&ust=1692801435201140&usg=AOvVaw28p73stewu8V3_zex7Fgcb).

[[528]](#ftnt_ref528) The below is basedon the review in: Hadshar, Rose. ‘Current UK Government Levers on AI Development’. EA Forum, 10 April 2023.[https://forum.effectivealtruism.org/posts/BFBf5yPLoJMGozygE/current-uk-government-levers-on-ai-development](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/BFBf5yPLoJMGozygE/current-uk-government-levers-on-ai-development&sa=D&source=editors&ust=1692801435439959&usg=AOvVaw3YapNM6sqbhjJvUsl4Oxzo).

[[529]](#ftnt_ref529) For suggestions here, I also thank James Ozden. See also Ozden, James, and Sam Glover. ‘Protest Movements: How Effective Are They?’ Social Change Lab, July 2022.[https://www.socialchangelab.org/\_files/ugd/503ba4\_052959e2ee8d4924934b7efe3916981e.pdf](https://www.google.com/url?q=https://www.socialchangelab.org/_files/ugd/503ba4_052959e2ee8d4924934b7efe3916981e.pdf&sa=D&source=editors&ust=1692801435076194&usg=AOvVaw3fkbEEhKWn8ou2LY3Cq8-W).; see also the taxonomy in: Beer, Michael A. ‘Civil Resistance Tactics in the 21st Century’. International Center on Nonviolent Conflict, 2021.[https://www.vredesmuseum.nl/download/civilresistance.pdf](https://www.google.com/url?q=https://www.vredesmuseum.nl/download/civilresistance.pdf&sa=D&source=editors&ust=1692801435076578&usg=AOvVaw13LxDZ1rMXGtpnpN-jw5lc).

[[530]](#ftnt_ref530) Belfield, Haydn. ‘Activism by the AI Community: Analysing Recent Achievements and Future Prospects’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 15–21. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375814](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375814&sa=D&source=editors&ust=1692801435095421&usg=AOvVaw06Giw63xjcUEiiscgT5fF4).

[[531]](#ftnt_ref531) Baum, Seth D. ‘On the Promotion of Safe and Socially Beneficial Artificial Intelligence’. AI & SOCIETY, 28 September 2016.[https://doi.org/10.1007/s00146-016-0677-0](https://www.google.com/url?q=https://doi.org/10.1007/s00146-016-0677-0&sa=D&source=editors&ust=1692801435098181&usg=AOvVaw1IR5Zu-32SvZh6xCylJ7R0).

[[532]](#ftnt_ref532) See generally; AI Now Institute. ‘Taking Algorithms To Court’. Medium (blog), 24 September 2018.[https://medium.com/@AINowInstitute/taking-algorithms-to-court-7b90f82ffcc9](https://www.google.com/url?q=https://medium.com/@AINowInstitute/taking-algorithms-to-court-7b90f82ffcc9&sa=D&source=editors&ust=1692801435096627&usg=AOvVaw1IKTcwWcfx5S9Z_iL-ZpeV).; for an overview of (US) cases, see Ethical Tech Initiative of DC. ‘AI Litigation Database’. Accessed 20 October 2022.[https://blogs.gwu.edu/law-eti/ai-litigation-database/](https://www.google.com/url?q=https://blogs.gwu.edu/law-eti/ai-litigation-database/&sa=D&source=editors&ust=1692801435097032&usg=AOvVaw2KUf1VI0U2fTyhS9sztnlD).

[[533]](#ftnt_ref533) See e.g. Vincent, James. ‘The Lawsuit That Could Rewrite the Rules of AI Copyright’. The Verge, 8 November 2022.[https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data](https://www.google.com/url?q=https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data&sa=D&source=editors&ust=1692801435129420&usg=AOvVaw0ADCZI5CQ1NNbVRl5zh75P).; Butterick, Matthew. ‘Stable Diffusion Litigation’, 13 January 2023.[https://stablediffusionlitigation.com/](https://www.google.com/url?q=https://stablediffusionlitigation.com/&sa=D&source=editors&ust=1692801435129870&usg=AOvVaw1v2ZUDnYPZ0dv0-yEP04AI).

[[534]](#ftnt_ref534) See generally: Hobson, Tom. ‘Kill the Bill to Save The Future’. Medium (blog), 29 December 2021.[https://medium.com/@t.hobson/kill-the-bill-to-save-the-future-e62689e02328](https://www.google.com/url?q=https://medium.com/@t.hobson/kill-the-bill-to-save-the-future-e62689e02328&sa=D&source=editors&ust=1692801435097589&usg=AOvVaw29uZrQlpe_D3ooThT6I2Dp) (discussing generally the importance of protest to existential risk mitigation).

[[535]](#ftnt_ref535) Cihon, Peter, Moritz J. Kleinaltenkamp, Jonas Schuett, and Seth D. Baum. ‘AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries’. IEEE Transactions on Technology and Society 2, no. 4 (December 2021): 200–209.[https://doi.org/10.1109/TTS.2021.3077595](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3077595&sa=D&source=editors&ust=1692801435096037&usg=AOvVaw3zr3c-6tfdwBPLvTAdBcir).

[[536]](#ftnt_ref536) See for example: Slaughterbots - If Human: Kill(), 2021.[https://www.youtube.com/watch?v=9rDo1QxI260](https://www.google.com/url?q=https://www.youtube.com/watch?v%3D9rDo1QxI260&sa=D&source=editors&ust=1692801435094923&usg=AOvVaw3r8EJBV7JrpUDmloZvIJFK).

[[537]](#ftnt_ref537) See generally: Dothan, Shai. ‘Social Networks and the Enforcement of International Law’. SSRN Scholarly Paper. Rochester, NY, 2 May 2017.[https://doi.org/10.2139/ssrn.2961715](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.2961715&sa=D&source=editors&ust=1692801435360733&usg=AOvVaw3vRMzsjNqskm8WUO0oROPa).

[[538]](#ftnt_ref538) See e.g. The Collective Intelligence Project. ‘Whitepaper’. The Collective Intelligence Project, 2023.[https://cip.org/whitepaper](https://www.google.com/url?q=https://cip.org/whitepaper&sa=D&source=editors&ust=1692801435434344&usg=AOvVaw0nXiwJzIGrhf_hxmSA2NwW).

[[539]](#ftnt_ref539) See generally: Eilstrup-Sangiovanni, Mette, and J. C. Sharman. Vigilantes beyond Borders: NGOs as Enforcers of International Law. Vigilantes beyond Borders. Princeton University Press, 2022.[https://doi.org/10.1515/9780691232249](https://www.google.com/url?q=https://doi.org/10.1515/9780691232249&sa=D&source=editors&ust=1692801435393381&usg=AOvVaw2b6GXmrBSPPv6unoYixCbI).

[[540]](#ftnt_ref540) Cihon, Peter. ‘Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development’. Technical Report. Oxford: Center for the Governance of AI, Future of Humanity Institute, University of Oxford, April 2019.[https://www.fhi.ox.ac.uk/wp-content/uploads/Standards\_-FHI-Technical-Report.pdf](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/Standards_-FHI-Technical-Report.pdf&sa=D&source=editors&ust=1692801434955949&usg=AOvVaw1lyR0nv9tMQ7ojV4RpiOEm).; O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801434956538&usg=AOvVaw3AfhKdfklElvvec7n9Vtty).

[[541]](#ftnt_ref541) I thank José Jaime Villalobos for input and suggestions on this section.

[[542]](#ftnt_ref542) Matz-Lück, Nele. ‘Framework Conventions as a Regulatory Tool’. Goettingen Journal of International Law 3 (2009): 439–58.[https://doi.org/10.3249/1868-1581-1-3-MATZ-LUECK](https://www.google.com/url?q=https://doi.org/10.3249/1868-1581-1-3-MATZ-LUECK&sa=D&source=editors&ust=1692801435373427&usg=AOvVaw3Wdz8YD9N0EwLQnIXGA6cH).

[[543]](#ftnt_ref543) Scharre, Paul, and Megan Lamberth. ‘Artificial Intelligence and Arms Control’. Center for a New American Security, 12 October 2022.[https://www.cnas.org/publications/reports/artificial-intelligence-and-arms-control](https://www.google.com/url?q=https://www.cnas.org/publications/reports/artificial-intelligence-and-arms-control&sa=D&source=editors&ust=1692801435331645&usg=AOvVaw3QycQMR0THN92OS-4cvgIT). Maas, Matthijs M. ‘How Viable Is International Arms Control for Military Artificial Intelligence? Three Lessons from Nuclear Weapons’. Contemporary Security Policy 40, no. 3 (6 February 2019): 285–311.[https://doi.org/10.1080/13523260.2019.1576464](https://www.google.com/url?q=https://doi.org/10.1080/13523260.2019.1576464&sa=D&source=editors&ust=1692801435332061&usg=AOvVaw1Zb-E2TG7J5DYMo_Z5I6U1).

[[544]](#ftnt_ref544) Brockmann, Kolja. ‘Applying Export Controls to AI: Current Coverage and Potential Future Controls’. In Armament, Arms Control and Artificial Intelligence: The Janus-Faced Nature of Machine Learning in the Military Realm, edited by Thomas Reinhold and Niklas Schörnig, 193–209. Studies in Peace and Security. Cham: Springer International Publishing, 2022.[https://doi.org/10.1007/978-3-031-11043-6\_14](https://www.google.com/url?q=https://doi.org/10.1007/978-3-031-11043-6_14&sa=D&source=editors&ust=1692801435332611&usg=AOvVaw1AlE6VIwuouq9SNYlnPyQF).

[[545]](#ftnt_ref545) See generally Howse, Robert, and Ruti Teitel. ‘Beyond Compliance: Rethinking Why International Law Really Matters’. Global Policy 1, no. 2 (2010): 127–36.[https://doi.org/10.1111/j.1758-5899.2010.00035.x](https://www.google.com/url?q=https://doi.org/10.1111/j.1758-5899.2010.00035.x&sa=D&source=editors&ust=1692801435333215&usg=AOvVaw0-ctU9evO9e6yhcEtXuhhu).

[[546]](#ftnt_ref546) Ruhl, Christian. ‘Risks from Autonomous Weapon Systems and Military AI’. Founders Pledge, 19 May 2022.[https://forum.effectivealtruism.org/posts/RKMNZn7r6cT2Yaorf/risks-from-autonomous-weapon-systems-and-military-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/RKMNZn7r6cT2Yaorf/risks-from-autonomous-weapon-systems-and-military-ai&sa=D&source=editors&ust=1692801435330166&usg=AOvVaw1mFgBtq3eEf6J2oi5UBjuR). See also: Horowitz, Michael C, and Paul Scharre. ‘AI and International Stability: Risks and Confidence-Building Measures’. Center for a New American Security, 12 January 2021.[https://www.cnas.org/publications/reports/ai-and-international-stability-risks-and-confidence-building-measures](https://www.google.com/url?q=https://www.cnas.org/publications/reports/ai-and-international-stability-risks-and-confidence-building-measures&sa=D&source=editors&ust=1692801435330668&usg=AOvVaw1W2J4PXuksDKA7HCPfK_m1).; Horowitz, Michael C., Lauren Kahn, and Casey Mahoney. ‘The Future of Military Applications of Artificial Intelligence: A Role for Confidence-Building Measures?’ Orbis, 14 September 2020.[https://doi.org/10.1016/j.orbis.2020.08.003](https://www.google.com/url?q=https://doi.org/10.1016/j.orbis.2020.08.003&sa=D&source=editors&ust=1692801435331086&usg=AOvVaw2p2pl-SQL17xdUg-nqa9gy).

[[547]](#ftnt_ref547) Guzman, Andrew T. ‘The Design of International Agreements’. European Journal of International Law 16, no. 4 (1 September 2005): 579–612.[https://doi.org/10.1093/ejil/chi134](https://www.google.com/url?q=https://doi.org/10.1093/ejil/chi134&sa=D&source=editors&ust=1692801435333820&usg=AOvVaw1SDa0qYzmeKnItiALVVcV2). For how new technologies may affect this, see: McGregor, Lorna. ‘Are New Technologies an Aid to Reputation as a Disciplinarian?’ AJIL Unbound 113 (ed 2019): 238–41.[https://doi.org/10.1017/aju.2019.54](https://www.google.com/url?q=https://doi.org/10.1017/aju.2019.54&sa=D&source=editors&ust=1692801435334275&usg=AOvVaw1ccp_i31sbtftHJ-vV5zxz).

[[548]](#ftnt_ref548) Dothan, Shai. ‘A Virtual Wall of Shame: The New Way of Imposing Reputational Sanctions on Defiant States’. Duke Journal of Comparative and International Law 27 (2017 2016): 141.

[[549]](#ftnt_ref549) Clark, Jack, and Gillian K Hadﬁeld. ‘Regulatory Markets for AI Safety’, Safe Machine Learning workshop at ICLR, 2019. 2019.[https://arxiv.org/abs/2001.00078](https://www.google.com/url?q=https://arxiv.org/abs/2001.00078&sa=D&source=editors&ust=1692801434970550&usg=AOvVaw3_LBPliYLQHUaFNaMRsiN8).

[[550]](#ftnt_ref550) Clark, Jack. ‘Information Markets and AI Development’. In The Oxford Handbook of AI Governance, edited by Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang, 0. Oxford University Press. Accessed 6 February 2023.[https://doi.org/10.1093/oxfordhb/9780197579329.013.21](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.21&sa=D&source=editors&ust=1692801435360056&usg=AOvVaw3z9uaBg1I0Zj4n_P2whlpS).

[[551]](#ftnt_ref551) See generally: Posner, Eric A., and Eric Glen Weyl. Radical Markets: Uprooting Capitalism and Democracy for a Just Society. Princeton ; Oxford: Princeton University Press, 2018.

[[552]](#ftnt_ref552) Buterin, Vitalik. ‘Why Cryptoeconomics and X-Risk Researchers Should Listen to Each Other More’. Medium, 5 July 2016.[https://medium.com/@VitalikB](https://www.google.com/url?q=https://medium.com/@VitalikButerin/why-cryptoeconomics-and-x-risk-researchers-should-listen-to-each-other-more-a2db72b3e86b&sa=D&source=editors&ust=1692801435133016&usg=AOvVaw3NM39IenVeULnLxzf-GaoA)[uterin/why-cryptoeconomics-and-x-risk-researchers-should-listen-to-each-other-more-a2db72b3e86b](https://www.google.com/url?q=https://medium.com/@VitalikButerin/why-cryptoeconomics-and-x-risk-researchers-should-listen-to-each-other-more-a2db72b3e86b&sa=D&source=editors&ust=1692801435133295&usg=AOvVaw3yVXQ3JHIrt6nWRbTaFsO8).

[[553]](#ftnt_ref553) For definition, see also Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023.

[[554]](#ftnt_ref554) Moës, Nicolas, and Caroline Jeanmaire. ‘A Map to Navigate AI Governance’. EA Forum, 14 February 2022.[https://forum.effectivealtruism.org/posts/tmxkRFx6HyhhvHdz4/a-map-to-navigate-ai-governance](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/tmxkRFx6HyhhvHdz4/a-map-to-navigate-ai-governance&sa=D&source=editors&ust=1692801434993831&usg=AOvVaw3KLYFCwpA8suMqqhq33TWN). (highlighting three major ‘governance pathways’--hard governance, industry-wide self-governance, and company self-governance–each with associated sub-activities; also mentions a range of additional governance pathways not mentioned on the map: Military & National Security governance, Supply Chain & Trade governance, Multilateral soft governance, Extralegal governance, and Academic governance).

[[555]](#ftnt_ref555) Clarke, Sam. ‘The Longtermist AI Governance Landscape: A Basic Overview’. EA Forum, 18 January 2022.[https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview&sa=D&source=editors&ust=1692801435117113&usg=AOvVaw0wyucqjK0AlvXklFqOPj7A). (“Clarke sketches a spectrum of activities, spanning strategy research, tactics research, policy development work, policy advocacy work, and policy implementation work–supported by field-building work at all levels.”).

[[556]](#ftnt_ref556) O’Keefe, Cullen. ‘Law-Following AI’. AI Alignment Forum, 4 August 2022.[https://www.alignmentforum.org/s/ZytYxd523oTnBNnRT](https://www.google.com/url?q=https://www.alignmentforum.org/s/ZytYxd523oTnBNnRT&sa=D&source=editors&ust=1692801435024108&usg=AOvVaw2oRzpxk07o5jzg_hqNS2k2).

[[557]](#ftnt_ref557) Hadfield-Menell, Dylan, and Gillian Hadfield. ‘Incomplete Contracting and AI Alignment’. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 2019.[http://arxiv.org/abs/1804.04268](https://www.google.com/url?q=http://arxiv.org/abs/1804.04268&sa=D&source=editors&ust=1692801435175862&usg=AOvVaw0blYSbQxjMVxLmAXobvFC-).

[[558]](#ftnt_ref558) Bajgar, Ondrej, and Jan Horenovsky. ‘Negative Human Rights as a Basis for Long-Term AI Safety and Regulation’. Journal of Artificial Intelligence Research, 2022, 30. [https://arxiv.org/abs/2208.14788](https://www.google.com/url?q=https://arxiv.org/abs/2208.14788&sa=D&source=editors&ust=1692801435024469&usg=AOvVaw2RRF-YjskKt-agNNs1n9oL); see also Bajgar, Ondrej, and Jan Horenovsky. ‘Narrow Rules are not Enough: Why artificial intelligence needs to understand human rights’. Verfassungsblog (blog), 11 August 2022.[https://verfassungsblog.de/narrow-rules-are-not-enough/](https://www.google.com/url?q=https://verfassungsblog.de/narrow-rules-are-not-enough/&sa=D&source=editors&ust=1692801435024928&usg=AOvVaw1LnsuHXFUA46m8NLNLYgFu).

[[559]](#ftnt_ref559) Gabriel, Iason. ‘Artificial Intelligence, Values, and Alignment’. Minds and Machines 30, no. 3 (1 September 2020): 411–37.[https://doi.org/10.1007/s11023-020-09539-2](https://www.google.com/url?q=https://doi.org/10.1007/s11023-020-09539-2&sa=D&source=editors&ust=1692801435035197&usg=AOvVaw1wvMmWyJhhHu7I5oJqbtWS).

[[560]](#ftnt_ref560) Aguirre, Anthony, Peter Bart Reiner, Harry Surden, and Gaia Dempsey. ‘AI Loyalty by Design: A Framework for Governance of AI’. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, 24 September 2021.[https://papers.ssrn.com/abstract=3930338](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3930338&sa=D&source=editors&ust=1692801435036298&usg=AOvVaw2DkJXUqPclSImkxgBwtDi9).; Benthall, Sebastian. ‘For Safe AI Tomorrow, Fiduciary Duties for Big Tech Today’. dli-cornell-tech, 11 May 2022.[https://www.dli.tech.cornell.edu/post/for-safe-ai-tomorrow-fiduciary-duties-for-big-tech-today](https://www.google.com/url?q=https://www.dli.tech.cornell.edu/post/for-safe-ai-tomorrow-fiduciary-duties-for-big-tech-today&sa=D&source=editors&ust=1692801435036842&usg=AOvVaw1rek_ZAWqqUBlLuHFpf4_w).

[[561]](#ftnt_ref561) Nay, John. ‘Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans’. SSRN Scholarly Paper. Rochester, NY, 13 September 2022.[https://doi.org/10.2139/ssrn.4218031](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4218031&sa=D&source=editors&ust=1692801435063060&usg=AOvVaw08XlaZq4QKjc-Z0lGRka-5).

[[562]](#ftnt_ref562) Perry, Brandon, and Risto Uuk. ‘AI Governance and the Policymaking Process: Key Considerations for Reducing AI Risk’. Big Data and Cognitive Computing 3, no. 2 (June 2019): 26.[https://doi.org/10.3390/bdcc3020026](https://www.google.com/url?q=https://doi.org/10.3390/bdcc3020026&sa=D&source=editors&ust=1692801434951584&usg=AOvVaw0px_HkcQ_7Ws-o8Wd55FFN).

[[563]](#ftnt_ref563) Leung, Jade. ‘How Can We See the Impact of AI Strategy Research?’ Presented at the EA Global: San Francisco 2019, 2019.[https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research&sa=D&source=editors&ust=1692801435176472&usg=AOvVaw1NCDBmtbJW6wpVVCYEJDyJ).

[[564]](#ftnt_ref564) Critch, Andrew. ‘Some AI Research Areas and Their Relevance to Existential Safety’. LessWrong, 19 November 2020.[https://www.lesswrong.com/posts/hvGoYXi2kgnS3vxqb/some-ai-research-areas-and-their-relevance-to-existential-1](https://www.google.com/url?q=https://www.lesswrong.com/posts/hvGoYXi2kgnS3vxqb/some-ai-research-areas-and-their-relevance-to-existential-1&sa=D&source=editors&ust=1692801435156895&usg=AOvVaw1bturx6Ij5UUdWMMftQmpP).

[[565]](#ftnt_ref565) Schiff, Daniel S. ‘Setting the Agenda for AI: Actors, Issues, and Influence in United States Artificial Intelligence Policy’. Georgia Institute of Technology, 2022.[https://osf.io/kw8xd/](https://www.google.com/url?q=https://osf.io/kw8xd/&sa=D&source=editors&ust=1692801434994508&usg=AOvVaw3biZOAkW5h1fsk0_W32MkL). (exploring the US policy context, and arguing that “policy entrepreneurs can use persuasive narratives to influence legislators about AI policy, and that these narratives are just as effective as technical information. [...] despite pervasive calls for public participation in AI governance, the public does not appear to play a key role in directing attention to AI's social and ethical implications nor in shaping concrete policy solutions, such that the emerging AI agenda remains primarily expert-driven.”).

[[566]](#ftnt_ref566) Leung, Jade. ‘How Can We See the Impact of AI Strategy Research?’ Presented at the EA Global: San Francisco 2019, 2019.[https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research&sa=D&source=editors&ust=1692801434995416&usg=AOvVaw0_LbqswTxcyMVlQL6sYhgD).

[[567]](#ftnt_ref567) See also Maas, Matthijs, AI Foundations Report 1: Disentangling Definitions in Transformative AI Governance. Legal Priorities Project. Forthcoming 2023. And previously: Maas, ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’., pg. 215-216. For general work on how legal analogies can influence rulings made, see: Crootof, Rebecca. ‘Autonomous Weapon Systems and the Limits of Analogy’. Harvard National Security Journal 9 (2018): 51–83.[https://doi.org/10.2139/ssrn.2820727](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.2820727&sa=D&source=editors&ust=1692801435037630&usg=AOvVaw1c5LTCSsQrF25vEmbwoW43).; Lakier, Genevieve. ‘The Problem Isn’t the Use of Analogies but the Analogies Courts Use’. Knight First Amendment Institute at Columbia University (blog), 26 February 2018.[https://knightcolumbia.org/content/problem-isnt-use-analogies-analogies-courts-use](https://www.google.com/url?q=https://knightcolumbia.org/content/problem-isnt-use-analogies-analogies-courts-use&sa=D&source=editors&ust=1692801435038115&usg=AOvVaw2nrMX3s9kyLVnA3b4cHZDo).

[[568]](#ftnt_ref568) Wiblin, Robert, and Keiran Harris. ‘Carl Shulman on the Common-Sense Case for Existential Risk Work and Its Practical Implications’. 80,000 Hours Podcast. Accessed 11 October 2021.[https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/](https://www.google.com/url?q=https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/&sa=D&source=editors&ust=1692801435158799&usg=AOvVaw0EmizVdxyH-JXEKehHL0zF).

[[569]](#ftnt_ref569) For an overview, see Rayfuse, Rosemary. ‘Public International Law and the Regulation of Emerging Technologies’. In The Oxford Handbook of Law, Regulation and Technology, 2017.[http://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199680832.001.0001/oxfordhb-9780199680832-e-22](https://www.google.com/url?q=http://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199680832.001.0001/oxfordhb-9780199680832-e-22&sa=D&source=editors&ust=1692801435355354&usg=AOvVaw0cn3lGpxypNjY3WRlpHg7C). Pg. 503:

(“the basic norms of international peace and security law, such as the prohibitions on the use of force and intervention in the domestic affairs of other states [...]; the basic principles of international humanitarian law, such as the requirements of humanity, distinction and proportionality [...]; the basic principles of international human rights law, including the principles of human dignity and the right to life, liberty, and security of the person [...]; and the basic principles of international environmental law, including the no-harm principle, the obligation to prevent pollution, the obligation to protect vulnerable ecosystems and species, the precautionary principle, and a range of procedural obligations relating to cooperation, consultation, notification, and exchange of information, environmental impact assessment, and participation [...]. The general customary rules on state responsibility and liability for harm also apply.”).

[[570]](#ftnt_ref570) Vöneky, Silja. ‘How Should We Regulate AI? Current Rules and Principles as Basis for “Responsible Artificial Intelligence”’, 19 May 2020.[https://papers.ssrn.com/abstract=3605440](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3605440&sa=D&source=editors&ust=1692801435356104&usg=AOvVaw1AI5j1LW825aLrzv4azorw).

[[571]](#ftnt_ref571) Martínez, Eric, and Christoph Winter. ‘Ordinary Meaning of Existential Risk’, [forthcoming].

[[572]](#ftnt_ref572) ÓhÉigeartaigh, Seán S., Jess Whittlestone, Yang Liu, Yi Zeng, and Zhe Liu. ‘Overcoming Barriers to Cross-Cultural Cooperation in AI Ethics and Governance’. Philosophy & Technology, 15 May 2020.[https://doi.org/10.1007/s13347-020-00402-x](https://www.google.com/url?q=https://doi.org/10.1007/s13347-020-00402-x&sa=D&source=editors&ust=1692801435044638&usg=AOvVaw0hx7YOQSELrvQNhBAhbu9x).

[[573]](#ftnt_ref573) Ash, Elliott, Daniel L. Chen, and Suresh Naidu. ‘Ideas Have Consequences: The Impact of Law and Economics on American Justice’. Working Paper. Working Paper Series. National Bureau of Economic Research, February 2022.[https://doi.org/10.3386/w29788](https://www.google.com/url?q=https://doi.org/10.3386/w29788&sa=D&source=editors&ust=1692801435043467&usg=AOvVaw0UmalM2HWyWhruX4vHYxQg). Discussed in: Matthews, Dylan, and Byrd Pinkerton. ‘How a Resort Weekend for Judges Made Courts More Conservative’. Vox, 1 June 2019.[https://www.vox.com/future-perfect/2019/6/1/18629859/judge-resort-weekend-naidu-manne-seminar-ginsburg](https://www.google.com/url?q=https://www.vox.com/future-perfect/2019/6/1/18629859/judge-resort-weekend-naidu-manne-seminar-ginsburg&sa=D&source=editors&ust=1692801435043945&usg=AOvVaw2_VGVGYjqAN7xFw6WRoVet).

[[574]](#ftnt_ref574) See generally Thompson, Neil, Brian Flanagan, Edana Richardson, Brian McKenzie, and Xueyun Luo. ‘Trial by Internet: A Randomized Field Experiment on Wikipedia’s Influence on Judges’ Legal Reasoning’. SSRN Scholarly Paper. Rochester, NY, 27 July 2022.[https://doi.org/10.2139/ssrn.4174200](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4174200&sa=D&source=editors&ust=1692801435045117&usg=AOvVaw0Zd4eJ1b3qxAdvCvFgPWeU).

[[575]](#ftnt_ref575) Wasil, Akash, Olivia Jimenez, and Thomas Larsen. ‘Ways to Buy Time’. LessWrong, 12 November 2022.[https://www.lesswrong.com/posts/bkpZHXMJx3dG5waA7/ways-to-buy-time](https://www.google.com/url?q=https://www.lesswrong.com/posts/bkpZHXMJx3dG5waA7/ways-to-buy-time&sa=D&source=editors&ust=1692801435392595&usg=AOvVaw3TNLQJLqPhNEOEzzgoT_aG). I thank Zach Stein-Perlman for this suggestion.

[[576]](#ftnt_ref576) Cihon, Peter, Jonas Schuett, and Seth D. Baum. ‘Corporate Governance of Artificial Intelligence in the Public Interest’. Information 12, no. 7 (July 2021): 275.[https://doi.org/10.3390/info12070275](https://www.google.com/url?q=https://doi.org/10.3390/info12070275&sa=D&source=editors&ust=1692801435159297&usg=AOvVaw2oEOTUq5V0Ytw_WlCJlEwA).

[[577]](#ftnt_ref577) Belfield, Haydn. ‘Activism by the AI Community: Analysing Recent Achievements and Future Prospects’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 15–21. New York NY USA: ACM, 2020.[https://doi.org/10.1145/3375627.3375814](https://www.google.com/url?q=https://doi.org/10.1145/3375627.3375814&sa=D&source=editors&ust=1692801435159803&usg=AOvVaw1tYi3QEcEESRlBjXrBuH8-).

[[578]](#ftnt_ref578) Vöneky, Silja. ‘How Should We Regulate AI? Current Rules and Principles as Basis for “Responsible Artificial Intelligence”’, 19 May 2020.[https://papers.ssrn.com/abstract=3605440](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3605440&sa=D&source=editors&ust=1692801435160260&usg=AOvVaw1I0r7nEz6ngSHSnGTcw1BD).

[[579]](#ftnt_ref579) Helfer, Laurence R., and Molly K. Land. ‘The Facebook Oversight Board’s Human Rights Future’. SSRN Scholarly Paper. Rochester, NY, 22 August 2022.[https://doi.org/10.2139/ssrn.4197107](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4197107&sa=D&source=editors&ust=1692801435160771&usg=AOvVaw0IONIz9fTnJ8Pn_wRHN9Y5).; Kulick, Andreas. ‘Corporations as Interpreters and Adjudicators of International Human Rights Norms – Meta‘s Oversight Board and Beyond’. SSRN Scholarly Paper. Rochester, NY, 22 September 2022.[https://papers.ssrn.com/abstract=4226521](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4226521&sa=D&source=editors&ust=1692801435161212&usg=AOvVaw3XaOVYGy-dTrzehC6TscI5).; Wong, David, and Luciano Floridi. ‘Meta’s Oversight Board: A Review and Critical Assessment’. SSRN Scholarly Paper. Rochester, NY, 22 October 2022.[https://papers.ssrn.com/abstract=4255817](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D4255817&sa=D&source=editors&ust=1692801435161653&usg=AOvVaw1v3edWG7N2RanEiCjvezAG).

[[580]](#ftnt_ref580) Hua, Shin-Shin, and Haydn Belfield. ‘AI & Antitrust: Reconciling Tensions Between Competition Law and Cooperative AI Development’. Yale Journal of Law and Technology 23 (Spring 2021): 127. [https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development](https://www.google.com/url?q=https://yjolt.org/ai-antitrust-reconciling-tensions-between-competition-law-and-cooperative-ai-development&sa=D&source=editors&ust=1692801435162149&usg=AOvVaw2nwjUivYaiKuXXlqIZ9UQH)

[[581]](#ftnt_ref581) Ashurst, Carolyn, Markus Anderljung, Carina Prunkl, Jan Leike, Yarin Gal, Toby Shevlane, and Allan Dafoe. ‘A Guide to Writing the NeurIPS Impact Statement’. Centre for the Governance of AI (Medium) (blog), 19 May 2020.[https://medium.com/@GovAI/a-guide-to-writing-the-neurips-impact-statement-4293b723f832](https://www.google.com/url?q=https://medium.com/@GovAI/a-guide-to-writing-the-neurips-impact-statement-4293b723f832&sa=D&source=editors&ust=1692801435034145&usg=AOvVaw2zynjuBueMgNI5TvK9zAvc).

[[582]](#ftnt_ref582) Baum, Seth D. ‘On the Promotion of Safe and Socially Beneficial Artificial Intelligence’. AI & SOCIETY, 28 September 2016.[https://doi.org/10.1007/s00146-016-0677-0](https://www.google.com/url?q=https://doi.org/10.1007/s00146-016-0677-0&sa=D&source=editors&ust=1692801434991777&usg=AOvVaw0Eqpx9WWXUyRNxVG4HFsn2).; on some form of pushback, see also lc. ‘What an Actually Pessimistic Containment Strategy Looks Like’. LessWrong, 5 April 2022.[https://www.lesswrong.com/posts/kipMvuaK3NALvFHc9/what-an-actually-pessimistic-containment-strategy-looks-like](https://www.google.com/url?q=https://www.lesswrong.com/posts/kipMvuaK3NALvFHc9/what-an-actually-pessimistic-containment-strategy-looks-like&sa=D&source=editors&ust=1692801434992386&usg=AOvVaw3kSfyobYLvFELNEpsRX6LY).

[[583]](#ftnt_ref583) Cremer, Carla Zoe, and Jess Whittlestone. ‘Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI’. International Journal of Interactive Multimedia and Artificial Intelligence 6, no. 5 (2021): 100–109. [https://www.ijimai.org/journal/sites/default/files/2021-02/ijimai\_6\_5\_10.pdf](https://www.google.com/url?q=https://www.ijimai.org/journal/sites/default/files/2021-02/ijimai_6_5_10.pdf&sa=D&source=editors&ust=1692801435034663&usg=AOvVaw2Klh7exCVbKbqNryExgDu9)

[[584]](#ftnt_ref584) Rosert, Elvira, and Frank Sauer. ‘How (Not) to Stop the Killer Robots: A Comparative Analysis of Humanitarian Disarmament Campaign Strategies’. Contemporary Security Policy 0, no. 0 (30 May 2020): 1–26.[https://doi.org/10.1080/13523260.2020.1771508](https://www.google.com/url?q=https://doi.org/10.1080/13523260.2020.1771508&sa=D&source=editors&ust=1692801435094403&usg=AOvVaw3IW6cc6PxyN7LTXFIYKQZj).

[[585]](#ftnt_ref585) Benvenisti, Eyal. ‘Upholding Democracy Amid the Challenges of New Technology: What Role for the Law of Global Governance?’ European Journal of International Law 29, no. 1 (23 July 2018): 9–82.[https://doi.org/10.1093/ejil/chy031](https://www.google.com/url?q=https://doi.org/10.1093/ejil/chy031&sa=D&source=editors&ust=1692801435171355&usg=AOvVaw2n3YNawCbfCF0rSh6glBoz).

[[586]](#ftnt_ref586) See generally: Koh, Harold Hongju. ‘Why Do Nations Obey International Law?’ Edited by Abram Chayes, Antonia Handler Chayes, and Thomas M. Franck. The Yale Law Journal 106, no. 8 (1997): 2599–2659.[https://doi.org/10.2307/797228](https://www.google.com/url?q=https://doi.org/10.2307/797228&sa=D&source=editors&ust=1692801435354649&usg=AOvVaw2xcURO2OsFlgkWrIlEIx0P).

[[587]](#ftnt_ref587) Orangias, Joseph. ‘The Nexus between International Law and Science: An Analysis of Scientific Expert Bodies in Multilateral Treaty-Making’. International Community Law Review 25, no. 1 (1 April 2022): 60–93.[https://doi.org/10.1163/18719732-bja10068](https://www.google.com/url?q=https://doi.org/10.1163/18719732-bja10068&sa=D&source=editors&ust=1692801435413737&usg=AOvVaw0MTK-zRhiP5io6ac6pteaa).

[[588]](#ftnt_ref588) Ingersleben-Seip, Nora von. ‘Competition and Cooperation in Artificial Intelligence Standard Setting: Explaining Emergent Patterns’. Review of Policy Research n/a, no. n/a. Accessed 25 January 2023.[https://doi.org/10.1111/ropr.12538](https://www.google.com/url?q=https://doi.org/10.1111/ropr.12538&sa=D&source=editors&ust=1692801435383812&usg=AOvVaw2fYyQbAeGl0AguE3ipWgEz).

[[589]](#ftnt_ref589) Center for Long-Term Cybersecurity. ‘Seeking Input and Feedback: AI Risk Management-Standards Profile for Increasingly Multi-Purpose or General-Purpose AI’. CLTC (blog), 2022.[https://cltc.berkeley.edu/seeking-input-and-feedback-ai-risk-management-standards-profile-for-increasingly-multi-purpose-or-general-purpose-ai/](https://www.google.com/url?q=https://cltc.berkeley.edu/seeking-input-and-feedback-ai-risk-management-standards-profile-for-increasingly-multi-purpose-or-general-purpose-ai/&sa=D&source=editors&ust=1692801435384739&usg=AOvVaw2NQUwyg2eytDKl_Ty9_rRC); O’Keefe, Cullen, Jade Leung, and Markus Anderljung. ‘How Technical Safety Standards Could Promote TAI Safety’. Effective Altruism Forum, 8 August 2022.[https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/zvbGXCxc5jBowCuNX/how-technical-safety-standards-could-promote-tai-safety&sa=D&source=editors&ust=1692801435385463&usg=AOvVaw0JEPbUYHXZYJ1jc-H9r0Aj).

[[590]](#ftnt_ref590) AI Impacts. ‘AI Vignettes Project’. AI Impacts, 12 October 2021.[https://aiimpacts.org/ai-vignettes-project/](https://www.google.com/url?q=https://aiimpacts.org/ai-vignettes-project/&sa=D&source=editors&ust=1692801435162710&usg=AOvVaw0KdQZ6xX3OKQAWTW1gVpmL).

[[591]](#ftnt_ref591) Brundage, Miles. ‘Guide to Working in Artificial Intelligence Policy and Strategy’. 80,000 Hours, 13 June 2017.[https://80000hours.org/articles/ai-policy-guide/](https://www.google.com/url?q=https://80000hours.org/articles/ai-policy-guide/&sa=D&source=editors&ust=1692801435117749&usg=AOvVaw1Kup66UElFDhfhO6tz6h9U).; For different country-level guides, see: Langosco, Lauro. ‘AI Policy Careers in the EU’. EA Forum, 11 November 2019.[https://forum.effectivealtruism.org/posts/XGPW25NZHq2WHbK9w/ai-policy-careers-in-the-eu](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/XGPW25NZHq2WHbK9w/ai-policy-careers-in-the-eu&sa=D&source=editors&ust=1692801435118306&usg=AOvVaw3kMj60O3Ee8YbuNG3jBJsb).; Bowerman, Niel. ‘The Case for Building Expertise to Work on US AI Policy’. 80,000 Hours, December 2020.[https://80000hours.org/articles/us-ai-policy/](https://www.google.com/url?q=https://80000hours.org/articles/us-ai-policy/&sa=D&source=editors&ust=1692801435118788&usg=AOvVaw1iPGwBv_iOIj0-RleX_j9S).; 80,000 Hours. ‘China-Related AI Safety and Governance Paths’. 80,000 Hours, February 2022.[https://80000hours.org/career-reviews/china-related-ai-safety-and-governance-paths/](https://www.google.com/url?q=https://80000hours.org/career-reviews/china-related-ai-safety-and-governance-paths/&sa=D&source=editors&ust=1692801435119309&usg=AOvVaw3NBlDhKHV-zV2DfCmQgXab).; Chua, Yi-Yang. ‘Singapore AI Policy Career Guide’. EA Forum, 21 January 2021.[https://forum.effectivealtruism.org/posts/umeMcbD4jDseLjsgT/singapore-ai-policy-career-guide](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/umeMcbD4jDseLjsgT/singapore-ai-policy-career-guide&sa=D&source=editors&ust=1692801435119927&usg=AOvVaw1fp_89PTXv09LI8MUsIUrI). See also the careers reading guide in: AGI Safety Fundamentals. ‘AI Governance Curriculum’. Accessed 20 October 2022.[https://www.agisafetyfundamentals.com/ai-governance-curriculum](https://www.google.com/url?q=https://www.agisafetyfundamentals.com/ai-governance-curriculum&sa=D&source=editors&ust=1692801435120452&usg=AOvVaw1CFUtMzbSy9f9l4S_PKZX-). (week 7).

[[592]](#ftnt_ref592) Irving, Geoffrey, and Amanda Askell. ‘AI Safety Needs Social Scientists’. Distill 4, no. 2 (19 February 2019): e14.[https://doi.org/10.23915/distill.00014](https://www.google.com/url?q=https://doi.org/10.23915/distill.00014&sa=D&source=editors&ust=1692801435121217&usg=AOvVaw246rfWjlHtMcHHdM262Hhx).

[[593]](#ftnt_ref593) Mauricio. ‘AI Governance Needs Technical Work’. EA Forum, 6 September 2022.[https://forum.effectivealtruism.org/posts/BJtekdKrAufyKhBGw/ai-governance-needs-technical-work](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/BJtekdKrAufyKhBGw/ai-governance-needs-technical-work&sa=D&source=editors&ust=1692801435121944&usg=AOvVaw21dfZlAWUKlKl9Jio3pA3o).

[[594]](#ftnt_ref594) Gutierrez, Carlos Ignacio. ‘The Unforeseen Consequences of Artificial Intelligence (AI) on Society: A Systematic Review of Regulatory Gaps Generated by AI in the U.S.’ Thesis, Pardee RAND Graduate School, 2020.[https://www.rand.org/pubs/rgs\_dissertations/RGSDA319-1.html](https://www.google.com/url?q=https://www.rand.org/pubs/rgs_dissertations/RGSDA319-1.html&sa=D&source=editors&ust=1692801435271742&usg=AOvVaw2i0m0GuDFTelPRXOXaMy-r).

[[595]](#ftnt_ref595) Meyer, Timothy, and Ganesh Sitaraman. ‘The National Security Consequences of the Major Questions Doctrine’. SSRN Scholarly Paper. Rochester, NY, 4 August 2022.[https://doi.org/10.2139/ssrn.4181908](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4181908&sa=D&source=editors&ust=1692801435391897&usg=AOvVaw3q_lpoV5nw3NxWuEDwFE8H). I thank Christoph Winter for this suggestion.

[[596]](#ftnt_ref596) Portman, Rob. ‘S.4488 - 117th Congress (2021-2022): Global Catastrophic Risk Management Act of 2022’. Legislation, 13 December 2022. 2021/2022.[https://www.congress.gov/bill/117th-congress/senate-bill/4488/text](https://www.google.com/url?q=https://www.congress.gov/bill/117th-congress/senate-bill/4488/text&sa=D&source=editors&ust=1692801435403790&usg=AOvVaw15pDuVcirw_QOqgOVxEDOj).

[[597]](#ftnt_ref597) See also: Hacker, Philipp. ‘The European AI Liability Directives -- Critique of a Half-Hearted Approach and Lessons for the Future’. arXiv, 23 January 2023.[https://doi.org/10.48550/arXiv.2211.13960](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2211.13960&sa=D&source=editors&ust=1692801435382316&usg=AOvVaw0GYXFhNSQUdemlYBb8q8x4).

[[598]](#ftnt_ref598) Hua, Shin-Shin, and Haydn Belfield. ‘Effective Enforceability of EU Competition Law Under Different AI Development Scenarios: A Framework for Legal Analysis’. Verfassungsblog (blog), 18 August 2022.[https://verfassungsblog.de/effective-enforceability-of-eu-competition-law-under-different-ai-development-scenarios/](https://www.google.com/url?q=https://verfassungsblog.de/effective-enforceability-of-eu-competition-law-under-different-ai-development-scenarios/&sa=D&source=editors&ust=1692801435408751&usg=AOvVaw1VZwVNjwm1Nm9Rs1IlpqLk).

[[599]](#ftnt_ref599) For a broader review of international norms applicable to emerging existential risks, including those from technology, see: Villalobos, José Jaime, Matthijs Maas, and Christoph Winter. ‘States Must Mitigate Existential Risk under International Law’, Legal Priorities Project working paper. Forthcoming 2023.

[[600]](#ftnt_ref600) See also: Kunz, Martina, and Seán Ó hÉigeartaigh. ‘Artificial Intelligence and Robotization’. In Oxford Handbook on the International Law of Global Security, edited by Robin Geiss and Nils Melzer. Oxford University Press, 2021.[https://papers.ssrn.com/abstract=3310421](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3310421&sa=D&source=editors&ust=1692801435272443&usg=AOvVaw21bRC26k-Rf303dnWGi3Pg).

[[601]](#ftnt_ref601) See also: Rayfuse, Rosemary. ‘Public International Law and the Regulation of Emerging Technologies’. In The Oxford Handbook of Law, Regulation and Technology, 2017.[http://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199680832.001.0001/oxfordhb-9780199680832-e-22](https://www.google.com/url?q=http://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199680832.001.0001/oxfordhb-9780199680832-e-22&sa=D&source=editors&ust=1692801435273230&usg=AOvVaw2oP9rT9yF1bJjgPQy9cVVP). Pg. 506. As discussed in: Maas, Matthijs M. ‘Artificial Intelligence Governance Under Change: Foundations, Facets, Frameworks’. University of Copenhagen, 2020. Pg. 97-98. For arguments in favour of the role of customary law in regulating new technologies, or large-scale risks, see respectively: Crootof, Rebecca. ‘Jurisprudential Space Junk: Treaties and New Technologies’. In Resolving Conflicts in the Law, edited by Chiara Giorgetti and Natalie Klein, 106–29, 2019.[https://brill.com/view/book/edcoll/9789004316539/BP000015.xml](https://www.google.com/url?q=https://brill.com/view/book/edcoll/9789004316539/BP000015.xml&sa=D&source=editors&ust=1692801435273837&usg=AOvVaw23tcy5jOBb_-lIpqKAAwf8).; Aaken, Anne van. ‘Is International Law Conducive To Preventing Looming Disasters?’ Global Policy 7, no. S1 (2016): 81–96.[https://doi.org/10.1111/1758-5899.12303](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12303&sa=D&source=editors&ust=1692801435274381&usg=AOvVaw19gUCNtfCoQWjB7smWnPKq).

[[602]](#ftnt_ref602) Vöneky, Silja. ‘How Should We Regulate AI? Current Rules and Principles as Basis for “Responsible Artificial Intelligence”’, 19 May 2020.[https://papers.ssrn.com/abstract=3605440](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3605440&sa=D&source=editors&ust=1692801435380826&usg=AOvVaw01X67RLhEXflYIbtk23fQp).; For an interesting angle: Bajgar, Ondrej, and Jan Horenovsky. ‘Negative Human Rights as a Basis for Long-Term AI Safety and Regulation’. Journal of Artificial Intelligence Research, 2022, 30. [https://arxiv.org/abs/2208.14788](https://www.google.com/url?q=https://arxiv.org/abs/2208.14788&sa=D&source=editors&ust=1692801435381152&usg=AOvVaw2Qqd5xV8jH6gi3TbvU7aDT)

[[603]](#ftnt_ref603) Though this is only recognized by some courts, and remains diffuse.

[[604]](#ftnt_ref604) Burri, Thomas. ‘International Law and Artificial Intelligence’. German Yearbook of International Law 60 (27 October 2017): 91–108. [http://dx.doi.org/10.2139/ssrn.3060191](https://www.google.com/url?q=http://dx.doi.org/10.2139/ssrn.3060191&sa=D&source=editors&ust=1692801435381649&usg=AOvVaw3uAG7inI7lzFMptFzTD53j)

[[605]](#ftnt_ref605) See also: Boutin, Bérénice. ‘State Responsibility in Relation to Military Applications of Artificial Intelligence’. Leiden Journal of International Law 36, no. 1 (March 2023): 133–50.[https://doi.org/10.1017/S0922156522000607](https://www.google.com/url?q=https://doi.org/10.1017/S0922156522000607&sa=D&source=editors&ust=1692801435386190&usg=AOvVaw1kseM6lHHeFbIZWhr4_AOy).

[[606]](#ftnt_ref606) Liu, Han-Wei, and Ching-Fu Lin. ‘Artificial Intelligence and Global Trade Governance: A Pluralist Agenda’. Harvard International Law Journal 61, no. 2 (2020).[https://papers.ssrn.com/abstract=3675505](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3675505&sa=D&source=editors&ust=1692801435383100&usg=AOvVaw1EiIX8w7sdS_g-99r5_JHL).

[[607]](#ftnt_ref607) See generally: McLaughlin, Mark. ‘Regulating Artificial Intelligence in International Investment Law’. The Journal of World Investment & Trade 24, no. 2 (5 April 2023): 256–300.[https://doi.org/10.1163/22119000-12340288](https://www.google.com/url?q=https://doi.org/10.1163/22119000-12340288&sa=D&source=editors&ust=1692801435441077&usg=AOvVaw3lWoJD9PUSSAMaRL8UNg3q).

[[608]](#ftnt_ref608) Araújo, Renan, and Leonie Koessler. ‘The Rise of the Constitutional Protection of Future Generations’. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, 30 September 2021.[https://papers.ssrn.com/abstract=3933683](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3933683&sa=D&source=editors&ust=1692801435435778&usg=AOvVaw04yYhl6_faPjOMlCPGR4P9).

[[609]](#ftnt_ref609) Hale, Thomas, Finlay Moorhouse, Toby Ord, and Anne-Marie Slaughter. ‘Toward a Declaration on Future Generations’, 12 January 2023.[https://www.bsg.ox.ac.uk/research/publications/toward-declaration-future-generations](https://www.google.com/url?q=https://www.bsg.ox.ac.uk/research/publications/toward-declaration-future-generations&sa=D&source=editors&ust=1692801435436414&usg=AOvVaw2lSsgZdW4chb6vsSLKbJhs).

[[610]](#ftnt_ref610) Schuett, Jonas. ‘Defining the Scope of AI Regulations’. Law, Innovation and Technology 0, no. 0 (3 March 2023): 1–23.[https://doi.org/10.1080/17579961.2023.2184135](https://www.google.com/url?q=https://doi.org/10.1080/17579961.2023.2184135&sa=D&source=editors&ust=1692801435040675&usg=AOvVaw2NoftwPX_aTzVQzNkH5MpN). See also Gutierrez, Carlos Ignacio, Anthony Aguirre, Risto Uuk, Claire C. Boine, and Matija Franklin. ‘A Proposal for a Definition of General Purpose Artificial Intelligence Systems’. Future of Life Institute - Working Paper, 5 October 2022.[https://doi.org/10.2139/ssrn.4238951](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4238951&sa=D&source=editors&ust=1692801435041101&usg=AOvVaw3uQNqJKN5VVfOh7dxPi2oa).

[[611]](#ftnt_ref611) Petit, Nicolas, and Jerome De Cooman. ‘Models of Law and Regulation for AI’. EUI Working Paper RSCAS 2020/63. Social Science Research Network, 1 October 2020.[https://doi.org/10.2139/ssrn.3706771](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.3706771&sa=D&source=editors&ust=1692801435038576&usg=AOvVaw27k5rHBzwgLpWjxox9LIfw).; Maas, Matthijs M. ‘Aligning AI Regulation to Sociotechnical Change’. In The Oxford Handbook of AI Governance, edited by Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young, and Baobao Zhang. Oxford University Press, 2022.[https://doi.org/10.1093/oxfordhb/9780197579329.013.22](https://www.google.com/url?q=https://doi.org/10.1093/oxfordhb/9780197579329.013.22&sa=D&source=editors&ust=1692801435038982&usg=AOvVaw0_MdYuaw5qQaGTn80XgLTZ).

[[612]](#ftnt_ref612) Whittlestone, Jess, and Jack Clark. ‘Why and How Governments Should Monitor AI Development’. ArXiv:2108.12427 [Cs], 31 August 2021.[http://arxiv.org/abs/2108.12427](https://www.google.com/url?q=http://arxiv.org/abs/2108.12427&sa=D&source=editors&ust=1692801435275052&usg=AOvVaw3ZiGTsqftrzZcpEU0ljil7).

[[613]](#ftnt_ref613) Cihon, Peter, Matthijs M. Maas, and Luke Kemp. ‘Fragmentation and the Future: Investigating Architectures for International AI Governance’. Global Policy 11, no. 5 (November 2020): 545–56.[https://doi.org/10.1111/1758-5899.12890](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12890&sa=D&source=editors&ust=1692801435039516&usg=AOvVaw02JVN2txtWD2dkUo0ULfjP).; Cihon, Peter, Matthijs M Maas, and Luke Kemp. ‘Should Artificial Intelligence Governance Be Centralised? Six Design Lessons from History’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 11, 2019.[https://www.cser.ac.uk/media/uploads/files/Cihon\_et\_al-\_2019-\_Should\_AI\_Governance\_be\_Centralised.pdf](https://www.google.com/url?q=https://www.cser.ac.uk/media/uploads/files/Cihon_et_al-_2019-_Should_AI_Governance_be_Centralised.pdf&sa=D&source=editors&ust=1692801435040055&usg=AOvVaw1WnLOxzF5blQpp3rTv3h4f).

[[614]](#ftnt_ref614) Casper, Stephen. ‘Avoiding Perpetual Risk from TAI’. LessWrong, 26 December 2022.[https://www.lesswrong.com/posts/FfTxEf3uFPsZf9EMP/avoiding-perpetual-risk-from-tai](https://www.google.com/url?q=https://www.lesswrong.com/posts/FfTxEf3uFPsZf9EMP/avoiding-perpetual-risk-from-tai&sa=D&source=editors&ust=1692801435203277&usg=AOvVaw3X5vKG4mI0XmWSsKqKFbFa).

[[615]](#ftnt_ref615) See generally the discussion of ‘adaptation-enabling’ TAI governance in: Maas, Matthijs. ‘Handbook on Approaches in Transformative AI Governance’; 2023 Forthcoming.

[[616]](#ftnt_ref616) Bowman, M. J. ‘The Multilateral Treaty Amendment Process—A Case Study’. International & Comparative Law Quarterly 44, no. 3 (July 1995): 540–59.[https://doi.org/10.1093/iclqaj/44.3.540](https://www.google.com/url?q=https://doi.org/10.1093/iclqaj/44.3.540&sa=D&source=editors&ust=1692801435166862&usg=AOvVaw26kmgtlsq_l0Zlcn2EyyYt).

[[617]](#ftnt_ref617) Vidigal, Geraldo. ‘Evolutionary Interpretation and International Law’. Journal of International Economic Law 24, no. 1 (1 March 2021): 203–19.[https://doi.org/10.1093/jiel/jgaa035](https://www.google.com/url?q=https://doi.org/10.1093/jiel/jgaa035&sa=D&source=editors&ust=1692801435168706&usg=AOvVaw3xLB-cKGOVm2iYTks53sHz).; Abi-Saab, Georges, Kenneth Keith, Gabrielle Marceau, and Clément Marquet, eds. Evolutionary Interpretation and International Law. S.l.: Hart Publishing, 2021.

[[618]](#ftnt_ref618) Smith, Bryant Walker. ‘New Technologies and Old Treaties’. AJIL Unbound 114 (ed 2020): 152–57.[https://doi.org/10.1017/aju.2020.28](https://www.google.com/url?q=https://doi.org/10.1017/aju.2020.28&sa=D&source=editors&ust=1692801435167474&usg=AOvVaw379DSQAlu1VqxCyjoOcC7-).

[[619]](#ftnt_ref619) Crootof, Rebecca. ‘Change Without Consent: How Customary International Law Modifies Treaties’. Yale Journal of International Law 41, no. 2 (2016): 65. [https://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article=1670&context=yjil](https://www.google.com/url?q=https://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article%3D1670%26context%3Dyjil&sa=D&source=editors&ust=1692801435169362&usg=AOvVaw1R9nxZDz3KStm8Xx8EwrgW)

[[620]](#ftnt_ref620) Matz-Lück, Nele. ‘Framework Conventions as a Regulatory Tool’. Goettingen Journal of International Law 3 (2009): 439–58.[https://doi.org/10.3249/1868-1581-1-3-MATZ-LUECK](https://www.google.com/url?q=https://doi.org/10.3249/1868-1581-1-3-MATZ-LUECK&sa=D&source=editors&ust=1692801435165629&usg=AOvVaw3nJL85ou5OJqGsRqjhor5I).

[[621]](#ftnt_ref621) Morin, Jean‐Frédéric, Hugo Dobson, Claire Peacock, Miriam Prys‐Hansen, Abdoulaye Anne, Louis Bélanger, Peter Dietsch, et al. ‘How Informality Can Address Emerging Issues: Making the Most of the G7’. Global Policy 10, no. 2 (May 2019): 267–73.[https://doi.org/10.1111/1758-5899.12668](https://www.google.com/url?q=https://doi.org/10.1111/1758-5899.12668&sa=D&source=editors&ust=1692801435164991&usg=AOvVaw2_mrd03T2GtWvq96yFXB4K).

[[622]](#ftnt_ref622) Israel, Brian. ‘Treaty Stasis’. AJIL Unbound 108 (ed 2014): 63–69.[https://doi.org/10.1017/S2398772300001860](https://www.google.com/url?q=https://doi.org/10.1017/S2398772300001860&sa=D&source=editors&ust=1692801435166254&usg=AOvVaw38YtjpInJT7yiZx1tOb6vk).

[[623]](#ftnt_ref623) Marchant, Gary E, and Yvonne A Stevens. ‘Resilience: A New Tool in the Risk Governance Toolbox for Emerging Technologies’. U.C. Davis Law Review 51, no. 1 (2017): 233–71. [https://lawreview.law.ucdavis.edu/issues/51/1/Symposium/51-1\_Marchant\_Stevens.pdf](https://www.google.com/url?q=https://lawreview.law.ucdavis.edu/issues/51/1/Symposium/51-1_Marchant_Stevens.pdf&sa=D&source=editors&ust=1692801435168007&usg=AOvVaw2E0q3n4h-XtFWYushTTUcf)

[[624]](#ftnt_ref624) Uuk, Risto. ‘General Purpose AI and the AI Act’. Future of Life Institute, May 2022.[https://artificialintelligenceact.eu/wp-content/uploads/2022/05/General-Purpose-AI-and-the-AI-Act.pdf](https://www.google.com/url?q=https://artificialintelligenceact.eu/wp-content/uploads/2022/05/General-Purpose-AI-and-the-AI-Act.pdf&sa=D&source=editors&ust=1692801435041942&usg=AOvVaw1Mnml8fT1lCf-jkQRvEVWT). See also: Gutierrez, Carlos Ignacio, Anthony Aguirre, Risto Uuk, Claire C. Boine, and Matija Franklin. ‘A Proposal for a Definition of General Purpose Artificial Intelligence Systems’. Future of Life Institute - Working Paper, 5 October 2022.[https://doi.org/10.2139/ssrn.4238951](https://www.google.com/url?q=https://doi.org/10.2139/ssrn.4238951&sa=D&source=editors&ust=1692801435042297&usg=AOvVaw0Pa1kyTCpTzAFI8UARxZea).

[[625]](#ftnt_ref625) See Canfil, Justin Key. ‘Yesterday’s Reach: How International Law Keeps Pace with Technological Change’. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, 2 January 2020.[https://papers.ssrn.com/abstract=3684991](https://www.google.com/url?q=https://papers.ssrn.com/abstract%3D3684991&sa=D&source=editors&ust=1692801435046640&usg=AOvVaw0FCxtHw8Cr70WG3bRng1FY).

[[626]](#ftnt_ref626) Koremenos, Barbara. The Continent of International Law: Explaining Agreement Design. Cambridge: Cambridge University Press, 2016.[https://doi.org/10.1017/CBO9781316415832](https://www.google.com/url?q=https://doi.org/10.1017/CBO9781316415832&sa=D&source=editors&ust=1692801435163306&usg=AOvVaw0gGZBqEpGeFr_Dy35MiWp5).; Helfer, Laurence R. ‘Flexibility in International Agreements’. In Interdisciplinary Perspectives on International Law and International Relations, edited by Jeffrey L. Dunoff and Mark A. Pollack, 175–96. Cambridge: Cambridge University Press, 2012.[https://doi.org/10.1017/CBO9781139107310.010](https://www.google.com/url?q=https://doi.org/10.1017/CBO9781139107310.010&sa=D&source=editors&ust=1692801435163821&usg=AOvVaw3UdlF1LF587yAZ2tv1WZ1e).; Boockmann, B., and Paul W. Thurner. ‘Flexibility Provisions in Multilateral Environmental Treaties’. International Environmental Agreements: Politics, Law and Economics 6, no. 2 (1 June 2006): 113–35.[https://doi.org/10.1007/s10784-006-9001-7](https://www.google.com/url?q=https://doi.org/10.1007/s10784-006-9001-7&sa=D&source=editors&ust=1692801435164335&usg=AOvVaw2_weLol2Ks7oGpemLVFZst).

[[627]](#ftnt_ref627) Búzás, Zoltán I, and Erin R Graham. ‘Emergent Flexibility in Institutional Development: How International Rules Really Change’. International Studies Quarterly 64, no. 4 (7 December 2020): 821–33.[https://doi.org/10.1093/isq/sqaa049](https://www.google.com/url?q=https://doi.org/10.1093/isq/sqaa049&sa=D&source=editors&ust=1692801435170028&usg=AOvVaw3PUKpZxWXUbch0eo_DJcPz).

[[628]](#ftnt_ref628) Dewey, Daniel. ‘Long-Term Strategies for Ending Existential Risk from Fast Takeoff’. In Risks of Artificial Intelligence. Chapman and Hall/CRC, 2015.[https://www.taylorfrancis.com/chapters/edit/10.1201/b19187-14/long-term-strategies-ending-existential-risk-fast-takeoff-daniel-dewey](https://www.google.com/url?q=https://www.taylorfrancis.com/chapters/edit/10.1201/b19187-14/long-term-strategies-ending-existential-risk-fast-takeoff-daniel-dewey&sa=D&source=editors&ust=1692801435134149&usg=AOvVaw0rw4XpFgIZm8-TppV39sen).

[[629]](#ftnt_ref629) Sotala, Kaj, and Roman Yampolskiy. ‘Responses to the Journey to the Singularity’. In The Technological Singularity, edited by Victor Callaghan, James Miller, Roman Yampolskiy, and Stuart Armstrong, 25–83. The Frontiers Collection. Berlin, Heidelberg: Springer Berlin Heidelberg, 2017.[https://doi.org/10.1007/978-3-662-54033-6\_3](https://www.google.com/url?q=https://doi.org/10.1007/978-3-662-54033-6_3&sa=D&source=editors&ust=1692801435427588&usg=AOvVaw0f8LDfhUNZAwpC9SMPHMN_).; and also Sotala, Kaj, and Roman V. Yampolskiy. ‘Responses to Catastrophic AGI Risk: A Survey.’ Technical Report. Berkeley, CA: Machine Intelligence Research Institute, 2013.[https://intelligence.org/files/ResponsesAGIRisk.pdf](https://www.google.com/url?q=https://intelligence.org/files/ResponsesAGIRisk.pdf&sa=D&source=editors&ust=1692801435428063&usg=AOvVaw1chSfLnqQik7HQoLl_35Jv).

[[630]](#ftnt_ref630) More detailed surveys of proposed AI interventions are given in: Maas, Matthijs. ‘Handbook on Approaches in Transformative AI Governance’; 2023 Forthcoming.

[[631]](#ftnt_ref631) Future of Life Institute. ‘Pause Giant AI Experiments: An Open Letter’. Future of Life Institute (blog), 30 March 2023.[https://futureoflife.org/open-letter/pause-giant-ai-experiments/](https://www.google.com/url?q=https://futureoflife.org/open-letter/pause-giant-ai-experiments/&sa=D&source=editors&ust=1692801435442351&usg=AOvVaw1m3iY6wN1xifBha1-24Y4b).

[[632]](#ftnt_ref632) Future of Life Institute. ‘Policymaking in the Pause: What Can Policymakers Do Now to Combat Risks from Advanced AI Systems?’ Future of Life Institute, 12 April 2023.[https://futureoflife.org/wp-content/uploads/2023/04/FLI\_Policymaking\_In\_The\_Pause.pdf](https://www.google.com/url?q=https://futureoflife.org/wp-content/uploads/2023/04/FLI_Policymaking_In_The_Pause.pdf&sa=D&source=editors&ust=1692801435441678&usg=AOvVaw1QOlW69OM21T9Ke8sNoUAq).

[[633]](#ftnt_ref633) Kemp, Luke, Peter Cihon, Matthijs Michiel Maas, Haydn Belfield, Zoe Cremer, Jade Leung, and Seán Ó hÉigeartaigh. ‘UN High-Level Panel on Digital Cooperation: A Proposal for International AI Governance’. Centre for the Study of Existential Risk and Leverhulme Centre for the Future of Intelligence, 26 February 2019.[https://www.cser.ac.uk/news/advice-un-high-level-panel-digital-cooperation/](https://www.google.com/url?q=https://www.cser.ac.uk/news/advice-un-high-level-panel-digital-cooperation/&sa=D&source=editors&ust=1692801435267325&usg=AOvVaw38Skbo_J0qa6IyQ98zIm21).

[[634]](#ftnt_ref634) Fischer, Sophie-Charlotte, and Andreas Wenger. ‘A Politically Neutral Hub for Basic AI Research’. Policy Perspectives. Zurich: CSS, ETH Zurich, March 2019.[http://www.css.ethz.ch/content/dam/ethz/special-interest/gess/cis/center-for-securities-studies/pdfs/PP7-2\_2019-E.pdf](https://www.google.com/url?q=http://www.css.ethz.ch/content/dam/ethz/special-interest/gess/cis/center-for-securities-studies/pdfs/PP7-2_2019-E.pdf&sa=D&source=editors&ust=1692801435268013&usg=AOvVaw0a2fyXHqu7bKANmltsp4vF).

[[635]](#ftnt_ref635) Korinek, Anton. ‘Why We Need a New Agency to Regulate Advanced Artificial Intelligence: Lessons on AI Control from the Facebook Files’. Brookings (blog), 8 December 2021.[https://www.brookings.edu/research/why-we-need-a-new-agency-to-regulate-advanced-artificial-intelligence-lessons-on-ai-control-from-the-facebook-files/](https://www.google.com/url?q=https://www.brookings.edu/research/why-we-need-a-new-agency-to-regulate-advanced-artificial-intelligence-lessons-on-ai-control-from-the-facebook-files/&sa=D&source=editors&ust=1692801435170837&usg=AOvVaw1Unfjm8PIlFMneanyM2N_a).

[[636]](#ftnt_ref636) Stix, Charlotte. ‘Foundations for the Future: Institution Building for the Purpose of Artificial Intelligence Governance’. AI and Ethics, 29 September 2021.[https://doi.org/10.1007/s43681-021-00093-w](https://www.google.com/url?q=https://doi.org/10.1007/s43681-021-00093-w&sa=D&source=editors&ust=1692801434952231&usg=AOvVaw2ofSXEpXK8MzS_OliRSZ51).

[[637]](#ftnt_ref637) Wilson, Grant. ‘Minimizing Global Catastrophic and Existential Risks from Emerging Technologies through International Law’. Va. Envtl. LJ 31 (2013): 307. [http://heinonline.org/hol-cgi-bin/get\_pdf.cgi?handle=hein.journals/velj31&section=12](https://www.google.com/url?q=http://heinonline.org/hol-cgi-bin/get_pdf.cgi?handle%3Dhein.journals/velj31%26section%3D12&sa=D&source=editors&ust=1692801435045542&usg=AOvVaw2Wc9jU84LY-Y5wj86tEHFs) ;  Carayannis, Elias G., and John Draper. ‘Optimising Peace through a Universal Global Peace Treaty to Constrain the Risk of War from a Militarised Artificial Superintelligence’. AI & SOCIETY, 11 January 2022.[https://doi.org/10.1007/s00146-021-01382-y](https://www.google.com/url?q=https://doi.org/10.1007/s00146-021-01382-y&sa=D&source=editors&ust=1692801435046054&usg=AOvVaw1WKLvJh5pSqfvdlweXQ7ns).

[[638]](#ftnt_ref638) Cihon, Peter, Jonas Schuett, and Seth D. Baum. ‘Corporate Governance of Artificial Intelligence in the Public Interest’. Information 12, no. 7 (July 2021): 275.[https://doi.org/10.3390/info12070275](https://www.google.com/url?q=https://doi.org/10.3390/info12070275&sa=D&source=editors&ust=1692801435263533&usg=AOvVaw2IPJI6ttPhqSpCYhc3fsn1).; Cihon, Peter, Moritz J. Kleinaltenkamp, Jonas Schuett, and Seth D. Baum. ‘AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries’. IEEE Transactions on Technology and Society 2, no. 4 (December 2021): 200–209.[https://doi.org/10.1109/TTS.2021.3077595](https://www.google.com/url?q=https://doi.org/10.1109/TTS.2021.3077595&sa=D&source=editors&ust=1692801435263950&usg=AOvVaw0rmdP6rDSg2EnnpWCz-a4i).

[[639]](#ftnt_ref639) Shavit, Yonadav. ‘What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training Via Compute Monitoring’, 2023.[https://paperswithcode.com/paper/what-does-it-take-to-catch-a-chinchilla](https://www.google.com/url?q=https://paperswithcode.com/paper/what-does-it-take-to-catch-a-chinchilla&sa=D&source=editors&ust=1692801435420060&usg=AOvVaw3Li13b899poyU-fM6OhrXd).

[[640]](#ftnt_ref640) Chan, Alan, Herbie Bradley, and Nitarshan Rajkumar. ‘Reclaiming the Digital Commons: A Public Data Trust for Training Data’. arXiv, 15 March 2023.[https://doi.org/10.48550/arXiv.2303.09001](https://www.google.com/url?q=https://doi.org/10.48550/arXiv.2303.09001&sa=D&source=editors&ust=1692801435419579&usg=AOvVaw3TQ11LZRGZd0iIL19EiAWe).

[[641]](#ftnt_ref641) Aguirre, Anthony. ‘Why Those Who Care about Catastrophic and Existential Risk Should Care about Autonomous Weapons’. EA Forum, 11 November 2020.[https://forum.effectivealtruism.org/posts/oR9tLNRSAep293rr5/why-those-who-care-about-catastrophic-and-existential-risk-2](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/oR9tLNRSAep293rr5/why-those-who-care-about-catastrophic-and-existential-risk-2&sa=D&source=editors&ust=1692801435264511&usg=AOvVaw0mYIMEBOvnN6U0k1hRnSPU).

[[642]](#ftnt_ref642) Kemp, Luke, Peter Cihon, Matthijs Michiel Maas, Haydn Belfield, Zoe Cremer, Jade Leung, and Seán Ó hÉigeartaigh. ‘UN High-Level Panel on Digital Cooperation: A Proposal for International AI Governance’. Centre for the Study of Existential Risk and Leverhulme Centre for the Future of Intelligence, 26 February 2019.[https://www.cser.ac.uk/news/advice-un-high-level-panel-digital-cooperation/](https://www.google.com/url?q=https://www.cser.ac.uk/news/advice-un-high-level-panel-digital-cooperation/&sa=D&source=editors&ust=1692801435265028&usg=AOvVaw2dl8oI_GdTWMusWtXvayBs).

[[643]](#ftnt_ref643) Jelinek, Thorsten, Wendell Wallach, and Danil Kerimi. ‘Policy Brief: The Creation of a G20 Coordinating Committee for the Governance of Artificial Intelligence’. AI and Ethics, 6 October 2020.[https://doi.org/10.1007/s43681-020-00019-y](https://www.google.com/url?q=https://doi.org/10.1007/s43681-020-00019-y&sa=D&source=editors&ust=1692801435265530&usg=AOvVaw0BT3zB7I__jEYgh2dfvthF).

[[644]](#ftnt_ref644) Moorhouse, Fin, and Avital Balwit. ‘Major UN Report Discusses Existential Risk and Future Generations (Summary)’. EA Forum, 17 September 2021.[https://forum.effectivealtruism.org/posts/Fwu2SLKeM5h5v95ww/major-un-report-discusses-existential-risk-and-future](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Fwu2SLKeM5h5v95ww/major-un-report-discusses-existential-risk-and-future&sa=D&source=editors&ust=1692801435266124&usg=AOvVaw0nGFN0zJJidDfGKdWylYqY).

[[645]](#ftnt_ref645) Belfield, Haydn. ‘The Rival AI Deployment Problem: A Pre-Deployment Agreement as the Least-Bad Response’. EA Forum, 23 September 2022.[https://forum.effectivealtruism.org/posts/uSH6DqjzggAYQGjxm/the-rival-ai-deployment-problem-a-pre-deployment-agreement](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/uSH6DqjzggAYQGjxm/the-rival-ai-deployment-problem-a-pre-deployment-agreement&sa=D&source=editors&ust=1692801435266692&usg=AOvVaw32e15c7frweYCwB2TWe4uV).

[[646]](#ftnt_ref646) Global Catastrophic Risk Policy. ‘Policy Ideas’. Accessed 6 December 2022.[https://www.gcrpolicy.com/ideas](https://www.google.com/url?q=https://www.gcrpolicy.com/ideas&sa=D&source=editors&ust=1692801435122535&usg=AOvVaw2n_wWvX-JyFbHF0_Lvnb3q).

[[647]](#ftnt_ref647) Dai, Wei. ‘AI Safety “Success Stories”’. AI Alignment Forum, 7 September 2019.[https://www.alignmentforum.org/posts/bnY3L48TtDrKTzGRb/ai-safety-success-stories](https://www.google.com/url?q=https://www.alignmentforum.org/posts/bnY3L48TtDrKTzGRb/ai-safety-success-stories&sa=D&source=editors&ust=1692801435066242&usg=AOvVaw0RK65AR8BOJN9ztqyLuz0b).

[[648]](#ftnt_ref648) Nanda, Neel. ‘My Overview of the AI Alignment Landscape: A Bird’s Eye View’. LessWrong, 16 December 2021.[https://www.lesswrong.com/posts/SQ9cZtfrzDJmw9A2m/my-overview-of-the-ai-alignment-landscape-a-bird-s-eye-view](https://www.google.com/url?q=https://www.lesswrong.com/posts/SQ9cZtfrzDJmw9A2m/my-overview-of-the-ai-alignment-landscape-a-bird-s-eye-view&sa=D&source=editors&ust=1692801435066857&usg=AOvVaw3bt0AtuyV5jnzparyO2dQf).

[[649]](#ftnt_ref649) Nanda, Neel. ‘A Longlist of Theories of Impact for Interpretability’. LessWrong, 11 March 2022.[https://www.lesswrong.com/posts/uK6sQCNMw8WKzJeCQ/a-longlist-of-theories-of-impact-for-interpretability](https://www.google.com/url?q=https://www.lesswrong.com/posts/uK6sQCNMw8WKzJeCQ/a-longlist-of-theories-of-impact-for-interpretability&sa=D&source=editors&ust=1692801435067491&usg=AOvVaw3ylIHlPaJoxlIHRlhtRKBR).

[[650]](#ftnt_ref650) Hubinger, Evan. ‘A Positive Case for How We Might Succeed at Prosaic AI Alignment’. LessWrong, 16 November 2021.[https://www.lesswrong.com/posts/5ciYedyQDDqAcrDLr/a-positive-case-for-how-we-might-succeed-at-prosaic-ai](https://www.google.com/url?q=https://www.lesswrong.com/posts/5ciYedyQDDqAcrDLr/a-positive-case-for-how-we-might-succeed-at-prosaic-ai&sa=D&source=editors&ust=1692801435068100&usg=AOvVaw3xbC64VGf4R41FvqvEHL3K).

[[651]](#ftnt_ref651) Dafoe, Allan. ‘AI Governance: Opportunity and Theory of Impact’, 17 September 2020.[https://www.allandafoe.com/opportunity](https://www.google.com/url?q=https://www.allandafoe.com/opportunity&sa=D&source=editors&ust=1692801434952809&usg=AOvVaw3grOa692LY4FVMtyHlXFDv). This model allows that there can be diverse views around which of the various assets, or what breadth of assets, are worth investing in today. Dafoe sketches a continuum between a narrow product model- and a broad field-building model of research, and argues that while there is much current emphasis on delivering concrete research projects, given the uncertainty over TAI’s technological trajectories and the prevailing political conditions around a future critical TAI moment, it is worth pursuing broad field-building activities for now. (“I believe the product model substantially underestimates the value of research in AI safety and, especially, AI governance; I estimate that the majority (perhaps ~80%) of the value of AI governance research comes from assets other than the narrow research product”).

[[652]](#ftnt_ref652) Leung, Jade. ‘How Can We See the Impact of AI Strategy Research?’ 2019.[https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Ae98k9d2gas32Yvmi/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research&sa=D&source=editors&ust=1692801434953648&usg=AOvVaw2HQ2Vxy83Gv4gJeeQz8it_).

[[653]](#ftnt_ref653) Ibid.

[[654]](#ftnt_ref654) Garfinkel, Benjamin. ‘AI Strategy: Pathways for Impact’. Accessed 6 April 2022.

[[655]](#ftnt_ref655) Seth Baum on AI Governance, 2021.[https://www.youtube.com/watch?v=G-8uEg7mCdA](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DG-8uEg7mCdA&sa=D&source=editors&ust=1692801434954412&usg=AOvVaw00nNIOCYMEpx1Pz6PUMkpV).

[[656]](#ftnt_ref656) Hobbhahn, Marius, Max Räuker, Yannick Mühlhäuser, Jasper Götting, and Simon Grimm. ‘What Success Looks Like’. Effective Altruism Forum, 28 June 2022.[https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like&sa=D&source=editors&ust=1692801434955262&usg=AOvVaw383mxVanzrgIFNRu5cBo0l).

[[657]](#ftnt_ref657) Karnofsky, Holden. ‘AI Strategy Nearcasting’. AI Alignment Forum, 25 August 2022.[https://www.alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting&sa=D&source=editors&ust=1692801434974266&usg=AOvVaw3_yEJ6t3ldptXctCtX7X1C). ("trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's.").

[[658]](#ftnt_ref658) Campos, Simon.‘AGI Timelines in Governance: Different Strategies for Different Timeframes’. EA Forum, 19 December 2022.[https://forum.effectivealtruism.org/posts/Pt7MxstXxXHak4wkt/agi-timelines-in-governance-different-strategies-for](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Pt7MxstXxXHak4wkt/agi-timelines-in-governance-different-strategies-for&sa=D&source=editors&ust=1692801435202225&usg=AOvVaw0Etk_9GUZ-doPamXZwnt_5).

[[659]](#ftnt_ref659) Stein-Perlman, Zach. ‘Framing AI Strategy’. AI Impacts, 6 February 2023.[https://aiimpacts.org/ framing-ai-strategy/](https://www.google.com/url?q=https://aiimpacts.org/framing-ai-strategy/&sa=D&source=editors&ust=1692801435361240&usg=AOvVaw09FI0-GnKJSFWmEECfUoE1).