What are some arguments for AI safety being less important?

Notes[[a]](#cmnt1)[[b]](#cmnt2)[[c]](#cmnt3)[[d]](#cmnt4)
=========================================================

Some of these arguments are substantially better than others, and some of the pieces here are rebuttals to worse arguments. Some of the best are in bold.

Also, the title of this document may be misleading, as many of these pieces are simply providing some ideas to consider, rather than giving a comprehensive and definitive argument.

It may be a useful exercise to consider how these arguments interact with arguments in[[e]](#cmnt5)[[f]](#cmnt6) various [introductions to AI safety](https://www.google.com/url?q=https://docs.google.com/document/d/1zx_WpcwuT3Stpx8GJJHcvJLSgv6dLje0eslVKvuk1yQ/edit%23&sa=D&source=editors&ust=1692801443558362&usg=AOvVaw2i65nFZBH3gvTVWw8lfsvX).

Author classification
=====================

* \* means the person was working ~full-time on [AI existential risk reduction](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence&sa=D&source=editors&ust=1692801443558907&usg=AOvVaw1miqan6DeTz2B3BKoVG2UQ) in some form
* ^ means the person was at least somewhat part of the AI existential risk community and/or related communities
* The asterisk or caret applies based on the date of publication. People only have an asterisk or caret if this designation was true reasonably close to the time of them publishing their critique. If someone critiques AI safety and then starts working on it 5 years later, they will not have an asterisk.
* Some classifications might be incorrect.

The list
========

* Some of the [reviews of "Is power-seeking AI an existential risk?"](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/GRv3KB2nPFRREXb5o/reviews-of-is-power-seeking-ai-an-existential-risk&sa=D&source=editors&ust=1692801443559568&usg=AOvVaw2uXfGayLfbO0iQ2mppVOzw) ( Joe Carlsmith\* ); see also [Carlsmith's report](https://www.google.com/url?q=https://arxiv.org/abs/2206.13353&sa=D&source=editors&ust=1692801443559793&usg=AOvVaw1oWrm_qe_Lg-ZXvPiYlKY1)
* [What do we think are the best arguments we’re wrong?](https://www.google.com/url?q=https://80000hours.org/problem-profiles/artificial-intelligence/%23best-arguments-against-this-problem-being-pressing&sa=D&source=editors&ust=1692801443560177&usg=AOvVaw2S8wF1avXSi6_mM8PtRmFm) and [Arguments against working on AI risk to which we think there are strong responses](https://www.google.com/url?q=https://80000hours.org/problem-profiles/artificial-intelligence/%23good-responses&sa=D&source=editors&ust=1692801443560442&usg=AOvVaw3h3rZhwMFJUoJu1X05wV3q) ( Benjamin Hilton\* )
* [Success without dignity: a nearcasting story of avoiding catastrophe by luck](https://www.google.com/url?q=https://www.lesswrong.com/posts/jwhcXmigv2LTrbBiB/success-without-dignity-a-nearcasting-story-of-avoiding&sa=D&source=editors&ust=1692801443560830&usg=AOvVaw00V_hd_G1-7N7Nv-tWQkm9) ( Holden Karnofsky\* )
* [My Objections to "We’re All Gonna Die with Eliezer Yudkowsky"](https://www.google.com/url?q=https://www.alignmentforum.org/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky&sa=D&source=editors&ust=1692801443561192&usg=AOvVaw0b_K8ggt4nD4HR5HJfNUos) ( Quintin Pope\* )
* [Counterarguments to the basic AI x-risk case](https://www.google.com/url?q=https://www.alignmentforum.org/posts/LDRQ5Zfqwi8GjzPYG/counterarguments-to-the-basic-ai-x-risk-case&sa=D&source=editors&ust=1692801443561583&usg=AOvVaw2IZftHt_NkqnkM818GEBVZ) ( Katja Grace\* ); also see [this response](https://www.google.com/url?q=https://www.alignmentforum.org/posts/GQat3Nrd9CStHyGaq/response-to-katja-grace-s-ai-x-risk-counterarguments&sa=D&source=editors&ust=1692801443561910&usg=AOvVaw32nvXvQJiV4Zg2BmjXmlfH))
* [Ben Garfinkel on scrutinising classic AI risk arguments](https://www.google.com/url?q=https://80000hours.org/podcast/episodes/ben-garfinkel-classic-ai-risk-arguments/&sa=D&source=editors&ust=1692801443562286&usg=AOvVaw1ebLhsaCeeSMGG_Xb1lek1) ( Rob Wiblin^, Ben Garfinkel\* ); also see [this slideshow](https://www.google.com/url?q=https://docs.google.com/presentation/d/1sHA3rwTHLIxyZPQObcw8mbNo2jffswH8uYV7N5PwqZE/edit%23slide%3Did.g6230db10d0_0_305&sa=D&source=editors&ust=1692801443562518&usg=AOvVaw2ed4s_a0vUGkBqhKIG99IF)
* [How big of a risk is misalignment?](https://www.google.com/url?q=https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/%23how-big-of-a-risk-is-misalignment&sa=D&source=editors&ust=1692801443562804&usg=AOvVaw1oFOHtPfhhOah8_dEg1otM) in “Why AI alignment could be hard with modern deep learning” ( Ajeya Cotra\* )
* [The Crux List](https://www.google.com/url?q=https://thezvi.substack.com/p/the-crux-list&sa=D&source=editors&ust=1692801443563091&usg=AOvVaw37oHRVLqBgI5LmPkln2IzG) + [To Predict What Happens, Ask What Happens](https://www.google.com/url?q=https://thezvi.substack.com/p/to-predict-what-happens-ask-what&sa=D&source=editors&ust=1692801443563296&usg=AOvVaw1H-VNU_WZqwl6KFtav3mwp) + [Stages of Survival](https://www.google.com/url?q=https://thezvi.substack.com/p/stages-of-survival&sa=D&source=editors&ust=1692801443563492&usg=AOvVaw3b2ReJYDqki57w9cp349Tf) ( Zvi Mowshowitz\* )
* [Does natural selection favor AIs over humans? Model this!](https://www.google.com/url?q=https://marginalrevolution.com/marginalrevolution/2023/04/does-natural-selection-favor-ais-over-humans-model-this.html&sa=D&source=editors&ust=1692801443563791&usg=AOvVaw20EMf4p9b5svfp3SUoma99) (Tyler Cowen)
* [Exaggerating the risks (Part 6: Introducing the Carlsmith report)](https://www.google.com/url?q=https://ineffectivealtruismblog.com/2023/04/08/exaggerating-risks-carlsmith-report/&sa=D&source=editors&ust=1692801443564110&usg=AOvVaw1Y7MzRNMNqSVfzICzvAMpq) + [part 7](https://www.google.com/url?q=https://ineffectivealtruismblog.com/2023/05/06/exaggerating-the-risks-part-7-carlsmith-on-instrumental-convergence/&sa=D&source=editors&ust=1692801443564350&usg=AOvVaw0827oUvYujd_lnV_V2-tds) + [part 8](https://www.google.com/url?q=https://ineffectivealtruismblog.com/2023/06/03/exaggerating-the-risks-part-8-carlsmith-wrap-up/&sa=D&source=editors&ust=1692801443564583&usg=AOvVaw2TgWNqnWYJ5WXpg0tFV39Y) ( David Thorstad^ )
* [What do XPT forecasts tell us about AI risk?](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/K2xQrrXn5ZSgtntuT/what-do-xpt-forecasts-tell-us-about-ai-risk-1&sa=D&source=editors&ust=1692801443564866&usg=AOvVaw1tKNYv6kfOJHjRR5HF1G1u) ( Forecasting Research Institute^, rosehadshar^ )
* [The bullseye framework: My case against AI doom](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/Nxtq2d8Xb3QuuHKE8/the-bullseye-framework-my-case-against-ai-doom&sa=D&source=editors&ust=1692801443565159&usg=AOvVaw3HV7eGUVPejsYpA4WK9XoP) ( titotal^ )
* [Where I agree and disagree with Eliezer](https://www.google.com/url?q=https://www.alignmentforum.org/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer&sa=D&source=editors&ust=1692801443565467&usg=AOvVaw2PLNgmYRYptVhqji3XZom0) ( Paul Christiano\* )
* [Frequent arguments about alignment](https://www.google.com/url?q=https://www.alignmentforum.org/posts/6ccG9i5cTncebmhsH/frequent-arguments-about-alignment&sa=D&source=editors&ust=1692801443565796&usg=AOvVaw3k0_eC4Sp9npdbX1WWXJFI) ( John Schulman^ )
* [Disagreements with the Yudkowskian future of AI](https://www.google.com/url?q=https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities?commentId%3D9ZhXbv8p2fr8mkXaa&sa=D&source=editors&ust=1692801443566165&usg=AOvVaw2yxEF9VgrGrS8miSe0F_cU) ( Matthew Barnett^ )
* [Passing the ideological Turing test](https://www.google.com/url?q=https://www.lesswrong.com/posts/tmCJyvaoRei4zSiJr/passing-the-ideological-turing-test-arguments-against&sa=D&source=editors&ust=1692801443566473&usg=AOvVaw2gJt8WQjZA5szMo8pkalv6) + [part 2](https://www.google.com/url?q=https://www.lesswrong.com/posts/q5GYAyXETnBATNCSw/arguments-against-existential-risk-from-ai-part-2%23comments&sa=D&source=editors&ust=1692801443566710&usg=AOvVaw2jgxavIGNnXSWJqIAqol9H) ( NinaR\* )
* [A tale of 2.5 orthogonality theses](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/kCAcrjvXDt2evMpBz/a-tale-of-2-75-orthogonality-theses&sa=D&source=editors&ust=1692801443566995&usg=AOvVaw18t8PXonNmYCPRkvGIr27t) ( Arepo )
* [Why transformative artificial intelligence is really, really hard to achieve](https://www.google.com/url?q=https://thegradient.pub/why-transformative-artificial-intelligence-is-really-really-hard-to-achieve/&sa=D&source=editors&ust=1692801443567310&usg=AOvVaw3Uhqtl9md2_QhJF-Kc6zl0) ( Arjun Ramani, Zhengdong Wang ); see [this reply](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/aXGDHeyhaep5sLzuG/link-post-interesting-shallow-round-up-of-reasons-to-be?commentId%3DBaK9HBmLjY57ev7MM&sa=D&source=editors&ust=1692801443567567&usg=AOvVaw2hDw1PEXeoacgmmvIpt4Pm)
* [Transformative AGI by 2043 is <1% likely](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely&sa=D&source=editors&ust=1692801443567894&usg=AOvVaw0U8TsQ2c90ZA1RHPRNWUlF) ( Ari Allyn-Feuer, Ted Sanders ); see [this comment](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely?commentId%3DB2oW9r5LjiWotEK2a&sa=D&source=editors&ust=1692801443568216&usg=AOvVaw3o4DUIMZVweBhYwZdjVO_K)
* [AI Doom and David Hume: A Defence of Empiricism in AI Safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/MDkYSuCzFbEgGgtAd/ai-doom-and-david-hume-a-defence-of-empiricism-in-ai-safety&sa=D&source=editors&ust=1692801443568582&usg=AOvVaw2bC4SSldn4vnFbRj3HbcfM) ( Matt Beard^ )
* [Order Matters for Deceptive Alignment](https://www.google.com/url?q=https://www.lesswrong.com/posts/CsjLDAhQat4PY6dsc/order-matters-for-deceptive-alignment-1&sa=D&source=editors&ust=1692801443568927&usg=AOvVaw0WD9aAaQO_JetCwsNMhO4H) ( DavidW^ )
* [[AN #80]: Why AI risk might be solved without additional intervention from longtermists](https://www.google.com/url?q=https://www.alignmentforum.org/posts/QknPz9JQTQpGdaWDp/an-80-why-ai-risk-might-be-solved-without-additional&sa=D&source=editors&ust=1692801443569356&usg=AOvVaw3hLCd-zdLjHrPPKO6yZWDd) ( Rohin Shah\* summarizing other people’s views )
* [But exactly how complex and fragile?](https://www.google.com/url?q=https://www.alignmentforum.org/posts/xzFQp7bmkoKfnae9R/but-exactly-how-complex-and-fragile&sa=D&source=editors&ust=1692801443569676&usg=AOvVaw1HbBINm77GkNJmXooYJVbX) ( Katja Grace\* )
* “[a long thread about why I'm personally not worried yet](https://www.google.com/url?q=https://twitter.com/WilliamAEden/status/1630690003830599680&sa=D&source=editors&ust=1692801443569974&usg=AOvVaw1_gGATHcm_iRJUytvQCsU_)” ( William Eden^ )
* [A Critique of AI Takeover Scenarios](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/j7X8nQ7YvvA7Pi4BX/a-critique-of-ai-takeover-scenarios&sa=D&source=editors&ust=1692801443570280&usg=AOvVaw2xS2Qhbx4j7CXmDXd1REuF) ( James Fodor^ )
* [Against a General Factor of Doom](https://www.google.com/url?q=https://aiimpacts.org/against-a-general-factor-of-doom/&sa=D&source=editors&ust=1692801443570590&usg=AOvVaw3npbQGaCeNg6C3E8iTbWs5) ( Jeffrey Heninger^ )
* [A list of good heuristics that the case for AI x-risk fails](https://www.google.com/url?q=https://www.alignmentforum.org/posts/bd2K3Jdz82csjCFob/a-list-of-good-heuristics-that-the-case-for-ai-x-risk-fails&sa=D&source=editors&ust=1692801443570946&usg=AOvVaw2zJG1qu-9qfCt7NbkMBRPT) ( David Krueger\* )
* [Concrete Reasons for Hope about AI](https://www.google.com/url?q=https://www.alignmentforum.org/posts/BfN88BfZQ4XGeZkda/concrete-reasons-for-hope-about-ai&sa=D&source=editors&ust=1692801443571265&usg=AOvVaw1i81sIbahymPnLv3QkYoBc) ( Zac Hatfield-Dodds\* )
* [My highly personal skepticism braindump on existential risk from artificial intelligence.](https://www.google.com/url?q=https://nunosempere.com/blog/2023/01/23/my-highly-personal-skepticism-braindump-on-existential-risk/&sa=D&source=editors&ust=1692801443571592&usg=AOvVaw2TqzpmHGcMdxX-w322atYN) ( Nuño Sempere^ ); see [Aaron Scher's summary](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/L6ZmggEJw8ri4KB8X/my-highly-personal-skepticism-braindump-on-existential-risk?commentId%3DhD4AvR8An2enTnQZC&sa=D&source=editors&ust=1692801443571952&usg=AOvVaw0q5I-Q2GjCe_MeYnnZkqf8)
* [AI will change the world, but won’t take it over by playing “3-dimensional chess”.](https://www.google.com/url?q=https://www.alignmentforum.org/posts/zB3ukZJqt3pQDw9jz/ai-will-change-the-world-but-won-t-take-it-over-by-playing-3&sa=D&source=editors&ust=1692801443572335&usg=AOvVaw0Q67dFLEZyua_FAlXc2q3P) ( Boaz Barak, Ben Edelman ); see [these comments](https://www.google.com/url?q=https://www.lesswrong.com/posts/zB3ukZJqt3pQDw9jz/ai-will-change-the-world-but-won-t-take-it-over-by-playing-3%23comments&sa=D&source=editors&ust=1692801443572632&usg=AOvVaw3d8QJvp2XpGvxIZeySrrJI)
* [Arguments for AI Risk](https://www.google.com/url?q=https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review%23Arguments_for_AI_risk&sa=D&source=editors&ust=1692801443572929&usg=AOvVaw3g0cFeQKd4chsAMRgYHofj) and[Arguments against AI risk](https://www.google.com/url?q=https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review%23Arguments_against_AI_risk&sa=D&source=editors&ust=1692801443573374&usg=AOvVaw1jUDkzql1nkv-VCyO4fphf) sections of "AI Alignment 2018-19 Review." ( Rohin Shah\* )
* [Reasons I’ve been hesitant about high levels of near-ish AI risk](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/5hprBzprm7JjJTHNX/reasons-i-ve-been-hesitant-about-high-levels-of-near-ish-ai-1&sa=D&source=editors&ust=1692801443573729&usg=AOvVaw3s25p0d99mksmOc2oW9Lwf) ( Eli Lifland\*^ )
* [AI Risk Skepticism](https://www.google.com/url?q=https://arxiv.org/abs/2105.02704&sa=D&source=editors&ust=1692801443573996&usg=AOvVaw1phr8P8V49jV76PcaR_m2s) ( Roman Yampolskiy\* )
* [How sure are we about this AI stuff?](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/9sBAW3qKppnoG3QPq/ben-garfinkel-how-sure-are-we-about-this-ai-stuff&sa=D&source=editors&ust=1692801443574358&usg=AOvVaw0hJOFw__4jTM4bN_sWReWX) ( Ben Garfinkel\* )
* [On Deference and Yudkowsky's AI Risk Estimates](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/NBgpPaz5vYe3tH4ga/on-deference-and-yudkowsky-s-ai-risk-estimates&sa=D&source=editors&ust=1692801443574766&usg=AOvVaw3TjGYljgS1U4bXtsXsFe5k) ( Ben Garfinkel\* )
* [The hot mess theory of AI misalignment: More intelligent agents behave less coherently](https://www.google.com/url?q=https://sohl-dickstein.github.io/2023/03/09/coherence.html&sa=D&source=editors&ust=1692801443575127&usg=AOvVaw0Y2KwGfJBQGGnZUn3WlkiQ) ( Jascha Sohl-Dickstein ); see [this response](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/CaCWfjGHdw7H4o6XL/notes-on-the-hot-mess-theory-of-ai-misalignment&sa=D&source=editors&ust=1692801443575430&usg=AOvVaw1vldc_GoRUamyPzqvc39WQ)
* [Artificial superintelligence and its limits: why AlphaZero cannot become a general agent](https://www.google.com/url?q=https://link.springer.com/article/10.1007/s00146-020-01070-3&sa=D&source=editors&ust=1692801443575715&usg=AOvVaw2mH2zJWdrNCNq8B_LGSj1y) ( Karim Jebari, Joakim Lundborg ); see [this response](https://www.google.com/url?q=https://www.alignmentforum.org/posts/rokpjK3jcy5aKKwiT/reply-to-jebari-and-lundborg-on-artificial-superintelligence-1&sa=D&source=editors&ust=1692801443576049&usg=AOvVaw01Cy15Oqvxky3r1OrmQ-vA)
* [Existential risk from artificial general intelligence: Skepticism](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence%23Skepticism&sa=D&source=editors&ust=1692801443576442&usg=AOvVaw1RZqvhz9m5geCAc3NEBGT9) ( Wikipedia )
* [Reframing Superintelligence: Comprehensive AI Services as General Intelligence](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/reframing/&sa=D&source=editors&ust=1692801443576927&usg=AOvVaw3P8kt216tG1M_0xN375RFQ) ( K Eric Drexler^ )
* [We Aren't Close To Creating A Rapidly Self-Improving AI](https://www.google.com/url?q=https://jacobbuckman.substack.com/p/we-arent-close-to-creating-a-rapidly&sa=D&source=editors&ust=1692801443577316&usg=AOvVaw1LfKKN2IIdjFobjnlLT1UC) ( Jacob Buckman )
* [AI Risk, Again](https://www.google.com/url?q=https://www.overcomingbias.com/p/ai-risk-again&sa=D&source=editors&ust=1692801443577666&usg=AOvVaw2DJAyRd9ty3QsoptDLg3B1) + [I Still Don't Get Foom](https://www.google.com/url?q=https://www.overcomingbias.com/p/30855html&sa=D&source=editors&ust=1692801443577881&usg=AOvVaw21e6xTR2pTytd4Wdmmhmun) + [How Does Brain Code Differ?](https://www.google.com/url?q=https://www.overcomingbias.com/p/how-does-brain-code-differhtml&sa=D&source=editors&ust=1692801443578126&usg=AOvVaw1ZjE8tVsydMlr3mf9-Ibo5) ( Robin Hanson^ )
* [What Are Reasonable AI Fears?](https://www.google.com/url?q=https://quillette.com/2023/04/14/what-are-reasonable-ai-fears/&sa=D&source=editors&ust=1692801443578431&usg=AOvVaw0aveI1uOdjhzRce5qHUaCQ) ( Robin Hanson^ ); see [this response](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/XnnfPC2gsgRFZezkE/linkpost-what-are-reasonable-ai-fears-by-robin-hanson-2023?commentId%3Db7upK9it4S7uBrkkC&sa=D&source=editors&ust=1692801443578688&usg=AOvVaw0ob0FuLX9oVK3y1GjT5zae)
* [Agency Failure AI Apocalypse?](https://www.google.com/url?q=https://www.overcomingbias.com/p/agency-failure-ai-apocalypsehtml&sa=D&source=editors&ust=1692801443578960&usg=AOvVaw3e43AHaFNTH9I4RQOAq540) ( Robin Hanson^ ); see[this response](https://www.google.com/url?q=https://www.lesswrong.com/posts/Z5ZBPEgufmDsm7LAv/what-can-the-principal-agent-literature-tell-us-about-ai&sa=D&source=editors&ust=1692801443579414&usg=AOvVaw1KzDnB62lLjNchNLn4pCBS)
* “I Object!” section of [Extinction Risk from Artificial Intelligence](https://www.google.com/url?q=https://aisafety.wordpress.com/&sa=D&source=editors&ust=1692801443579686&usg=AOvVaw3aHhlrwpZowS_adKbqGsug) ( Michael Cohen\* )
* [Blake Richards on Why he is Skeptical of Existential Risk from AI](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/BqokXcCQrvkk2BktH/blake-richards-on-why-he-is-skeptical-of-existential-risk&sa=D&source=editors&ust=1692801443579990&usg=AOvVaw2v5OjH_orLIa_k1CsTXPSy) on [The Inside View](https://www.google.com/url?q=https://theinsideview.ai/blake&sa=D&source=editors&ust=1692801443580253&usg=AOvVaw1890xhb7X2nKB0EuaskVPZ) (Blake Richards, Michaël Trazzi\* )
* [Gary Marcus and Stuart Russell discuss AI risk on the Sam Harris podcast](https://www.google.com/url?q=https://www.samharris.org/podcasts/making-sense-episodes/312-the-trouble-with-ai&sa=D&source=editors&ust=1692801443580549&usg=AOvVaw3p-eNm9VMeF3xKRYS1vsX8) ( Gary Marcus, Sam Harris )
* [Alignment By Default](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Nwgdq6kHke5LY692J/alignment-by-default&sa=D&source=editors&ust=1692801443580814&usg=AOvVaw1krUOYRnQh-9go3YzeJjYg) ( John Wentworth\* )
* [Melanie Mitchell and Stuart Russell debate](https://www.google.com/url?q=https://munkdebates.com/podcast/the-rise-of-thinking-machines&sa=D&source=editors&ust=1692801443581089&usg=AOvVaw1EVs3Kg3Pbkf-hu_aTY3cf) for Munk Debates ( Melanie Mitchell, Stuart Russell\* )
* [Why AI is Harder Than We Think](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/C94JhsbSfZ8iPNedy/why-ai-is-harder-than-we-think-melanie-mitchell&sa=D&source=editors&ust=1692801443581410&usg=AOvVaw2hD_koGS-T4vVJPoD9bk3n) ( Melanie Mitchell ); also see [Richard Ngo's comment](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/C94JhsbSfZ8iPNedy/why-ai-is-harder-than-we-think-melanie-mitchell?commentId%3DmkNwocFphweG8hHrE&sa=D&source=editors&ust=1692801443581690&usg=AOvVaw07Q7mDRx63yhXEGHwQ1Nkb)
* [A shift in arguments for AI risk](https://www.google.com/url?q=https://fragile-credences.github.io/prioritising-ai/&sa=D&source=editors&ust=1692801443582001&usg=AOvVaw3s_9gEPDhoGsyFE-p6KW7j) ( Tom Adamczewski )
* [How to navigate the AI apocalypse as a sane person](https://www.google.com/url?q=https://erikhoel.substack.com/p/how-to-navigate-the-ai-apocalypse&sa=D&source=editors&ust=1692801443582307&usg=AOvVaw183hUCMkS8TRPDUBCjKCGT) ( Eric Hoel )
* [Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More](https://www.google.com/url?q=https://www.alignmentforum.org/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell&sa=D&source=editors&ust=1692801443582602&usg=AOvVaw1hsO5CtSuutcDrSjybPZ_A) ( Yann LeCun, Stuart Russell\*, Yoshua Bengio, Tony Zador, and more )
* [Why I Am Not (As Much Of) A Doomer (As Some People)](https://www.google.com/url?q=https://astralcodexten.substack.com/p/why-i-am-not-as-much-of-a-doomer&sa=D&source=editors&ust=1692801443582870&usg=AOvVaw1K8SqZEKzhC9HkWhGUMggl) ( Scott Alexander^ )
* [My Bet: AI Solves Flubs](https://www.google.com/url?q=https://astralcodexten.substack.com/p/my-bet-ai-size-solves-flubs&sa=D&source=editors&ust=1692801443583198&usg=AOvVaw1Nsz_Bz3kL8Qj8rfDMK2F9) ( Scott Alexander^ )
* [AI Researchers On AI Risk](https://www.google.com/url?q=https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/&sa=D&source=editors&ust=1692801443583465&usg=AOvVaw3jITDQd8oCu4VxlcLqUizU) ( Scott Alexander^ )
* [Contra Acemoglu On...Oh God, We're Doing This Again, Aren't We?](https://www.google.com/url?q=https://astralcodexten.substack.com/p/contra-acemoglu-onoh-god-were-doing&sa=D&source=editors&ust=1692801443583745&usg=AOvVaw3NooTFUfRlmsQ3mjWI0tAR) + [highlights from the comments](https://www.google.com/url?q=https://astralcodexten.substack.com/p/highlights-from-the-comments-on-acemoglu&sa=D&source=editors&ust=1692801443583993&usg=AOvVaw1O6k7BJoj-jsaUvCQKG1rC) ( Scott Alexander^ responding to Daron Acemoglu )
* [Maybe The Real Superintelligent AI Is Extremely Smart Computers](https://www.google.com/url?q=https://slatestarcodex.com/2018/01/15/maybe-the-real-superintelligent-ai-is-extremely-smart-computers/&sa=D&source=editors&ust=1692801443584292&usg=AOvVaw0jl122PKkMVlOKyG2dFQHq) ( Scott Alexander^ responding to Ted Chiang )
* [10 Reasons to Ignore AI Safety](https://www.google.com/url?q=https://www.youtube.com/watch?v%3D9i1WlcCudpU&sa=D&source=editors&ust=1692801443584644&usg=AOvVaw3PbeyvEf_qvdAtdijJSk0y) ( Rob Miles\* )
* [A Response to Steven Pinker on AI](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DyQE9KAbFhNY&sa=D&source=editors&ust=1692801443584930&usg=AOvVaw2BECWP9PY36hRRmRO2zILA) ( Rob Miles\* )
* [How much EA analysis of AI safety as a cause area exists?](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/u3ePLsbtpkmFdD7Nb/how-much-ea-analysis-of-ai-safety-as-a-cause-area-exists-1&sa=D&source=editors&ust=1692801443585298&usg=AOvVaw3Mnw0MWIvPmCnLLjvl57b7) ( Richard Ngo\* )
* [Heretical Thoughts on AI | Eli Dourado](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/9XGhNArGhiZ3daQws/heretical-thoughts-on-ai-or-eli-dourado&sa=D&source=editors&ust=1692801443585609&usg=AOvVaw0kRbnuKoUCQ1fqjmT-iqLr) ( Cinera^ )
* [Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment](https://www.google.com/url?q=https://www.alignmentforum.org/posts/vT4tsttHgYJBoKi4n/some-abstract-non-technical-reasons-to-be-non-maximally&sa=D&source=editors&ust=1692801443585908&usg=AOvVaw2zSxexmXpz64elZNDBtUeA) ( Rob Bensinger\* )
* [Can a Paperclip Maximizer Overthrow the CCP?](https://www.google.com/url?q=https://richardhanania.substack.com/p/can-a-paperclip-maximizer-overthrow&sa=D&source=editors&ust=1692801443586200&usg=AOvVaw3S-S103NFMJOJZzbLcWrLq) + [Pinker on Alignment and Intelligence as a "Magical Potion"](https://www.google.com/url?q=https://richardhanania.substack.com/p/pinker-on-alignment-and-intelligence&sa=D&source=editors&ust=1692801443586405&usg=AOvVaw1wuerxxzlVDQEFKjFgjThx) ( Richard Hanania )
* [Superintelligence: The Idea That Eats Smart People](https://www.google.com/url?q=https://idlewords.com/talks/superintelligence.htm&sa=D&source=editors&ust=1692801443586687&usg=AOvVaw2mlhf9qAMm8mivx2m8tmcD) + [video](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DkErHiET5YPw&sa=D&source=editors&ust=1692801443586899&usg=AOvVaw1-WyKDqvPW5miZ5LxFOFm8) ( Maciej Ceglowski )
* [How I failed to form views on AI safety](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/ST3JjsLdTBnaK46BD/how-i-failed-to-form-views-on-ai-safety-3&sa=D&source=editors&ust=1692801443587180&usg=AOvVaw2p6ztJJq27VOg2ZEIG7Z70) ( Ada-Maaria Hyvärinen^ )
* [There are no coherence theorems](https://www.google.com/url?q=https://www.alignmentforum.org/posts/yCuzmCsE86BTu9PfA/there-are-no-coherence-theorems&sa=D&source=editors&ust=1692801443587438&usg=AOvVaw1W3T8EDofbkNZEJSGzzCy1) ( Elliott Thornley\* )
* [The Singularity is not coming](https://www.google.com/url?q=https://fchollet.com/blog/the-singularity-is-not-coming.html&sa=D&source=editors&ust=1692801443587684&usg=AOvVaw2rL5DIVXrMNZQ1PBNgd2XN), [On the Measure of Intelligence](https://www.google.com/url?q=https://arxiv.org/abs/1911.01547&sa=D&source=editors&ust=1692801443587851&usg=AOvVaw0jPmzHqVuYff5Yge_-WQrw), [The implausibility of intelligence explosion](https://www.google.com/url?q=https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec&sa=D&source=editors&ust=1692801443588096&usg=AOvVaw2oXz8lYJ3Mm_G5yAzEwwIT) ( François Chollet ); also see [Eliezer Yudkowsky’s reply to the last piece](https://www.google.com/url?q=https://intelligence.org/2017/12/06/chollet/&sa=D&source=editors&ust=1692801443588294&usg=AOvVaw2E4pX_z6M-ulpjP--ifV49)
* ["Bad AI DontKillEveryoneism Takes"](https://www.google.com/url?q=https://thezvi.substack.com/p/ai-3%23%25C2%25A7bad-ai-dontkilleveryoneism-takes&sa=D&source=editors&ust=1692801443588567&usg=AOvVaw0v1ulOpshMAIxkbsi_Untp) section of [AI #3](https://www.google.com/url?q=https://thezvi.substack.com/p/ai-3&sa=D&source=editors&ust=1692801443588765&usg=AOvVaw1-f1g6e3S8Rr4hTgnzoj_3) ( Zvi Mowshowitz\* )
* [Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment](https://www.google.com/url?q=https://www.alignmentforum.org/posts/vT4tsttHgYJBoKi4n/some-abstract-non-technical-reasons-to-be-non-maximally&sa=D&source=editors&ust=1692801443589082&usg=AOvVaw2aEN52fNvKXUzxvqHUxtkQ) ( Rob Bensinger\* )
* [Bad alignment take bingo with replies](https://www.google.com/url?q=https://twitter.com/robbensinger/status/1503220020175769602&sa=D&source=editors&ust=1692801443589362&usg=AOvVaw0lt68YMyRZSh22IlCRjkfG) ( Rob Bensinger\* ); also see the card [“with some 2023 additions”](https://www.google.com/url?q=https://twitter.com/carad0/status/1632427388230422529&sa=D&source=editors&ust=1692801443589553&usg=AOvVaw3e1zspumPYhFtndnR6zvS6); note these bingo cards are mostly designed to be funny, although they do contain some core ideas of various arguments
* [The Preference Fulfillment Hypothesis](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Kf6sKZudduhJmykTg/the-preference-fulfillment-hypothesis&sa=D&source=editors&ust=1692801443589882&usg=AOvVaw3aZBEGrQWi8bXio2YX5yvN) ( Kaj Sotala^ )
* [How Organisms Come to Know the World: Fundamental Limits on Artificial General Intelligence](https://www.google.com/url?q=https://www.frontiersin.org/articles/10.3389/fevo.2021.806283/full&sa=D&source=editors&ust=1692801443590156&usg=AOvVaw3A1X8MiUu0ZXSQlozqAqsL) ( Andrea Roli, Johannes Jaeger, Stuart Kauffman )
* [Don’t Worry about Superintelligence](https://www.google.com/url?q=https://jetpress.org/v26.1/agar.htm&sa=D&source=editors&ust=1692801443590391&usg=AOvVaw1iNzp8yDbbfm_zNK7RM5dT) ( Nicholas Agar )
* [Where I'm at with AI risk: convinced of danger but not (yet) of doom](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/fXkCcsyF8M6dp6sXx/where-i-m-at-with-ai-risk-convinced-of-danger-but-not-yet-of&sa=D&source=editors&ust=1692801443590754&usg=AOvVaw2X7r88H4aGOLtKKsSPYpMx) ( Amber Dawn^ )
* [How Much Should You Freak out About AI?](https://www.google.com/url?q=https://fakenous.substack.com/p/how-much-should-you-freak-out-about&sa=D&source=editors&ust=1692801443591087&usg=AOvVaw12endGdeUJmd6T77R_64zv) ( Michael Huemer )
* [Reasons you might think human-level AI is unlikely to happen soon](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/kJzPDbgmA8nrLqTgH/asya-bergal-reasons-you-might-think-human-level-ai-is&sa=D&source=editors&ust=1692801443591435&usg=AOvVaw1iJXA-ZT8g0AYloTFirTmN) ( Asya Bergal\* )
* [Existential risk, AI, and the inevitable turn in human history](https://www.google.com/url?q=https://marginalrevolution.com/marginalrevolution/2023/03/existential-risk-and-the-turn-in-human-history.html&sa=D&source=editors&ust=1692801443591731&usg=AOvVaw3gWJN1nV-NZm7AsifQd26F) ( Tyler Cowen ); also see [this response from Scott Alexander](https://www.google.com/url?q=https://astralcodexten.substack.com/p/mr-tries-the-safe-uncertainty-fallacy&sa=D&source=editors&ust=1692801443591930&usg=AOvVaw1wx3SJ9AvuCuBhVl8vrKFJ) (and read the comments on it from Tyler and Scott) + [this response from Zvi Mowshowitz](https://www.google.com/url?q=https://thezvi.substack.com/p/response-to-tyler-cowens-existential&sa=D&source=editors&ust=1692801443592172&usg=AOvVaw3KWhOQrw2NVzDiXaU5EZti) + [this response from Leopold Aschenbrenner](https://www.google.com/url?q=https://www.forourposterity.com/response-to-tyler-cowen-on-ai-risk/&sa=D&source=editors&ust=1692801443592374&usg=AOvVaw3aV0loqYmRD81_uhiLu3r9))
* [Don’t Fear the Terminator](https://www.google.com/url?q=https://blogs.scientificamerican.com/observations/dont-fear-the-terminator/&sa=D&source=editors&ust=1692801443592703&usg=AOvVaw3pRYqxyMSXhb6US6_FAVRE) ( Yann LeCun, Anthony Zador )
* [Is Human Intelligence Simple? Part 3: Disambiguating Types of Simplicity](https://www.google.com/url?q=https://sarahconstantin.substack.com/p/is-human-intelligence-simple-part-75c&sa=D&source=editors&ust=1692801443592995&usg=AOvVaw3NVzFGK0hClTx-kV-Vwg5j) ( Sarah Constantin^ )
* [More AI debate between me and Steven Pinker!](https://www.google.com/url?q=https://scottaaronson.blog/?p%3D6593&sa=D&source=editors&ust=1692801443593234&usg=AOvVaw3LGoaMSnpVSuZYBwfIn44O) + [Steven Pinker and I debate AI scaling!](https://www.google.com/url?q=https://scottaaronson.blog/?p%3D6524&sa=D&source=editors&ust=1692801443593403&usg=AOvVaw2e2aHnCmPfp0GAqeEo5ZPo) + [Reform AI Alignment](https://www.google.com/url?q=https://scottaaronson.blog/?p%3D6821&sa=D&source=editors&ust=1692801443593572&usg=AOvVaw054DwfhZ99npBiKu6FlLNj) ( Scott Aaronson^ )
* [The Prospect of an AI Winter](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/i7aKatsck7x3aLoiH/the-prospect-of-an-ai-winter&sa=D&source=editors&ust=1692801443593852&usg=AOvVaw0CzxirELV_5ajtlj52diZp) ( Erich Grunewald )
* [Alan Chan discussing alignment with Tim Scarfe of Machine Learning Street Talk](https://www.google.com/url?q=https://www.youtube.com/watch?v%3DXBMnOsv9_pk&sa=D&source=editors&ust=1692801443594128&usg=AOvVaw0eVAymMIqRCH_3QkBu5FRe) ( Alan Chan\*, Tim Scarfe )
* [Against AI Doomerism, For AI Progress](https://www.google.com/url?q=https://maxmore.substack.com/p/against-ai-doomerism-for-ai-progress&sa=D&source=editors&ust=1692801443594425&usg=AOvVaw3j4qo2VTIsXLeplYl6Y5jl) ( Max More )
* [What are some objections to the importance of AI alignment](https://www.google.com/url?q=https://aisafety.info?state%3D7794_&sa=D&source=editors&ust=1692801443594705&usg=AOvVaw13bX865hqieZEF7NMVjVz2)? ( Søren Elverlin\* )
* [But Have They Engaged with the Arguments?](https://www.google.com/url?q=https://philiptrammell.com/blog/46&sa=D&source=editors&ust=1692801443595073&usg=AOvVaw3d3QSX0z4Na7oM1uSXDBJM) ( Philip Trammell^ )
* [On The Impossibility of AI Alignment](https://www.google.com/url?q=https://lacker.io/ai/2023/04/03/on-the-impossibility-of-ai-alignment.html&sa=D&source=editors&ust=1692801443595378&usg=AOvVaw05tJE0hQQZYfjU2WioEyMK) ( Kevin Lacker )
* [How to know if artificial intelligence is about to destroy civilization](https://www.google.com/url?q=https://www.technologyreview.com/2020/02/25/906083/artificial-intelligence-destroy-civilization-canaries-robot-overlords-take-over-world-ai/&sa=D&source=editors&ust=1692801443595700&usg=AOvVaw3De8WtxdG4M4U0CQuVB2HK) ( Oren Etzioni ); see [this summary](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/7x2BokkkemjnXD9B6/new-article-from-oren-etzioni&sa=D&source=editors&ust=1692801443595935&usg=AOvVaw3t-xYRfcykAxqjSGhXvg0H)
* [The AI Messiah](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/r72wjMns9wyaAhWhc/the-ai-messiah&sa=D&source=editors&ust=1692801443596210&usg=AOvVaw0BJZf-NculYS9GSVbr7azx) ( ryancbriggs^ )
* [How good is humanity at coordination?](https://www.google.com/url?q=https://www.lesswrong.com/posts/y3jDSoTTdBD9Nj3Gx/how-good-is-humanity-at-coordination&sa=D&source=editors&ust=1692801443596625&usg=AOvVaw1cJHmHK1H66aqfzYnB-2qC) ( Buck Shlegeris\* )
* [The Orthogonality Thesis is Not Obviously True](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/e2dK25iWou3irqFss/the-orthogonality-thesis-is-not-obviously-true&sa=D&source=editors&ust=1692801443597024&usg=AOvVaw2T7NFoSMU1Q908Hu1J3njy) ( Omnizoid^ )
* [Superintelligence Is Not Omniscience](https://www.google.com/url?q=https://aiimpacts.org/superintelligence-is-not-omniscience/&sa=D&source=editors&ust=1692801443597338&usg=AOvVaw3a1ipTyiyWaU8WuOjp2LWJ) ( Jeffrey Heninger^, Aysja Johnson)
* [Memes that caricature some common counterarguments](https://www.google.com/url?q=https://twitter.com/AISafetyMemes/status/1647308848506744833&sa=D&source=editors&ust=1692801443597682&usg=AOvVaw0zuISLZeD2aaAMiANGGqH6) (these are obviously misleading and lacking nuance since they are memes, but they are pretty fun)
* [High-level hopes for AI alignment](https://www.google.com/url?q=https://www.cold-takes.com/high-level-hopes-for-ai-alignment/&sa=D&source=editors&ust=1692801443598010&usg=AOvVaw0SFgqW0lYFkTUvRQ5vbi2P) ( Holden Karnofsky\* )
* [Response to e/acc arguments](https://www.google.com/url?q=https://twitter.com/DanHendrycks/status/1651740865159901184&sa=D&source=editors&ust=1692801443598341&usg=AOvVaw05kK0_vU65atYtp4Zo7Wjl) ( Dan Hendrycks\* )
* [The 'Wild' and 'Wacky' Claims of Karnofsky’s ‘Most Important Century’](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/TCsanzwKGqfBBTye9/the-wild-and-wacky-claims-of-karnofsky-s-most-important&sa=D&source=editors&ust=1692801443598742&usg=AOvVaw2FMXijix0O5Jg8uPR4ZjqC) ( Spencer Becker-Kahn\* )
* [The 'Don't Look Up' Thinking That Could Doom Us With AI](https://www.google.com/url?q=https://time.com/6273743/thinking-that-could-doom-us-with-ai/&sa=D&source=editors&ust=1692801443599003&usg=AOvVaw15tDIslY62pAJV2uDv9K-L) ( Max Tegmark\* )
* [Why do some AI researchers dismiss the potential risks to humanity?](https://www.google.com/url?q=https://www.newscientist.com/article/2369626-why-do-some-ai-researchers-dismiss-the-potential-risks-to-humanity/&sa=D&source=editors&ust=1692801443599442&usg=AOvVaw1SKgMQjgpuNTwGVGswhrm6) ( David Krueger\* ); paywalled
* [The “General Problem Solver” Does Not Exist: Mortimer Taube and the Art of AI Criticism](https://www.google.com/url?q=https://www.shunryugarvey.com/wp-content/uploads/2021/03/Garvey_2021_The-General-Problem-Solver-Does-Not-Exist-1.pdf&sa=D&source=editors&ust=1692801443599904&usg=AOvVaw0R0SNzBUF3QIpXRc5c1qRo) ( Shunryu Colin Garvey )
* [Unsavory medicine for technological civilization: Introducing ‘Artificial Intelligence & its Discontents’](https://www.google.com/url?q=https://www.shunryugarvey.com/wp-content/uploads/2021/03/YISR_I_46_1-2_TEXT_P-1.pdf&sa=D&source=editors&ust=1692801443600199&usg=AOvVaw3OySb7_3EqkoUhSeQppD5V) ( Shunryu Colin Garvey )
* [Is Superintelligence Impossible?](https://www.google.com/url?q=https://www.edge.org/conversation/john_brockman-is-superintelligence-impossible&sa=D&source=editors&ust=1692801443600528&usg=AOvVaw29jDa7r0gWqAgQT5xeOzGR) ( Daniel Dennett and David Chalmers)
* [“Literally every conversation I have on Twitter about long-term risk leaves me more worried than when I started.”](https://www.google.com/url?q=https://twitter.com/GaryMarcus/status/1657420851556519936&sa=D&source=editors&ust=1692801443600856&usg=AOvVaw2kOiN_j2m30DrTEoPr6zUf) ( Gary Marcus )
* [Steelman arguments against the idea that AGI is inevitable and will arrive soon](https://www.google.com/url?q=https://www.lesswrong.com/posts/3kijTbgfizDgSgst3/steelman-arguments-against-the-idea-that-agi-is-inevitable&sa=D&source=editors&ust=1692801443601304&usg=AOvVaw0TH4WWXp4g7qxpncXaNkPL) ( RomanS^ )

Other collections
=================

* ["Alternate Perspectives" in STS 10SI: Intro to AI Alignment](https://www.google.com/url?q=https://docs.google.com/document/d/1NX0DlZRzD3NP7tBeLjMh76w7-w2s8SxV3wj0P7EYpKY/edit%23heading%3Dh.3bexxn1cvdu&sa=D&source=editors&ust=1692801443601891&usg=AOvVaw2ewidj8r56leQtm5642yLg) ( Stanford AI Alignment )
* [Object-level AI risk skepticism](https://www.google.com/url?q=https://www.lesswrong.com/tag/object-level-ai-risk-skepticim&sa=D&source=editors&ust=1692801443602195&usg=AOvVaw1zes87aDp59ftbts92famY) tag ( LessWrong )
* [“AI Success Models”](https://www.google.com/url?q=https://www.lesswrong.com/tag/ai-success-models?sortedBy%3Dtop&sa=D&source=editors&ust=1692801443602513&usg=AOvVaw1Vnq8-s8RmAWNPEcO0hrH5) tag ( LessWrong )
* [A contra AI FOOM reading list](https://www.google.com/url?q=https://magnusvinding.com/2017/12/16/a-contra-ai-foom-reading-list/&sa=D&source=editors&ust=1692801443602877&usg=AOvVaw1AdFuowSjXaZSyn9NH6CGD) ( Magnus Vinding^ )
* [Richard Ngo Critiques Collection](https://www.google.com/url?q=https://docs.google.com/document/d/1zGxvxccxNap4KL70iqHiOBw8voB38Bhp-Z5WpvPQm-w/edit&sa=D&source=editors&ust=1692801443603260&usg=AOvVaw176XEGCByAAJti9iFblL4J) ( Richard Ngo\* )
* [AI Alignment 2018-19 Review](https://www.google.com/url?q=https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review%23Arguments_against_AI_risk&sa=D&source=editors&ust=1692801443603582&usg=AOvVaw1Tijo5OZDGWZYpQxdT4RDQ) ( Rohin Shah\* )
* [Has anyone written a case against AI x-risk where it is clear they understand the case \*for\* x-risk superbly well](https://www.google.com/url?q=https://twitter.com/michael_nielsen/status/1658622154093764610&sa=D&source=editors&ust=1692801443603929&usg=AOvVaw16hN8PilTRfzTlnNhDY7Wk) ( Michael Nielsen )
* [Who are some prominent reasonable people who are confident that AI won't kill everyone?](https://www.google.com/url?q=https://www.lesswrong.com/posts/rCZ9fruWriD6uGNLp/who-are-some-prominent-reasonable-people-who-are-confident&sa=D&source=editors&ust=1692801443604302&usg=AOvVaw0cS1_nfJSPwrd1-B1Ksmle) ( LessWrong )
* [Future Fund Worldview Prize](https://www.google.com/url?q=https://forum.effectivealtruism.org/topics/future-fund-worldview-prize&sa=D&source=editors&ust=1692801443604652&usg=AOvVaw393XpeYseNV8L_SBHC1i_i) tag (EA Forum)
* [Open Philanthropy AI Worldviews Contest](https://www.google.com/url?q=https://www.openphilanthropy.org/open-philanthropy-ai-worldviews-contest/&sa=D&source=editors&ust=1692801443604976&usg=AOvVaw39s04X5QZPlahdpXsp9BRC) ( Open Philanthropy )
* [AI skepticism](https://www.google.com/url?q=https://forum.effectivealtruism.org/topics/ai-skepticism&sa=D&source=editors&ust=1692801443605305&usg=AOvVaw3kf-ouuunJdi1TGHTLgPQI) tag (EA Forum)
*

[[a]](#cmnt_ref1)maybe change title to "for AI safety being unimportant" or "against AI safety being important" or "for AI safety being less important than we think" or "for AI safety being less important than other issues" (as it stands, not clear less important than what)

[[b]](#cmnt_ref2)"(as it stands, not clear less important than what)"

What do you mean?

[[c]](#cmnt_ref3)"for AI safety being less important than we think"

^this was the meaning I intended when I wrote the title

[[d]](#cmnt_ref4)I just meant if I imagine reading the phrase "arguments for AI safety being less important" for the first time, I'd be left wondering in the back of my mind if I'd missed some context like "for AI safety being less important than other risks" or "for AI safety being less important than it used to be". But the intended meaning does seem like the most obvious one, so maybe the title is okay the way it is.

[[e]](#cmnt_ref5)I'm not sure what you mean by this; do you mean something like "consider whether parts of the introductions are shown to be wrong by the counterarguments"?

[[f]](#cmnt_ref6)Just changed the wording. Does it make more sense? I mean it's useful to explicitly consider opposing viewpoints, because there are likely some contradictory claims to find between opposing viewpoints, and these claims can be cruxes that generate the overall disagreement.


The State of AI in various countries - an Overview


Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

Syllabus: Artificial Intelligence and China

1

In recent years, China’s ambitious development of artificial intelligence (AI) has attracted

much attention in policymaking and academic circles. This syllabus aims to broadly cover the

research landscape surrounding China’s AI ecosystem, including the context, components,

capabilities, and consequences of China’s AI development. The materials presented range

from blogs to books, with an emphasis on English translations of Mandarin source materials.

The reading list is not exhaustive, and it will benefit from feedback and revisions.

1
 We hope to regularly update this syllabus. Please send comments and suggestions to

chinainewsletter@gmail.com

. Thanks to Emmie Hine for excellent assistance with formatting and

organization of the syllabus. For recommendations and feedback, thanks go to Julia Chen, Yunxin

Fan, Rebecca Kagan, Jade Leung, Helen Toner, and Remco Zwetsloot.

1

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

1. General Introduction and Overview Pieces
3

A. Assessments of China’s AI Capabilities
3

B. China’s AI Development and International Competition
4

C. Longer Reads on China’s AI Landscape
4

D. Background Readings on China’s Scientific and Technological Ambitions
4

2. Official Documents and Government White Papers
5

3. China’s Approach to AI Governance
5

A. Chinese AI Ethics Documents and Statements
6

B. Analyses of Chinese AI Governance Efforts
6

C. Background Readings on China’s Governance of Technology
6

4. AI and the Chinese Economy
6

A. Impact of AI on China’s Economy
7

B. Profiles of Chinese AI Firms
7

C. Background Readings on China’s Innovation Landscape
7

5. AI in the Military Domain
7

A. International Security
7

B. Civil-Military Fusion
8

C. Background Readings on China’s Military Modernization
8

6. AI and Social Governance
8

A. Social Credit System
9

B. Privacy
9

C. Surveillance
9

D. Propaganda
9

7. Party-Firm Relations
10

A. AI Firms and the CCP
10

B. Background Readings on Party-Firm Relations
10

8. Continuously Updated Sources
10

2

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

1. General Introduction and Overview Pieces

This section includes readings that provide an overview of China’s AI landscape. They will

serve as good launching pads for more in-depth research of particular aspects of China’s AI

development. Parts of these sources that are especially relevant to other topics will be

referenced in those sections as well.

●
Allen, Gregory. (2019). “Understanding China’s AI Strategy.”

Center for a New

American Security

. [

PDF

]

●
China Institute for Science and Technology Policy at Tsinghua University. (2018).

“China AI Development Report.”  [

PDF

]

●
Ding, Jeffrey. (2018). “Deciphering China’s AI Dream: The context, components,

capabilities, and consequences of China’s strategy to lead the world in AI.”

Future of

Humanity Institute.

 [

PDF

]

●
Fischer, Sophie-Charlotte. (February 2018). “Artificial Intelligence: China’s High-Tech

Ambitions.”

Center for Security Studies, ETH Zurich

. [

PDF

]

●
He, Yujia. (2017). “How China is Preparing for an AI-Powered Future.”

Wilson

Center

. [

PDF

]

○
Yujia He’s piece provides an excellent, concise summary of China’s AI-related

policies.

●
Webster, Graham [ed.]. (October 29, 2019) “AI Policy and China: Realities of

State-Led Development.”

Stanford-New America DigiChina Project

. [

link

]

A. Assessments of China’s AI Capabilities

●
Acharya, Ashwin and Zachary Arnold. (December 2019). “Chinese Public AI R&D

Spending: Provisional Findings.”

CSET Issue Brief

. [

PDF

]

●
Ding, Jeffrey. (June 7, 2019). “China’s Current Capabilities, Policies, and Industrial

Ecosystem in AI.”

Testimony before the U.S.-China Economic and Security Review

Commission.

 [

PDF

]

○
Jeffrey Ding’s written testimony systematically assesses China’s AI

capabilities versus those of the U.S. in an effort to address the “AI abstraction

problem.”

●
Ding, Jeffrey. (2018). “Deciphering China’s AI Dream.”

Future of Humanity Institute.

Part III: Capabilities. [

PDF

]

●
MacroPolo. “ChinAI Digital Project.” [

link

]; Sheehan, Matt. (July 16, 2019). “Much Ado

about Data: How America and China Stack Up.” MacroPolo [

link

] ; Ma, Joy Dantong.

(April 15, 2019). “The AI Race is Wide Open, If America Remains Open.”  [

link

]; Ma,

Joy Dantong. (July 30, 2019). “China’s AI Talent Base Is Growing, and then Leaving.”

[

link

]

3

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

B. China’s AI Development and International Competition

●
Ahmed et al. “AI, China, Russia, and the Global Order: Technological, Political,

Global, and Creative Perspectives.” (2018). White paper,

A Strategic Multilayer

Assessment (SMA) Periodic Publication

.

[

PDF

]

●
Hannas, Wm. C. and Huey-meei Chang. (September 2019). "China's Access to

Foreign AI Technology." Center for Security and Emerging Technology. [

link

]

●
Horowitz, Michael C. (September 12, 2018). “The Algorithms of August.”

Foreign

Policy.

 [

link

]

○
Michael Horowitz highlights how the obsession with U.S.-Sino competition in

AI may miss a more important trend of AI propelling emerging and smaller

powers to the forefront of defense innovation.

●
Zwetsloot, Remco, Helen Toner, and Jeffrey Ding. (November 16, 2018). “Beyond

the AI Arms Race: America, China, and the Dangers of Zero-Sum Thinking.”

Foreign

Affairs

. [

link

]

C. Longer Reads on China’s AI Landscape

●
Lee, Kai-Fu. (2018).

AI Superpowers: China, Silicon Valley, and the New World

Order.

Houghton Mifflin Harcourt. [

link

]

○
Kai-Fu Lee’s book offers an insider’s perspective on the startup ecosystem

producing China’s AI unicorns.

●
Tencent Research Institute Legal Research Center of the China Information

Communications Research Institute, Tencent AI Lab, and Tencent Open Platform.

(2017).

Artificial Intelligence: A National Strategic Initiative

. Renmin University Press.

Informal translations of select chapters can be found here:

[

link

]

●
Xiang, Nina. (2019).

Red AI: Victories and Warnings From China’s Rise In Artificial

Intelligence.

[

link

]

●
Yang, Lan. (2017).

In Search of Artificial Intelligence [
人工智能真的来了
]

. Jiangsu

Phoenix Literature and Art Publishing. Chinese edition only: [

link

]

D. Background Readings on China’s Scientific and

Technological Ambitions

●
Schell, Orville and John Delury. (2013).

Wealth and Power: China’s Long March to

the 21st Century

. Abacus. [

link

]

●
Segal, Adam. (2003).

Digital Dragon: High-Technology Enterprises in China.

 Cornell

University Press. [

link

]

●
Shi-Kupfer, Kristen, and Mareike Ohlberg. (2019).

“China’s Digital Rise: Challenges

for Europe.”

MERICS

.

[

PDF

]

○
Mareike Ohlberg and Kristin Shi-Kupfer analyze China’s digitization strategy,

including implications for how AI will graft onto this overall transformation.

●
Wang, Jieshu. (October 17, 2018). “The Early History of Artificial Intelligence in China

(1950s – 1980s).”

Paper presented in the Graduate Student Workshop at 2018

4

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

Annual Meeting of the Society for the History of Technology (SHOT), St. Louis,

MO.

 [

link

]

2. Official Documents and Government White

Papers

This section includes documents and white papers put forward by the Chinese government.

They are crucial for understanding how the central government views domestic AI

development, which has implications for both local governments and the industry as a whole.

For a must-read, see “A Next Generation Artificial Intelligence Development Plan.” For those

looking for a deep dive into Chinese policy considerations of the challenges AI presents on

topics from cybersecurity to social stability, see the “Artificial Intelligence Security White

Paper.”

●
China Academy for Information and Communications Technology. (2019). “Artificial

Intelligence Security White Paper [excerpts].” (E. Kania, D. Peterson, L. Laskai, and

G. Webster, Trans.).

Stanford-New America DigiChina Project.

 [

link

]

●
China Ministry of Industry and Information Technology. (2017). “Three-Year Action

Plan for Promoting Development of a New Generation Artificial Intelligence Industry

(2018–2020).” (P. Triolo, E. Kania, and G. Webster, Trans.).

Stanford-New America

DigiChina Project.

 [

link

]

●
China State Council. (2017). “A Next Generation Artificial Intelligence Development

Plan.” (G. Webster, R. Creemers, P. Triolo, and E. Kania, Trans.).

Stanford-New

America DigiChina Project

. [

link

]

●
Ding, Jeffrey. (August 6, 2018). “ChinAI Newsletter #22: An Open Source AI Strategy

- China's New White Paper on AI Open Source Software.,”

ChinAI Newsletter

.( [

link

]

○
Includes translated excerpts of White Paper on AI Open Source Software.

●
People’s Republic of China. (April 11, 2018). “Position Paper of China.” Submitted at

the United Nations Group of Governmental Experts of the High Contracting Parties to

the Convention on Prohibitions or Restrictions on the Use of Certain Conventional

Weapons Which May Be Deemed to Be Excessively Injurious or to Have

Indiscriminate Effects. [

PDF

]

●
Standards Administration of China. (2018). “White Paper on Artificial Intelligence

Standardization [excerpts].” (J. Ding and P. Triolo, Trans.).

Stanford-New America

DigiChina Project.

 [

link

]

●
Wang, Zhaobing, and Chang Sheng. (November 14, 2018). “Shaping the Political

Attributes of Military Applications of Artificial Intelligence.” (E. Kania, Trans.).

Study

Times

. [

link

]

3. China’s Approach to AI Governance

This section includes documents and analyses of Chinese AI ethics and governance efforts.

Some major developments from 2019 have been documented in this list. For example, the

piece on Governance Principles includes a profile of the newly established “New Generation

5

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

AI Governance Expert Committee.” The “Beijing AI Principles” manifesto is arguably the

most significant document on AI ethics to come out of China so far.

A. Chinese AI Ethics Documents and Statements

●
Beijing Academy of Artificial Intelligence. “Beijing AI Principles.” (2019). [

link

]

●
Hickert, Cameron and Jeffrey Ding. (November 29, 2018). “Read What Top Chinese

Officials Are Hearing About AI Competition and Policy.”

 Stanford-New America

DigiChina Project.

 [

link

]

●
Laskai, Lorand, and Graham Webster. “Chinese AI Alliance Drafts Self-Discipline

'Joint Pledge'.”

Stanford-New America DigiChina Project.

 (English translation). [

link

]

●
Webster, Graham. (2019). “Chinese Expert Group Offers 'Governance Principles' for

'Responsible AI'.”

Stanford-New America DigiChina Project.

(English translation).

[

link

]

B. Analyses of Chinese AI Governance Efforts

●
Ding, Jeffrey, Paul Triolo, and Samm Sacks. (June 20, 2018). “Chinese Interests

Take a Big Seat at the AI Governance Table.”

Stanford-New America DigiChina

Project

. [

link

]

●
Gal, Danit. (June 7, 2019). “Perspectives and Approaches in AI Ethics: East Asia.”

I

n

M. Dubber, F. Pasquale, and S. Das (Eds.),

Oxford Handbook of Ethics of Artificial

Intelligence

. Oxford University Press. [

link

]

○
 Danit Gal’s piece compares China’s approach to AI ethics with that of Japan

and South Korea.

●
Kania, Elsa. (April 17, 2018). "China’s Strategic Ambiguity and Shifting Approach to

Lethal Autonomous Weapons Systems."

Lawfare

.

[link

]

●
Roberts et al. (December 2019). “The Chinese Approach to Artificial Intelligence: an

Analysis of Policy and Regulation.” Oxford Internet Institute Working Paper.

[PDF

]

C. Background Readings on China’s Governance of

Technology

●
Ahmed, Shazeda and Steven Weber. (May 7, 2018). “China’s Long Game in

Techno-nationalism.”

First Monday

. [

link

]

●
Arnaldi et al. (August 2015). “Responsible Governance in Science and Technology

Policy: Reflections from Europe, China, and India.”

Technology in Society.

 [

link

]

●
Feng, Coco. (December 16, 2019). “A Year After Scandal in China, Gene-Editing

Technology Advances.”

South China Morning Post

. [

link

]

4. AI and the Chinese Economy

This section outlines a few reports that give a high-level overview of the possible effects of

AI on China’s economic development. It places particular emphasis on the firms that are

driving the invention, innovation, and diffusion of AI technologies in China.

6

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

A. Impact of AI on China’s Economy

●
Barton et al. (2017).

Artificial Intelligence: Implications for China

. McKinsey Global

Institute

.

 [

PDF

]

●
CSIS ChinaPower Project. “Is China Ready for Intelligent Automation?” [

link

]

○
The CSIS China Power Project’s piece gives a data-packed evaluation of

China’s readiness for automation.

●
Chen et al. (2019). “Automation Impacts on China's Polarized Job Market.”

ArXiv

preprint

. [

PDF

]

●
Purdy, Mark, Serena Qiu, and Frank Chen. (2017).

How Artificial Intelligence Can

Drive China’s Growth

. Accenture

.

 [

PDF

]

B. Profiles of Chinese AI Firms

●
Cave, Danielle, Fergus Ryan, and Vicky Xiuzhong Xu. (November 28, 2019).

“Mapping More of China’s Tech Giants: AI and Surveillance.”

Australian Strategic

Policy Institute

. [

link

]

●
CB Insights. (March 20, 2018). "China in AI Series."

CB Insights Research Brief

. [

link

]

●
Hao, Karen. (January 22, 2019). "Three charts show how China's AI Industry is

propped up by three companies."

 MIT Technology Review.

 [

link

]

●
Min, Liu. (October 12, 2018). “
那些
给
人工智能打工的人
 [Those Who Work for AI].”

GQ China.

[

original in Mandarin

 |

ChinAI informal English translation

]

○
This article discusses data annotation workshops and the people who make a

living in them. .

C. Background Readings on China’s Innovation Landscape

●
Bound, Kirsten, Tom Saunders, James Wilsdon, and Jonathan Adams. (November 5,

2013).

China’s Absorptive State: Innovation and Research in China.

 Nesta. [

PDF

]

●
Fuller, Douglas B. (2016).

Paper Tigers, Hidden Dragons. Firms and the Political

Economy of China's Technological Development

. Oxford University Press. [

link

].

●
Fu, Xiaolan. (2016).

China’s Path to Innovation.

Cambridge University Press. [

link

]

5. AI in the Military Domain

This section focuses on China’s strategy for developing and implementing AI for military

applications. Some questions that are highlighted in the readings include how AI fits into

China's military modernization goals,how the government tries to leverage commercial AI

resources for military purposes, and what China’s military AI ambitions could mean for the

future of U.S.-China strategic competition.

A. International Security

●
Allen, Gregory C. (2019).

Understanding China's AI Strategy

. CNAS. [

PDF

]

7

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

●
Fu, Ying. (2019). “
人工智能
对
国
际
关系的影响初析
 [A Preliminary Analysis of the

Impact of AI on International Relations].” [

original in Mandarin

 |

ChinAI informal

English translation

].

●
Horowitz, Michael. (May 2018). "Artificial Intelligence, International Competition, and

the Balance of Power."

Texas National Security Review

. [

link

]

●
Kania, Elsa B. (November 2017). “Battlefield Singularity: Artificial Intelligence, Military

Revolution, and China’s Future Military Power.” [

PDF

]

B. Civil-Military Fusion

●
Kania, Elsa. (July 28, 2017). “The Dual-Use Dilemma in China’s New AI Plan:

Leveraging Foreign Innovation Resources and Military-Civil Fusion.”

Lawfare

. [

link

]

●
Laskai, Lorand. (April 9, 2018). “Civil-Military Fusion and the PLA’s Pursuit of

Dominance in Emerging Technologies.”

China Brief

 6(18). [

link

]

●
Nouwens, Meia. and Helena Legarda. (December 2018).

Emerging technology

dominance: what China’s pursuit of advanced dual-use technologies means for the

future of Europe’s economy and defence innovation

. IISS and MERICS. [

PDF

]

○
This piece offers a comparison between China’s pursuit of dual-use

technologies, including AI,and European responses in defense innovation.

●
Tate et al. (May 12, 2018).

China’s Advanced Weapons Systems

. IHS Markit for the

US-China Economic and Security Review Commission. [

PDF

]

C. Background Readings on China’s Military Modernization

●
Cheung, Tai Ming. (2014).

Forging China’s Military Might: A New Framework for

Assessing Innovation

. Baltimore, MD: Johns Hopkins University Press. [

link

]

●
Feigenbaum, Evan. (2003).

China’s Techno-warriors: National Security and Strategic

Competition from the Nuclear to the Information Age

. Palo Alto, CA: Stanford

University Press. [

link

]

●
Fisher, Richard D. (2010).

China’s Military Modernization: Building for Regional and

Global Reach

. Palo Alto, CA: Stanford University Press. [

link

]

●
Fravel, M. Taylor. (2019).

Active Defense: China’s Military Strategy Since 1949

.

Princeton, NJ: Princeton University Press. [

link

]

●
Levesque, Greg and Mark Stokes (December 2016).

Blurred Lines: Military-Civil

Fusion and the “Going Out” of China’s Defense Industry

. Pointe Bello. [

PDF

]

6. AI and Social Governance

This section attempts to untangle the impact of AI advances on a range of social issues,

including the social credit system, privacy, surveillance, and propaganda. It includes

analyses of the important distinction between “privacy” in the context of consumer

protections against overstepping of tech companies and “privacy” in the sense of protections

against government surveillance. Some pieces analyze how mass surveillance in Xinjiang

takes the form of an integrated policing platform that aggregates data and flags potentially

threatening people.

8

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

A. Social Credit System

●
Ahmed, Shazeda. (May 1, 2019). “The Messy Truth About Social Credit.”

Logic

. [

link

]

●
Creemers, Rogier. (2018). "China's Social Credit System: An Evolving Practice of

Control."  [

PDF

]

●
Dai, Xin. (2018). "Toward a Reputation State: The Social Credit System Project of

China."  [

link

]

○
Dai Xin, a law professor at Ocean University in Qingdao, explores the social

credit system as a reputational mechanism to tackle governance problems.

●
Hoffman, Samantha. (June 28, 2018). “Social Credit: Technology-enhanced

authoritarian control with global consequences.”

Australian Strategic Policy Institute

.

[

link

]

●
Zhang, Yuzhe, and Han Wei. (April 1, 2019). "In Depth: China’s Burgeoning Social

Credit System Stirs Controversy." Caixin. [

link

]

B. Privacy

●
Ding, Jeffrey. (March 25, 2019). “ChinAI #47: The Sensenet Data Leak - What

Actually Happened.”

ChinAI Newsletter

. [

link

]

●
Sacks, Samm and Lorand Laskai. (February 7, 2018). “China’s Privacy Conundrum.”

Slate

. [

link

]

●
Webster, Graham and Scarlet Kim. (August 14, 2018). “The Data Arms Race is No

Excuse for Abandoning Privacy.” [

link

]

C. Surveillance

●
Buckley, Chris andPaul Mozur. (May 22, 2019). “How China Uses High-Tech

Surveillance to Subdue Minorities.”

The New York Times.

 [

link

]

●
Ding, Jeffrey. (September 22, 2018). “ChinAI Newsletter #29: Complicit - China's AI

Unicorns and the Securitization of Xinjiang.”

ChinAI Newsletter

. [

link

]

●
Ding, Jeffrey. (January 21, 2019). “ChinAI #44: Public Security Bureaus and 8 Future

Applications of AI.”

ChinAI Newsletter

. [

link

]

●
Wang, Maya. (May 1, 2019). "China's Algorithms of Repression: Reverse

Engineering a Xinjiang Police Mass Surveillance App."

Human Rights Watch

. [

link

]

D. Propaganda

●
Ables, Kelsey. (June 21, 2018). “What happens when China’s state-run media

embraces AI?”

Columbia Journalism Review

. [

link

]

●
Bolsover, Gillian. (2017).

Computational Propaganda in China: An Alternative Model

of a Widespread Practice

. University of Oxford, Oxford Internet Institute. [

PDF

]

●
Cadell, Cate. (March 4, 2019). “Propaganda 2.0 - Chinese Communist Party's

message gets tech upgrade.”

Reuters

. [

link

]

●
Frisch, Nick. (May 2, 2019). “We Should Worry About How China Uses Apps like

TikTok.”

The New York Times (The Privacy Project)

. [

link

]

9

Jeffrey Ding, Sophie-Charlotte Fischer, Brian Tse, Chris Byrd

January 2020

7. Party-Firm Relations

The set of readings in this section provides insights into the complex relations between the

state and private enterprises in China. The analyses also discuss some implications of

party-firm relations for technology innovation in China, as well as for U.S. national security.

For instance, enhanced CCP control over AI firms may be at odds  with the realization of

China's ambitious AI strategy.

A. AI Firms and the CCP

●
Feng, Ashley. (February 7, 2019). “We Can’t Tell if Chinese Firms Work for the

Party.”

Foreign Policy

. [

link

]

○
This article provides a nuanced analysis of how Chinese Communist Party

(CCP) influence over Chinese firms has evolved since 2015.

●
Kania, Elsa B. (August 2, 2018). “China’s AI Giants Can’t Say No to the Party.”

Foreign Policy

. [

link

]

●
Jing,

Meng

 and Sarah Dai. (November 21, 2017). “China recruits Baidu, Alibaba and

Tencent to AI ‘national team’.”

South China Morning Post

. [

link

]

B. Background Readings on Party-Firm Relations

●
Dickson, Bruce J. (2008).

Wealth into Power: The Communist Party’s Embrace of the

Private Sector

. Cambridge: Cambridge University Press. [

link

]

●
Milhaupt, Curtis J. and Wentong Zheng. (2015). “Beyond Ownership: State

Capitalism and the Chinese Firm.”

The Georgetown Law Journal

, 665(103), 665-722.

[

PDF

]

●
Norris, William J. (2016).

Chinese Economic Statecraft: Commercial Actors, Grand

Strategy and State Control

. Cornell University Press. [

link

]

●
Wu, Mark. (2016). "The 'China, Inc.' Challenge to Global Trade Governance."

Harvard International Law Journal

 2(57). [

PDF

]

○
Focus on Part 1: The Origin and Nature of the Challenge, especially its

discussion of “six elements [that] render China’s current economic structure

distinct.”

8. Continuously Updated Sources

●
ChinAI, a weekly newsletter by Jeffrey Ding, which features translations of writings

from Chinese thinkers on China's AI landscape. [

link

]

●
Stanford-New America DigiChina Project, a collaborative effort to analyze data

governance, AI, and internet law developments in China through translating and

analyzing Chinese-language sources. [

link

]

●
Tech Buzz China, a biweekly SupChina podcast on China’s innovation landscape,

co-hosted by Ying-Ying Lu and Rui Ma. [

link

]

●
Yuan Yang’s weekly tech column for the Financial Times. [

link

]

10


AGI safety from first principles: Superintelligence


The role of philanthropic funding in politics


 How technical safety standards
 could promote TAI safety
 Cullen O’Keefe, Jade Leung, Markus Anderljung  1  2
 Summary
 Standard-setting is often an important component of technology safety
 regulation. However, we suspect that existing standard-setting infrastructure
 won’t by default adequately address transformative AI (TAI) safety issues. We are
 therefore concerned that, on our default trajectory, good TAI safety best practices
 will be overlooked by policymakers due to the lack or insigniﬁcance of eorts
 which identify, reﬁne, recommend, and legitimate TAI safety best practices in
 time for their incorporation into regulation.
 Given this, we suspect the TAI safety and governance communities should invest
 in capacity to inﬂuence technical standard setting for advanced AI systems. There
 is some urgency to these investments, as they move on institutional timescales.
 Concrete suggestions include  deepening engagement with relevant standard
 setting organizations (SSOs) and AI regulation  ,  translating emerging TAI safety
 best practices into technical safety standards,  and  investigating what an ideal
 SSO for TAI safety would look like  .
 Standards Help Turn Technical Safety Discoveries Into
 Legal Safety Requirements
 A plausible high-level plan for achieving TAI safety is to (a) identify
 state-of-the-art technical safety and security measures that reduce the
 probability of catastrophic AI failures, then (b) ensure (such as by legal mandate)
 2  This post is written in our individual capacities, rather than in our capacities of employment
 or aliation with particular organizations.  1  Thanks to Jonas Schuett, Joslyn Barnha rt, Miles Brundage, and Will Hunt for comments on
 earlier drafts of this post. All views and errors our own.
 that actors at the frontier of AI development and deployment adopt those
 measures.
 This general structure of ﬁrst identifying and then mandating safety measures is
 obviously not unique to AI. How do lawmakers choose which substantive safety
 measures to legally mandate for other technologies? Several options are possible
 and used in practice, including encoding such requirements directly into
 legislation, or delegating such decisions to regulatory agencies.  One common
 strategy  is to have the law  incorporate by reference  (i.e., “point” to) existing
 technical safety standards  3  previously developed by private standard-setting
 organizations (“SSOs”). Another strategy, common in the EU, is to ﬁrst pass
 generally-phrased regulation, and later have the regulation operationalized via
 standards developed by SSOs.  4
 Standardization accomplishes several important things. First, it provides a
 structured process for a consensus of technical safety experts  to identify and
 recommend the best, well-tested technical safety ideas. As a result, policymakers
 have to spend less time developing governmental standards and exercise less
 non-expert judgment about which safety requirements should be adopted.
 Notably, standards can also be updated more rapidly than regulation, due to lower
 bureaucratic and legal overhead, therefore making it possible to keep more apace
 with technical developments. Second, standardization takes emerging safety
 practices that are under-speciﬁed or heterogeneous and restates them in a
 4  In the EU, this responsibility typically falls on the “European Standards Organizations”
 (ESOs), some of which work on requests from the EU Commission, e.g. in preparation of
 forthcoming regulation such as the  AI Act  . The most important ones are the European
 Committee for Standardisation (CEN), European Committee for Electrotechnical
 Standardisation CENELEC) and European Telecommunications Standards Institute (ETSI). The
 EU’s recent  Strategy on Standardisation  is a good place to get an overview of EU
 standard-setting and approach to enga gement with international SSOs.  3  “Standardization” is deﬁne d as “[c]ommon and repeated use of rules, conditions, guidelines
 or characteristics for products or related processes and production methods, and related
 management systems practices.” O. of Mgmt. & Budget, Exec. O. of the President, OMB Circ.
 No. A-119, Federal Participation in the Development and Use of Voluntary Consensus
 Standards and in Conformity Assessment Activities § 3(a) (1998) (hereinafter “1998 Circular
 A-119”),  https://perma.cc/Y32D-R2JQ  . Standards can include “[t]he deﬁnition of terms;
 classiﬁcation of components; delineation of procedures; speciﬁcation of dimensions,
 materials, performance, designs, or operations; measurement of quality and quantity in
 describing materials, processes, products, systems, services, or practices; test methods and
 sampling procedures; or descriptions of ﬁt and measurements of size or strength.”  Id.  We are
 here primarily focused on standards that attempt to improve safety. Other standards (perhaps
 most) are focused on promoting interoperability or reducing information costs.
 precise, consistent, and systematized form  that is more readily adoptable by new
 actors and appropriately clear for a legal requirement. Supranational SSOs provide
 a routinized and reliable infrastructure for facilitating international
 harmonization and regulation via standards. Finally, well-structured
 standard-setting organizations (“SSOs”) operate on the basis of multistakeholder
 consensus, and therefore both aim to  generate and provide evidence of politically
 viable standards  .
 In the US, the path from standardization often roughly follows a pattern of:
 1.  Informal, loose networks of industry safety experts identify, develop, and
 converge on safety-promoting best practices.
 2.  Private  5  SSOs elevate some of these best practices into standards, through a
 well-deﬁned, multistakeholder, consensus-driven process with  procedural
 safeguards  (such as open and equitable participation, a balance of
 represented parties, and opportunities for appeal).  6
 3.  Assuming the government passes regulation for which some of these
 standards are appropriate, this then provides a route via which these
 standards are incorporated into domestic law.  7  8
 8  To be clear, the US federal government retains the option to develop their own standards
 outside of this framework.  See  2016 Circular A-119 § 5(c).  7  See also  O. of Mgmt. & Budget, Exec. O. of the President, OMB Circ. No. A-119, Federal
 Participation in the Development and Use of Voluntary Consensus Standards and in
 Conformity Assessment Activities § 2(e) (2016) (hereinafter “2016 Circular A-119”),
 https://perma.cc/KUV8-VWN8  ; National Techno logy Transfer And Advancement Act Of 1995,
 Pub. L. No. 104–113, 110 Stat. 775 (1996).  6  Some might worry that these due process requirements pose a possible risk as a source of
 distraction, obfuscation, or delay in setting TAI safety standards. We share this concern, which
 is why we propose investigating the creation of a new SSO that could retain a strong focus on
 TAI, with corresponding application of other due process requirements (such as faster
 turnaround times than achieved by most SSOs).  5  In some other countries, governments take a much more active role in standard-setting.
 4.  International bodies like the ISO attempt to harmonize standards across
 countries, as well as between SSOs; via these mechanisms, standards
 developed in e.g. the US could eventually have international impact.
 To be clear, we do not necessarily think this is the best way to approach
 technology regulation. Our claim is primarily empirical: that privately developed
 standards are one of the main (and in the US, legally preferred) sources of
 mandated safety measures, and are likely to remain as such. There are substantial
 downsides with this approach, such as:
 ●  Increased risk of industry capture, since industry employees are heavily
 represented in SSOs.
 ●  A built-in preference for uniformity over experimentation and  competition
 in regulatory approaches.
 ●  Slow and bureaucratic processes for setting standards (though less slow and
 bureaucratic than many governmental processes).
 ●  Reduced democratic accountability and participation, since SSOs are private
 organizations.
 ●  Lack of access to the incorporated standards, since the standards often cost
 hundreds of dollars each to access.  9  Importantly, we also think that
 standardization can be a  useful lever for safety  even if those standards are
 not incorporated into hard law. Established safety standards can establish a
 natural normative “ﬂoor” against which AI developers (especially those
 represented in the standard-setting process) can be evaluated. Special
 antitrust protections for bona ﬁde standard-setting activities makes
 9  SSOs defend the costs to access as  necessary  to recoup the costs of standards development
 and maintenance. Standards incorporated by reference into US regulations can be  freely
 viewed  . One important goal for the TAI safety and governance communities is ensuring that
 existentially important AI safety standards are freely available, unlike most safety standards.
 standard-setting a less risky way for labs to jointly work on safety.  10
 Standardization of informal and heterogeneous safety best practices can
 lower the cost of adopting such practices, leading to broader coverage.  11
 Standards can also form the substantive primitives for private  certiﬁcation
 and  auditing  schemes.
 Emergence of Consensus AI Safety Best Practices
 Part of what excites us about standardization as a tractable approach to TAI
 governance is the increasing emergence of best practices in AI safety with
 increasingly broad buy-in. For example, a number of industry, academic, and civil
 society actors appear to endorse and/or are willing to discuss some fairly concrete
 measures to improve alignment, safety, and social impact throughout the AI
 lifecycle, including (but not limited to):
 ●  A  variety of social and technical measures  to limit harms from
 commercialized large language models
 ●  Use of  reinforcement learning from human feedback  to align model
 behavior with human preferences  12
 ●  Responsible  publication  of advances in AI, including  discussion of ethical
 impacts  and  disclosure of ethically relevant model construction and
 behavior  .
 We think these measures may be good candidates for formalization into standards
 in the near future. As AI safety and policy research matures, currently theoretical,
 12  E.g., at  Anthropic  ,  DeepMind  ,  OpenAI  , and elsewhere.  11  For example, companies can reduce the amount of discovery and tinkering required to
 achieve some goal by referencing an appropriate standard. Consistent standards can also foster
 an ecosystem of actors specialized in the relevant standards, who can transfer those skills to
 other appropriate contexts.  10  See  15 U.S.C. § 4302.
 vague, or nascent  ideas  may mature into consensus best practices, adding to the
 list of candidates for standardization. Of course, the goal of existential-risk
 focused AI safety research is to eventually produce training and testing methods
 that can, when applied to an AI system, reliably improve that system’s alignment
 with human values. We hope that such methods will be (or could be made)
 suciently clear and universalizable to make into legally appropriate standards.
 AI Standardization Today
 Standardization may be an appropriate next step for some (but by no means all)  13
 consensus best practices.
 A number of SSOs currently develop standards relevant to AI safety. For example,
 the  International Organization for Standardization  (“ISO”) and  International
 Electrotechnical Commission  (“IEC”) run a  joint subcommittee on AI  , which has
 promulgated standards on AI  trustworthiness  ,  robustness  ,  bias  , and  governance  .
 The  Institute of Electrical and Electronics Engineers  (“IEEE”) has also
 promulgated a number of  AI standards  . The U.S. National Institute of Standards
 and Technology (“NIST”) is developing an  AI Risk Management Framework  .
 Best practices, standardization, and the complementary process of  conformity
 assessment  are beginning to play an important role in the regulation of AI. The
 Federal Trade Commission has  repeatedly  implied  that compliance with best
 practices and “independent standards” in ethical AI may be required by—or at
 least help evidence conformity with—various laws they enforce. In its  Inaugural
 Joint Statement  , the U.S.–EU Trade and Technology Council announced an intent
 to prioritize collaboration on AI standard-setting.  Standardization  and  conformity
 assessments  for certain high-risk AI systems play an important role in the
 proposed  EU Artiﬁcial Intelligence Act  . In short, governments appear poised to
 rely heavily on standardization for AI regulation.
 Actionable Implications for the TAI Safety and
 Governance Communities
 13  In particular, standard-setting is a time-consuming and expensive process. These costs may
 not always be worth the beneﬁts of standardization.
 Our core thesis is that technical AI safety standards can and will be the building
 blocks for many forms of future AI regulation. We’ve laid out the case brieﬂy
 above; additional analysis and reﬁnement of this thesis could be valuable. If this
 thesis is true of the most existentially important forms of AI regulation, this has
 important and actionable implications for the TAI safety and governance
 communities, many of which were presciently identiﬁed by  Cihon (2019)  . Thus,
 this post serves as a renewed call to take AI safety standardization seriously.
 Concretely, we have several ideas on how to do this in the near- and
 medium-term.
 First, safety-conscious AI practitioners should consider advancing
 standardization of TAI-relevant safety best practices. Although we are aware and
 appreciative of several TAI-concerned individuals who have participated in AI
 safety standard-setting, we suspect that TAI-focused perspectives are still
 underrepresented in the processes of the various SSOs already developing AI
 safety, security, and governance standards. While this might not be a problem
 today, if those standards are increasingly relied upon by policymakers for
 substantive AI regulation, TAI perspectives and priorities might not be adequately
 represented or considered legitimate, and we won’t have routes to promote TAI
 safety best practices once they are discovered. We therefore renew Cihon (2019)’s
 call for strategic engagement between the TAI safety communities and AI SSOs.
 For example, (more) AI safety researchers may consider joining the membership
 of such SSOs, and serving on relevant committees.  14
 For similar reasons, TAI safety researchers and practitioners should consider
 engaging seriously with regulatory eorts in jurisdictions where regulation
 typically precedes standards. This notably includes forthcoming EU AI regulation
 and accompanying standard-setting processes, especially if we should expect
 such regulation to diuse globally.
 As the TAI safety community converges on best practices for frontier systems, we
 should proactively push for them to be reﬁned into technical standards. An
 intermediate step here might look like creating fora where safety practitioners
 from across organizations can easily share and reﬁne safety best practices and
 other lessons learned,  15  then sharing these publicly in concrete form.
 15  In so doing, they will have to take care not to run afoul of antitrust laws.  14  To be clear, we do not consider such involvement to be an obvious unalloyed good. The time
 of AI safety researchers and engineers is very valuable, and they should not reallocate it lightly.
 We’d also encourage proper analysis of the adequacy of current AI-relevant SSOs.
 If it seems they might be inadequate at dealing with TAI safety issues, we should
 get to work investigating what new SSOs tailored to TAI safety issues might look
 like. Ideal features of such an SSO would likely include:
 ●  Disciplined focus on TAI safety issues, with supporting institutional rules
 (e.g., bylaws) and aligned leadership to maintain that focus.
 ●  Exceptionally high transparency and accessibility (e.g., technical standards
 are freely available, including in multiple relevant languages, for easy use,
 reference, and critique).  16
 ●  Ability to very rapidly initiate or update standards.
 ●  Calibrated communication of the safety value of standards (i.e.,
 communicating by how much does this standard, when properly applied,
 reduce worst-case risks).
 ●  Design of multiple layers of standards, including organization- and
 process-level standards (e.g., organizations are required to make extensive
 good-faith eorts to identify and remedy safety issues, and are not
 permitted to infer that their systems are safe merely because they’ve
 checked o a list of object-level system requirements).
 ●  Low implementation costs for safety standards., such as through provision
 of how-to guides for implementation.
 ●  ANSI accreditation  , for credibility and legitimacy reasons.
 Like many things in governance, learning to inﬂuence and implement
 standardization well will require iteration and experience. We shouldn’t assume
 that we can simply “tack on” standardization after discovering AI safety solutions.
 We suspect that such solutions will be more consistently, quickly, and smoothly
 adopted and eventually legally codiﬁed if there is already a nimble,
 16  NB: This is not the case with most existing SSOs!
 well-functioning, respected, legitimate, TAI-oriented standardization
 infrastructure to translate our best collective safety measures into standards.
 Creating such an infrastructure will take time, but seems tractable if we invest our
 eorts eciently and strategically in this space.
 Conclusion
 To summarize:
 1.  Technical standards form the fundamental building blocks of many
 technology regulation regimes, and could plausibly form the fundamental
 building blocks of TAI-relevant regulation.
 2.  Given (1), the TAI safety and governance communities should ensure that
 there exists SSOs that can eciently elevate AI safety best practices into
 technical standards that are, in substance and form, appropriate for legal
 and regulatory use.
 3.  It’s not clear that long-term safety priorities are currently well represented
 in existing AI standard-setting eorts, or that the current structure and
 procedures of AI-relevant SSOs are appropriate to the challenges that TAI
 may pose.
 4.  If existing SSOs are inadequate, we should have a plan for improving
 existing SSOs or creating new ones, particularly to ensure they are focused
 on and nimble to the evolving challenges of advanced AI safety. This would
 take several years.
 5.  We may therefore wish to start investing in answering (3)—and then
 possibly working on (4)—soon.
 Concrete steps we propose include:
 1.  TAI safety researchers and practitioners should consider joining and
 inﬂuencing existing TAI-relevant SSOs, both for the object-level reason of
 improving AI safety standard-setting and for the purpose of learning more
 about AI safety standard-setting.
 2.  For similar reasons, TAI safety researchers and practitioners should
 consider engaging with regulatory eorts in jurisdictions where regulation
 typically precedes standards.
 3.  TAI safety researchers should actively drive for convergence on what best
 safety practices are for frontier systems, and reﬁne those best practices into
 technical standards that would be suitable for integration into law.
 4.  TAI safety and governance researchers and practitioners should analyze
 whether existing AI SSOs are adequate for the needs of TAI
 standard-setting, including analyzing which standardization processes are
 going to be most important to inﬂuence today  17  . If existing eorts seem
 likely to be inadequate, we should design and possibly build new
 standard-setting infrastructure. If you are interested in working on this and
 think that we could help, or have valuable insight regarding AI safety
 standard-setting, please reach out to us at
 tai-standards[at]googlegroups.com
 17  Relevant questions include: Which standards are likely to be most relevant for future frontier
 systems? Which order are standards from inﬂuential standard setting bodies going to come
 out? Which standards are most likely to see global diusion? For example, will the EU AI Act,
 and its accompanying standards, diuse globally? Should we expect the NIST AI Risk
 Management Framework to aect relevant ISO, IEC, or ESO standards?

Cullen O'Keefe
Jade Leung
Markus Anderljung


Activism by the AI Community