Specification gaming: the flip side of AI ingenuity

Victoria Krakovna
Jonathan Uesato
Vladimir Mikulik
Matthew Rahtz
Tom Everitt
Ramana Kumar
Zac Kenton
Jan Leike
Shane Legg


Primer on Safety Standards and Regulations for Industrial-Scale AI Development


The other alignment problem: mesa-optimisers and inner alignment


Overview of how AI might exacerbate long-running catastrophic risks


Researcher perceptions of Current and Future AI


Revisiting Algorithmic Progress


A template for Arxiv Style
††thanks: Citation:
Authors. Title. Pages…. DOI:000000/11111.

1 Introduction
---------------

Many of the remarkable advances of the past 5 years in deep learning have been driven by a continuous increase in the quantity of *training compute* used to develop cutting-edge models [kaplan2020scaling, hoffman2022training, sevilla2022compute].
Such large-scale training has been made possible through the concurrent use of hundreds or thousands of specialized accelerators with high inter-chip communication bandwidth (such as Google TPUs, NVIDIA A100 and H100 GPUs, or AMD MI250 GPUs), employed for a span of weeks or months to compute thousands or millions of gradient updates.
We refer to these specialized accelerators as *ML chips*, which we distinguish from consumer-oriented GPUs with lower interconnect bandwidth (e.g., the NVIDIA RTX 4090, used in gaming computers).

This compute scaling trend has yielded models with ever more useful capabilities.
However, these advanced capabilities also bring with them greater dangers from misuse [brundage2018malicious].
For instance, it is increasingly plausible that criminals may soon be able to leverage heavily-trained code-generation-and-execution models to autonomously identify and exploit cyber-vulnerabilities, enabling ransomware attacks on an unprecedented scale.
111The 2017 NotPetya worm was estimated to have caused $10B in damages, and was both far less capable and far more straightforward to disable [greenberg\_2018].
Even absent malicious intent, rival companies or countries trapped in an AI “race dynamic” may face substantial pressure to cut corners on testing and risk-mitigation, in order to deploy high-capability ML systems in the hopes of outmaneuvering their competitors economically or militarily.
The edge-case behaviors of deep learning models are notoriously difficult to debug [hendrycks2021unsolved], and without thorough testing and mitigation, such bugs in increasingly capable systems may have increasingly severe consequences.
Even when rival parties would all prefer to individually do more testing and risk-mitigation, or even forgo developing particularly dangerous types of ML models entirely [urbina2022dual], they may have no way to verify whether their competitors are matching their levels of caution.

In the event that such risks do emerge, governments may wish to enforce limits on the large-scale development of ML models.
While law-abiding companies will comply, criminal actors, negligent companies, and rival governments may not, especially if they believe their rule-violations will go unnoticed.
It would therefore be useful for governments to have methods for reliably *verifying* that large-scale ML training runs comply with agreed rules.

These training runs’ current need for large quantities of specialized chips leaves a large physical and logistical footprint, meaning that such activities are generally undertaken by sizable organizations (e.g., corporate or governmental data-center operators) well-equipped to comply with potential regulations.
Yet even if the relevant facilities are known, there is no easily-observable difference between training a model for social benefit, and training a model for criminal misuse — they require the same hardware, and at most differ in the code and data they use.
Given the substantial promise of deep learning technologies to benefit society, it would be unfortunate if governments, in a reasonable attempt to curtail harmful use-cases but unable to distinguish the development of harmful ML models, ended up repressing the development of beneficial applications of ML as well.
Such dynamics are already appearing: the US Department of Commerce’s rationale for its October 2022 export controls denying the sale of high-performance chips to the People’s Republic of China, while not specific to ML, was based in part on concern that those chips might be used to develop weapons against the United States or commit human rights abuses [bis2022controls].
If the US and Chinese governments could reach an agreement on a set of permissible beneficial use-cases for export-controlled chips, and had a way to verify Chinese companies’ compliance with that agreement, it may be possible to prevent or reverse future restrictions.

Such a system of verification-based checks and balances, distinguishing between “safe” and “dangerous” ML model training, might seem infeasible.
Yet a similar system has been created before.
At the dawn of the nuclear age, nations faced an analogous problem:
reactor-grade uranium (used for energy) and weapons-grade uranium (used to build nuclear bombs) could be produced using the same types of centrifuges, just run for longer and in a different configuration.
In response, in 1970 the nations of the world adopted the Treaty on the Non-Proliferation of Nuclear Weapons (NPT) and empowered the International Atomic Energy Agency (IAEA) to verify countries’ commitments to limiting the spread of nuclear weapons, while still harnessing the benefits of nuclear power.
This verification framework has helped the world avoid nuclear conflict for over 50 years, and helped limit nuclear weapons proliferation to just 9 countries while spreading the benefits of safe nuclear power to 33 [wikipedia\_2023].
If future progress in machine learning creates the domestic or international political will for enacting rules on large-scale ML development, it is important that the ML community is ready with technical means for verifying such rules.

###
1.1 Contributions

In this paper, we propose a monitoring framework for enforcing rules on the *training* of ML222Throughout the text, we use “ML” to refer to deep-learning-based machine learning, which has been responsible for much of the progress of recent years. models using large quantities of specialized ML chips.
Its goal is to enable governments to verify that companies and other governments have complied with agreed guardrails on the development of ML models that would otherwise pose a danger to society or to international stability.
The objective of this work is to lay out a possible system design, analyze its technical and logistical feasibility, and highlight important unsolved challenges that must be addressed to make it work.

The proposed solution has three parts:

1. To prove compliance, an ML chip owner employs firmware that logs limited information about that chip’s activity, with their employment of that firmware attested via hardware features.
We propose an activity logging strategy that is both lightweight, and maintains the confidentiality of the chip-owner’s trade secrets and private data, based on the NN weights present in the device’s high-bandwidth memory.
2. By inspecting and analyzing the logs of a sufficient subset of the chips, inspectors can provably determine whether the chip-owner executed a rules-violating training run in the past few months, with high probability.
3. Compute-producing countries leverage supply-chain monitoring to ensure that each chip is accounted for, so that actors can’t secretly acquire more ML chips and then underclaim their total to hide from inspectors.

The system is compatible with many different rules on training runs (see Section [2.1](#S2.SS1 "2.1 What types of rules can we enforce by monitoring ML training? ‣ 2 The Problem: Detecting Violations of Large-Scale ML Training Rules ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")), including those based on the total chip-hours used to train a model, the type of data and algorithms used, and whether the produced model exceeds a performance threshold on selected benchmarks.
To serve as a foundation for meaningful international coordination, the framework aspires to reliably detect violations of ML training rules *even in the face of nation-state hackers attempting to circumvent it*.
At the same time, the system does not force ML developers to disclose their confidential training data or models.
Also, as its focus is restricted to specialized data-center chips, the system does not affect individuals’ use of their personal computing devices.

Section [2](#S2 "2 The Problem: Detecting Violations of Large-Scale ML Training Rules ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") introduces the problem of verifying rules on large-scale ML training. Section [3](#S3 "3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") provides an overview of the solution, and describes the occasional inspections needed to validate compliance.
Sections [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), and [6](#S6 "6 At the supply chain ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") discuss the interventions at the chip-level, data-center-level, and supply-chain respectively.
Section [7](#S7 "7 Discussion ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") concludes with a discussion of the proposal’s benefits for different stakeholders, and lays out near-term next steps.

###
1.2 Limitations

The proposed system’s usefulness depends on the continued importance of large-scale training to produce the most advanced (and thus most dangerous) ML models, a topic of uncertainty and ongoing disagreement within the ML community.
The framework’s focus is also restricted only to training runs executed on specialized data-center accelerators, which are today effectively necessary to complete the largest-scale training runs without a large efficiency penalty.
In Appendix [A](#A1 "Appendix A Discussion on future training requirements ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), we discuss whether these two trends are likely to continue.
Additionally, hundreds of thousands of ML chips have already been sold, many of which do not have the hardware security features required by the framework, and may not be retrofittable nor even locatable by governments.
These older chips’ importance may gradually decrease with Moore’s Law.
But combined with the possibility of less-efficient training using non-specialized chips, these unmonitored compute sources present an implicit lower bound on the minimum training run size that can be verifiably detected by the proposed system.
Still, it may be the case that frontier training runs, which result in models with new emergent capabilities to which society most needs time to adapt, are more likely to require large quantities of monitorable compute.

More generally, the framework does not apply to small-scale ML training, which can often be done with small quantities of consumer GPUs.
We acknowledge that the training of smaller models (or fine-tuning of existing large models) can be used to cause substantial societal harm (e.g., computer vision models for autonomous terrorism drones [pledger2021role]).
Separately, if a model is produced by a large-scale training run in violation of a future law or agreement, that model’s weights may from then on be copied undetectably, and it can be deployed using consumer GPUs [sheng2023high] (as ML inference requires far lower inter-chip communication bandwidth).
Preventing the proliferation of dangerous trained models is itself a major challenge, and beyond the scope of this work.
More broadly, society is likely to need laws and regulations to limit the harms from bad actors’ misusing such ML models.
However, exhaustively *enforcing* such rules at the hardware-level would require surveilling and policing individual citizens’ use of their personal computers, which would be highly unacceptable on ethical grounds.
This work instead focuses attention upstream, regulating whether and how the most dangerous models *are created in the first place*.

Lastly, rather than proposing a comprehensive shovel-ready solution, this work provides a high-level solution design.
Its contribution is in isolating a set of open problems whose solution would be sufficient to enable a system that achieves the policy goal.
If these problems prove unsolvable, the system’s design will need to be modified, or its guarantees scaled back.
We hope that by providing a specific proposal to which the community can respond, we will initiate a cycle of feedback, iteration, and counter-proposals that eventually culminates in an efficient and effective method for verifying compliance with large-scale ML training rules.

###
1.3 Related Work

This paper joins an existing literature examining the role that compute may play in the governance of AI.
Early work by Hwang [hwang2018computational] highlighted the potential of computing power to shape the social impact of ML.
Concurrent work by Sastry et al. [sastry2023computing] identifies attributes of compute that make it a uniquely useful lever for governance, and provides an overview of policy options.
Closely-related work by Baker [baker2023nuclear] draws lessons from nuclear arms control for the compute-based verification of international agreements on large-scale ML.

Rather than focusing on specific policies, the work proposes a technical platform for verifying many possible regulations and agreements on ML development.
Already, the EU AI Act has proposed establishing risk-based regulations on AI products [veale2021demystifying], while US senators have proposed an “Algorithmic Accountability Act” to oversee algorithms used in critical decisions [chu\_2022], and the Cyberspace Administration of China (CAC) has established an “algorithm registry” for overseeing recommender systems [cac\_2022].
Internationally, many previous works have discussed the general feasibility and desirability of AI arms control [reinhold2022arms, docherty\_2020, mittelsteadt2021ai], with [scharre2022artificial] highlighting the importance of verification measures to the success of potential AI arms control regimes.
Past work has also explored the benefits of international coordination on non-military AI regulation [erdelyi2018regulating].

The proposed solution involves proving that a rule-violating ML training run was *not* done, in part by proving which other training runs *were* done.
The analysis of the latter problem is heavily inspired by the literature on Proof-of-Learning [jia2021proof, fang2022fundamental] (discussed further in Section [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")).
Other works has have used tools from cryptography to train NN models securely across multiple parties [wagh2018securenn], and to securely prove the correctness of NN inference [lee2020vcnn].
However, these approaches suffer large efficiency penalties and cannot yet be scaled to cutting-edge model training, rendering them nonviable as a method for verifying rules on large-scale training runs.

2 The Problem: Detecting Violations of Large-Scale ML Training Rules
---------------------------------------------------------------------

We focus on the setting in which one party (the “Verifier”) seeks to verify that a given set of ML training rules is being followed, and another party (the “Prover”) is developing the ML system and wants to prove to the Verifier that it is complying with those rules.
The Verifier can request that the Prover take actions, such as disclosing information on training runs, in order to help the Verifier determine the Prover’s compliance.
The Prover is a “covert adversary” [aumann2007security] – they may benefit from *violating* the ML training rule, but will only seek to violate the rule *if they can still appear compliant* to the Verifier.
There are two real-world Prover-Verifier relationships we are particularly interested in:

* *Domestic Oversight*:
Governments have a clear interest that the ML systems developed by companies operating within their borders comply with certain rules.
Regulators can level both civil and criminal penalties on organizations caught violating rules, and often require organizations to maintain records that prove regulatory compliance (e.g., financial transaction record-keeping requirements).
* *International Oversight*:
The most significant types of ML training rules may be those enforced internationally (on companies and governments in multiple countries), and verified by other governments or international bodies.
These include enforcing globally-beneficial rules (e.g., combatting disinformation), and verifying arms control agreements (e.g., limiting the development of autonomous code-generating cyberweapons).
There is precedent for countries abiding by international agreements with strict monitoring regimes when they stand to benefit, such as Russia’s historically allowing random U.S. inspections of its missiles as a part of the START treaties, in exchange for certainty that the U.S. was abiding by the same missile limits [schenck2011start].

Thus, the problem we address is: what minimal set of verifiable actions can the Verifier require the Prover to take that would enable the Verifier to detect, with high probability, whether the Prover violated any training rules?

###
2.1 What types of rules can we enforce by monitoring ML training?

It is important that standards and agreements on ML training focus on preventing concrete harm, and otherwise leave society free to realize the broad benefits of highly-capable ML systems.
Indeed, there are many types of ML models that should not only be legal to train, but that should open-sourced so that all of society can benefit from them [solaiman2023gradient].
The proposed framework focuses only on enforcing rules on the training of those more dangerous models whose creation and distribution would substantially harm society or international security.
Indeed, as mentioned in Section [1.2](#S1.SS2 "1.2 Limitations ‣ 1 Introduction ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), this framework *could not* prevent smaller-scale training of ML models, and thus limits the risk of overreach by authoritarian Verifiers.
Below are some informative properties that a Verifier could determine by monitoring the training process of an ML model:

* *Total training compute*, which has proven to be an indicator for ML models’ capabilities [kaplan2020scaling, sutton2019bitter].
* *Properties of the training data*, such as whether a language model’s text dataset contains code for cybersecurity exploits.
* *Properties of the hyperparameters*, such as the fraction of steps trained via reinforcement learning.
* *The resulting model’s performance on benchmarks designed to elicit its capabilities*, including whether the model’s capabilities exceed agreed-on thresholds, and including interactive benchmarks (e.g. finetuning the model on a particular task).
* Combinations of the above — for example, “if a model was trained on RL-for-code-generation for greater than X FLOPs, then it should not be trained beyond Y performance on Z benchmarks.”

Ultimately, these rule thresholds should be selected based on the model capabilities that would result.
Current “scaling law” extrapolations are not yet able to reliably predict ML models’ downstream capabilities [ganguli2022predictability], so finding principled methods for deciding on rule-thresholds that achieve desired policy outcomes is an important area for future work.

If a Verifier can reliably detect the aforementioned training run properties, that would allow them to mandate several types of rules, such as:

* *Reporting requirements* on large training runs, to make domestic regulators aware of new capabilities or as a confidence-building measure between companies/competitors [horowitz2021ai].
* *Bans or approval-requirements* for training runs considered overly likely to result in models that would threaten society or international stability. Approval could be conditioned on meeting additional requirements (e.g., willingness to comply with downstream regulations on model use, increased security to prevent model-theft, greater access for auditors).
* *Requiring that any trained model be modified to include post-hoc safety mitigations* if the unmodified model could be expected to pose a severe accident risk absent those mitigations.
Such safety assessments and mitigations (such as “Helpful and Harmless” finetuning [bai2022training]) may involve a prohibitive upfront cost that companies/governments would otherwise avoid.
However, once they have been forced to make the investment and built a less accident-prone model, they may then prefer to use the safer version.
Such rules allow all parties to coordinate spending more resources on safe and responsible innovation, without fearing that their competitors may secretly undercut them by rushing ahead without addressing negative externalities.

###
2.2 Other Practical Requirements

There are several other considerations for such a monitoring system to be practical. Its cost should be limited, both by limiting changes to current hardware, and by minimizing the ongoing compliance costs to the Prover and enforcement costs to the Verifier.
The system should also not pose a high risk of leaking the Prover’s proprietary information, including model weights, training data, or hyperparameters.
Most importantly, the system must be robust to cheating attempts, even by highly-resourced adversaries such as government hacking groups, who may be willing to employ sophisticated hardware, software, and even supply-chain attacks.

3 Solution Overview
--------------------

In this section, we outline a high-level technical plan, illustrated in Figure [1](#S3.F1 "Figure 1 ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), for Verifiers to monitor Provers’ ML chips for evidence that a large rule-violating training occurred.

![Overview of the proposed monitoring framework.](https://media.arxiv-vanity.com/render-output/7480550/x1.png)
Figure 1: Overview of the proposed monitoring framework.

The framework revolves around chip inspections: the Verifier will inspect a sufficient random sample of the Prover’s chips (Section [3.2](#S3.SS2 "3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")), and confirm that none of these chips contributed to a rule-violating training run.
For the Verifier to ascertain compliance from simply inspecting a chip, we will need interventions at three stages: on the chip, at the Prover’s data-center, and in the supply chain.

* *On the chip* (Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")): When the Verifier gets access to a Prover’s chip, they need to be able to confirm whether or not that chip was involved in a rule-violating training run.
Given that rule violation depends only on the code that was run, our solution will necessitate that ML chips logging infrequent traces of their activity, with logging done via hardware-backed firmware.
We suggest that ML chips’ firmware occasionally log a copy of the current state of the chip’s high-bandwidth memory to long-term storage, and in particular, that it logs the shard of the NN’s weights stored in memory.
These *weight-snapshots* can serve as a fingerprint of the NN training that took place on each chip.
* *At the data-center* (Section [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")): The Verifier needs a way to interpret the chips’ logs, and determine whether or not they are evidence for a rule-violating training run.
To that end, the Prover, who is training the model, will be required to store a transcript of the training process — including training data, hyperparameters, and intermediate weight checkpoints — for each model they train.
Using protocols similar to “Proof-of-Learning” [jia2021proof], these training transcripts may serve as provenance for the logged weight-snapshots, which are themselves the result of the same training process.
In practice, for each (hash of a) weight-snapshot logged by a chip, the Prover provides the Verifier (the hashed version of) the matching training transcript.
Then the Prover and Verifier jointly and securely verify that, with high probability, retraining using the training transcript would have indeed resulted in the logged weight-snapshot (and that no other valid training transcript could have resulted in that snapshot).
Finally, now that the Verifier knows an approximate training transcript of the training run that had been executed on that chip at that time, they can examine properties of the training transcript to confirm that the Prover has complied with the agreed upon rules.
* *At the supply chain* (Section [6](#S6 "6 At the supply chain ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")): The Verifier needs to know which ML chips the Prover owns, so that the Verifier can randomly inspect a representative sample of those chips, to confirm their ownership and that their logging mechanisms are reporting correctly.
Without this chip-ownership verification step, a Prover might covertly acquire a large quantity of chips and use them for training without ever notifying the Verifier.
Keeping track of chip-ownership is viable because the cutting-edge data-center chip supply chain is highly concentrated, meaning that chips originate from a few initial chokepoints and can be monitored thereafter.

These steps, put together, enable a chain of guarantees.

* When any organization wants to train a large rule-violating ML model, they must do so using chips that the Verifier is aware of.
* These chips will occasionally log weight-snapshots.
Each time a chip logs a weight-snapshot, the Prover must report the log to the Verifier, along with (hashes of) training transcripts that establish the provenance of that weight-snapshot.
* If the Prover tries to hide the logs from every chip involved in the training run, the Verifier will eventually find out, because it will sample and physically inspect at least one of those chips with high probability.
* Also, the hashed training transcripts that the Prover provides along with the logged weight-snapshot need to be authentic, reflecting the training run that was run on the chip.
If they are not authentic, they will be caught by the Verifier’s transcript verification procedure.
* If the training transcript *is* authentic, and the run violated the rules, then the Verifier can tell, and will catch the Prover.

Thus, so long as the Prover complies with the Verifier’s steps, the Verifier will detect the Prover’s rule-violation with high probability.
Just as in financial audits, a Prover’s refusal to comply with the verification steps would itself represent an indication of guilt.

###
3.1 Chip inspections

Ideally, chips could remotely report their logs, with on-chip firmware and remote attestation being sufficient to guarantee that those logs were truthfully reported (see Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")).
Such guarantees may be sufficient to deter most Provers from misbehavior.
Unfortunately, these measures may not restrain well-resourced adversarial Provers:
it is generally understood within the hardware security world that, given unlimited physical access to a chip (which the Prover certainly would have), a chip’s attestation and signed-firmware-enforcement mechanisms can be undermined [skorobogatov2005semi].
However, by physically inspecting chips after the fact, it may be possible to reliably detect such tampering.
This is because most hardware attacks require physically damaging the chip in ways that would be nearly-impossible to hide from a Verifier’s subsequent inspection, especially if ML chips are supplemented with tamper-evident packaging measures [waksman2010tamper].
Luckily, rather than needing to inspect every chip, we only need to inspect enough to catch at least one chip from each large training run (Section [3.2](#S3.SS2 "3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")).

A chip inspection can be done in one of two ways.
The Verifier can send an inspector to the facility in which a sampled chip is stored, which may be cost-effective if the facility contains multiple chips selected for sampling.
Alternatively, the ML chips can be *transported a neutral inspection site*, as these chips are generally housed in easily-removable servers. (Sensitive components like storage can be removed prior to transport.)
The transporting option may be more attractive to security-conscious Provers who would prefer not to allow Verifier inspectors into their data-centers.

A chip inspection involves several steps.
First, the Verifier confirms the chip’s serial number (often burned into chips and accessible both digitally and physically) matches the serial number they requested from the Prover.
This confirms that this is the correct randomly-sampled chip, and also that the Prover still owns the chip and has not secretly resold it to an untrusted party.
Second, the Verifier checks the on-chip logs, from which they extract the weight snapshot hashes which they will check against the Prover’s reported training transcripts (Section [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")).
Finally, the Verifier checks that the chip’s logging mechanisms, such as its secure firmware authentication mechanism (Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")), have not been physically tampered with.

###
3.2 How many ML chips does the Verifier need to inspect?

Sections [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), and [6](#S6 "6 At the supply chain ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") will together guarantee that if the Verifier observes even a single ML chip that logged a weight snapshot hash during a rules-violating training run, then the Verifier can use that weight snapshot to prove that the Prover violated the rules.
For the Verifier to make sure they’d observe at least one such snapshot from a large training run with high probability (or alternatively, catch a chip that’d been tampered with), they would need to inspect at least s of the Prover’s C chips every T days.
We now seek to calculate s.

We define a large-scale training run as any training run using more than H FLOPs.
We only seek to verify rules on training runs using >H FLOPs, meaning that any weight-snapshots from the first H FLOPs are not incriminating (as the Prover could have stopped the training run after that snapshot).
Catching a chip that happened to save a weight-snapshot at time H+ϵ is vanishingly unlikely for small ϵ, so we instead provide guarantees on detecting the case where the Prover’s training run is executed for ≥2H, substantially exceeding H.333We can always keep the detection threshold the same by cutting H in half.
Also, the literature on neural scaling laws [kaplan2020scaling, rosenfeld2021scaling, epoch2023scalinglawsliteraturereview] suggests that model loss decreases logarithmically at best given additional compute, so failing to catch a less-than-2x violation may have limited consequences.
This means that the training run was executed for an additional second period of at least H FLOPs, during which any weight snapshot would capture evidence of an ML model that had already been trained for at least >H FLOPs.
From hereon, when we describe detecting a snapshot of a “large-scale training run”, we refer to this second stage of the training run.
If a Verifier wanted to catch a Prover *before* they had completed a H rule-violating training run, they can simply shrink their target threshold to a fraction of H, and sample chips on a rolling basis.

The Verifier, whether due to specific suspicions or routine due diligence, wants to audit a particular Prover’s C chips.
Let a be the FLOPs per day for those chips.
We will conservatively assume that NN training can be perfectly parallelized and utilize all the FLOPs on each chip.
Let f be the expected number of weight-snapshots saved by the chip per day (Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")) snapshots are Poisson distributed.
The Verifier wants to catch a training run within at most T days of its concluding, though the training run itself may have lasted for a longer period
444For simplicity, we assume T is divisible by Tm
T>Tm.
This implies that the Prover must have used a minimum
of HaT chips.
One might expect that by increasing T, the Prover could use fewer chips and therefore evade detection, but this turns out to not to meaningfully improve the Prover’s evasion probability (a <5% increase) because each chip is sampled T/Tm times as often. (See Appendix [C](#A3 "Appendix C Analysis of prover evasion strategies ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") for details.)
Thus for simplicity, we assume the Verifier will choose s as though T is fixed.

To detect at least one chip with a snapshot from a rule-violating training run with probability p, the Prover needs to sample s chips each time period T, where:

|  |  |  |  |
| --- | --- | --- | --- |
|  | s≥log(1−p)log(1−HaTC(1−e−fTm))⋅TmT |  | (1) |

We provide a derivation for Equation [1](#S3.E1 "(1) ‣ 3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") in Appendix [B](#A2 "Appendix B Derivation of Sampling Rate ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").

In Table [1](#S3.T1 "Table 1 ‣ 3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), we provide the total required samples-per-year 365⋅s/Tm to catch several past and projected training runs, for Provers who own different quantities of chips C.

| Model | Training FLOPs H | H100-days H/a | H100s to train in 1 yr | Samples/yr if C=103 | Samples/yr if C=105 | Samples/yr if C=107 |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-3 [BMR+20] | 3.14e+23 | 3.64e+3 | 10 | 243 | 2.43e+4 | 2.43e+6 |
| Chinchilla [hoffman2022training] | 5.76e+23 | 6.67e+3 | 19 | 132 | 1.33e+4 | 1.33e+6 |
| PaLM [chowdhery2022palm] | 2.56e+24 | 2.96e+4 | 82 | 29 | 2.98e+3 | 2.99e+5 |
| Chinchilla-280B † | 9.9e+24 | 1.15e+5 | 314 | 7 | 771 | 7.72e+4 |
| Chinchilla-1T † | 1.27e+26 | 1.47e+6 | 4.03e+3 | — | 60 | 6.02e+3 |
| Chinchilla-10T † | 1.3e+28 | 1.5e+8 | 4.12e+5 | — | — | 58 |

Table 1:

Example numbers of required total *annual* samples 365⋅s/Tm to catch a chip from every large-scale training run within T=30 days,
given a=1015⋅24⋅3600 (the daily 16-bit Tensor Core FLOPs of an NVIDIA H100 SXM GPU [nvidiah100]),
f=0.1 weight snapshots per day (see Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")),
and the Verifier’s desired probability of catching a rule-violating training run p=0.9.
Models marked with † are projections for future training requirements [hoffman2022training].

As the table shows, the number of inspected chips remains approximately at a fixed fraction of the full stockpile, decreasing with training run size H.

An additional difficulty is the possibility that multiple Provers collude and each contribute a small fraction of their chips to a shared training run, such that if only one of the Provers is audited, the fraction of their C chips they used for the training run is much smaller than HaTmC.
This is particularly relevant in the International Oversight setting, where a government Prover might force the participation of multiple firms.
One solution to this is for an international Verifier to do background monitoring of *all* Provers, in effect treating them all as one large Prover with access to the global chip stockpile.
Predictably, this would require a much greater number of samples, and is likely to only be feasible for regulating the largest-scale training runs (which can be caught with fewer samples).
Yet, for training runs at the scale of 1025 FLOPs or greater, monitoring could be done with a bureaucracy similar in size to the IAEA.555
We want to estimate the number of inspectors needed to catch a Chinchilla-280B-sized training run, with 1025 FLOPs, given several more years of hardware progress and global production.
Given C=107 worldwide chips (>5× global stocks as of 2022), each of which can output a=3⋅1015⋅86400 FLOPs per day (3× more FLOP/s than the NVIDIA H100), detecting a Chinchilla-280B-sized run within T=30 days of its completion anywhere on earth with 90% probability would require roughly 232,000 worldwide chip samples per year.
A single inspector might be expected to verify at least 1000 chips a year, especially if those chips are brought to a central location (see Section [3.1](#S3.SS1 "3.1 Chip inspections ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")).
This would require ≈232 inspectors, slightly smaller than the 280 active IAEA inspectors as of 2021.
[dayinthelife].

As individual chips become more powerful (a grows) and algorithmic efficiency increases (H shrinks), the number of required samples would need to grow in order to verify rules on a fixed-capability training run.
Still, it may be that the training runs of greatest societal concern are those *near the frontier* of scaling, and those are likely to remain detectable with few samples for the foreseeable future, even if Provers attempt to undermine them with physical tampering.

4 On the chip
--------------

In an NN training cluster, a large number of ML devices (each containing an ML chip, and connected to a “host” CPU chip) collaborate to simultaneously compute gradient updates.
Large NNs have too many weights to fit in the high-bandwidth memory of a single ML device, so efficiently training a model requires splitting the model’s weights
into *weight shards*, and placing a shard on each chip [smith2022computation].
Each such shard is a vector of millions or billions of values.
Shards are updated after each gradient step so that new gradients can always be computed with respect to the latest weight values.
This means that at every timestep, the chip contains an up-to-date result from the training run, which we will show in Section [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") can serve as a type of checksum for the training run.

In this section, we will describe how it may be possible to have ML chips log the values of these weight-shard vectors, for Verifiers to retroactively inspect.
We will assume the presence of certain hardware features666These include standard components of a hardware security module: firmware verification and attestation, firmware anti-rollback protection, and the ability to sign, authenticate, and encrypt messages.
It also includs the ability for firmware to raise interrupts, read device memory, and (to efficiently hash weight snapshots) execute code on the chip’s processor.
on the ML chip, though the details of the approach may need to be modified depending on the specifics of the device.
For example, if the ML chip does not have the hardware required for remote firmware attestation, the chip may need to be permanently mated to a host CPU that does have such hardware, with the CPU then orchestrating the logging operations.

A chip’s weight-shards can be captured by *snapshotting* the chip’s high-bandwidth memory, which contains the weights.
These snapshots must be saved at random times, as otherwise the Prover can swap out the true weights for a fake set of weights right before the scheduled logging.
With some small probability at every clock cycle, the firmware present on each of the Prover’s chips should raise an interrupt and briefly freeze the chip’s operations.
The chip should then send a copy of the weight shard regions ω to its host CPU to save in Prover-controlled long-term storage.
The firmware also computes a hash h=H(ω) of the relevant memory regions,
777Such hashing can be done cheaply so long as the firmware can leverage the ML chip’s processing power.
using a publicly-agreed cryptographic hash function H.
This hash h is then saved for the Verifier to find it later, either by having the firmware store it on-chip888Keeping the hash in local NVRAM is useful if the ML chip is in a secure data-center and is restricted from communication with the outside world, such as via air-gapping. in nonvolatile memory writable only by the firmware, or by having the chip sign the hash with its private key and then sending it to a Verifier-trusted central server (which must return an acknowledgement of receipt before the chip can resume operating).
Later, a Verifier can use h to ask the Prover to provide them with the original weight shard ω, as this is the only input to H whose hash matches H(ω)=h.

More frequent snapshots mean more performance overhead for the Prover, especially if all the other synchronized chips in the training run must hang until the single chip’s snapshotting concludes.
This freezing may become expensive if the rest of the training run must hang until it completes, especially if it’s done by many chips.
999A possible alternative would be to orchestrate less-frequent snapshots via a Verifier-trusted central server, though this would require a solution to the problem of the Prover swapping out the chip’s weights before propagating the Verifier’s signal.
The frequency of snapshots should be set as low as possible while still ensuring that used chips reliably contain a relevant weight snapshot if they were employed in a large-scale training run (see Eq. ([1](#S3.E1 "(1) ‣ 3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."))).

Though the Prover may claim to have used firmware that logged snapshots as described above, the Verifier may not trust the Prover to have indeed used the appropriate logging firmware.
For this reason, ML chips also need hardware mechanisms for enforcing that firmware be signed, and in turn, chipmakers will need to only sign firmware if it implements appropriate logging mechanisms.
101010A similar firmware mechanism was used by NVIDIA to enforce cryptocurrency mining limits on its GPUs [gartenberg\_2021].
111111Assuming that logging-free versions have been signed in the past, ML chips may need to include anti-rollback functionality [krishnamurthy\_odonoghue\_bhatia].
The Prover might also delay updating the chip’s firmware until shortly before inspection.
To avoid this, ML chips may need to be made to remotely attest at an earlier time that their firmware has been updated.
121212Additional challenges arise if the Verifier suspects that the chipmaker (who authors the firmware) and Prover have colluded to create firmware with backdoors that disable logging.
Increasing Verifiers’ confidence in the firmware may be an important consideration when verifying the operations of powerful nation-state Provers.

An obstacle to logging all weight-shards stored in ML device memory is that different ML training code will store a model’s weights in different regions of memory.
The chip’s firmware must be able to determine which region of memory the weights are stored in.
It may be possible to determine the weight-regions retroactively, by logging the regions of memory claimed to correspond to the weights, along with a copy of the compiled on-device code, which can then be matched to Prover-provided source code and its memory allocation pattern analyzed.
131313It may even be possible to modify standard libraries for generating chip-level ML training code (e.g., PyTorch-generated CUDA) to make their memory allocation processes more easily checkable by a subsequent Verifier.
141414Revealing the Prover’s source code to the Verifier directly may be unacceptable, demanding a more complicated verification procedure like that described in Section [5.2](#S5.SS2 "5.2 Real-world implementation ‣ 5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").
As a more invasive alternative, the Prover could proactively certify that its chip-level ML training code stores the weights in a specific memory region, by having its chip-code verified and then signed by a Verifier-trusted server before it can be executed by the firmware.
151515The iOS App Store uses a similar method to ensure Apple devices only run signed programs [codesigning].

A more challenging problem is that “ML chips” are general-purpose hardware accelerators, and thus are frequently used for other high-performance computing workloads like climate modeling.
There is no straightforward way to determine whether an ML chip is running a neural network training job (and therefore should log weight snapshots), or an unrelated type of job exempt from Verifier oversight.
161616Potential avenues for addressing this may include be requiring non-ML-training code compilers to also sign their results, or improving methods for distinguishing between ML training code and other code.
If the types of code can be retroactively distinguished, then ML chips could all occasionally save memory/code snapshots, and then retroactively determine whether they belonged to a large-scale training run and thus deserve further scrutiny.
One particularly straightforward to address case is ML inference: the model’s in-memory weights could be snapshotted and retroactively verified in a similar way to that described in Section [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").

Relatedly, it would also be desirable to be able to exempt a fraction of chips from monitoring, for use in education or small-scale R&D where the overhead of compliance would be prohibitive.
171717This might be addressable by having these ML chips’ interconnect topology restricted to prevent their use in large-scale training.
However, methods for Verifiers to retroactively confirm the topology that ML chips were configured in at a data-center are beyond the scope of this work.
Until we find methods for reliably distinguishing large-scale ML training, some fraction of ML chips must be exempted from using weight-logging firmware.
The larger system’s guarantee would then be that *for those of the Provers’ chips that the Prover allows the Verifier to monitor*, no rule-violating training occurred.
The Verifier may then use their leverage to incentivize Provers into opting-in a larger and larger fraction of their chips over time.

5 At the data-center
---------------------

From Sections [3.2](#S3.SS2 "3.2 How many ML chips does the Verifier need to inspect? ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") and [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111."), we know how a Verifier can get access to at least one weight shard ω snapshotted during a large-scale training run.
In this section, we will show how the Verifier can use this weight shard to determine whether the associated training run followed the agreed-upon rules.

On its own, knowing ω is useless to the Verifier, since the Verifier does not know the values of the other weight-shards comprising the NN, or even how many other weight shards there are.
Yet ω holds the key to the training run’s history.
Any given weight-shard vector is incredibly unlikely to be yielded by the training process (there are ≫2106 possibilities even if we round to 1-bit precision).
But this particular ω is the nearly-deterministic result of the training run’s hyperparameters M (including its random seed) and sequence of training data D.
We refer to this information, along with information yielded throughout the training run W (such as intermediate weight snapshots), as the model’s *training transcript* T={M,D,W}.
It is plausible that it would be very difficult for the Prover to find a second “legitimate” training transcript T′={M′,D′,W′}≠T that would exactly yield a full set of weights that included ω as a contiguous weight-shard.
It is also plausible that, if the Prover tried to lie and provide an H′,D′, and M′ that would *not* result in ω but which they *claimed* would result in ω, then the Verifier might find a suite of tests that would detect such a faked transcript reliably.
If the Verifier could reliably detect such fakes,
then by asking the Prover to provide a training transcript that would yield each logged snapshot ω, the Verifier could force the Prover to reveal the true data D, and hyperparameters M used in every large-scale training run (along with the full weights of the resulting model, not just ω).
This would be sufficient information to check that the training run complied with any of the training rule types listed in Section [2.1](#S2.SS1 "2.1 What types of rules can we enforce by monitoring ML training? ‣ 2 The Problem: Detecting Violations of Large-Scale ML Training Rules ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").

Unfortunately, having the Verifier simply re-run the full training process to verify a transcript’s correctness is unworkable for several reasons.
First, the Prover would likely not be willing to reveal their training data, model weights, and hyperparameters, so the Verifier must do any verification without direct access to the data. (We address this in Section [5.2](#S5.SS2 "5.2 Real-world implementation ‣ 5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").)
Second, the compute cost to fully re-run the training transcript would be massive, as large as every original training run.
Third, the training run would likely not be perfectly reproducible: due to hardware-level noise, even two repetitions of the same sequence of training updates would gradually diverge.
Fourth, the Prover *may* be able to construct a second “spoof” training transcript, that yields an exact match for ω but differs from the original training run that yielded ω in the first place.181818As a trivial example, the Prover could claim that ω was simply the value of the random initialization, and no training had happened at the time of the snapshot.

Thanfully, a close variant of this problem has already been studied in the literature, known as “Proof of Learning” [jia2021proof].
The goal of a Proof-of-Learning (PoL) schema is to establish proof of ownership over a model Wt (e.g., to corroborate IP claims) by having the model-trainer save the training transcript T (including hyperparameters M, data sequence D, and a series of intermediate full-model weight checkpoints191919We use “weight checkpoints” as shorthand, but if using an optimizer like Adam [KingmaB14], the optimizer state should also be included. W={W0,Wk,W2k…}) which only the original model trainer would know.
Jia et al. [jia2021proof] propose a verification procedure that makes it difficult for any third party to construct a spoofed transcript T′, if they only have access to Wt and the unordered dataset.

The solution of [jia2021proof] is as follows: once a Prover reports a training transcript T, the Verifier checks that the initialization appears random, and then chooses a number of pairs of adjacent weight snapshots that are k gradient steps apart (Wi,Wi+k),…,(Wj,Wj+k).
Then, rather than re-running all of training, the Verifier only reruns the training of these specific segments, starting at Wi and progressing through the appropriate data batches Di…Di+k to yield a W′i+k.
The Verifier then confirms that the resulting checkpoint is approximately correct: ∥Wi+k−W′i+k∥2<ϵ for some small ϵ that depends on the reproduction noise and the number of steps k.
The original paper proposed a heuristic for choosing which segments to verify,
though subsequent papers have demonstrated spoofing attacks that evade the original verification approach [zhang2022adversarial, fang2022fundamental].
Thus far, the literature on PoL has been heuristic-based, and discovering PoL protocols with higher degrees of reliability is the subject of ongoing work [fang2022fundamental].

###
5.1 Proof-of-Training-Transcript

Our requirement, which we will define as “Proof-of-Training-Transcript” (PoTT), differs from Proof-of-Learning because our Prover knows a true training transcript that leads to the final weight-snapshot (since they executed the original training run), and the Verifier wants to verify that the Prover has reported the *original* transcript that yielded ω (or at least one very close to the original).
Further, unlike PoL, we allow that an adversarial Prover might be willing to spend *more than the original training compute* H in order to construct a spoof (e.g., an adversary may be willing to pay double the cost in order to execute a rule-violating training run without being detected).
Thus, we must hope to push the cost to construct a spoof to be as high as possible, so as to make evasion impractical.

######
Definition 1.

A PoTT protocol (R,V) is defined by a Prover training and reporting protocol R and a Verifier verification protocol V.
A target weight vector Wt is produced as the true result of a training run which complies with a training and reporting protocol R, and which can be summarized via a training transcript T∈T, where T={M,D,W}, M is the metadata required to reproduce the training run (including hyperparameters, random seed, and loss function), D is an ordered sequence of sets of training data batches, and W is a collection of intermediate weight checkpoints resulting from the training procedure.
The verification protocol V should accept any such true training transcript with high probability, Pr[V(T,Wt)=accept]>1−δ1 for some small δ1.

A “spoofed” training transcript T′={M′,D′,W′} is a transcript, which may not correspond to any valid training run, and which is substantially different from the original transcript T in its data or hyperparameters: d1(D,D′)≥δ3 for some edit distance d1 quantifying the number of data point insertions/deletions, and/or d2(M,M′)≥δ4 for some hyperparameter distance d2.
A reporting/verification protocol pair (R,V) is J-efficient and F-hard if V runs in at most J time, and there does not exist any spoof-generation algorithm A∈A:T→T such that Pr[V(A(T),Wt)=accept]>1−δ2 where A runs in less than F time.

Colloquially, we want a Prover training and reporting protocol and Verifier verification protocol such that the Verifier only accepts *original* training transcripts that would result in a final weight checkpoint which contains a shard matching our on-chip weight-shard snapshot ω.
We leave the problem of finding provably secure, efficient methods for PoTT as an important avenue for future work, but we discuss a few promising directions below.

PoTT appears to be strictly harder than PoL, as it requires robustness to a better-resourced adversary has additional information (i.e., they know the true transcript T) and has more compute-time to construct a spoof.
Given that existing PoL schemes are still heuristic-based and not yet provably secure, there may be a long way to go until we have PoTT methods that are both efficient and hard to spoof.
Still, one dynamic favoring the Verifier is that the Prover must *commit* to a training transcript without knowing the Verifier’s verification strategies.
Thus, Verifiers can amass secret collections of verification heuristics, much as the IAEA does not disclose all its methods for detecting nuclear activity.
Even if PoTTs are only ever heuristic-based, the presence of this dynamic may dissuade Provers from taking the risk of being detected by an unexpected test.

Defining conditions on the types of legitimate training runs is another useful source of leverage.
For example, one Prover cheating strategy could be for the Prover to report one long training run as many shorter training runs, each claimed to be “initialized” where the previous training run ended.
A simple prevention would be for the training-and-reporting protocol R to require the Prover to initialize every training run’s weights via a known pseudorandom generator and a short seed.
This means that the randomness of the initial weights can later be confirmed by the Verifier.

Another promising strategy may be to require the Prover to *pre-commit* to portions of its training transcript (e.g., the hyperparameters M and training batches D) at the start of training.
This could be done by having the ML chip firmware log a hash of this precommitment, which would prove that the precommitment preceded the chip’s snapshot ω.
At the time of precommitment, the Prover does not know what trajectory the training run will follow or at what time it will be snapshotted, as the training has not yet been performed.
The Prover would be unable to construct a spoofed training transcript that would end at ω and precommit to it, because ω isn’t known yet.
However, it is not obvious how to extend this approach to online learning settings like online RL, where the training data cannot be known ahead of time.

A final complication of our setting derives from the fact that the Verifier observes only a shard of the weights ω, and not the full weight vector Wt.
It could be easier to construct a spoofed training transcript for some ^W which contains a shard matching ω, but which differs from the true original weights Wt≠^W on the portion of the weight vector outside the shard.
We briefly describe an intuition for why this is likely to be as hard as general PoTT-spoofing.
Assuming ω must contain weights from more than a single linear layer, any Prover must at minimum construct a valid PoTT for this smaller NN represented by ω, except without any original training transcript to start from (making it similarly hard to the original Proof of Learning problem).
Alternatively, if the Prover tries to reuse the original training transcript, it will require them to synthesize fake data and labels to exactly reproduce the true inputs/gradients, and these must correspond to a reasonable loss function that is actually decreasing over time.
If the Prover alternatively attempts to fake the surrounding layers’ weights, then the weights in these surrounding layers must also have their own valid training trajectories as part of the training transcript *and* must generate the appropriate inputs/gradients to yield ω, simultaneously.

###
5.2 Real-world implementation

As the literature uncovers new methods for verifying training transcripts, the Verifier can run these protocol to check that, for each of the logged weight snapshots on the Prover’s chips that the Verifier inspects, the Prover has provided a legitimate corresponding training transcript.202020
Note that this requires the Prover to save and report training transcripts for all training runs corresponding to chips sampled by the Verifier, not just the largest-scale ones.
This is because, without a matching training transcript, it’s impossible for the Verifier to determine whether a given weight-shard was part of a large-scale training run or a small-scale one.
Alternate methods for proving that a chip was only involved in a short training run are an important direction for future work.

The Verifier can then examine the data, hyperparameters, and resulting models as reported by the trianing transcript, and thereby detect whether the Prover complied with the training rules.

A key design constraint is that the Prover does not disclose the raw transcripts (which likely contain confidential data that the Prover would prefer the Verifier did not see).
We propose a method for the Verifier to verify the Prover’s previously-reported transcript’s correctness without ever observing the raw data.
First, the Prover and Verifier utilize a ‘‘trusted cluster’’: a set of chips that both parties trust, which is air-gapped to prevent the Prover’s sensitive data from leaking, and which is equipped to execute small amounts of ML inference and training (so as to segment-wise approximately reproduce the Prover’s original training process).
212121Maintaining such compatible training clusters may prove quite challenging.
One desirable direction for future work is in verification methods by which the Verifier does not need to directly reexecute model training.
For example, it may be possible for the Verifier to interactively query the Prover for additional information on segments of the training run beyond what was included in the training transcript.
There may be queries that have valid answers *only if* the original training transcript was authentic (e.g., a series of weight sub-checkpoints between two checkpoints, each with progressively lower loss), and the Prover could dynamically recompute answers to these queries using their own data-center.
While some properties of the verification would still need to be confirmed using a neutral cluster to maintain the confidentiality of the query-responses, such clusters may not need to be equipped for large-scale training, and thus be much easier to maintain.

Second, instead of reporting the full transcript, the Prover initially only provides the Verifier with hashes of the training transcript T.
Then, at a later verification stage, the Prover supplies the true M,D, and W within the trusted cluster (which must match the reported hashes), executes the verification protocol, and reports whether both steps succeeded.
Finally, the Verifier can use this same trusted cluster to run programs checking that the Prover’s training transcript complies with all agreed ML training rules (e.g., measuring the performance of the final model weights on agreed benchmarks).
We provide more details on a possible procedure in Appendix [D](#A4 "Appendix D Logistics for Confidentially Verifying Proofs-of-Training-Transcript ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").

When evaluating which training transcript verification protocols to pursue, Verifiers should prioritize those verification strategies that get the most detection-probability, for the lowest cost.
Beyond the upfront costs of building trusted clusters or modifying chip hardware, the system has three ongoing operating costs:
the efficiency loss from pausing to save weight checkpoints and the weight-shard snapshots (as described in Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")), the storage costs for maintaining training transcripts (and in particular the weight-checkpoints, each of which may require terabytes) until the Verifier inspects them, and the compute costs to execute the training-transcript verification protocols on the trusted clusters.
These costs seem likely to scale linearly with the total compute used by the Prover, and will ultimately depend on the efficiency with which training transcripts can be verified.
Even though governments could in principle pressure Provers into paying the costs of compliance, a 1% overhead for each dollar spent on training compute would be much easier for Provers to comply with than a 20% overhead.
Indeed, for International Verifiers, the history of arms control suggests that maximally-stringent verification measures may have limited utility, as they may reduce the likelihood of compliance [reddie2019governing].
One important avenue for future work is finding cheaper, lower-fidelity alternatives to NN-retraining-based verification, which need only establish limited properties of the weight-shard’s corresponding training run, and which could prompt more expensive verification methods if needed.

6 At the supply chain
----------------------

We need supply-chain monitoring to accomplish two goals: to construct a “chip directory” of who owns each ML chip, for the purposes of sampling; and to ensure that each chip has the hardware features needed to provably log its training activity as in Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").
Unlike the chip and data-center interventions (Sections [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") and [5](#S5 "5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.")), monitoring the international ML chip supply chain cannot be done by a single Verifier.
Instead, an international consortium of governments may need to implement these interventions on behalf of other Verifiers (much as the IAEA runs inspections on behalf of member states).

###
6.1 Creating a chip-owner directory

For a Verifier to be confident that a Prover is reporting the activity of all the Prover’s ML chips, they need to know both which ML chips the Prover owns, and that there are no secret stockpiles of chips beyond the Verifier’s knowledge.
Such ownership monitoring would represent a natural extension of existing supply chain management practices, such as those used to enforce U.S. export controls on ML chips.
It may be relatively straightforward to reliably determine the total number of cutting-edge ML chips produced worldwide, by monitoring the production lines at high-end chip fabrication facilities.
The modern high-end chip fabrication supply chain is extremely concentrated, and as of 2023 there are fewer than two dozen facilities worldwide capable of producing chips at a node size of 14nm or lower [semiconductorwiki], the size used for efficient ML training chips.
As [baker2023nuclear] shows, the high-end chip production process may be monitorable using a similar approach to the oversight of nuclear fuel production (e.g., continuous video monitoring of key machines).

As long as each country’s new fab can be detected by other countries (e.g., by monitoring the supply chain of lithography equipment), an international monitoring consortium can require the implementation of verification measures at each fab, to provide assurances for all Verifiers.
After processing, each wafer produced at a fab is then sent onward for dicing and packaging.
Since the facilities required for postprocessing wafers are less concentrated, it is important for the wafers (and later the dies) to be securely and verifiably transported at each step.
If these chip precursors ever go missing, responsibility for the violation would lie with the most recent holder.
This chain of custody continues until the chip reaches its final owner, at which point the chip’s unique ID is associated with that owner in a *chip owner directory* trusted by all potential Verifiers and Provers.
This ownership directory must thereafter be kept up-to-date, e.g., when chips are resold or damaged.222222In the rare scenario where a large number of chips owned by the same Prover are lost or destroyed beyond recognition, the Verifier or international consortium can launch an investigation to determine whether the Prover is lying to evade oversight.
The continued accuracy of this registry can be validated as part of the same random sampling procedure discussed in Section [3.1](#S3.SS1 "3.1 Chip inspections ‣ 3 Solution Overview ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").
As a second layer of assurance, chips could also be discovered by inspecting datacenters, if those datacenters are detectable via other signals [baker2023nuclear].

Given the high prices and large power and cooling requirements of these ML chips, they are largely purchased by data-center operators.
These organizations are well-suited to tracking and reporting transfers of their ML chips, and complying with occasional inspections.
Though a small fraction of data-center ML chip purchases are made by individuals, so long as these are a small fraction of chips they may be exempted from the overall monitoring framework.

###
6.2 Trusting secure hardware

We require in Section [4](#S4 "4 On the chip ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.") that each ML chip produced by the semiconductor supply chain is able to provably log traces of its usage.
The second goal of supply-chain monitoring is to provide Verifiers with high confidence in the reliability of these on-chip activity-logging mechanisms.
This requires ML chip designers to integrate security features into their hardware and firmware designs, especially in ways that make them externally-legible to Verifiers that may not trust the chip-designer.
Key priorities include the immutability of the chip’s burned-in ID, the integrity of the hardware-backed mechanism for only booting signed firmware, and the resilience of the on-chip hardware-roots-of-trust to side-channel attacks that could steal the chip’s encryption keys [crypto-1996-1469, 6271612] and thus fake its logs.

A concern for Verifiers checking the conduct of powerful Provers (e.g., states verifying each others’ ML training runs) is the possibility of supply-chain attacks [robertson\_riley\_2021], which could enable a Prover to undetectably disable/spoof the ML chips’ logging functionality.
Fully mitigating the threat of supply-chain attacks is a major global issue and beyond the scope of this paper.
However, one particularly useful step for building trust in ML chip mechanisms’ integrity would be for ML chip designers to use open-source Hardware-Roots-of-Trust.
This transparency means that chips’ designs can be validated by untrusting Verifiers to confirm there are no backdoors.
For example, Google’s Project OpenTitan has produced such an HRoT [opentitan], and many major ML chip designers (Google, Microsoft, NVIDIA, and AMD) have agreed to integrate the Open Compute Project’s “Caliptra” Root of Trust. [caliptra]

7 Discussion
-------------

The described additions to the production and operation of ML training chips, if successfully implemented, would enable untrusting parties (like a government and its domestic companies, or the US and Chinese governments) to verify rules and commitments on advanced ML development using these chips.
There are many useful measures that governments and companies could begin taking today to enable future implementation of such a framework if it proved necessary, and that would simultaneously further businesses’ and regulators’ other objectives.

* Chipmakers can include improved hardware security features in their data-center ML chips, as many of these are already hardware security best practices (and may already be present in some ML chips [nvidiah100]).
These features are likely to be independently in-demand as the costs of model training increase, and the risk of model theft becomes a major consideration for companies or governments debating whether to train an expensive model that might simply be stolen.
* Similarly, many of the security measures required for this system (firmware and code attestation, encryption/decryption modules, verification of produced models without disclosing training code) would also be useful for “cloud ML training providers”, who wish to prove to security-conscious clients that the clients’ data did not leave the chips, and that the clients’ models did not have backdoors inserted by a third party [liu2017trojaning].
Procurement programs like the US’s FedRAMP could encourage such standards for government contracts, and thereby incentivize cloud providers and chipmakers to build out technical infrastructure that could later be repurposed for oversight.
* Individual companies and governments can publicly commit to rules on ML development that they would like to abide by, if only they could have confidence that their competitors would follow suit.
* Responsible companies can log and publicly disclose (hashed) training transcripts for their large training runs, and assist other companies in verifying these transcripts using simple heuristic.
This would not prove the companies *hadn’t also* trained undisclosed models, but the process would prove technical feasibility and create momentum around an industry standard for (secure) training run disclosure.
* Companies and governments can build trusted neutral clusters of the sort described in Section [5.2](#S5.SS2 "5.2 Real-world implementation ‣ 5 At the data-center ‣ A template for Arxiv Style Citation: Authors. Title. Pages…. DOI:000000/11111.").
These would be useful for many other regulatory priorities, such as enabling third-party auditors to analyze companies’ models without leaking the model weights.
232323For similar reasons, the US Census Bureau operates secured Federal Statistical Research Data Centers to securely provide researchers access to sensitive data [uscensusbureau\_2022].
* Governments can improve tracking of ML chip flows via supply-chain monitoring, to identify end-users who own significant quantities of ML chips.
In the West, such supply-chain oversight is already likely to be a necessary measure for enforcing US-allied export controls.
* Responsible companies can work with nonprofits and government bodies to practice the physical inspection of ML chips in datacenters.
This could help stakeholders create best practices for inspections and gain experience implementing them, while improving estimates of implementation costs.
* Researchers can investigate more efficient and robust methods for detecting spoofed training transcripts, which may be useful for in proving that no backdoors were inserted into ML models.

For the hardware interventions, the sooner such measures are put in place, the more ML chips they can apply to, and the more useful any verification framework will be.
Starting on these measures early will also allow more cycles to catch any security vulnerabilities in the software and hardware, which often require multiple iterations to get right.

###
7.1 Politics of Implementation

Given the substantial complexity and cost of a monitoring and verification regime for large-scale ML training runs, it will only become a reality if it benefits the key stakeholders required to implement it.
In this last section, we discuss the benefits of this proposal among each of the required stakeholders.

* *The global public*: Ordinary citizens should worry about the concentration of power associated with private companies possessing large quantities of ML chips, without any meaningful oversight by the public.
Training run monitoring is a way to make powerful companies’ advanced ML development accountable to the public, and not just the free market.
Most importantly, ordinary people benefit from the security and stability enabled by laws and agreements that limit the most harmful applications of large-scale ML systems.
* *Chipmakers and cloud providers*: Absent mechanisms for verifying whether ML chips are used for rule-violating training runs, governments may increasingly resort to banning the sale of chips (or even cloud-computing access to those chips) to untrusted actors [bis2022controls].
By enabling provable monitoring of large-scale ML training runs, chipmakers may reverse this trend and may even be able to resume sales to affected markets.
* *AI companies*: Responsible AI companies may themselves prefer not to develop a particular capability into their products, but may feel they have no choice due to competitive pressure exerted by less-scrupulous rivals.
Verifying training runs would allow responsible AI companies to be recognized for the limits they impose on themselves, and would facilitate industry-wide enforcement of best practices on responsible ML development.
* *Governments and militaries*: Governments’ and militaries’ overarching objective is to ensure the security and prosperity of their country.
The inability to coordinate with rivals on limits to the development of highly-capable ML systems is a threat to their own national security.
There would be massive benefit to a system that enabled (even a subset of) countries to verify each others’ adherence with ML training agreements, and thus to maintain an equilibrium of responsible ML development.

Even if only a subset of responsible companies and governments comply with the framework, they still benefit from verifiably demonstrating their compliance with self-imposed rules by increasing their rivals’ and allies’ confidence in their behavior [horowitz2021ai] (and thus reducing their rival’s uncertainty and incentive towards recklessness).

Finally, we highlight that the discussed verification framework requires continuous participation and consent by the Prover.
This makes the framework fundamentally non-coercive, and respects national sovereignty much as nuclear nonproliferation and arms control agreements respect national sovereignty.
Indeed, the ongoing success of such a system relies on all parties’ self-interest in continuing to live in a world where no one – neither they, nor their rivals – violates agreed guardrails on advanced ML development.

Acknowledgements
----------------

The author would like to thank
Tim Fist,
Miles Brundage,
William Moses,
Gabriel Kaptchuk,
Cynthia Dwork,
Lennart Heim,
Shahar Avin,
Mauricio Baker,
Jacob Austin,
Lucy Lim,
Andy Jones,
Cullen O’Keefe,
Helen Toner,
Julian Hazell,
Richard Ngo,
Jade Leung,
Jess Whittlestone,
Ariel Procaccia,
Jordan Schneider,
and Rachel Cummings Shavit
for their helpful feedback and advice in the writing of this work.

\printbibliography

As advanced machine learning systems' capabilities begin to play a significant role in geopolitics and societal order, it may become imperative that (1) governments be able to enforce rules on the development of advanced ML systems within their borders, and (2) countries be able to verify each other's compliance with potential future international agreements on advanced ML development. This work analyzes one mechanism to achieve this, by monitoring the computing hardware used for large-scale NN training. The framework's primary goal is to provide governments high confidence that no actor uses large quantities of specialized ML chips to execute a training run in violation of agreed rules. At the same time, the system does not curtail the use of consumer computing devices, and maintains the privacy and confidentiality of ML practitioners' models, data, and hyperparameters. The system consists of interventions at three stages: (1) using on-chip firmware to occasionally save snapshots of the the neural network weights stored in device memory, in a form that an inspector could later retrieve; (2) saving sufficient information about each training run to prove to inspectors the details of the training run that had resulted in the snapshotted weights; and (3) monitoring the chip supply chain to ensure that no actor can avoid discovery by amassing a large quantity of un-tracked chips. The proposed design decomposes the ML training rule verification problem into a series of narrow technical challenges, including a new variant of the Proof-of-Learning problem [Jia et al. '21].


Introduction to Compute Governance


Nuclear proliferation (and Nonproliferation) Explained


Career Resources on U.S. AI Policy