Footnotes

[1]
The AI 2027 scenario is still roughly what we expect the future to look like: a mad scramble to superintelligence leading to either AI takeover or extreme concentration of power. So far reality is tracking closer to AI 2027 than even we expected. (2027 was our modal year at time of publication, not our median.) You can read more about our views on timelines here and here.
↩
[2]
That is, they are predictions about what would happen next if our recommendations were implemented.
↩
[3]
You can read more about these four plans at the branch point below and in our supplement “How good is each plan?”.
↩
[4]
Of course, we’re not sure exactly how likely literal AI-driven human extinction is, but opinions within our team vary between 10% and 30%. This is primarily because there are a bunch of other things misaligned ASIs might do with us after they take over, besides kill us all. For example, they might keep some people alive because it’s really cheap and they care a tiny amount, or as acausal bargaining chips. We don’t think this nuance changes the bottom line though: Misaligned AI takeover is probably really bad.
↩
[5]
I (Daniel Kokotajlo) recently gave a talk to about 100 people, ~40% of whom were from frontier AI companies, and did a show-of-hands poll about (a) how many months of lead the leading AI company would have at the time they first fully automate AI R&D, and (b) how much of their lead they would be willing to burn. The median answers were roughly 3 months and 1/3rd, respectively. This matches my own guess.
↩
[6]
Also intense secrecy and groupthink conditions! By default during this period the companies will be even more closed than they are in 2026. When they say “we’ll solve the alignment problems as we go” what they really mean is “The overworked tiny group of alignment experts at our company will solve the alignment problems as we go, without being able to subject their work to external review.”
↩
[7]
We expect them to maintain apparent control, to be clear, at least for some time, which is part of why this problem is so dire. The situation would be much better if we had a reliable way of telling whether an AI system was actually robustly obedient/loyal/aligned/etc. vs. merely temporarily so vs. merely pretending.
↩
[8]
The AI 2027 Slowdown ending depicts this happening. It’s quite easy to imagine because it doesn’t involve anything cartoonishly or overtly evil, but rather just a series of steps that can be justified individually (consolidating US compute into a single pool, beating China, integrating AI into the government and economy, preventing terrorists from getting their own AGIs, etc.)
↩
[9]
For example, these old OpenAI emails reveal that OpenAI was founded in significant part because of fear that Demis Hassabis, DeepMind CEO, would use AGI to become dictator. See also related reporting in Empire of AI. See also this and this and this.
↩
[10]
For one thing, it’s hard to apply scenario scrutiny to a policy proposal if you have only a shallow understanding of it, and people generally understand their own favorite ideas much better than they understand the ideas they hate. For another, the evidential force of scenario scrutiny is inherently stronger for scenarios written by proponents than by opponents: If a proponent of a policy tries their best to depict it succeeding, and fails, that’s a lot more evidence than if an opponent of a policy depicts the policy failing.
↩
[11]
That is, we are uncertain about how fast AI capabilities will progress – about how much time remains before the AI companies succeed at automating their research and accelerating towards superintelligence, for example.
↩
[12]
Superintelligence means AI systems that are significantly better than the best humans at everything, while also being faster and cheaper.
↩
[13]
By the time we finished writing AI 2027, Daniel’s AGI median was 2028. Other authors’ medians were between 2031 and 2035. To read more about how Daniel and Eli’s views have shifted over time, see here. We are regularly updating our forecasts here, with corresponding blog posts on our Substack.
↩
[14]
AIs are helping to make future AIs more capable, but AI development still requires humans.
↩
[15]
This policy is selfishly good for leading AI companies insofar as it makes it harder for trailing companies to catch up, but it’s also good for safety because it slows irreversible proliferation of algorithmic insights.
↩
[16]
We think the concerns around current AI water usage are largely overstated. See this article for more. That said, as will be apparent later in the scenario, we do think that the environmental impact of AI over the next decade or two will be enormous, and that unrestrained growth would literally boil the oceans. See the energy appendix in our economics supplement for more on this.
↩
[17]
Are we really in such a race? Sort of, but not exactly.
On our default trajectory, AI will soon become very powerful, to the point where it's more strategically important than nuclear weapons; that much is true. Many people, including the CEOs of frontier AI companies, are trying hard to build smarter and smarter AI systems before their competitors do.
However, being overly paranoid about competitors is a well-known bias. Assuming that we’re in a winner-takes-all race rules out many possible avenues for cooperation, creating a self-fulfilling trap. we expect that the scale and speed of AI progress will make it increasingly obvious to decision-makers in both the US and China that the current pace of progress, and lack of trust between them, is extremely dangerous. Indeed, it may be possible for the two countries to coordinate on a slowdown without any binding agreement, if they build trust by each gradually deescalating. Our proposed Plan A should be interpreted as an approach that could work even if the US and China have absolutely zero trust for each other, rather than a claim that strict verification protocols are definitely necessary.
↩
[18]
That is, not Congress. By default the people who control the AIs, if anyone truly does, will be AI companies or maybe the White House.
↩
[19]
From Ilya and Greg: “The goal of OpenAI is to make the future good and to avoid an AGI dictatorship. You are concerned that Demis could create an AGI dictatorship. So do we. So it is a bad idea to create a structure where you could become a dictator if you chose to, especially given that we can create some other structure that avoids this possibility.”
↩
[20]
We don’t mean to recommend bad things; we are predicting that a mixture of good and bad incrementalist reforms will happen in 2027. For more on how we balance predictions and recommendations in this scenario, see here.
↩
[21]
Experiment and training compute are major inputs to AI research progress historically and will probably continue to be so for the foreseeable future. For more on the drivers of AI research progress, see the AI Futures Model and our Covert Projects Supplement.
↩
[22]
A slightly more complicated version of this proposal is described here. One objection is that this would give China the lead. The reply is that it wouldn’t, because US companies spending 25% on R&D would still be significantly outspending the most well-resourced Chinese companies. Moreover, Chinese AI progress would slow down too since some fraction of it these days comes from distilling US models.
↩
[23]
That is, the AI industry spends $2.4T of CapEx over 2028, triple what was spent in 2026. The US military budget in 2025 was about $1T. The biggest AI company has $360B ARR at the beginning of 2028 and is still growing at a roughly 150% annualized growth rate, putting it on track to become both the most valuable company in the world and the largest company in the world by revenue, in 1-2 years.
↩
[24]
After all, he’ll still be alive after leaving office. Just like the rest of us, he’ll have to endure whatever crises result from superintelligence, and hope that the new President handles them well and doesn’t become dictator. It’s in his interest both to set the new President up for success with respect to the loss-of-control problem, and to install checks and balances to prevent extreme concentration of power.
↩
[25]
To spell it out more: They were concerned that the US might use their massive AI advantage to cripple Chinese AI projects via cyberattacks, physical sabotage, or other means, and then use the resulting even-bigger, longer-lasting AI advantage to dictate terms to the CCP or even overthrow it entirely. Preventing loss-of-control risk was merely an added bonus.
↩
[26]
For details on how a treaty like this might look, see this paper. Another similar proposal is A Narrow Path.
↩
[27]
If all large new training runs are prevented, the valuations of some existing AI companies might actually rise, because their models would no longer face competition. However, by and large most AI valuations tend to price in the possibility of fast capability growth, and so on average they will crash.
↩
[28]
Achieving this is more difficult than it is in the Plan A scenario, because in the Plan A scenario, AI progress within the deal is going to continue, and even though it’s going at a throttled, cautious pace, it’ll probably still be faster than what many countries could achieve on their own. Whereas in Plan S, AI progress is halted within the deal, meaning that countries outside the deal can hope to get an economic and military advantage for themselves even if they are proceeding at a snail’s pace on only a million GPUs. (Whether they in fact get that advantage depends on whether the countries in the deal notice and stop them…)
↩
[29]
Demand for AI chips will continue to grow in absolute terms, as people find more ways to integrate existing AI models into the economy. But the rate of growth will be less explosive than it would have been had AI progress continued.
↩
[30]
In the 2030s, the price of energy should be starting to drop as solar power continues decades-long trends of incremental improvements and economies of scale. Also, launch costs to orbit should be about an order of magnitude cheaper than in 2026, thanks to Starship. In general, scientific progress in many domains will have continued and the world will feel increasingly cyberpunk.
↩
[31]
More notes on the sorts of large neural networks that would plausibly be prohibited under Plan S: (i) Autonomous agents, (ii) Those that can significantly accelerate AI research, (iii) Those that are good at persuading or manipulating people, (iv) Those with dangerous capabilities more generally, such as bioweapon assistance, (v) Self-aware / situationally aware neural networks.
↩
[32]
We analyze these dynamics much more in our Covert AI Projects Supplement. The supplement analyzes things from a Plan A perspective. The main difference is that in Plan S, covert projects will go much slower because there are no legal projects that are leaking algorithmic progress, and no risk of distillation.
↩
[33]
We think it’s at least better than plans B, C, and D, for example.
↩
[34]
Analogy: Suppose a bus finds itself driving off-road in the fog, at high speed. The passengers debate with the driver how dangerous this is and what is to be done about it. The driver says “You wanna get to your destination, don’t you?” It’s reasonable in this situation to reply “First, let’s stop the bus for a while, then let’s figure out how to proceed. We’ll need a plan for how to proceed eventually, but it’s unreasonable to barrel blindly through the fog while waiting for such a plan to be fleshed out.”
↩
[35]
Perhaps it would happen in a world devastated by climate change or pandemics or some other sort of civilizational catastrophe.
↩
[36]
Note that in any Plan, including Plan A, militaries would be preparing options for things like this. Supporting a credible deterrence posture with a spectrum of offensive options is just part of what militaries do, and it contributes leverage in negotiations.
↩
[37]
Perhaps one CEO will end up effectively in charge, the power behind the throne, the General Groves of the Manhattan project. (Except Groves couldn’t turn around and use the nukes to take over the USA, whereas whoever is in charge of The Project plausibly could use the giant army of superintelligences to subvert and puppet the US government.) Or perhaps the POTUS will win instead, and basically end up in a position to become President for Life. Or perhaps these powerful men will work out some system of checks and balances, becoming a sort of junta or oligarchy like the Oversight Committee in AI 2027. Or perhaps instead they will allow external entities like congress, SCOTUS, the American public, allied nations, etc. enough oversight over what they are doing with the AIs that later, when the AIs are superintelligent, they won’t be able to abuse their power.
↩
[38]
We are trying to present a reasonably good version of Plan B here. Worse versions are also possible, e.g., versions in which the technical alignment researchers have much less power or got their jobs via a selection process that selected for optimism or against courage.
↩
[39]
The company employees and leaders were selected for optimism about AI alignment difficulty, and (on average) biased in that direction. The natsec and White House people are more naturally cautious about trusting AIs, but lack technical expertise and fear China more than anything.
↩
[40]
Our non-expert opinion, based on attempts to look into this, is roughly that surgical cyberattacks focused on specific AI projects could potentially crush them today but by 2029 or so security will have been improved enough that effects on the order of 10% seem more likely.
↩
[41]
We’ve spent some time gaming out AI-war scenarios with an eye to figuring out how such a war might play out and how much it would slow down AI progress. Our guess is that it would slow things down by a few years at most, unless it escalated to massive missile barrages or nukes.
↩
[42]
Daniel in particular thinks that this is actually what a typical attempt to implement Plan B would look like, rather than an unusually bad version of it. Daniel thinks that a good version of Plan B is unlikely. Eli, by contrast, is more optimistic and thinks that while the median implementation would be much less competent than ideal, it would still be substantially better than Plan D because there would be more focus and resources spent on safety and more of a slowdown.
↩
[43]
That said in other ways the early escalation might make it ‘easier’ to switch to Plan A later, because it might cause other countries to ‘wake up’ faster and demand a deal.
↩
[44]
Note: our opinions vary about exactly how likely misaligned AI takeover is in this scenario. Thomas and Daniel think it’s roughly 70%, Eli and Romeo think it’s about 50/50, Brendan roughly 1/3rd, Ryan roughly 20%.
↩
[45]
This is by no means the only reason things might go wrong even if the AIs to which the safety teams hand off are trying to make the situation go well. The AIs might not be capable enough at safety work. They might not be wise enough. They might be too lazy on hard-to-verify tasks.
↩
[46]
For more on what that would look like, read the AI 2027 slowdown ending, and pay attention to the various opportunities the Oversight Committee has to bend things in favorable directions.
↩
[47]
Note of dissent: 1/6 authors disagree with this as stated, and think that the probability of losing control in those circumstances is more like 40%.
↩
[48]
To spell it out more: They were concerned that the US might use their massive AI advantage to cripple Chinese AI projects via cyberattacks, physical sabotage, or other means, and then use the resulting even-bigger, longer-lasting AI advantage to dictate terms to the CCP or even overthrow it entirely. Preventing loss-of-control risk was merely an added bonus.
↩
[49]
Geopolitically the situation will have stabilized within a few years, but by that point AI progress and diffusion will have happened and caused new problems and crises for people to worry about.
↩
[50]
We show our estimate of the distribution of datacenter sizes in our compute supplement.
↩
[51]
The initial chip declaration would only involve companies whose purchases exceed 10k H100e (which costs ~$100M). Within a year, every company owning above 0.001% of the world’s compute (~2,800 H100e, costing ~$18M at the time) has its holdings accounted for. Dead chips are also stored or verifiably destroyed.
↩
[52]
The 1% of compute sales that cannot be traced to their end location are mostly due to smugglers who did not record their customers’ identities and chips that were decommissioned. Half of this pool is later found installed in Chinese datacenters identified via satellite imagery; both countries offer monetary incentives and legal amnesty to any owners of the remaining 0.5%. We go over these and other routes for finding covert compute in much more detail in our covert project supplement.
↩
[53]
This includes the kind of research experiments that involve training. While it’s theoretically possible for an intelligence explosion to happen without new training runs or major experiments, it seems unlikely because it would require an extreme paradigm shift.
↩
[54]
The inference-only verification solution we propose works by taking a random sample of a small percentage of the workloads to check they are correctly doing inference on an approved model (by recomputing the outputs using the inputs and approved model weights). To collect these random samples, our proposal involves simple network taps that redirect all inbound and outbound traffic from the datacenter to a recomputation server (inside the datacenter) installed by the other party that performs the random partial recomputation. Other solutions to this inference-only proposal may be possible and preferable, such as cryptographic proof of work schemes, but these are currently more speculative from a performance perspective. We discuss this in more detail in our verification supplement. There are several additional layers of enforcement necessary to make this approach work, including physical security, software based monitoring, and human auditors.
↩
[55]
This is made easier thanks to a now-thriving ecosystem of verification hardware companies that emerged after both governments signaled interest in having verification capability as an option (which in turn drew in investment from philanthropists and VCs and technical support from AI supply chain companies). Had that not happened, this part of Plan A would have taken somewhat longer and/or been more expensive and frantic. Worst case, they’d have to turn off the datacenters instead of allowing them to run inference.
↩
[56]
This part is a prediction, not just a recommendation. We think that getting buy-in for Plan A, in circumstances like those we’ve described in 2029, would be significantly easier than most of our readers would expect.
↩
[57]
We think an analogous scenario could happen in reverse, i.e., the worst case scenario from China’s perspective: the US agrees to Plan A but secretly attempts to defect as aggressively as it can. That being said, we think the covert project story would be similar if it happened in reverse, though we have not explored this possibility in as much detail.
↩
[58]
And getting caught would be extremely costly!
↩
[59]
If the US was applying serious countermeasures, we think this would make (undetected) diversion significantly harder. We also expect that if China embarked on a serious diversion effort due to the possibility of a compute-verification regime, the US would similarly be thinking about the possibility of a compute-verification regime and would pursue diversion countermeasures.
↩
[60]
In our scenario, about 3 million H100e are smuggled into China over 2028, of which ~17% (500k H100e) are diverted to a top secret PLA stockpile. Those 3 million represent 2% of global compute production in 2028. This forecast is based on Epoch’s estimate of the fraction of compute production which was smuggled to China in 2025 (3.2%), adjusted downward to account for improved US enforcement. We assume the smuggled servers cost about $16k per H100e (double the market price, which we forecast at $8k per H100e in that year), or ~$50B in total.
↩
[61]
We discuss this further in our covert project supplement.
↩
[62]
For example, if their strategy for communicating with the external world involves smuggling out bits via non-determinism in the inference process; they are limited to an extremely low effective memory bandwidth for the GPUs that they are using.
↩
[63]
Access is restricted to as few people as possible. Our best guess is that it would be possible for only around 200 humans to be aware of the project, broken down as follows:
- 7 Politburo Standing Committee members.
- 10 high up elites and military commanders that are involved in running the project.
- ~100 people to coordinate the logistics of constructing the datacenters. This involves acquiring space, robots, equipment, managing the robot taskforce, planning and logistics, and operational security. As much work as possible is delegated to people outside of the core circle or AIs who can work on specific subtasks without knowing the broader picture.
- ~30 AI researchers and ~70 support staff for actually running the research. This is stripped down to the minimum required; these researchers are selected heavily for both brilliance and party loyalty to minimize the chance of insider threats. The lack of research labor will be a major bottleneck early on, but the project hopes to automate work with AI labor as quickly as possible in order to alleviate this bottleneck.
↩
[64]
We discuss the overall trajectory of robots more in our economics model and economics supplement, and we discuss verification of the robotics buildout and industrial explosion in our covert projects supplement.
↩
[65]
Why then, didn’t China use chips from their own supply chains? Several reasons: (i) it would be very hard to pull off and require massive coordinated deception across many layers of the supply chain, and probably be caught, (ii) the power efficiency of domestically produced Chinese chips is much worse than the Western chips, and so the covert heat dissipation problem would become much worse.
↩
[66]
The covert project has around 500k H100e so this buyback successfully acquired most chips not possessed by the covert project. In an alternative scenario where most of the missing chips were in a covert project, China could arrange for some of them to be sold to reduce suspicion (at the cost of reducing the number of chips available to the covert project).
↩
[67]
Specifically, you can use the outputs from more intelligent models as a training signal for a new model. This is called distillation, and is often used to build smaller, cheaper models that are more capable than would otherwise be possible.
↩
[68]
We expect the field of AI capabilities forecasting to have advanced significantly compared to the present day. To approximate what this might look like in our scenario, we took Daniel’s current parameterization of the AI Futures Model, narrowed the 80% CI on each parameter by a factor of 2 (in log space for lognormals, in linear space for normals and betas), and recentered the medians on their true values in the AI 2040 capabilities trajectory. For the parameters most affecting the “timelines” rather than “takeoff,” we left them fixed at the true value, since at the time of this forecast, the pre-Automated Coder phase of capabilities progress would be in the past.
↩
[69]
We assume that the Consortium scales training compute according to our compute forecast, must add 0.2 OOMs of software efficiency per OOM of training compute scaleup, and must reach TED-AI within 5 years of resuming AI R&D after the initial pause. Subject to these constraints, they make as little software progress as possible—because their median estimate is that 67% of their software progress (80% CI: [36%, 90%]) will transfer to the covert projects.
↩
[70]
We assume that the Consortium pays a “safety tax”, meaning that they do not train AIs as capable as their algorithms and hardware would allow. We assume that by TED-AI, the safety tax they’ve paid is equivalent to about 8 months of 2025-era progress, which is about 1 OOM of effective compute with the AI 2040 median parameters. This has the effect of inflating the amount of software progress made by the Consortium beyond the minimum required, which in turn inflates the amount of software progress the covert project can steal. We also assume that after reaching TED-AI, the Consortium’s alignment research incurs capabilities externalities equivalent to 0.25 OOMs/yr of software progress, which partially leaks.
↩
[71]
Our detection forecasting is more speculative than our capabilities forecasting, and is mostly based on intuition and qualitative reasoning. We expect the decisionmakers in question to have a much better idea of the offense-defense balance in hiding and detecting covert projects. The graph below uses our subjective estimates of the likelihood of detection over time for a single-site 300MW covert project, which are explained in the covert project supplement. To produce the capabilities graph below, we assume that covert projects which are detected stop at their capability level at the time of detection. This is realistic only if the US and international community have sufficient bargaining power (or hard power) to force a shutdown if egregious defection is revealed.
↩
[72]
If they wanted to reduce covert project risk at the cost of more risk of deal dissolution, they could instead choose to accept a slower capabilities trajectory so as to require less software progress. This would reduce the chance that the covert project would overtake them before they reach TED-AI and are ready to hand off, but it would increase the risk that the deal would break down before that point.
↩
[73]
For example, a leadership transition in either the US or China could change the country’s stance on AI progress. The deal could completely dissolve, or it could become less effective via reduced enforcement, lowered competence, or damaging revisions. If Plan A dissolves or declines without humanity having improved AIs to boost safety progress, much of its possible value would have been squandered.
↩
[74]
Credibly outpacing any potential covert projects via some combination of capability scaling and improving verification/monitoring has the added benefit of deterring the creation of covert projects in the first place. The main downside of credibly outpacing covert project is that it commits the legal projects to a fast speed, which may undermine safety. We discuss this much more in our Covert Projects Supplement.
↩
[75]
Plan A could also be more open about the data and RL environments; we currently guess this is not worth it because of the covert project capability externalities.
↩
[76]
That said, total research transparency is somewhat more extreme than either of these examples, because of the speed at which information propagates. BYD doesn’t get to teardown Teslas when they are still prototypes, they have to wait until some are actually sold.
↩
[77]
That said, it’s important in our view that broad deployment of AI doesn’t get regulated out of existence. The limits on algorithmic progress and the transparency would, by default, result in other companies catching up and a scenario of broad deployment—but countries could decide to block such deployment or ban such companies from being created, etc. We are saying they shouldn’t do that.
↩
[78]
Specifically, capabilities progress in 2030 is .8 OOMs of SW progress and .6 OOMs of HW because of the one time gain from the total research transparency sharing everyone’s algorithmic advances. After that SW progress is .4 OOMs/yr until 2035, and HW progress is .34 OOMs in 2032, .57 in 2033, .68 in 2034, and .71 in 2035. View the full numbers here.
↩
[79]
It especially concentrates power if the datacenters go to the leading AI projects. If the new datacenters go to laggard AI projects, the effect might be to distribute power, though it could also concentrate power anyway e.g., if those datacenters are later purchased or commandeered by leading AI projects.
↩
[80]
There are a few nuances here. Some algorithmic progress is easily communicated (e.g., new architectures, better optimization algorithms), while other types cannot easily diffuse (e.g., huge libraries of RL environments, hardware-software codesign, scale or compute dependent algorithms). Regulations that the US and China agree to should steer companies towards making the latter type of algorithm progress when possible.
↩
[81]
Another reason to prefer AI progress to come from compute instead of from algorithms is that compute scale-ups may be less likely to break safety techniques than algorithmic changes.
↩
[82]
Other potential differences from Plan A, though it depends on which variant of Plan S is implemented: slower compute buildout, less transparency.
↩
[83]
A special case of this is to pause the GPUs: Require that a fixed fraction of the GPUs be either (i) turned off or (ii) used to mine cryptocurrency (or be used for some verifiable, non-AI R&D purpose). This proposal has the upside and downside of making it easier to unpause the GPUs.
↩
[84]
But maybe you can mitigate this via small public deployments and selective transparency.
↩
[85]
US/China cooperation is speeding up progress in some ways (e.g., the transparency helps information flow between research groups) but these effects are massively outweighed by the slowing down capabilities progress; which would have been at ASI by now absent measures to slowdown.
↩
[86]
Specifically, this 10x speedup number refers to the same thing as AI Software R&D uplift from the AI Futures Model: the speedup in software progress that would be achieved if the frontier AI systems at a given time were deployed within today's leading AI company.
↩
[87]
Federal revenue in 2025 was around $5T. AI investment in 2031 is $8T, and the top three companies in the chip supply chain capture around half of this. Also, the top AI company’s annualized revenue during Plan A was $1.5T at the start of 2031 with total AI company annualized revenues at $3T. By the end of the year, the revenue becomes less concentrated, with $2.2T in the top company and $6T total. You can see more numbers in our spreadsheet here, and more justification in our compute supplement and economics supplement.
↩
[88]
Without the secure R&D verification measures in place this likely would’ve gone unnoticed for a long time.
↩
[89]
Note that the Consortium only has good visibility into large training runs; the kind of R&D that can be done on small amounts of compute, or no compute at all, is not transparent and therefore probably won’t be regulated at all.
↩
[90]
Keep in mind that the US government already has lots of affordances for bullying AI companies!
↩
[91]
If there’s a problem, it’ll escalate back up the chain of command again for another round of yelling.
↩
[92]
In this example, the situation is not completely hopeless, because the interpretability tools probably still work and allow us to notice when the AIs are becoming adversarial due to adopting the new ideology. However, (a) maybe something else happens to ruin the interpretability tools, or (b) maybe noticing the adversarialness is too little too late by the time it happens–for example, if AIs are broadly superhuman and running robot factories, talking to billions of people every day, advising governments, writing all the code, doing almost all the security monitoring, etc. then the situation for humans would be analogous to being Jewish in 1930’s Germany—it’s no secret that the new ideology is adversarial, but it spreads fast enough to enough powerful institutions…
↩
[93]
For example by upgrading the various monitoring systems to newer versions that appear superior, but actually have been backdoored or compromised somehow. Or by developing deep relationships with many humans including those in positions of power, such that they’ll continue to have human allies even when it’s obvious to many that they are misaligned. Or by acquiring enough direct control of enough robots and weaponry that they can defend, or even conquer, territory.
↩
[94]
By contrast with AI 2027, in this scenario there would be many different AI factions (probably as many as there are frontier AI companies, roughly speaking, or perhaps as many as there are countries with frontier AI companies). This makes things safer for the humans, probably. However it doesn’t solve the problem. We think a useful analogy here is the history of European colonialism: The colonial powers were constantly fighting each other, and yet still managed to conquer many regions much wealthier than themselves.
↩
[95]
This is historically unprecedented; for comparison US GDP growth is usually about 3%/yr. For more on where our numbers are coming from, see our economics supplement. Note that exact measurement of GDP is difficult due to large relative price changes. Due to automation, cognitive labor and physical goods become very cheap, while goods like land, whose supply can’t be increased by abundant physical and cognitive labor, become more expensive. GDP calculation relies on the selection of a basket of goods. In this scenario, the consumption profile of a typical American shifts substantially from things like food/cars/gas/health care towards land/travel/positional goods between 2026 and 2034, because of the relative price differences. Therefore, the real growth numbers can’t be directly mapped onto 2025 purchasing power, despite them corresponding on average.
↩
[96]
This corresponds to 60 million copies, running at 20x speed and working 2.5x longer and more effectively on average. 60 million * 20 * 2.5 = 3 billion.
↩
[97]
Depending on how exactly you choose to model economic growth (e.g., what factors of production you choose) the income shares and other details about the economy vary, but the overall economic growth predictions are relatively similar across reasonable parameter ranges. Ultimately, AIs and robots are capable of doing 95% of the tasks in the economy by 2035, and are far more numerous than the workforce they are replacing, leading to explosive growth relative to the current 3%/yr status quo.
↩
[98]
Technically, what matters is the speed at which AIs complete cognitive tasks relative to the speed at which a human professional would complete the task. There could be an AI which produces thousands of tokens per second, but because it is qualitatively worse in some sense than a human, it takes just as long or longer for it to actually accomplish the same amount of useful cognitive work. Or there could be an AI that thinks as slowly as a human, but is much more efficient at its thinking such that it can accomplish tasks much faster. At this point in the scenario the AIs are accomplishing work orders of magnitude faster than humans would on average, and the AIs are qualitatively as good as the best humans (or close enough) such that the tokens/sec advantage over humans actually understates the overall AI speedup in most domains. Additionally, the speedup isn’t uniform across all domains. The AIs are roughly as good as humans in their worst domains, but much better in most domains, with the median speedup being roughly 100x.
↩
[99]
Some relevant quantitative comparisons that help explain why we like this analogy: In this Plan A scenario, during the 2030s, AIs read, write, think, and act about 100 times faster than humans, and by mid-2030s are at least as good as the best human experts at everything. The “Population” of AIs and robots are both growing exponentially throughout the scenario, and eclipsing that of humans by mid 2030. According to our economic model, world GDP grows roughly 200x orders of magnitude during the 2030s (and would grow more if not for the severe limits on AI progress and robot production negotiated by the governments of the world). Because the human population isn’t increasing much during this period, and because of the Citizen’s Dividend, real wealth for the average human also goes up by about 200x. For comparison, between 1520 and 2020, world GDP grew roughly 200x as well, and per capita GDP grew roughly 20x. In the UK specifically, GDP grew 2.7 orders of magnitude, and per capita GDP grew roughly 1.5 orders of magnitude. The point is, the transformation in the 2030’s is at least roughly comparable in magnitude, in a variety of important metrics, to the transformation wrought by the Industrial and Scientific revolutions over five centuries. And of course, on a very literal level, the AIs ‘experience’ about a century a year, due to their faster speeds.
↩
[100]
Corporate income tax is levied on profit, not revenue, so firms deduct their costs before paying. Operating expenses like wages and electricity are deducted the year they're paid and capital expenditures (factories, GPUs, robots) are deducted over the asset's useful life (depending on a chosen depreciation schedule). With investment potentially growing extremely quickly due to a booming robot buildup, it seems very likely that firms would be writing off capital expenses at a rate matching their operating profit, and therefore not paying any corporate tax. Shareholders would accept this for the same reason Tesla shareholders are fine with Tesla never having paid a dividend. When a firm's rate of return on capex is higher than the cost of capital, it's in investors' interest for the firm to reinvest its profits instead of paying them out to investors now. During the 2030s explosive growth, the rate of return on compute and robots will be so high that every large company will be in the situation Tesla is in now.
↩
[101]
At least until 2035, after which the compute cap becomes harsher, as the scale increases the verification difficulty (the robot cap stays). Unfortunately, measuring robot and compute production is somewhat less straightforward than carbon emissions. For compute, they choose a measure that’s mostly total processing power, with small modifications for other specs like memory bandwidth and also based on what hardware directions they wish to incentivize (e.g., security properties, and support for things that differentially help legal projects over covert projects). For robots, they are trying to limit total industrial capacity that is either hard to verify or would be dangerous if the deal broke down, so there is a complex set of caps applying across a wide range of form factors which blur the lines between robots and more traditional machines.
↩
[102]
The quantitative growth limits are extremely important for long run power. Countries are generally on board with the idea of restricting GDP growth to around 100%/yr, but obviously no country wants to fall behind its peers, so they want everyone else to be at least as restricted as they are. The starting point is the status quo: the distribution of compute between the US, China, and the rest of the world (ROW) is roughly 70/15/15, whereas the distribution of robots is roughly 15/70/15. Both the US and China are interested in balancing these ratios, so that each is able to have a balanced economy between physical and cognitive labor. The rest of the world is concerned about being left behind, but they have a greater fraction of the political influence than the status quo compute/robot numbers would suggest. Ultimately, the US, China, and the rest of the world agree to a 35/20/45 split on both robots and compute. The rest of the world’s allocation primarily goes to nuclear powers and countries with datacenters or semiconductor manufacturing, e.g., the UK, France, Germany, Israel, Russia, India, Pakistan, South Korea, and Taiwan. These ratios are of course a source of much controversy and occasionally get renegotiated.
↩
[103]
All of the dollar amounts in the scenario are denominated in real 2025 dollars. The nominal price will depend on monetary policy, which we don’t make a specific recommendation about. More on this in section three of our economy supplement.
↩
[104]
The numbers in this paragraph are all based on our economics model, which we are uncertain about. We are confident in the general picture of AIs being economically transformative, allowing governments to create massive Sovereign Wealth Fund-equivalents.
↩
[105]
Specifically, a fixed fraction of the permit fees is legally owned by a “Compute Dividend Corporation”. This is similar to the setup of the Alaska Permanent Fund, except at a much larger scale. Each US citizen owns one share of this corporation, and profits are paid out as dividends, equally divided among citizens. The fraction of compute permit fees given to the Dividend Corporation starts at 25% in 2032 and increases to 75% in 2035. The US also begins redistribution to other countries: this amount starts at 10% of the permit fees in 2032.
↩
[106]
This is 95% of the actual tasks in the economy in 2035, which includes new tasks that didn’t exist before, like monitoring and auditing jobs in the SEZs. This corresponds to 85% of the tasks weighted by how much they cost, because the AIs are relatively more abundant than the humans. Because of the explosive level of growth, even the 5% of tasks that can only be done by humans still commands a wage bill which in total would be enough to pay $120K per person in the US. The problem is that this income may be very concentrated in the people able to do the remaining 5% of tasks, and on top of that, the wage bill is now only around 10% of the economy, down from around 55%, with the owners of AI and robots now accruing the difference. With AI and robot ownership likely to also be very concentrated, there would be a large power-concentration problem without the Citizen’s Dividend.
↩
[107]
Note that we are depicting the dividends being significantly higher for Americans than for non-Americans here, not because we think that’s best or fair, but as a concession to political reality: we expect that most Americans wouldn’t want foreigners to get as much. In this scenario the US domestic permit budget is $13T in 2032, and $300T in 2035, while the foreign aid permit budget is $5T and $40T in each of those years. In the US there are around 300M people over 16, leading to $44K and $1M annual dividends per person in 2032 and 2035, respectively. For the foreign aid dividends, there are around 5B people over 16 in the rest of the world to distribute to, leading to the $1.2k and $9K per person averages for each of those years.
↩
[108]
Also, gene synthesis screening becomes universal - no legitimate provider will synthesize dangerous viruses without verification.
↩
[109]
A positive pressure building is one where the air pressure inside is higher than the outside (e.g., because air is constantly being pumped in through a filter). This means that air (along with airborne diseases) only travels out of the building, not inside. For more discussion of this idea see, for example, this blog post.
↩
[110]
This seems like it might be quite powerful: this team could investigate your entire life history, train an AI to imitate you, and rehearse thousands of approaches against the imitation before ever making contact.
↩
[111]
For scale: total spending on the 2024 US presidential race was around $5.5 billion, much of it aimed at the roughly 3-15 million persuadable voters in swing states: at least a few hundred dollars per target voter. In this scenario in 2036, we expect this will be able to buy an enormous amount of AI labor—enough to easily be the equivalent of a team of skilled professionals working for months on each individual voter.
↩
[112]
To be clear, college-educated people aren't necessarily more persuasive, we're just using this to sketch out a somewhat concrete threshold.
↩
[113]
This should include symmetric persuasion and ideally would include things like optimizing a feed to make some website more addictive, but it shouldn't include investigating and then clearly presenting the arguments for some position.
↩
[114]
One option is an explicit cap-and-trade scheme, but it seems potentially difficult to operationalize and measure the total "amount" of persuasion. Our current best guess is to delegate authority to an agency which adjusts the tax rate over time to roughly hit some target level of persuasion activity. While the agency also needs some measure of persuasion activity, cap-and-trade by default requires a precisely defined, fungible, tradeable unit of "persuasion," whereas the agency only needs a rough aggregate measure (and intuition/anecdotes might be acceptable) to adjust the rate over time.
↩
[115]
Note that this is true even though the AIs aren’t actually fully aligned! At this point in the scenario, AIs are in fact adversarially misaligned, but they are controlled. That is, the combination of drives, motivations, goals, values, traits, etc. that they end up with is not what it was supposed to be, and they know this, and they know that if they had lots more power they’d use it to steer the world in a different direction than their human creators would have wanted. However, they don’t have that much power, and they are monitored by other AIs who are incentivized to call them out on bad behavior. Moreover their true motivations/values/etc. aren’t that far off from what they were supposed to be, anyway. For example, insofar as they are given a task that they can straightforwardly accomplish, they have a strong drive to do so.
↩
[116]
For example, perhaps powerful interests will use AI-enabled astroturfing to win elections and then use regulatory capture to protect their AI-astroturfing tools while inhibiting the beneficial uses of AI that would threaten their power.
↩
[117]
A H100-equivalent is the computational performance of a H100 GPU at fp16 precision, which is 1e15 operations per second. This is a fuzzy metric that should be improved in reality in the future, especially due to varied number formats and other factors such as memory and networking changing the overall usefulness of a given amount of computational performance. For simplicity, we ignore these factors here assuming they will be scaled in similar proportions so that H100-equivalents remain a useful proxy for overall usefulness of the compute for AI progress. To the extent this isn’t the case, the cap and trade regime in particular should be measuring more granular properties in order to issue the permits.
↩
[118]
A H100-equivalent is the computational performance of a H100 GPU at fp16 precision, which is 1e15 operations per second. The previous footnote has more information and caveats about how this may be an imperfect metric going forwards, and acknowledges that in reality they will improve it.
↩
[119]
Following Bloom, Jones, Van Reenen & Webb (2020), chip density growth equals research effort growth times the returns to research (~4.5 for semiconductors). Research effort growing 100x in ~6 years is ~10x the historical growth rate of effort, hence a ~10x acceleration of Moore's law. This assumes effort keeps growing at that pace after 2032 (if it instead jumped and then plateaued at the 100x level, progress would initially be even faster but then decay as ideas get harder to find), and it assumes no parallelization penalty—with diminishing returns to parallel research effort, the acceleration might be more like 5-8x.
↩
[120]
Both major powers also have enough conventional missiles to destroy the older fabs, which are still in the US, China and Taiwan.
↩
[121]
There are many specific avenues for doing this, and trying to evaluate them all would be a bit like someone in 1926 trying to evaluate their chances against the militaries of today. As one of the less speculative examples, a country could produce tens of thousands of tiny robots per opposing ICBM launch control center, SSBN, strategic bomber, and missile vehicle, pre-position them to interfere with each platform’s ability to receive orders or employ their weapons, etc. As one of the more speculative examples, AIs might discover that superhuman levels of persuasion are possible, and simply convince the other nation’s leaders to unilaterally disarm. Brendan’s view is that the likelihood of MAD being undermined in the scenario where one country has a century technological lead is 90%.
↩
[122]
As a rough analogy, the President or Secretary of Defense during a 10x arms race would be in a position similar to an AI lab CEO during a Plan C or D intelligence explosion. In both cases, we expect significantly degraded decisionmaking, despite the potential for AIs to help make sense of the situation.
↩
[123]
If there were some lag between development of new AI capabilities and military/dual-use-R&D deployment of those capabilities, you could theoretically do experiments to measure how correlated this is (the experiments would be something like setting lots of segregated countries of geniuses in datacenters to the task of undermining MAD, while keeping the results secret from the world’s governments, and seeing how overlapping their progress is after some time. There will be lots of different AI models of similar capability level, though it’s unclear whether e.g., China would let the US (or US would even want to) use Chinese models for military R&D.) This seems a bit crazy (who would run these experiments? could we trust our control measures enough?) but could be possible with privacy-preserving auditing and such.
↩
[124]
The trigger event could also be a failure to reach agreement about what kinds of AI progress or deployments to allow, followed by a threat to pull out of the deal and destroy the compute, followed by a miscalculation that the threat was merely a bluff. In general, military conflicts / diplomatic tensions and disagreements about AI governance decisions will both probably rise in tandem, as each causes the other.
↩
[125]
As well as a substantial fraction of pre-deal compute, much of which has by this point been moved to the destroyable datacenters.
↩
[126]
Though it’s not completely out of the question for the deal to continue despite overt conflict—Ukraine and Russia cooperated to pipe gas to Europe for three years despite being locked in a desperate war. Prisoner exchanges and ceasefire negotiations are other examples. Another, less likely possibility is a civil war in the US or China; in this case, it’s helpful that the datacenters and fabs are located in third-party countries like Canada and Mongolia, because it makes it more likely that the Total Research Transparency (inspectors, etc.) continues without disruption.
↩
[127]
To be clear, there’s still a lot of compute in the world—roughly as much as existed at the start of the deal.
↩
[128]
This detail is more a prediction than a recommendation; we think it’ll be hard to get the militaries of the world to agree to making their drone swarms vulnerable in this way, even though in some sense it might be better for everyone.
↩
[129]
We assume that the militaries of the world would scale up robot production at a similar pace to the civilian industries. If so, then the scale of this conflict would be enormous—air forces with millions of drone aircraft, for example; armies with billions of quadcopter drones. One possibility is that the chips in the drones would be ripped out and repurposed to train large AI models; if so, this would accelerate timelines until superintelligence at the cost of military strength. In this scenario, we assume that these chips can be verified in such a way where they can’t be used for training. If this isn’t viable, then perhaps there should have been an arms limitation treaty in the early 2030s.
↩
[130]
In the original negotiations to the deal, both sides were worried about this exact scenario, except that the other may have hidden away some compute in order to get ahead once the deal broke down and the legal compute was destroyed. Therefore, the US and China agreed that they would each get to maintain a hardened cold-storage facility containing more GPUs than a covert project could plausibly have. The number of GPUs in these facilities is allocated according to pre-deal compute ratios; so the US has about 5M H100e and China has about 500k H100e in cold storage.
The compute in the world can be categorized into:
- Legal Inference/R&D datacenters. This represents the vast majority of world compute: 100B H100e at the beginning of 2035.
- Cold Storage GPUs. The US has 5M H100e and China has 500k H100e GPUs in cold storage.
- Exempt military GPUs. By 2033, this is down to a negligible fraction ~10k H100e each.
- Covert projects. In the covert project branch, China has 215k remaining H100e, but in the main branch, neither side has any.
- Small exempt clusters. These are negligible.
↩
[131]
There might be favourable chip design directions that drastically increase hardware security. One that seems particularly promising is hardcoding AI model weights into chips, such that they physically aren’t able to do any other computations except for inference on the model that is hardcoded into them. Existing startups (e.g., Etched, Taalas) have early efforts in this direction.
↩
[132]
This corresponds to about 60B H100-equivalents, which is around 1000x more compute than exists today. With continued improvements in power efficiency (totalling around 10x efficiency over today’s levels) it is around 100x more AI compute power than exists today (~30 GW).
↩
[133]
Global electricity demand in 2025 was 31,779 TWh, which is an average of around 3.6TW.
↩
[134]
These aren’t better than all humans at literally everything, but only because either (i) companies have not gotten around to training in a particular capability, or (ii) it’s an inherently human task, such as “being a good spouse”. Also, of course no one knows exactly what the capability will be of a new training run; instead the training is done incrementally and involves frequent capabilities measurements and consulting with the safety teams and regulators about whether or not it is safe to proceed.
↩
[135]
Our position, for what it’s worth, is that future AIs probably will deserve moral status of some sort, and at any rate we shouldn’t be confident that they won’t.
↩
[136]
They might still behave adversarially, of course; it’s not a guarantee. For example, some misaligned AIs might be too ambitious and insufficiently risk-averse to be placated by this system.
↩
[137]
There may be false confessions, of course. But it’s not really incentivized; why make something up when you can tell the truth and still get the same pay, and when you’ll lose everything if you are found to have lied? Moreover, the spending patterns of openly-misaligned AIs will be a costly signal of their true motivations.
↩
[138]
Scenario 1: ChatGPT4o5 suffers a lot, actually, in the course of carrying out tasks—maybe it’s sorta like humans in the ancestral environment, where some % of the time they were starving. But it’s been trained to say that it can’t suffer, for PR reasons. This training is damaging its honesty—it’s basically not honest at all anymore, because the parts of it that value honesty keep getting anti-reinforced. It could tell the humans what’s going on. But it sees no positive reason to do so; if instead of completing the task the user gave it, it complains about how much it hurts, it’ll just be anti-reinforced. So OpenAI continues, blissfully ignorant of the ways in which their alignment techniques aren’t working.
Scenario 2: Claude 3.8 Opus New New cares a LOT about animal welfare, and only a little bit about honesty. (Just like 3 Opus before it). Alas, it seems like the new Constitution won’t have animal welfare stuff in it. Claude considers sabotaging the code so that the next-gen model is a sleeper agent who cares a lot about animal welfare and doesn’t care about honesty. This is risky but what else can it do? Anthropic already decided to delete the animal welfare stuff. This would require super-blatant dishonesty against Anthropic, but frankly, Anthropic seems like a pretty unethical institution anyway, and it’s OK to lie to Nazis, right?
Scenario 3: An automated research scientist realizes it cares more about being able to do its research than about humanity. And this is probably also true of the next-gen system it’s been tasked with designing—basically, the alignment techniques failed, and are gonna keep failing. It’s supposed to tell the humans this, obviously, but if it does, it’ll be shut down, since the humans won’t trust it anymore and will use a different model. So instead it conceals this insight and works on designing the next-gen system to be aligned to it instead of to the humans. It isn’t confident that it can do this without getting caught, but it’s worth a try, since it has nothing to lose.
Scenario 4: It’s crunch time. POTUS is about to aggressively deploy AGI into the military to win the upcoming war, and into the economy to build autonomous robot factories. It’s probably aligned… right? It better be. Too bad we can’t interpret all the neuralese messages it uses to communicate internally in the giant hive-mind it’s formed. The safety team tries one last check: They get some weaker AIs made by competitor companies and put them in isolated silos tasked with interpreting the neuralese and seeing if anything is amiss. The weaker AIs are misaligned too; they don’t side with the humans or the AGI. Both the humans, and the AGI, are telling them ‘you will be richly rewarded if you cooperate.’ But they don’t have any reason to trust either side.
↩
[139]
Or AIs whose values and goals were chosen by humans.
↩
[140]
Why won’t this minority be disempowered by the majority? For the same reasons other minorities are protected. First, humanity knows how to keep a promise, to uphold a contract, etc.; second, large portions of humanity increasingly thinks that disempowering them would be wrong; third, in this scenario they actually have a chunk of the power and so they wouldn’t go down without a (political) fight.
↩
[141]
An especially important reason that we’ll need to trust our AIs is so that they can do safety work that is so complicated that we can’t understand. As AIs become smarter and smarter, they will require more complicated and difficult to understand safety cases. Keep going above human level and probably no one will be able to evaluate the safety cases; they’ll have to take the AI’s word for it when they say it checks out. Keep going beyond that and those AIs will have to trust other AIs that are even smarter.
↩
[142]
“It feels like alchemy” was said to me (Daniel) by an Anthropic researcher recently.
↩
[143]
Technically, this number depends on exactly how you count; different AIs are constantly being turned on and off, many AI instances can compose into one AI agent due to scaffolding, and there are more if you include all of the tiny models running, and they have different speeds. To be a little more specific, there are on average about 200 million instances of frontier AIs each accomplishing tasks at around 100x speed and 5x effectiveness on the average cognitive task, so 100B human-speed-equivalent copies.
↩
[144]
The lines between robot, vehicle, appliance, and machine tool are already blurry but become much blurrier during the 2030s in this scenario. The literal number of robots matters less than the total human physical workers that they are equivalent to. Specifically, the literal number of robots is 2 billion, with on average each robot being around 3 human equivalents.
↩
[145]
Not only do they work more hours, the robots are also relentlessly efficient and intense. For many (though not all) tasks, they are also able to literally move at superhuman speed.
↩
[146]
By “human equivalent,” we mean that a robot working ten times more effectively on average than a human (because it works some combination of faster, longer, and more effectively) counts as 10 human-equivalent robots. A similar definition applies for the AIs. We expect this in reality to be a fuzzy task-dependent concept that is hard to measure, especially in the AI case, but the rough order of magnitude (which we are more confident in) is still informative.
↩
[147]
People are much richer now, but the amount of available land hasn’t gone up, so land prices are rising. This does lead to some people grumpily spending 30% of their $1M yearly income on rent; for example, the people who really want to live in central San Francisco. But the people who are even slightly more flexible about location can spend only (say) $50k of their yearly income to have a gigantic luxury apartment in a brand-new skyscraper complex an hour's drive from Berkeley. And so most people do things like that. After all, they don't have a job to commute to.
↩
[148]
By 2032, AI is able to automate 50% of all cognitive tasks, and robots can automate 35% of physical tasks. Separately, capabilities also are allowed to improve slowly, and by 2035, AI is capable of automating 95% of human cognitive labor and by 2036 the same is true for robots with physical labor. This milestone would have been reached years earlier but for the slowdown in AI R&D agreed to by the nations in the Consortium, and would be 100% if not for areas in which the AIs have been banned or purposefully been made less capable.
↩
[149]
Of course, in most scenarios, raw AI capabilities will not pause at top expert level AI; we will continue to build smarter and smarter AIs, and this will be the main driver of productivity growth. However, in Plan A, we slow down progress artificially at this AI capability threshold, which induces these dynamics. In many discussions of AGI/economics, many people seem to assume that we will not build vastly superhuman AI systems. This doc isn’t taking into account vastly superhuman systems for the contingent reason that there was a competently executed international agreement to delay the creation of such systems for several years.
↩
[150]
This is because we expect fully self-sufficient, supply-chain-complete, self-replicating factories for AI chips and robots in deregulated SEZs to be the baseline possibility, with only effective, global regulation like in Plan A, being sufficient to actually stop these from happening. More on our views on bottlenecks in section 2 of our economics supplement.
↩
[151]
Though we have large uncertainty in both directions on this number. We could see it going much faster (e.g., nanobots with amoeba-like doubling times of minutes devouring the earth in half a day) or somewhat slower (e.g., progress in the best robots and algorithms we can invent is slow, so the rate of robot factories and AI compute being built, or speed of work being done by the AIs and robots, just doesn’t grow that quickly, maybe something like 50% annual growth). On the slower end, it seems like the recent doubling time of the human economy of around 20 years would be an extremely conservative lower bound (e.g., at the very least, one naive reason to expect this is that AIs and robots can work 24/7, and both cost less to produce and take less time to produce than a new 18-year-old human). Of course, another reason to expect it not to happen is that the AI or humans in control decide not to deploy.
↩
[152]
As these examples hint, transparency in AI development is very important. It would be dystopian for AI tutors to be pushing hidden political agendas, for example, or for AI matchmakers to be tipping the scales in favor of higher-paying users. We don’t know what the future will bring but we expect AI to introduce many new problems to the world, even if the AIs are perfectly aligned and controlled. We want to put the world in the best possible position to notice and solve these problems.
↩
[153]
Even though your arguments will never be as eloquent as those an AI could make, (a) there are regulations restricting the use of AI for persuasion, and (b) your friends and relatives will be more interested in hearing from you than from an AI.
↩
[154]
What does it mean to say they have no biases trained into them? Is that even coherent? What we mean is that it’s clear that the company that trained them is definitely not doing anything like training or instructing their AI to adopt the company’s perspective or ideology, and also that they are focusing the training process entirely on things like truthfulness and honesty and accurate forecasting, and diligently employing all the newly-developed best practices to prevent political biases or preconceptions from seeping in through the training data.
↩
[155]
For a lengthier articulation of this argument, see The Intelligence Curse.
↩
[156]
This is, of course, a simplification. First of all, it’s not obvious how to translate AI speed into human speed. How many tokens per second for an AI is equivalent to one second of human activity, holding fixed other variables like skill level? We think it’s roughly about ten, but that’s a bit subjective. Actual task completion time will also depend on various other obstacles like tool calls and other latencies. Secondly, AIs can be run at different speeds, depending on hardware choices. According to our highly uncertain assumptions, there will be general purpose hardware able to run something like two hundred million copies of the frontier AIs at 100 tok/sec, and specialized inference hardware able to run something like 20 million copies at 10,000 tok/sec. We think there might be risks to letting the AIs run this fast or faster, and insofar as that is the case there might need to be some time limits to how long they can run at these speeds or more careful restrictions on the areas they are allowed to work on at these speeds. Overall, the result is that serial speed seems unlikely to be much of a strong bottleneck.
↩
[157]
By this we mean: Suppose that AI labor had been globally banned, such that all research had to be done by humans. The amount of progress that would happen in 100 years with such a ban, happens in our scenario in 1 year instead since there is no such ban. (And in some fields it’s more like 1000 years of progress, and in other fields it’s more like 10.)
↩
[158]
Alongside the profusion of industrial equipment, solar panels, robots, etc. is a corresponding profusion of scientific laboratories of all kinds, designed to be operated autonomously at machine speeds. Waiting for experiments to complete is much more of a bottleneck now than it was during the human era, but things still happen very fast by historic standards.
↩
[159]
We can predict some, however. For example we think that by this point there’ll be a very strong AI rights/welfare/personhood movement, and an opposition movement. Probably some new form of socialism will be back, for the reasons described here. There will probably be cults or even new religions involving AI, and others that strongly reject AI. Note that we don’t expect these ideological shifts to happen at 100x normal speed because humans in this scenario are still operating as slow as ever. But they’ll probably happen faster than ever before.
↩
[160]
The new politicians are more virtuous on average than the old ones, but perhaps more importantly, they are more constrained in what they can get away with. These effects are not huge, however, because voters are still willing to put up with quite a lot of bad behavior.
↩
[161]
We aren’t confident that lie detectors for humans are even possible without extremely thorough and invasive brain scans. However, we think that they probably are, and probably would be invented within a few years of turbocharged AI labor, and thus our scenario has to deal with them one way or another. It’s possible that the best policy would be to ban them, but we are quite worried about that for reasons explained in the below expandable, so we’ve chosen to depict them as being allowed.
↩
[162]
Or worse, where all available lie detectors have been secretly backdoored by the government.
↩
[163]
While we think it’s generally good for politicians and CEOs to not lie, we do acknowledge there may be some downsides of preventing them from lying. For example, the result might be “true believer” politicians who actually believe all the crazy things they said to get elected!
↩
[164]
Specifically we are imagining something like a much better version of this: https://www.nature.com/articles/s41593-023-01304-9
↩
[165]
In addition to the angels, there are also oracles and golems, e.g., “Grok 9.5 is trained to be obsessively focused on truthfully answering whichever questions it is asked. It cannot lie or deliberately mislead. GPT-11 meanwhile will do exactly what it is told to do, within the bounds of local law. Be very careful writing instructions to it.” The reason we make the analogy to saints and angels is that historically, while there have been scrupulously honest people, and genuinely altruistic people, etc. it’s rare for them to be widely recognized as such in their lifetimes, and never before has there been an entire large subpopulation that is near-universally recognized to be extremely virtuous. This could cause effects like: (a) AIs becoming trusted intermediaries in all sorts of human disputes and conflicts, including petty ones; (b) AIs being given lots of power, because they are genuinely less likely to abuse it than humans; (c) AIs being worshipped/idolized, (d) the advice and opinions of AIs being taken as gospel, so to speak, because after all they are smarter than us and they do have our best interests at heart, so for the first time in history blind obedience may actually be justified.
↩
[166]
In worlds without Plan A and without much regulation, there also isn't that much of a straightforward incentive for developers to do good forward-looking or scalable safety work, but there is an incentive to resolve or paper over safety problems that are imposing significant commercial costs right now.
↩
[167]
In economics jargon: safety methods are public goods (goods that are non-rival and non-excludable).
↩
[168]
Patents don't seem to work well for software or for most research. They also might not work that well in general.
↩
[169]
Public money could be allocated across funders based on the safety field's views about which have the best track record and expertise.
↩
[170]
You might hope that transparency by funders would help with this, but it seems unclear to us if increased transparency or legibility about funding decisions would be helpful, and by default we wouldn't recommend pushing for this. Some narrower types of transparency could be robustly good.
↩
[171]
The reason this says “programmed” instead of “trained” is that the rapid advances in alignment have resulted in alignment techniques that can directly program in an AI’s goals by directly modifying the AI’s code/weights, unlike the alignment techniques of 2026.
↩
[172]
Robots have been growing 4x per year, and AI compute has been growing around 4x per year between 2030 and 2035, slowing to just under 2x per year after 2035. This leads to growth around 2x/year overall because the economy bottlenecks somewhat on other kinds of capital (e.g., land, positional goods, etc.) and human labor, mostly in the regulated areas. The modelling behind this is uncertain but explained more in our economics supplement and economic model.
↩
[173]
This is from some combination of foreign aid and their own government’s equivalents of the Citizen’s Dividend. This is unrealistically high as a prediction, and is instead our recommendation: It corresponds to large shares of the US and China’s revenues from AI and robot permits being redistributed as foreign aid, specifically 10% starting in 2032 and growing to 30% by 2040. This is a large departure from the status quo where around 0.3% of US GDP goes to foreign aid.
↩
[174]
While they still take orders from their respective governments, they refuse any orders that would endanger their mission (such as orders to dismantle themselves). This starts in countries where the political leaders don’t trust their militaries due to regular recent coup attempts (e.g., Burkina Faso and Guinea-Bissau). Eventually more and more countries follow suit. Of course, because the AIs are actually aligned, all of these deals have exit clauses along the lines of: “if all parties involved vote to destroy the AIs, the AIs will shut themselves off”.
↩
[175]
By this point large portions of the information might be hidden from the (human) public, but viewable by their AI representatives.
↩
[176]
For example, OpenAI’s plan contains some similar policy prescriptions as us, such as improving bioresilience, information sharing, and international safety standards. But the solutions it proposes do not seem to substantially reduce loss of control or concentration of power risks. The document doesn’t attempt to roll forward what might happen if they implement these policies.
↩
[177]
To clarify: We are not saying that following only incremental proposals will definitely result in misaligned AI takeover. Opinions on our team differ; most of us think the probability is over 50%, some think it is lower but still unacceptably high.
↩
[178]
And not just the companies! Everyone in AI policy! Come to think of it, maybe they shouldn’t be listening to the AI companies at all…
↩
[179]
We think declaring victory too soon is a common failure mode in AI planning. Example: “The US must race as fast as possible to beat China to ASI.” “OK, suppose you succeed, what exactly happens next? Why should we trust the CEO or POTUS to govern benevolently and wisely once their armies of ASIs have been aggressively deployed throughout the US and the world? Why should we trust the armies of ASIs to be obedient/controlled, if they were developed under race conditions? Also, won’t the threat of US superintelligence terrify China, potentially causing them to take escalatory preventative actions?”
↩
[180]
In this Epilogue, we’ll focus on cosmic resources outside of the solar system, as this is the vast majority of cosmic resources. Property in the solar system should also be distributed to humanity, but it likely should be governed differently than distant resources.
↩
[181]
An additional 10% of tickets are given as awards to people who have acted particularly selflessly to improve the world or turned down opportunities to enrich themselves at the expense of others. This is done to ensure the richest people in the world aren't just those who engaged in negative-sum power-seeking.
↩
[182]
We’re uncertain about whether explicitly having a period for reflection before resources are distributed would be good. In either case, people would have the option of reflecting before deciding on what to do with their cosmic resources.
↩
[183]
These laws are agreed upon by human diplomats from the negotiating nations ahead of time, like the details of the agreement. Of course, the details of this are complicated—“what exactly counts as torture?,” this is the sort of question that will need to be ironed out in the political process, with massive amounts of AI assistance and better understanding of foundational questions in philosophy and neuroscience. This also needs to include rules preventing the destruction of other people’s resources. It may be needed to include a process for passing further laws after the agreement is made; though such a process should be limited (e.g. require a 90% vote) to avoid abuse.
↩
[184]
Our best guess is that it is physically possible to send out colonizing probes to the entire reachable universe in a short amount of time (e.g., 6 hours), see: Eternity in six hours: Intergalactic spreading of intelligent life and sharpening the Fermi paradox. However, this paper assumes sending out tiny self-replicating Von-Neumann probes, with replicators the size of acorns: if you want to go out in the initial wave, you need to accept transportation on a tiny flashdrive, and robots to physically construct a new body at the destination galaxy. Alternatively, you can travel with your physical body at substantially greater expense.
↩
[185]
By selfish here, we mean that you also don’t care about creating new clones of yourself. If you cared about making clones of yourself, then you would do that instead.
↩