Paul Christiano – We could be 12 months away from AIs that can overthrow the government
Dwarkesh Podcast | Nov 16 2026 | 01:38:45

Paul Christiano is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
Paul thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).
He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.
He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.
And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.
Watch on YouTube; read the transcript.
Sponsors
* Jane Street has finally come around on formal methods. Ron Minsky, who co-heads Jane Street’s technology group, told me that up until recently, formal methods just weren’t worth the cost. But coding agents have made them easier to apply AND increased the importance of code verification. So now Jane Street is hiring Formal Methods Engineers in both their London and New York offices. If you’re interested, or if you want to watch my full conversation with Ron, go to janestreet.com/dwarkesh
* Antithesis is NOT another LLM-based code-review tool. It doesn’t try to reason abstractly about how a PR might break your system. Instead, it just runs your code. You start by defining the rules that your system must never break. Then, whenever an agent proposes a change, Antithesis runs your code through thousands of adversarial worlds, testing whether or not it still adheres to your rules. If you want to try it, go to antithesis.com/dwarkesh and use code DWARKESH for 50% off your first month.
* WorkOS lets you give agents their own sign-up flow. This matters because agents tend to pick tools they can use right away, rather than ones that require humans to register. Fortunately, providing this kind of dedicated path is easy: if you already use WorkOS for login, you just toggle on “Agent Registration” in the WorkOS dashboard. Once it’s enabled, you can decide what agents are allowed to do on their own, and what needs to wait until the person they’re working for gets involved, all with the permissions you’ve already set up. Learn more at workos.com/authkit
Timestamps
(00:00:00) – We could be 12 months away from AIs that can overthrow the government
(00:11:44) – Why an AI might want to takeover
(00:20:49) – How an AI takeover could happen
(00:35:35) – If labs keep racing without security, China will just steal the weights of superintelligence
(00:53:15) – The case for pausing now instead of later
(00:59:31) – Superintelligence will be good at learning from real-world experience
(01:11:32) – Crossing the sim-to-real gap
(01:22:23) – METR says AI progress doubled, so why is revenue growth slowing?
(01:27:57) – The best plan we have is to build the plane as we fly it
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com
