Jaan Tallinn would like us to survive

“We’re pulling a trigger on Russian roulette with a planet. And every shot that doesn’t kill us will make us stronger, potentially, and wealthier and better off. But when do you stop pulling the trigger?” – Jaan Tallinn

Jaan Tallinn is a founding engineer of Skype and Kazaa, and a co-founder of the Cambridge Centre for the Study of Existential Risk and Future of Life Institute. For more than fifteen years, he has been among the most consistent voices arguing that advanced AI poses an extinction-level risk to humanity.

Jaan became interested in AI risk in 2008 after reading the work of Eliezer Yudkowsky, a founding researcher of the field of AI alignment. In 2009, after a four-hour meeting with Yudkowsky, he made his first donation to what became the Machine Intelligence Research Institute (MIRI). He went on to become an early investor in DeepMind and led the first round into Anthropic – despite believing that AI development, if unchecked, could kill us all. His reasoning: his money displaced VC money rather than bringing Anthropic into existence, and the influence and proceeds that come with it help fund safety work.

In this conversation, we explore what Jaan believes the OpenAI/Hugging Face attack revealed about AI motivation and why he was less surprised than most. We discuss his evolving risk estimate, his case for an immediate moratorium on frontier model training, the technical and geopolitical case for hardware-based verification as a foundation for global AI governance, and why he asked Dario Amodei to collect competing term sheets before accepting his own investment.

From my notebook

We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these!

1. The strange preferences of superintelligence. I’ve spent a good deal of time thinking about what a superintelligence might do, and what it might want at a high level. I hadn’t really considered what it might simply prefer, and the impact its preferences could have on us. Jaan’s thoughts on how superintelligence might reprogram our climate to suit its desires were especially chilling.

2. The unexplored paths of the tech tree. An AI “pause” does not have to spell the end of AI progress. As Jaan explains, there are other branches of the technology tree that we’ve hardly explored, and that may yield benefits without equivalent risk.

3. A cryptographic solution to trust-building. Jaan outlined how zero-knowledge proofs could be used to monitor chips to ensure that different parties – including nation-states – are complying with future regulations. A sharp way to permit observability without full technical disclosures.

4. China and the “big blob.” When Jaan first read Dario Amodei’s “big blob of compute” essay outlining the importance of scaling compute, his immediate thought was that China would be first to achieve AGI, given the country’s engineering prowess. Concerned, he decided to visit the country to speak to researchers there and assess the state of play.

5. The weirdest VC round. Usually VCs want as little competition as possible for the companies they want to invest in. When it came to funding Anthropic, Jaan took the opposite approach, asking Dario Amodei to find multiple term sheets so he could be certain he was just displacing VC money and not bringing another frontier lab into existence.

6. Jaan Tallinn loves to dance. I’m not sure why necessarily, but I did not expect one of the foremost experts on AI safety to be a fanatical street-dancing advocate. Great lore.

Tangents

What I’m reading, watching, and into this week:

• Robert Caro’s Working. Caro had been researching The Power Broker for two years before he got to interview its subject, Robert Moses. True patience.

• Colin from Accounts’ new season. A charming, excellently written and acted Aussie comedy. Great stuff

—Mario

Every timestamp

(00:00) Intro

(02:28) Growing up in Soviet-occupied Estonia

(05:54) How Jaan assesses AI risk and why he’s more optimistic now

(07:16) AI’s preferences and motivations

(12:44) The OpenAI Hugging Face incident

(16:12) Meeting with Eliezer Yudkowsky

(18:20) Everything as computation

(20:22) The Coxon moment

(23:05) Who dismisses AI alignment risks

(26:17) The AI race, the case for a pause, and safer alternatives

(32:47) AI 2027

(33:44) Investing in DeepMind and Anthropic

(40:03) The “big blob of compute” and competing approaches to AI

(44:42) What effective AI safeguards could look like

(48:15) Verifying countries’ compliance with AI safeguards

(50:15) AI kill switches and their limits

(52:11) Jaan’s love of dance and musical influences

(59:50) Jaan’s favorite movies

(1:01:41) What Jaan likes about AI

Jaan’s complete reading list

• Shadows of the Mind: A Search for the Missing Science of Consciousness: https://www.amazon.com/dp/0195106466

Other resources

• Skype: https://en.wikipedia.org/wiki/Skype

• Life is NOT a Journey - Alan Watts: https://www.youtube.com/watch?v=rBpaUICxEhk&t=10s

• AI Billionaire on Existential Risk: Jaan Tallinn – #112: https://www.manifold1.com/episodes/ai-billionaire-on-existential-risk-jaan-tallinn

• The Basic AI Drives: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf

• The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead

• Rogue AI didn’t breach Hugging Face, human decisions did: https://thebulletin.org/2026/09/rogue-ai-didnt-breach-hugging-face-human-decisions-did

• MIRI: https://intelligence.org

• Jacob Coxon on X, “I resigned from Anthropic today.”: https://x.com/hilbertspaess/status/2097476196791709843

• AI 2027: https://ai-2027.com

• Google DeepMind: https://deepmind.google

• Inside the Biggest Feud in Artificial Intelligence: https://www.theatlantic.com/technology/2026/09/openai-v-anthropic-inside-biggest-rivalry-tech/688819

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• AI 2040: https://ai-2040.com

• The Glass Bead Game: https://www.amazon.com/dp/B00DO994WY

• Jaan’s favorite movies: https://jaan.info/favmovies

• The Odyssey: https://www.imdb.com/title/tt33764258

• On the Edge: https://www.imdb.com/title/tt0221559

• Limitless: https://www.imdb.com/title/tt1219289

• Ex Machina: https://www.imdb.com/title/tt0470752

• Colossus: The Forbin Project: https://www.imdb.com/title/tt0064177

Where to find Jaan Tallinn

Website: https://jaan.info

Subscribe to the show

I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation.

YouTube

Spotify

Apple

Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected].