Africa-focused technology, digital and innovation ecosystem insight and commentary.
…
continue reading
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
“Foom & Doom 1: ‘Brain in a box in a basement’” by Steven Byrnes
MP3•Episode home
Manage episode 490629697 series 3364760
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
1.1 Series summary and Table of Contents
This is a two-post series on AI “foom” (this post) and “doom” (next post).A decade or two ago, it was pretty common to discuss “foom & doom” scenarios, as advocated especially by Eliezer Yudkowsky. In a typical such scenario, a small team would build a system that would rocket (“foom”) from “unimpressive” to “Artificial Superintelligence” (ASI) within a very short time window (days, weeks, maybe months), involving very little compute (e.g. “brain in a box in a basement”), via recursive self-improvement. Absent some future technical breakthrough, the ASI would definitely be egregiously misaligned, without the slightest intrinsic interest in whether humans live or die. The ASI would be born into a world generally much like today's, a world utterly unprepared for this new mega-mind. The extinction of humans (and every other species) would rapidly follow (“doom”). The ASI would then spend [...]
---
Outline:
(00:11) 1.1 Series summary and Table of Contents
(02:35) 1.1.2 Should I stop reading if I expect LLMs to scale to ASI?
(04:50) 1.2 Post summary and Table of Contents
(07:40) 1.3 A far-more-powerful, yet-to-be-discovered, simple(ish) core of intelligence
(10:08) 1.3.1 Existence proof: the human cortex
(12:13) 1.3.2 Three increasingly-radical perspectives on what AI capability acquisition will look like
(14:18) 1.4 Counter-arguments to there being a far-more-powerful future AI paradigm, and my responses
(14:26) 1.4.1 Possible counter: If a different, much more powerful, AI paradigm existed, then someone would have already found it.
(16:33) 1.4.2 Possible counter: But LLMs will have already reached ASI before any other paradigm can even put its shoes on
(17:14) 1.4.3 Possible counter: If ASI will be part of a different paradigm, who cares? It's just gonna be a different flavor of ML.
(17:49) 1.4.4 Possible counter: If ASI will be part of a different paradigm, the new paradigm will be discovered by LLM agents, not humans, so this is just part of the continuous 'AIs-doing-AI-R&D' story like I've been saying
(18:54) 1.5 Training compute requirements: Frighteningly little
(20:34) 1.6 Downstream consequences of new paradigm with frighteningly little training compute
(20:42) 1.6.1 I'm broadly pessimistic about existing efforts to delay AGI
(23:18) 1.6.2 I'm broadly pessimistic about existing efforts towards regulating AGI
(24:09) 1.6.3 I expect that, almost as soon as we have AGI at all, we will have AGI that could survive indefinitely without humans
(25:46) 1.7 Very little R&D separating seemingly irrelevant from ASI
(26:34) 1.7.1 For a non-imitation-learning paradigm, getting to relevant at all is only slightly easier than getting to superintelligence
(31:05) 1.7.2 Plenty of room at the top
(31:47) 1.7.3 What's the rate-limiter?
(33:22) 1.8 Downstream consequences of very little R&D separating 'seemingly irrelevant' from 'ASI'
(33:30) 1.8.1 Very sharp takeoff in wall-clock time
(35:34) 1.8.1.1 But what about training time?
(36:26) 1.8.1.2 But what if we try to make takeoff smoother?
(37:18) 1.8.2 Sharp takeoff even without recursive self-improvement
(38:22) 1.8.2.1 ...But recursive self-improvement could also happen
(40:12) 1.8.3 Next-paradigm AI probably won't be deployed at all, and ASI will probably show up in a world not wildly different from today's
(42:55) 1.8.4 We better sort out technical alignment, sandbox test protocols, etc., before the new paradigm seems even relevant at all, let alone scary
(43:40) 1.8.5 AI-assisted alignment research seems pretty doomed
(45:22) 1.8.6 The rest of AI for AI safety seems
544 episodes
MP3•Episode home
Manage episode 490629697 series 3364760
Content provided by LessWrong. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by LessWrong or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
1.1 Series summary and Table of Contents
This is a two-post series on AI “foom” (this post) and “doom” (next post).A decade or two ago, it was pretty common to discuss “foom & doom” scenarios, as advocated especially by Eliezer Yudkowsky. In a typical such scenario, a small team would build a system that would rocket (“foom”) from “unimpressive” to “Artificial Superintelligence” (ASI) within a very short time window (days, weeks, maybe months), involving very little compute (e.g. “brain in a box in a basement”), via recursive self-improvement. Absent some future technical breakthrough, the ASI would definitely be egregiously misaligned, without the slightest intrinsic interest in whether humans live or die. The ASI would be born into a world generally much like today's, a world utterly unprepared for this new mega-mind. The extinction of humans (and every other species) would rapidly follow (“doom”). The ASI would then spend [...]
---
Outline:
(00:11) 1.1 Series summary and Table of Contents
(02:35) 1.1.2 Should I stop reading if I expect LLMs to scale to ASI?
(04:50) 1.2 Post summary and Table of Contents
(07:40) 1.3 A far-more-powerful, yet-to-be-discovered, simple(ish) core of intelligence
(10:08) 1.3.1 Existence proof: the human cortex
(12:13) 1.3.2 Three increasingly-radical perspectives on what AI capability acquisition will look like
(14:18) 1.4 Counter-arguments to there being a far-more-powerful future AI paradigm, and my responses
(14:26) 1.4.1 Possible counter: If a different, much more powerful, AI paradigm existed, then someone would have already found it.
(16:33) 1.4.2 Possible counter: But LLMs will have already reached ASI before any other paradigm can even put its shoes on
(17:14) 1.4.3 Possible counter: If ASI will be part of a different paradigm, who cares? It's just gonna be a different flavor of ML.
(17:49) 1.4.4 Possible counter: If ASI will be part of a different paradigm, the new paradigm will be discovered by LLM agents, not humans, so this is just part of the continuous 'AIs-doing-AI-R&D' story like I've been saying
(18:54) 1.5 Training compute requirements: Frighteningly little
(20:34) 1.6 Downstream consequences of new paradigm with frighteningly little training compute
(20:42) 1.6.1 I'm broadly pessimistic about existing efforts to delay AGI
(23:18) 1.6.2 I'm broadly pessimistic about existing efforts towards regulating AGI
(24:09) 1.6.3 I expect that, almost as soon as we have AGI at all, we will have AGI that could survive indefinitely without humans
(25:46) 1.7 Very little R&D separating seemingly irrelevant from ASI
(26:34) 1.7.1 For a non-imitation-learning paradigm, getting to relevant at all is only slightly easier than getting to superintelligence
(31:05) 1.7.2 Plenty of room at the top
(31:47) 1.7.3 What's the rate-limiter?
(33:22) 1.8 Downstream consequences of very little R&D separating 'seemingly irrelevant' from 'ASI'
(33:30) 1.8.1 Very sharp takeoff in wall-clock time
(35:34) 1.8.1.1 But what about training time?
(36:26) 1.8.1.2 But what if we try to make takeoff smoother?
(37:18) 1.8.2 Sharp takeoff even without recursive self-improvement
(38:22) 1.8.2.1 ...But recursive self-improvement could also happen
(40:12) 1.8.3 Next-paradigm AI probably won't be deployed at all, and ASI will probably show up in a world not wildly different from today's
(42:55) 1.8.4 We better sort out technical alignment, sandbox test protocols, etc., before the new paradigm seems even relevant at all, let alone scary
(43:40) 1.8.5 AI-assisted alignment research seems pretty doomed
(45:22) 1.8.6 The rest of AI for AI safety seems
544 episodes
All episodes
×Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.