How should we think about artificial general intelligence (AGI), and the risks it might pose? What constraints exist on technical solutions to the problem of aligning superhuman AI systems with human intentions? In this episode, I talk to Richard Ngo about his report analyzing AGI safety from first principles, and recent conversations he had with Eliezer Yudkowsky about the difficulty of AI alignment. Topics we discuss, and timestamps: 00:00:40 - The nature of intelligence and AGI 00:01:18 - The nature of intelligence 00:06:09 - AGI: what and how 00:13:30 - Single vs collective AI minds 00:18:57 - AGI in practice 00:18:57 - Impact 00:20:49 - Timing 00:25:38 - Creation 00:28:45 - Risks and benefits 00:35:54 - Making AGI safe 00:35:54 - Robustness of the agency abstraction 00:43:15 - Pivotal acts 00:50:05 - AGI safety concepts 00:50:05 - Alignment 00:56:14 - Transparency 00:59:25 - Cooperation 01:01:40 - Optima and selection processes 01:13:33 - The AI alignment research community 01:13:33 - Updates from the Yudkowsky conversation 01:17:18 - Corrections to the community 01:23:57 - Why others don't join 01:26:38 - Richard Ngo as a researcher 01:28:26 - The world approaching AGI 01:30:41 - Following Richard's work The transcript Richard on the Alignment Forum Richard on Twitter The AGI Safety Fundamentals course Materials that we mention: AGI Safety from First Principles Conversations with Eliezer Yudkowsky The Bitter Lesson Metaphors We Live By The Enigma of Reason Draft report on AI timelines, by Ajeya Cotra More is Different for AI The Windfall Clause Cooperative Inverse Reinforcement Learning Imitative Generalisation Eliciting Latent Knowledge Draft report on existential risk from power-seeking AI, by Joseph Carlsmith The Most Important Century