Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

Training Data

#Tech #Business #Sequoia Capital

39:30

In this episode of The Innovators & Investors Podcast, host Kristian Marquez sits down with David Brem, Managing Director of the University of Michigan’s Zell Lurie Commercialization Fund. David offers a rare glimpse into the inner workings of a student-led endowment fund focused on early-stage, sector-agnostic investments primarily in the Michigan ecosystem. He shares insights on their unique, founder-first investment approach, how they navigate pre-seed to Series A venture opportunities, and the rigorous due diligence process involving qualitative analysis over pure numbers. David also discusses his roles with global VC networks including Electro Ventures, the London Venture Capital Network, and Level Up Ventures, illustrating how he bridges U.S., European, and Australian venture ecosystems with a special focus on mobility and transportation tech. Highlights include deep dives into emerging trends like eVTOLs (electric vertical takeoff and landing aircraft), smart city infrastructure, and safety innovations in aviation technology. Listeners will gain valuable perspectives on how diverse expertise—from military intelligence and management consulting to academic ventures—shapes David’s investment thesis and community-building efforts. The episode also explores the importance of networking, adding value in the startup ecosystem, and practical advice for aspiring investors or entrepreneurs navigating the venture capital world. With stories of successes, challenges, and future outlooks, this episode is a must-listen for innovators, founders, and investors aiming to understand the intersection of academia, technology, and venture capital in today’s dynamic landscape. Learn more about David's work at https://zli.umich.edu/zell-lurie-commercialization-fund/ Connect with David on LinkedIn at https://www.linkedin.com/in/david-lowell-brem/ Think you'd be a great guest on the show? Apply at https://finstratmgmt.com/innovators-investors-podcast/ Want to learn more about Kristian Marquez's work? Check out his website at https://finstratmgmt.com…

about a year ago 49:13

MP3•Episode home

AI researcher Jim Fan has had a charmed career. He was OpenAI’s first intern before he did his PhD at Stanford with “godmother of AI,” Fei-Fei Li. He graduated into a research scientist position at Nvidia and now leads its Embodied AI “GEAR” group. The lab’s current work spans foundation models for humanoid robots to agents for virtual worlds.

Jim describes a three-pronged data strategy for robotics, combining internet-scale data, simulation data and real world robot data. He believes that in the next few years it will be possible to create a “foundation agent” that can generalize across skills, embodiments and realities—both physical and virtual. He also supports Jensen Huang’s idea that “Everything that moves will eventually be autonomous.”

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital

Mentioned in this episode:

World of Bits: Early OpenAI project Jim worked on as an intern with Andrej Karpathy. Part of a bigger initiative called Universe
Fei-Fei Li: Jim’s PhD advisor at Stanford who founded the ImageNet project in 2010 that revolutionized the field of visual recognition, led the Stanford Vision Lab and just launched her own AI startup, World Labs
Project GR00T: Nvidia’s “moonshot effort” at a robotic foundation model, premiered at this year’s GTC
Thinking Fast and Slow: Influential book by Daniel Kahneman that popularized some of his teaching from behavioral economics
Jetson Orin chip: The dedicated series of edge computing chips Nvidia is developing to power Project GR00T
Eureka: Project by Jim’s team that trained a five finger robot hand to do pen spinning
MineDojo: A project Jim did when he first got to Nvidia that developed a platform for general purpose agents in the game of Minecraft. Won NeurIPS 2022 Outstanding Paper Award
ADI: artificial dog intelligence
Mamba: Selective State Space Models, an alternative architecture to Transformers that Jim is interested in (original paper here)

00:00 Introduction

01:35 Jim’s journey to embodied intelligence

04:53 The GEAR Group

07:32 Three kinds of data for robotics

10:32 A GPT-3 moment for robotics

16:05 Choosing the humanoid robot form factor

19:37 Specialized generalists

21:59 GR00T gets its own chip

23:35 Eureka and Issac Sim

25:23 Why now for robotics?

28:53 Exploring virtual worlds

36:28 Implications for games

39:13 Is the virtual world in service of the physical world?

42:10 Alternative architectures to Transformers

44:15 Lightning round

55 episodes

Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

Training Data

16 subscribers

published about a year ago

MP3•Episode home

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital

Mentioned in this episode:

World of Bits: Early OpenAI project Jim worked on as an intern with Andrej Karpathy. Part of a bigger initiative called Universe
Fei-Fei Li: Jim’s PhD advisor at Stanford who founded the ImageNet project in 2010 that revolutionized the field of visual recognition, led the Stanford Vision Lab and just launched her own AI startup, World Labs
Project GR00T: Nvidia’s “moonshot effort” at a robotic foundation model, premiered at this year’s GTC
Thinking Fast and Slow: Influential book by Daniel Kahneman that popularized some of his teaching from behavioral economics
Jetson Orin chip: The dedicated series of edge computing chips Nvidia is developing to power Project GR00T
Eureka: Project by Jim’s team that trained a five finger robot hand to do pen spinning
MineDojo: A project Jim did when he first got to Nvidia that developed a platform for general purpose agents in the game of Minecraft. Won NeurIPS 2022 Outstanding Paper Award
ADI: artificial dog intelligence
Mamba: Selective State Space Models, an alternative architecture to Transformers that Jim is interested in (original paper here)

00:00 Introduction

01:35 Jim’s journey to embodied intelligence

04:53 The GEAR Group

07:32 Three kinds of data for robotics

10:32 A GPT-3 moment for robotics

16:05 Choosing the humanoid robot form factor

19:37 Specialized generalists

21:59 GR00T gets its own chip

23:35 Eureka and Issac Sim

25:23 Why now for robotics?

28:53 Exploring virtual worlds

36:28 Implications for games

39:13 Is the virtual world in service of the physical world?

42:10 Alternative architectures to Transformers

44:15 Lightning round

55 episodes

#Tech #Business #Sequoia Capital

All episodes

Training Data

1
Mapping the Mind of a Neural Net: Goodfire’s Eric Ho on the Future of Interpretability 47:07

3 days ago47:07

47:07

Eric Ho is building Goodfire to solve one of AI’s most critical challenges: understanding what’s actually happening inside neural networks. His team is developing techniques to understand, audit and edit neural networks at the feature level. Eric discusses breakthrough results in resolving superposition through sparse autoencoders, successful model editing demonstrations and real-world applications in genomics with Arc Institute's DNA foundation models. He argues that interpretability will be critical as AI systems become more powerful and take on mission-critical roles in society. Hosted by Sonya Huang and Roelof Botha, Sequoia Capital Mentioned in this episode: Mech interp : Mechanistic interpretability, list of important papers here Phineas Gage : 19th century railway engineer who lost most of his brain’s left frontal lobe in an accident. Became a famous case study in neuroscience. Human Genome Project : Effort from 1990-2003 to generate the first sequence of the human genome which accelerated the study of human biology Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs Zoom In: An Introduction to Circuits : First important mechanistic interpretability paper from OpenAI in 2020 Superposition : Concept from physics applied to interpretability that allows neural networks to simulate larger networks (e.g. more concepts than neurons) Apollo Research : AI safety company that designs AI model evaluations and conducts interpretability research Towards Monosemanticity: Decomposing Language Models With Dictionary Learning . 2023 Anthropic paper that uses a sparse autoencoder to extract interpretable features; followed by Scaling Monosemanticity Under the Hood of a Reasoning Model : 2025 Goodfire paper that interprets DeepSeek’s reasoning model R1 Auto-interpretability : The ability to use LLMs to automatically write explanations for the behavior of neurons in LLMs Interpreting Evo 2: Arc Institute's Next-Generation Genomic Foundation Model . (see episode with Arc co-founder Patrick Hsu ) Paint with Ember : Canvas interface from Goodfire that lets you steer an LLM’s visual output in real time (paper here ) Model diffing : Interpreting how a model differs from checkpoint to checkpoint during finetuning Feature steering : The ability to change the style of LLM output by up or down weighting features (e.g. talking like a pirate vs factual information about the Andromeda Galaxy) Weight based interpretability : Method for directly decomposing neural network parameters into mechanistic components, instead of using features The Urgency of Interpretability: Essay by Anthropic founder Dario Amodei On the Biology of a Large Language Model: Goodfire collaboration with Anthropic…

Training Data

1
ElevenLabs’ Mati Staniszewski: Why Voice Will Be the Fundamental Interface for Tech 59:53

10 days ago59:53

59:53

Mati Staniszewski, co-founder and CEO of ElevenLabs, explains how staying laser-focused on audio innovation has allowed his company to thrive despite the push into multimodality from foundation models. From a high school friendship in Poland to building one of the fastest-growing AI companies, Mati shares how ElevenLabs transformed text-to-speech with contextual understanding and emotional delivery. He discusses the company's viral moments (from Harry Potter by Balenciaga to powering Darth Vader in Fortnite), and explains how ElevenLabs is creating the infrastructure for voice agents and real-time translation that could eliminate language barriers worldwide. Hosted by: Pat Grady, Sequoia Capital Mentioned in this episode: Attention Is All You Need : The original Transformers paper Tortoise-tts : Open source text to speech model that was a starting point for ElevenLabs (which now maintains a v2 ) Harry Potter by Balenciaga : ElevenLabs’ first big viral moment from 2023 The first AI that can laugh : 2022 blog post backing up ElevenLab’s claim of laughter (it got better in v3 ) Darth Vader's voice in Fortnite : ElevenLabs used actual voice clips provided by James Earl Jones before he died Lex Fridman interviews Prime Minister Modi : ElevenLabs enabled Fridman to speak in Hindi and Modi to speak in English. Time Person of the Year 2024 : ElevenLabs-powered experiment with “conversational journalism” Iconic Voices : Richard Feynman, Deepak Chopra, Maya Angelou and more available in ElevenLabs reader app SIP trunking : a method of delivering voice, video, and other unified communications over the internet using the Session Initiation Protocol (SIP) Genesys : Leading enterprise CX platform for agentic AI Hitchhiker’s Guide to the Galaxy : Comedy/science-fiction series by Douglas Adams that contains the concept of the Babel Fish instantaneous translator, cited by Mati FYI : communication and productivity app for creatives that Mati uses, founded by will.i.am Lovable : prototyping app that Mati loves…

Training Data

1
From DevOps ‘Heart Attacks’ to AI-Powered Diagnostics With Traversal’s AI Agents 40:32

17 days ago40:32

40:32

Anish Agarwal and Raj Agrawal, co-founders of Traversal, are transforming how enterprises handle critical system failures. Their AI agents can perform root cause analysis in 2-4 minutes instead of the hours typically spent by teams of engineers scrambling in Slack channels. Drawing from their academic research in causal inference and gene regulatory networks, they’ve built agents that systematically traverse complex dependency maps to identify the smoking gun logs and problematic code changes. As AI-generated code becomes more prevalent, Traversal addresses a growing challenge: debugging systems where humans didn’t write the original code, making AI-powered troubleshooting essential for maintaining reliable software at scale. Hosted by Sonya Huang and Bogomil Balkansky, Sequoia Capital Mentioned in this episode: SRE : Site reliability engineering. The function within engineering teams that monitors and improves the availability and performance of software systems and services. Golden signals : four key metrics used by Site Reliability Engineers (SREs) to monitor the health and performance of IT systems: latency, traffic, errors and saturation. MELT data : Metrics, events, log, and traces. A framework for observability. The Bitter Lesson : Another mention of Nobel Prize winner Rich Sutton’s influential post.…

Training Data

1
The Breakthroughs Needed for AGI Have Already Been Made: OpenAI Former Research Head Bob McGrew 48:51

24 days ago48:51

48:51

As OpenAI's former Head of Research, Bob McGrew witnessed the company's evolution from GPT-3’s breakthrough to today's reasoning models. He argues that there are three legs of the stool for AGI—Transformers, scaled pre-training, and reasoning—and that the fundamentals that will shape the next decade-plus are already in place. He thinks 2025 will be defined by reasoning while pre-training hits diminishing returns. Bob discusses why the agent economy will price services at compute costs due to near-infinite supply, fundamentally disrupting industries like law and medicine, and how his children use ChatGPT to spark curiosity and agency. From robotics breakthroughs to managing brilliant researchers, Bob offers a unique perspective on AI’s trajectory and where startups can still find defensible opportunities. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode : Solving Rubik’s Cube with a robot hand : OpenAI’s original robotics research Computer Use and Operator : Anthropic and OpenAI reasoning breakthroughs that originated with OpenAi researchers Skild and Physical Intelligence : Robotics-oriented companies Bob sees as well-positioned now Distyl : AI company founded by ex-Palintir alums to create enterprise workflows driven by proprietary data Member of the technical staff : Title at OpenAI designed to break down barriers between AI researchers and engineers Howie.ai : Scheduling app that Bob uses…

Training Data

1
OpenAI Codex Team: From Coding Autocomplete to Asynchronous Autonomous Agents 37:44

4 weeks ago37:44

37:44

Hanson Wang and Alexander Embiricos from OpenAI's Codex team discuss their latest AI coding agent that works independently in its own environment for up to 30 minutes, generating full pull requests from simple task descriptions. They explain how they trained the model beyond competitive programming to match real-world software engineering needs, the shift from pairing with AI to delegating to autonomous agents, and their vision for a future where the majority of code is written by agents working on their own computers. The conversation covers the technical challenges of long-running inference, the importance of creating realistic training environments, and how developers are already using Codex to fix bugs and implement features at OpenAI. Hosted by Sonya Huang and Lauren Reeder, Sequoia Capital Mentioned in this episode : The Culture : Sci-Fi series by Iain Banks portraying an optimistic view of AI The Bitter Lesson : Influential paper by Rich Sutton on the importance of scale as a strategic unlock for AI.…

Training Data

1
Google I/O Afterparty: The Future of Human-AI Collaboration, From Veo to Mariner 53:51

5 weeks ago53:51

53:51

Fresh off impressive releases at Google’s I/O event, three Google Labs leaders explain how they’re reimagining creative tools and productivity workflows. Thomas Iljic details how video generation is merging filmmaking with gaming through generative AI cameras and world-building interfaces in Whisk and Veo. Jaclyn Konzelmann demonstrates how Project Mariner evolved from a disruptive browser takeover to an intelligent background assistant that remembers context across multiple tasks. Simon Tokumine reveals NotebookLM’s expansion beyond viral audio overviews into a comprehensive platform for transforming information into personalized formats. The conversation explores the shift from prompting to showing and telling, the economics of AI-powered e-commerce, and why being “too early” has become Google Labs’ biggest challenge and advantage. Hosted by Sonya Huang, Sequoia Capital 00:00 Introduction 02:12 Google's AI models and public perception 04:18 Google's history in image and video generation 06:45 Where Whisk and Flow fit 10:30 How close are we to having the ideal tool for the craft? 13:05 Where do the movie and game worlds start to merge? 16:25 Introduction to Project Mariner 17:15 How Mariner works 22:34 Mariner user behaviors 27:07 Temporary tattoos and URL memory 27:53 Project Mariner's future 29:26 Agent capabilities and use cases 31:09 E-commerce and agent interaction 35:03 Notebook LM evolution 48:26 Predictions and future of AI Mentioned in this episode: Whisk : Image and video generation app for consumers Flow : AI-powered filmmaking with new Veo 3 model Project Mariner : research prototype exploring the future of human-agent interaction, starting with browsers NotebookLM : tool for understanding and engaging with complex information including Audio Overviews and now a mobile app Shop with AI Mode : Shopping app with a virtual try-on tool based on your own photos Stitch : New prompt-based interface to design UI for mobile and web applications. ControlNet paper : Outlined an architecture for adding conditional language to direct the outputs of image generation with diffusion models…

Training Data

1
From Data Centers to Dyson Spheres: P-1 AI's Path to Hardware Engineering AGI 38:25

6 weeks ago38:25

38:25

Former Airbus CTO Paul Eremenko shares his vision for bringing AI to physical engineering, starting with Archie—an AI agent that works alongside human engineers. P-1 AI is tackling the challenge of generating synthetic training data to teach AI systems about complex physical systems, from data center cooling to aircraft design and beyond. Eremenko explains how Archie breaks down engineering tasks into primitive operations and uses a federated approach combining multiple AI models. The goal is to progress from entry-level engineering capabilities to eventually achieving engineering AGI that can design things humans cannot. Hosted by Sonya Huang and Pat Grady, Sequoia Capital…

Training Data

1
Gong’s Amit Bendov: From Meeting Recordings to Revenue AI 42:05

7 weeks ago42:05

42:05

CEO Amit Bendov shares how Gong evolved from a meeting transcription tool to an AI-powered revenue platform that's increasing sales capacity by up to 60%. He explains why task-specific AI agents are the key to enterprise adoption, and why human accountability will remain crucial even as AI takes over routine sales tasks. Amit also reveals how Gong survived recent market headwinds by expanding their product suite while maintaining their customer-first approach. Hosted by Sonya Huang and Pat Grady, Sequoia Capital Mentioned in this episode : “ New paradigm of AI architectures ”: Yann LeCun’s talk at Davos where he talks about the world beyond transformers and LLMs in the next 3-5 years. The Beginning of Infinity : Book by David Deutsch that Amit says is “mind-changing.”…

Training Data

1
LIVE: Google's Jeff Dean on the Coming Transformations in AI 30:59

8 weeks ago30:59

30:59

At AI Ascent 2025, Jeff Dean makes bold predictions. Discover how the pioneer behind Google's TPUs and foundational AI research sees the technology evolving, from specialized hardware to more organic systems, and future engineering capabilities.

Training Data

1
LIVE: Ambient Agents and the New Agent Inbox ft. Harrison Chase of LangChain 8:28

8 weeks ago8:28

8:28

Recorded live at Sequoia’s AI Ascent 2025: LangChain CEO Harrison Chase introduces the concept of ambient agents, AI systems that operate continuously in the background responding to events rather than direct human prompts. Learn how these agents differ from traditional chatbots, why human oversight remains essential and how this approach could dramatically scale our ability to leverage AI.…

Training Data

1
LIVE: How AI is Reinventing Software Business Models ft. Bret Taylor of Sierra 34:30

8 weeks ago34:30

34:30

Recorded live at Sequoia’s AI Ascent 2025: Sierra co-founder Bret Taylor discusses why AI is driving a fundamental shift from subscription-based pricing to outcomes-based models. Learn why this transition is harder for incumbents than startups, why applied AI and vertical specialization represent the biggest opportunities for entrepreneurs and how to position your AI company for success in this new paradigm.…

Training Data

1
LIVE: Sam Altman of OpenAI on Building the ‘Core AI Subscription’ for Your Life 32:27

8 weeks ago32:27

32:27

Recorded live at Sequoia’s AI Ascent 2025: Sam reflects on OpenAI’s evolution from a 14-person research lab to a dominant AI platform. He envisions transforming ChatGPT into a deeply personal AI service that remembers your entire life's context—from conversations to emails—while working seamlessly across all services. Sam describes the generation gap in how users engage with ChatGPT, and makes surprisingly specific predictions for the next 2-3 years of AI evolution.…

Training Data

1
Workday CEO Carl Eschenbach: Building the System of Record for the AI Era 47:49

9 weeks ago47:49

47:49

Workday CEO and Sequoia partner Carl Eschenbach explains how the company is evolving its platform to handle both human and AI workers. He shares Workday’s three-pronged approach to monetizing AI through seat-based pricing, role-based agents and consumption-based API access. Eschenbach discusses why domain-specific agents with curated data will be more valuable than general-purpose models in the enterprise, and how Workday is helping enterprises navigate the transition to an AI-powered workplace while maintaining human connection. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital…

Training Data

1
The Quest to ‘Solve All Diseases’ with AI: Isomorphic Labs’ Max Jaderberg 55:40

10 weeks ago55:40

55:40

After pioneering reinforcement learning breakthroughs at DeepMind with Capture the Flag and AlphaStar, Max Jaderberg aims to revolutionize drug discovery with AI as Chief AI Officer of Isomorphic Labs, which was spun out of DeepMind. He discusses how AlphaFold 3's diffusion-based architecture enables unprecedented understanding of molecular interactions, and why we're approaching a "Move 37 moment" in AI-powered drug design where models will surpass human intuition. Max shares his vision for general AI models that can solve all diseases, and the importance of developing agents that can learn to search through the whole potential design space. Hosted by Stephanie Zhan, Sequoia capital Mentioned in this episode : Playing Atari with Deep Reinforcement Learning : Seminal 2013 paper on Reinforcement Learning Capture the Flag : 2019 DeepMind paper on the emergence of cooperative agents AlphaStar : 2019 DeepMind paper on attaining grandmaster level in StarCraft II using multi-agent RL AlphaFold Server : Web interface for AlphaFold 3 model for non-commercial academic use…

Training Data

1
Pricing in the AI Era: From Inputs to Outcomes, with Paid CEO Manny Medina 45:29

11 weeks ago45:29