sergey levine

Just as PCs and LLMs enabled anyone to build software applications, a general-purpose robotic foundation model will let anyone build robot applications without solving the intelligence stack from scratch. This shifts robotics from vertically integrated silos to a platform model.

Multimodal LLMs contain world knowledge that can be grounded in physical situations via chain-of-thought reasoning, solving the 'common sense' bottleneck for handling novel scenarios. This is the key advance enabling generalization beyond training distribution.

The key to solving robotics data scarcity is not pre-collecting massive datasets but deploying systems that are useful enough to operate in the real world, where they autonomously gather diverse data. Tesla's FSD demonstrates this: usefulness precedes data scale.

Physical intelligence is embodiment-agnostic: the same foundation model can control humanoids, arms, bulldozers, or drone swarms. The fundamentals of physical interaction, causality, and object dynamics are conserved across embodiments, requiring only adaptation/fine-tuning per body.

Generative AI (LLMs) captures human knowledge but mimics human performance; deep RL (AlphaGo) discovers superhuman strategies but lacks world knowledge. Robotics needs both: web-scale knowledge plus the ability to exceed human dexterity/speed through autonomous practice.

Tasks easy for humans but hard for robots (dexterity, manipulation) become tractable with ML when data collection is straightforward. The remaining hard problems are where data is scarce and multi-level reasoning is needed — elderly care, childcare, unstructured homes.

Robots now fail at task interpretation, not low-level control. Adding semantic labels (coaching) to autonomous experience improves generalization without new teleoperation data. This means non-experts can improve robots via natural language, dramatically lowering deployment friction.

Two divergent paradigms exist: humanoid acrobatics rely on heavy simulation with near-zero real data, while manipulation relies on massive real-world datasets with little simulation. The field hasn't resolved whether one wins or a synthesis emerges, creating strategic uncertainty for robotics investments.