rohan anil

Writing high-performance kernels (e.g., QR factorization) requires rare human expertise and months of effort; automating this with AI models would unlock orders-of-magnitude faster architecture iteration and enable novel optimizers like Shampoo at scale.

Current chain-of-thought scaling spends compute one token at a time, making inference costly and limiting frontier model access to a subset of users; architectural changes that increase computational depth without token-by-token generation are needed to expand the addressable market for AI.

Current separate pre-training (perplexity minimization) and RL (chain-of-thought) pipelines are suboptimal; combining them with second-order optimizers like Shampoo and architecture-optimizer co-design can yield 10x+ compute efficiency gains by aligning training objectives with inference-time computation patterns.

Human brains build custom circuits during development; matching biological efficiency likely needs analog compute with error correction, not just scaling current GPU/TPU architectures.

Novel architectures require custom high-performance kernels (e.g., 60x speedup for QR factorization), but current models cannot write them; automating kernel generation via AI-assisted search is the critical inner loop enabling rapid architecture experimentation on modern hardware like B200 GPUs.