jerry tworek

Current transformers only learn during lab training; real-world deployment requires models that adapt to new tasks, codebases, and tools at test time without human-in-the-loop retraining, which demands architectural breakthroughs beyond in-context learning and fine-tuning.

Transformers are trained in the lab on static data but deployed in the real world where distributions shift; they cannot learn at test time beyond limited in-context learning or inefficient fine-tuning, creating a ceiling on real-world utility that requires new architectures with meta-learned test-time adaptation.

Building labs where AI agents write kernels, run experiments, and iterate architectures daily — targeting 10-200 experiments per day vs. current human-limited pace — will dramatically shorten the search for transformer replacements by maximizing researcher agency and iteration speed.

Current coding agents already give individual researchers 10x leverage; a natively automated lab targeting 10-200 architecture experiments per day could outpace large labs constrained by organizational inertia and release-cycle pressure.