dan biderman

Externalized memory (RAG/context engineering) hits scaling limits with token costs and retrieval failures, while training adapters on private data enables 100x inference reduction and implicit knowledge associations that retrieval cannot achieve. Breakthroughs in continual learning will unlock personalized models for every team and individual.

KV caches for long contexts consume massive HBM (80GB for a single Wikipedia article vs 100GB for entire Llama-70B weights), proving current inference architecture is bit-inefficient. Offline training compute to compress context into small adapters requires new infrastructure for training many small models rather than one big run.