wuk kuan

Three structural forces make inference harder: models scaling to multi-trillion parameters requiring massive sharding; explosion of model architectures (sparse attention, linear attention) and chip variants; and agentic workloads introducing unpredictable KV cache lifetimes from tool use and human-in-the-loop delays.

The complexity of matching diverse model architectures (sparse attention, linear attention, varying context lengths) to diverse hardware (H100, B200, TPU, etc.) for each use case makes a single proprietary stack insufficient; open source enables the combinatorial innovation needed across the M×N problem of models × chips.

Each Nvidia chip generation (H100, B200, GB200) demands different model architectures for optimal performance, while TPUs and other accelerators diverge further; this vertical integration complexity necessitates a horizontal software layer (VLM) that abstracts hardware for models, similar to how OS abstracted CPU/memory and databases abstracted storage.