misha

Inference workloads are fundamentally heterogeneous — prefill is compute-bound while decode is memory-bandwidth-bound — making it economically optimal to disaggregate these phases onto different accelerator architectures (GPUs for prefill, SRAM-based accelerators for decode) rather than running both on uniform GPU clusters. This specialization extends to attention/MLP separation and speculative decoding offload, with TCO benefits emerging at sufficient output lengths.

Decode is memory-bandwidth bound; SRAM machines (SambaNova, Groq, etc.) keep entire weight matrices on-die, delivering bytes/cycle bandwidth and microsecond latency. They extend interactive latency regimes where GPUs collapse. However, capacity is limited by reticle size (hundreds of MB to tens of GB), requiring model sharding and heterogeneous co-design to maintain benefit.

Decode is severely memory-bandwidth-bound (arithmetic intensity far below machine balance) because weights must be fetched from HBM for every token. SRAM-based accelerators (Groq, Sambanova, etc.) keep entire weight matrices on-die, delivering orders-of-magnitude higher bandwidth and lower latency for decode. However, reticle-limited die area caps capacity at hundreds of MB to tens of GB, requiring model sharding across chips. Heterogeneous co-design must characterize when SRAM acceleration remains beneficial post-sharding.

Deploying GPUs, SRAM accelerators, and CPUs in the same facility creates hard systems problems: power density spikes, liquid cooling requirements, brake configuration complexity, and inter-accelerator networking topology. Performance modeling and simulation become essential to navigate design space. This favors vertically integrated infra players and specialized data center operators.