reiner pope

The fundamental constraint in chip design is data movement cost (muxes, register files) scaling linearly with register file size, while compute (multiply-accumulate) scales quadratically with array dimensions. Systolic arrays solve this by storing weights locally and streaming activations, turning O(xy) data movement into O(x) for O(xy) compute. This principle applies from ALU level to full chip architecture.

Larger scale-up domains (NVL72, Rubin) solve the memory bandwidth bottleneck by allowing parallel weight loading across more GPUs, directly reducing latency and enabling longer context lengths. Pipelining solves capacity but not bandwidth. The 20ms HBM drain time is a fundamental hardware constant.

Expert parallelism within a single scale-up rack (64-72 GPUs) is the optimal mapping for MoE layers. Pipeline parallelism across racks adds latency without reducing per-GPU KV cache memory. Tensor parallelism is obsolete as experts shrink. The 300×sparsity rule (batch size) emerges from hardware FLOPs/bandwidth ratio.

The same compute-vs-communication tradeoff appears at every level: ALU (muxes vs MAC), systolic array (register file vs matrix unit), chip (HBM vs compute), and cluster (interconnect vs GPU). Architectures that amortize data movement over more compute (systolic arrays, scratchpads, large batches) win on efficiency. This is a first-principle design constraint, not a temporary bottleneck.

The progression from 8-GPU to 72-GPU to 500+ GPU scale-up domains matches the all-to-all communication needs of Mixture-of-Experts layers. Each generation expands the expert parallelism limit within a single rack, avoiding the 8x slower scale-out network. Cable density and power/cooling are the physical constraints.

Equalizing pre-training, RL, and inference compute costs implies models should be trained on ~100x more tokens than Chinchilla predicts (200T vs 2T tokens). This explains why parameter counts haven't grown as fast as compute: models are smaller but trained vastly longer to minimize inference cost per token.

Pipeline registers increase clock frequency but consume area that could be used for compute. Throughput = (compute per cycle) × (cycles per second). Pushing clock too high (e.g., register after every gate) spends all area on synchronization, lowering compute per cycle. Optimal design targets ~10-30 gate delays per cycle, mirroring the batch-size latency/throughput tradeoff in inference.

An FPGA lookup table (4-input LUT) implements any 4→1 boolean function using a 16:1 mux costing ~32 gates vs 3 gates for a hardwired 4-input AND. The 10x area/energy penalty is the price of field programmability. FPGAs win only when tape-out cost ($30M) and redesign frequency (monthly) outweigh the per-unit penalty — a narrow niche (HFT, prototyping).