AI Infrastructure · GPU networking is the new bottleneck: multi-GPU kernels, in-network compute, NVL72 scale-up

episode briefing

extracted intelligence

Y Combinator

Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

now viewing▶ 7:55

GPU networking is the new bottleneck: multi-GPU kernels, in-network compute, NVL72 scale-up

2026-07-29 · 8 company · 8 thematic

sentiment

7 bull0 bear1 neu

speakers

mark

CTO of AMD, leading chip design using thousands of AI sub-agents; sees agentic AI driving 1:1 CPU:GPU ratio and transforming end-to-end workflows.

john

Launched a YouTube channel around 2020 focused on Silicon Valley history, reached ~500K subscribers; treats media as a blog/hobby rather than a business; co-hosts TBPN podcast interviewing entrepreneurs.

misha

Misha (Mishas Manski) spent two decades on hardware-software co-design, most recently running AI infrastructure at NVIDIA and working on co-design at Meta. He recently joined Marlo to build workload-optimized heterogeneous inference infrastructure.

stuart

Stuart is a CS PhD student at Stanford and researcher at Cursor where he trains the Composer model. He developed Parallel Kittens, a CUDA framework for multi-GPU kernels, and Thunder Kittens for single-GPU kernels.

now playing · AI Infrastructure

AI Hardware & Chip Architecturetailwindscore 9/10unknown

Chip specialization accelerating: training vs inference, prefill vs decode, SRAM for latency

The semiconductor landscape is fragmenting into specialized architectures: training chips (high bandwidth, all-to-all), prefill GPUs (compute-bound), decode SRAM accelerators (memory-bound)…

AI Economics & Business Modelstailwindscore 9/10john

Intelligence per joule compounding 18x in 16 months: local inference captures 80-90% of queries

Local open-source models on consumer accelerators (Apple M4 Max, NVIDIA DGX Spark) now deliver 88.7% of frontier capability at radically lower energy/cost. Intelligence per watt improved 3x…

AI Infrastructuretailwindscore 8/10stuart

GPU networking is the new bottleneck: multi-GPU kernels, in-network compute, NVL72 scale-up

Networking consumes up to 50% of runtime for LLM prefill; fine-grained overlap of compute and communication at tile/token granularity is critical. In-network compute offloads collectives to…

Memory & Storagetailwindscore 8/10misha

SRAM-based GMV accelerators disrupt decode: on-die weights eliminate HBM bandwidth wall

Decode is memory-bandwidth bound; SRAM machines (SambaNova, Groq, etc.) keep entire weight matrices on-die, delivering bytes/cycle bandwidth and microsecond latency. They extend interactive…

AI Agentstailwindscore 7/10unknown

Voice agents demand batch-size-one inference: new chip architectures for latency-critical workloads

Voice agents require sub-second latency at batch size one, which is infeasible on throughput-optimized GPUs (run out of GPUs, too expensive). This creates a distinct silicon category: ultra…

Networking & Optical Infrastructuretailwindscore 7/10stuart

In-network compute and fine-grained NVLink overlap redefine GPU cluster architecture

Collective operations (all-reduce, all-gather) moving into the network fabric frees GPU SMs for compute. Asynchronous bulk device-initiated networking and tile-granularity overlap (KB not M…

Open Source AItailwindscore 7/10john

Small open models rapidly closing frontier gap: 3x intelligence per watt in 2 years

Open models (Gemma, Qwen, Granite, GPT-OSS) in 1-200B parameter range are advancing faster than frontier models on capability per watt. Combined with better quantization, longer pre-trainin…

Data Center Infrastructuremixedscore 7/10misha

Heterogeneous data centers: power density, cooling, networking co-design for mixed accelerators

Deploying GPUs, SRAM accelerators, and CPUs in the same facility creates hard systems problems: power density spikes, liquid cooling requirements, brake configuration complexity, and inter-…