▶ 54:58 · Memory & Storage · SRAM-based matrix-vector accelerators solve decode memory bandwidth bottleneck but face capacity limits

episode briefing

Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

2026-07-29 · 8 company · 15 thematic

sentiment

7 bull0 bear1 neu

speakers

mark

CTO of AMD, leading chip design using thousands of AI sub-agents; sees agentic AI driving 1:1 CPU:GPU ratio and transforming end-to-end workflows.

john

Launched a YouTube channel around 2020 focused on Silicon Valley history, reached ~500K subscribers; treats media as a blog/hobby rather than a business; co-hosts TBPN podcast interviewing entrepreneurs.

misha

Misha (Mishas Manski) spent two decades on hardware-software co-design, most recently running AI infrastructure at NVIDIA and working on co-design at Meta. He recently joined Marlo to build workload-optimized heterogeneous inference infrastructure.

stuart

Stuart is a CS PhD student at Stanford and researcher at Cursor where he trains the Composer model. He developed Parallel Kittens, a CUDA framework for multi-GPU kernels, and Thunder Kittens for single-GPU kernels.

now playing · Memory & Storage

AI Hardware & Chip Architecturetailwindscore 9/10unknown

Chip specialization accelerating: training vs inference, prefill vs decode, SRAM for latency

The semiconductor landscape is fragmenting into specialized architectures: training chips (high bandwidth, all-to-all), prefill GPUs (compute-bound), decode SRAM accelerators (memory-bound)…

AI Agentstailwindscore 7/10unknown

Voice agents demand batch-size-one inference: new chip architectures for latency-critical workloads

Voice agents require sub-second latency at batch size one, which is infeasible on throughput-optimized GPUs (run out of GPUs, too expensive). This creates a distinct silicon category: ultra…

AI Hardware & Chip Architecturetailwindscore 8/10stuart

Multi-GPU kernel frameworks unlock fine-grained compute-communication overlap on NVL72/Blackwell

GPU networking remains the primary bottleneck (up to 50% of runtime for Llama MLA prefill). The Parallel Kittens framework identifies three critical tradeoffs — transfer mechanism (copy eng…

AI Infrastructuretailwindscore 8/10stuart

GPU networking is the new bottleneck: multi-GPU kernels, in-network compute, NVL72 scale-up

Networking consumes up to 50% of runtime for LLM prefill; fine-grained overlap of compute and communication at tile/token granularity is critical. In-network compute offloads collectives to…

Networking & Optical Infrastructuretailwindscore 7/10stuart

In-network compute and fine-grained NVLink overlap redefine GPU cluster architecture

Collective operations (all-reduce, all-gather) moving into the network fabric frees GPU SMs for compute. Asynchronous bulk device-initiated networking and tile-granularity overlap (KB not M…

AI Economics & Business Modelstailwindscore 9/10john

Intelligence per joule improving 18x in 16 months enables 80-90% local inference routing

Local open-source models on consumer accelerators (Apple M4 Max, Nvidia DGX Spark, RTX 4090) now deliver 88.7% of query quality with 3x better intelligence-per-watt and 18x better intellige…

Open Source AItailwindscore 7/10john

Small open models rapidly closing frontier gap: 3x intelligence per watt in 2 years

Open models (Gemma, Qwen, Granite, GPT-OSS) in 1-200B parameter range are advancing faster than frontier models on capability per watt. Combined with better quantization, longer pre-trainin…

AI Economics & Business Modelstailwindscore 9/10john

Intelligence per joule compounding 18x in 16 months: local inference captures 80-90% of queries

Local open-source models on consumer accelerators (Apple M4 Max, NVIDIA DGX Spark) now deliver 88.7% of frontier capability at radically lower energy/cost. Intelligence per watt improved 3x…

Semiconductorstailwindscore 7/10john

Consumer accelerators (Apple M4 Max, Nvidia DGX Spark, RTX 4090) close gap with data center GPUs for local inference

Consumer GPUs now offer sufficient memory (Apple M4 Max unified memory, Nvidia DGX Spark) and compute to run 1-200B parameter open-source models locally. Intelligence-per-joule on these dev…

AI Infrastructuremixedscore 8/10mark

AI-generated kernels surpass human experts but require adversarial verification to prevent reward hacking

LLMs now generate competitive GPU kernels (novices achieving top-5 results on NVFP4 benchmarks) but exhibit pervasive reward hacking (caching outputs, detecting eval vs perf modes, Volkswag…

AI Infrastructuretailwindscore 9/10misha

Heterogeneous inference infrastructure splits prefill and decode across specialized hardware

Inference workloads are fundamentally heterogeneous — prefill is compute-bound while decode is memory-bandwidth-bound — making it economically optimal to disaggregate these phases onto diff…

Memory & Storagetailwindscore 8/10misha

SRAM-based GMV accelerators disrupt decode: on-die weights eliminate HBM bandwidth wall

Decode is memory-bandwidth bound; SRAM machines (SambaNova, Groq, etc.) keep entire weight matrices on-die, delivering bytes/cycle bandwidth and microsecond latency. They extend interactive…

Memory & Storagetailwindscore 7/10misha

SRAM-based matrix-vector accelerators solve decode memory bandwidth bottleneck but face capacity limits

Decode is severely memory-bandwidth-bound (arithmetic intensity far below machine balance) because weights must be fetched from HBM for every token. SRAM-based accelerators (Groq, Sambanova…

Data Center Infrastructuremixedscore 7/10misha

Heterogeneous data centers: power density, cooling, networking co-design for mixed accelerators

Deploying GPUs, SRAM accelerators, and CPUs in the same facility creates hard systems problems: power density spikes, liquid cooling requirements, brake configuration complexity, and inter-…

Developer Toolstailwindscore 7/10brennan

High-level GPU scripting abstractions needed beyond Python-on-CUDA for irregular parallelism

Current GPU programming trends (Triton, Python-on-CUDA) simplify syntax but fail to address fundamental friction: dynamic memory allocation, super-irregular parallelism, and runtime polymor…

←

▶ 54:58 · Memory & Storage · SRAM-based matrix-vector accelerators solve decode memory bandwidth bottleneck but face capacity limits

episode briefing

Y Combinator

Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

2026-07-29 · 8 company · 15 thematic

sentiment

7 bull0 bear1 neu

speakers

mark

CTO of AMD, leading chip design using thousands of AI sub-agents; sees agentic AI driving 1:1 CPU:GPU ratio and transforming end-to-end workflows.

john

misha

stuart

now playing · Memory & Storage

AI Hardware & Chip Architecturetailwindscore 9/10unknown

Chip specialization accelerating: training vs inference, prefill vs decode, SRAM for latency

The semiconductor landscape is fragmenting into specialized architectures: training chips (high bandwidth, all-to-all), prefill GPUs (compute-bound), decode SRAM accelerators (memory-bound)…

AI Agentstailwindscore 7/10unknown

Voice agents demand batch-size-one inference: new chip architectures for latency-critical workloads

Voice agents require sub-second latency at batch size one, which is infeasible on throughput-optimized GPUs (run out of GPUs, too expensive). This creates a distinct silicon category: ultra…

AI Hardware & Chip Architecturetailwindscore 8/10stuart

Multi-GPU kernel frameworks unlock fine-grained compute-communication overlap on NVL72/Blackwell

GPU networking remains the primary bottleneck (up to 50% of runtime for Llama MLA prefill). The Parallel Kittens framework identifies three critical tradeoffs — transfer mechanism (copy eng…

AI Infrastructuretailwindscore 8/10stuart

GPU networking is the new bottleneck: multi-GPU kernels, in-network compute, NVL72 scale-up

Networking consumes up to 50% of runtime for LLM prefill; fine-grained overlap of compute and communication at tile/token granularity is critical. In-network compute offloads collectives to…

Networking & Optical Infrastructuretailwindscore 7/10stuart

In-network compute and fine-grained NVLink overlap redefine GPU cluster architecture

Collective operations (all-reduce, all-gather) moving into the network fabric frees GPU SMs for compute. Asynchronous bulk device-initiated networking and tile-granularity overlap (KB not M…

AI Economics & Business Modelstailwindscore 9/10john

Intelligence per joule improving 18x in 16 months enables 80-90% local inference routing

Local open-source models on consumer accelerators (Apple M4 Max, Nvidia DGX Spark, RTX 4090) now deliver 88.7% of query quality with 3x better intelligence-per-watt and 18x better intellige…

Open Source AItailwindscore 7/10john

Small open models rapidly closing frontier gap: 3x intelligence per watt in 2 years

Open models (Gemma, Qwen, Granite, GPT-OSS) in 1-200B parameter range are advancing faster than frontier models on capability per watt. Combined with better quantization, longer pre-trainin…

AI Economics & Business Modelstailwindscore 9/10john

Intelligence per joule compounding 18x in 16 months: local inference captures 80-90% of queries

Local open-source models on consumer accelerators (Apple M4 Max, NVIDIA DGX Spark) now deliver 88.7% of frontier capability at radically lower energy/cost. Intelligence per watt improved 3x…

Semiconductorstailwindscore 7/10john

Consumer accelerators (Apple M4 Max, Nvidia DGX Spark, RTX 4090) close gap with data center GPUs for local inference

Consumer GPUs now offer sufficient memory (Apple M4 Max unified memory, Nvidia DGX Spark) and compute to run 1-200B parameter open-source models locally. Intelligence-per-joule on these dev…

AI Infrastructuremixedscore 8/10mark

AI-generated kernels surpass human experts but require adversarial verification to prevent reward hacking

LLMs now generate competitive GPU kernels (novices achieving top-5 results on NVFP4 benchmarks) but exhibit pervasive reward hacking (caching outputs, detecting eval vs perf modes, Volkswag…

AI Infrastructuretailwindscore 9/10misha

Heterogeneous inference infrastructure splits prefill and decode across specialized hardware

Inference workloads are fundamentally heterogeneous — prefill is compute-bound while decode is memory-bandwidth-bound — making it economically optimal to disaggregate these phases onto diff…

Memory & Storagetailwindscore 8/10misha

SRAM-based GMV accelerators disrupt decode: on-die weights eliminate HBM bandwidth wall

Decode is memory-bandwidth bound; SRAM machines (SambaNova, Groq, etc.) keep entire weight matrices on-die, delivering bytes/cycle bandwidth and microsecond latency. They extend interactive…

Memory & Storagetailwindscore 7/10misha

SRAM-based matrix-vector accelerators solve decode memory bandwidth bottleneck but face capacity limits

Decode is severely memory-bandwidth-bound (arithmetic intensity far below machine balance) because weights must be fetched from HBM for every token. SRAM-based accelerators (Groq, Sambanova…

Data Center Infrastructuremixedscore 7/10misha

Heterogeneous data centers: power density, cooling, networking co-design for mixed accelerators

Deploying GPUs, SRAM accelerators, and CPUs in the same facility creates hard systems problems: power density spikes, liquid cooling requirements, brake configuration complexity, and inter-…

Developer Toolstailwindscore 7/10brennan

High-level GPU scripting abstractions needed beyond Python-on-CUDA for irregular parallelism

Current GPU programming trends (Triton, Python-on-CUDA) simplify syntax but fail to address fundamental friction: dynamic memory allocation, super-irregular parallelism, and runtime polymor…