$NVDA · SemiAnalysis benchmarks confirm Nvidia inference leadership across cost and throughput

episode briefing

extracted intelligence

SemiAnalysis

Bryan Shan x Cameron Quilici | Researcher Conversations at GTC

now viewing▶ 0:44

SemiAnalysis benchmarks confirm Nvidia inference leadership across cost and throughput

2026-04-07 · 3 company · 3 thematic

sentiment

1 bull0 bear2 neu

speakers

bryan shan

Co-host of SemiAnalysis podcast and co-developer of InferenceX, an open-source AI inference benchmarking suite. Attended GTC 2025 and works directly with Nvidia and AMD engineers on benchmark optimization.

cameron quilici

Co-host of SemiAnalysis podcast and co-developer of InferenceX. Focuses on agentic benchmark design, KV cache offloading strategies, and real-world inference performance modeling.

quote

“across all of our benchmarks, Nvidia chips like get the highest scores, the lowest cost for each make per token, etc. So it's really uh the king of inference, I guess.”

—bryan shan

now playing · $NVDA

$NVDA···bullish· highbryan shan

SemiAnalysis benchmarks confirm Nvidia inference leadership across cost and throughput

Independent open-source InferenceX benchmarks show Nvidia chips achieve highest throughput and lowest cost per token across the board, with Nvidia's customer intimacy enabling opt…

$GROQneutral· lowbryan shan

Groq LPU strengths recognized in disaggregated inference but Nvidia acquisition claim is inaccurate

Groq's LPU architecture surpasses GPUs for specific disaggregated inference tasks like fast interactivity, leading to co-packaging with Nvidia's Rubin system, though the claim tha…

$AMD···neutral· mediumcameron quilici

AMD MI355X HBM advantage may close gap with prefix caching but trails Nvidia today

AMD's MI355X offers 1.5x HBM capacity versus Nvidia's B200, which could enable better prefix caching performance in multi-turn workloads, but current InferenceX benchmarks show Nv…