shruti kulkarni

Evaluating AI infrastructure on input metrics like $/GPU-hour or FLOPS/$ is a fundamental mismatch because businesses run on token output; cost per token incorporates both input costs and delivered throughput, revealing true ROI.

As token costs drop (reasoning models, mixture-of-experts), new agentic use cases emerge where AI takes turns with AI and tools, multiplying LLM calls and token demand far beyond conversational workloads, increasing total GPU demand rather than reducing it.

Nvidia's Vera Rubin platform exemplifies extreme co-design — seven chips plus full software stack (CUDA kernels, runtimes, serving software, Dynamo disaggregated serving) co-optimized for lowest token cost, extending to ecosystem partners and OSS frameworks for compounding advantage.

The Nvidia ecosystem (vLLM, SGLang, TensorRT, disaggregated serving, KV cache offloading, speculative decoding) stacks optimizations that compound — delivering 8x throughput improvement in six months on the same infrastructure, continuously driving down token cost.

Businesses monetize tokens through four primary models: selling tokens directly (Fireworks, Together AI), building AI-native products (Perplexity, Cursor), enhancing existing products with AI (Adobe Firefly in Photoshop, Shopify, Airbnb), and improving internal operations — each requiring cost-per-token discipline and demand-distribution-aware pricing.