The team behind continuous batching says your idle GPUs should be running inference, not sitting dark
Every GPU cluster has dead time. Training jobs finish, workloads shift and hardware sits dark while power and cooling costs keep running. For neocloud operators, those empty cycles are lost margin.The obvious workaround is spot GPU markets — renting spare capacity to whoever needs it. But spot instances mean the cloud vendor is still the one doing the renting, and engineers buying that capacity are still paying for raw compute with no inference stack attached. FriendliAI's answer is different: run inference directly on the unused hardware, optimize for token throughput, and split the revenue with the operator. FriendliAI was founded by Byung-Gon Chun, the researcher whose paper on continuous batching became foundational to vLLM, the open source inference engine used across most prod [...]
Rating
Innovation
Pricing
Technology
Usability
We have discovered similar tools to what you are looking for. Check out our suggestions for similar AI tools.
5% GPU utilization: The $401 billion AI infrastructure problem enterprises can't keep ignoring
For the last 24 months, one narrative justified every over-provisioned data center and bloated IT budget: the GPU scramble. Silicon was the new oil, and H100s traded like contraband. Reserve capacity [...]
FOMO is why enterprises pay for GPUs they don't use — and why prices keep climbing
Enterprises can't fix their GPU waste problem because the fix makes the problem worse. Releasing idle capacity would improve utilization, but the same shortage driving GPU prices up is exactly wh [...]
Baseten takes on hyperscalers with new AI training platform that lets you own your model weights
Baseten, the AI infrastructure company recently valued at $2.15 billion, is making its most significant product pivot yet: a full-scale push into model training that could reshape how enterprises wean [...]
Together AI's ATLAS adaptive speculator delivers 400% inference speedup by learning from workloads in real-time
Enterprises expanding AI deployments are hitting an invisible performance wall. The culprit? Static speculators that can't keep up with shifting workloads.Speculators are smaller AI models that w [...]
Inference is splitting in two — Nvidia’s $20B Groq bet explains its next act
Nvidia’s $20 billion strategic licensing deal with Groq represents one of the first clear moves in a four-front fight over the future AI stack. 2026 is when that fight becomes obvious to enterprise [...]
AI inference costs dropped up to 10x on Nvidia's Blackwell — but hardware is only half the equation
Lowering the cost of inference is typically a combination of hardware and software. A new analysis released Thursday by Nvidia details how four leading inference providers are reporting 4x to 10x redu [...]
Open source Mamba 3 arrives to surpass Transformer architecture with nearly 4% improved language modeling, reduced latency
The generative AI era began for most people with the launch of OpenAI's ChatGPT in late 2022, but the underlying technology — the "Transformer" neural network architecture that allows [...]
Train-to-Test scaling explained: How to optimize your end-to-end AI compute budget for inference
The standard guidelines for building large language models (LLMs) optimize only for training costs and ignore inference costs. This poses a challenge for real-world applications that use inference-tim [...]