OpenAI quietly found an optimization that cuts inference costs in half on the models it touched, and the first visible payoff showed up in logged-out ChatGPT traffic where GPU count dropped to just a few hundred. The work came from engineers squeezing more out of existing servers rather than chasing new chips, a detail that stayed internal until The Information surfaced it.
OpenAI has found a way to cut inference costs in half.
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
The change only targeted logged-out traffic so far, leaving open how much headroom remains for the rest of the service or other products.
Anthropic and Google are chasing the same server-level gains, yet no public benchmarks or code have appeared, so the exact trick and its broader applicability stay unknown for now.
Many users hail OpenAI's optimization for halving ChatGPT inference costs as a huge efficiency leap that could lower the price of intelligence, while a few blame it for recent drops in model quality.
No Digg Deeper questions have been answered for this story yet.
OpenAI reportedly found new inference optimizations that more than halved the cost of running its models!
According to The Information, engineers told colleagues this month that the techniques helped power ChatGPT for visitors without free or paid accounts using only a couple hundred Nvidia GPUs at one point.
The exact method is unclear. It could involve quantization, KV caching, batching, routing simpler queries to cheaper models, or some mix of all of those.
The business angle is bigger than the technical detail: OpenAI ended Q1 with a 39% gross margin and wants to reach 52% by year-end. Lower inference costs give it room to either improve margins, raise ChatGPT usage limits, or cut API pricing pressure on developers.
OpenAI's moat is increasingly becoming inference and cost advantage, especially against Anthropic.
yet they can't cut costs for users in half
intelligence too cheap to meter has died long ago when they saw their first billion
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
wow did they discover speculative decoding or something? margins go up again!
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
The Information reports that OpenAI has cut inference costs by more than half on some existing models, while logged-out ChatGPT traffic ran on only a couple hundred Nvidia GPUs.
The obvious guesses include quantization, KV-cache changes, batching, speculative decoding, and routing easy queries cheaper.
If true, it will be a huge core competitive lever, lower cost can raise margins, expand usage limits, or reduce pressure on API pricing.
For some context, OpenAI’s adjusted gross margin fell to 33% in 2025 from 40% in 2024, after inference costs quadrupled.
Some reporting now puts Q1-2026 at 39%, with a 52% target by year-end.
Anthropic looks similar at roughly 44%, so frontier labs remain far below mature software economics.
---
theinformation .com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half
@AndrewCurran_ probably under a narrow-ish set of cirmcumstances or something, and then it gets reported like this
OpenAI has found a way to cut inference costs in half.
Um this seems big. @steph_palazzolo
https://www.theinformation.com/articles/openai-discovers-new-way-cut-inference-costs-half?utm_source=ti_app&rc=hwneun
https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half?rc=bfliih
OpenAI reportedly found new inference optimizations that more than halved the cost of running its models!
According to The Information, engineers told colleagues this month that the techniques helped power ChatGPT for visitors without free or paid accounts using only a couple hundred Nvidia GPUs at one point.
The exact method is unclear. It could involve quantization, KV caching, batching, routing simpler queries to cheaper models, or some mix of all of those.
The business angle is bigger than the technical detail: OpenAI ended Q1 with a 39% gross margin and wants to reach 52% by year-end. Lower inference costs give it room to either improve margins, raise ChatGPT usage limits, or cut API pricing pressure on developers.
OpenAI's moat is increasingly becoming inference and cost advantage, especially against Anthropic.
More on the optimization + what it could mean for OpenAI's gross margins or usage limits here:
https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half
Crazy how quickly better hardware has turned the AI business from lighting piles of money on fire, to one of the highest margin businesses on the planet.
This puts inferance margins at 75%
@natolambert Yes, need much more information.
@AndrewCurran_ probably under a narrow-ish set of cirmcumstances or something, and then it gets reported like this
OpenAI Halves Inference Costs???
What do you AI experts think about this latest development.. more incremental gains or something bigger?
Discuss over at @theinformation
https://www.theinformation.com/forum/posts/1205
@iruletheworldmo Looks like they figured it out a few weeks ago.
@AndrewCurran_ do you know if this is part of the reported price wars?
@steph_palazzolo ohh so that's why 5.5 became so catastrophically stupid
@steph_palazzolo worded a bit poorly >visitors who didn't have a free or paid account what kind of account type is left then? i'm guessing grant/research/trial? would be better to know certainly instead of inferring
@AndrewCurran_ do you know if this is part of the reported price wars?
OpenAI engineers figured out a way to more than halve the cost of inference, thanks to some newly-discovered optimizations.
Read more in today's AI Agenda: https://thein.fo/4gjQ42l
@AndrewCurran_ How we train and run AI is probably really really inefficient given the make it up as we went process.
Lot of gains to come on running the existing physical tech better / algorithmic improvements alone...
@AndrewCurran_ awfully convenient timing is all i'm going to say, not doubting OpenAI at all but this is a hilarious coincidence
what better way to cut inference costs in half than to double throughput
@RitsFur you mean like charging 30$ for a model that costs like 2$ to serve?
@steph_palazzolo >it's quantization again