The public conversation about the cost of artificial intelligence has been a conversation about training: the capital, the chips, the energy, the scale of the facilities. That framing made sense while the frontier was the story and deployment was mostly pilots.
Inside companies that have shipped, the number that matters is the other one. Inference, the cost of actually running a model each time someone uses the feature, is recurring, scales with adoption, and behaves in ways that traditional software cost models do not prepare anyone for.
Success is the thing that raises the bill
Conventional software has near-zero marginal cost per use. A feature that becomes ten times more popular costs approximately what it cost before, which is why software companies are structured the way they are and why product teams have spent thirty years being told to drive engagement.
Model inference inverts that instinct. Every invocation consumes compute, and usage growth converts directly into cost growth. Teams that shipped a well-received feature have found themselves in the disorienting position of watching adoption metrics and unit economics move in opposite directions.
This has pushed cost into architecture decisions that were previously made on capability alone. Routing simple requests to smaller models and reserving frontier capacity for cases that need it is now standard practice rather than an optimization. Caching, batching, shortening context and truncating conversation history all matter in a way they did not when the same team was running a pilot with fifty users and an unmetered credit balance.
The forecasting problem is worse than the cost itself. Finance can accommodate an expensive line item it can predict. Inference spend depends on usage patterns, prompt lengths, retry behavior and how much context each request carries, and small changes in any of those move the total substantially. Companies describe budget variances that would be scandals in any other category, and they are mostly a measurement failure rather than a spending one, closely related to the general evaluation gap that leaves organizations unable to characterize their own systems.
The strategic consequence is a reopening of a question that looked settled. The infrastructure buildout assumed inference would run in the same rented environments as training. Steady, high-volume inference of open-weight models is a different profile, predictable and continuous, which is precisely the workload the repatriation analysis targets. Several organizations have run that comparison and reached conclusions their cloud vendors would prefer they had not.
Pricing the feature is the part product teams have handled worst. A capability whose marginal cost is real cannot be given away at unlimited volume inside a flat-rate plan, and several companies have had to introduce usage limits after launch, which is a considerably worse experience than having set them at the outset.
Prices per token continue to fall, and that is the argument for waiting the problem out. It is a weaker argument than it appears, because the same period has seen applications adopt reasoning approaches that consume far more tokens per task. Unit prices falling while units per task rise is not obviously a saving, and it is the arithmetic most budgets have failed to do.
Earlier Cranberry Journal coverage examined Model Deprecation Becomes an Enterprise Risk, the other recurring cost of building on someone else's model.
