Instrumentation used to be the cheap part. A team ran its application, emitted some logs, kept a dashboard, and the cost of watching the system was a rounding error against the cost of the system itself.
That relationship has inverted at a surprising number of companies. Engineering organisations are reporting observability bills that run to a double-digit percentage of total infrastructure spend, and in a few cases exceed what they pay to serve production traffic.
Nobody decided this; it accumulated
The mechanism is not mysterious once it is described. Every new service emits logs by default. Distributed tracing multiplies events per request rather than per deployment. Metric cardinality explodes the first time somebody adds a user identifier as a label. Retention defaults to a generous period because storage was cheap when the default was set, and nobody revisits a default that is not causing an outage.
Each of those decisions is individually reasonable and none of them is visible on its own. What is visible is the invoice, arriving quarterly, growing faster than headcount and faster than traffic.
The responses have a familiar shape. Sampling comes first, because it is a configuration change: keep every error trace, keep a fraction of the successful ones. Tiering comes next, moving older data to cheaper storage on the correct observation that almost nothing older than a fortnight is ever queried. The harder step is deciding which signals to stop collecting at all, and that is where the conversation gets uncomfortable, because the argument for keeping any individual stream is always the incident it might have explained.
Vendors have not helped their own case. Pricing that scales on ingested volume and unique metric series means the customer's bill rises precisely as their architecture becomes more granular — so a company doing the modernisation the vendor recommends is penalised for it. Several buyers now describe the model as misaligned in the same terms they use about other consumption pricing, and it is producing the same reaction that has pushed workloads back onto owned hardware.
It also fits a pattern buyers have been working through elsewhere. Once a cost becomes legible it becomes a procurement question, and procurement asks who owns the line and what the standard is — which is exactly the discipline that arrived for the software supply chain and is arriving here a year behind it.
The comparison engineering leaders reach for is the one they already had about model serving. Inference was the first cost that behaved this way — invisible while it was small, then suddenly structural, and requiring the same shift from an engineering preference to a line item somebody defends in a budget review. Teams that went through that argument once tend to recognise this one faster, and they arrive at the same answer: the spend is fine, the absence of an owner is not.
The organisations handling it best have stopped treating telemetry as free and started treating it as a product with a budget and an owner. That means a service team sees its own observability spend next to its compute spend, and the trade between them becomes a decision somebody makes rather than a default nobody set.
What remains unresolved is the incident that the sampled-away trace would have explained. Nobody can price that in advance, which is why the argument recurs after every outage and why the ratchet has historically only turned one way.
The honest position is that some of this spending is buying nothing. Dashboards nobody opens, alerts routed to channels nobody reads, retention on data whose only reader is the retention policy itself. Teams that have audited it usually find a third of the volume has no consumer at all — which is not a technology problem and never was.
Topics technologycloudcost
