Inference Is Cost of Goods Sold: Gross Margin for a Product With AI Inside

Software became the most valuable kind of business for one boring reason: the second copy is almost free. Once the product is built, serving one more customer costs a little hosting, a little support and a payment fee, and gross margins of 75 to 85 percent follow. Investors pay for that margin, founders plan around it, and most of the financial habits of a software startup quietly assume it.

A product with a language model inside it breaks that assumption. Every time a customer asks the product to summarise, draft, classify or answer, somebody pays for the computation, and the bill scales with use rather than with headcount. The marginal cost is back. It is usually smaller than founders fear and larger than their pricing assumes, and it is almost always less visible than it should be, because it arrives as one line on a cloud or API invoice that nobody has broken down.

This post is about putting that cost where it belongs, measuring it in a unit that means something, and making sure the price of the product knows about it. None of it requires a new tool. It requires deciding a few definitions early and reading one sheet every month.

Where inference sits on the P&L

The first decision is accounting, and it is the one most often made by default. When the model provider's invoice arrives, it tends to land in whatever account the engineering team's other software lands in: tools, R&D, sometimes general overhead. That is wrong for most of the money, and the error flatters the company in exactly the place an investor will look.

The rule I use is simple. Inference that runs because a customer used the product is a cost of revenue, alongside hosting and the payment processor, and it reduces gross margin. Inference that runs because the team is building the product (prompt experiments, evaluation runs against a test set, fine-tuning, a new feature that has not shipped) is research and development. Inference the team uses for its own work, such as a coding assistant or an internal chat tool, is an ordinary operating expense, no different from any other subscription.

The rule is easy; applying it is the work. A single API key used for production, staging and a developer's laptop produces one invoice that cannot be split after the fact. The fix is to separate keys or projects by purpose from the first week, and to tag each production call with the feature that made it. That is the same discipline as keeping raw data that is never edited: if the split is not captured when the cost is incurred, no spreadsheet will recover it later.

Getting this right usually makes the reported gross margin worse, sometimes by several points. That is the point. A gross margin that excludes the cost of serving customers is not a margin, it is a forecast of trouble, and the first serious investor to open the data room will reclassify it anyway, with less goodwill than if you had done it yourself.

The unit is the task, not the token

Model providers price in tokens, so teams track tokens. Tokens are the right unit for the invoice and the wrong unit for the business. A customer does not buy tokens; they buy a summary of a contract, a categorised expense, an answer to a question. The number that connects cost to value is the cost per completed task.

The gap between cost per call and cost per task is where most surprises come from. A single task can involve several calls: one to retrieve context, one to draft, one to check the draft, and a retry when the output fails validation. Retrieval quietly grows the input, because each new document a feature can see is more context sent with every request. An agent-style feature that plans, acts and checks its own work in a loop can make ten or twenty calls for what the user experiences as one click. And tasks that fail still cost money; if a fifth of attempts are discarded, the cost of each successful one is a quarter higher than the cost of the average attempt.

So the measurement that matters is total production inference for a feature in a month, divided by the number of tasks that feature completed successfully. It takes an hour to set up if calls are already tagged by feature, and it immediately answers the questions a cost per token never will: which feature is expensive, whether it is getting cheaper, and what it costs to serve a customer who uses it heavily.

The average user hides the margin

Here is a worked example with round numbers. A B2B product sells at $50 per seat per month. Before the AI feature, serving a seat costs about $3 in hosting, $1.50 in payment fees and $4 in support, so gross margin is roughly 83 percent. The team adds an assistant that costs two cents per completed task, and the average seat runs 300 tasks a month. That is $6 of inference per seat, and gross margin falls to about 71 percent. Lower, but still a software business, and the board deck says so.

The average is the problem. Usage of AI features is rarely spread evenly. In a distribution like this one, the median seat might run 80 tasks a month and cost $1.60, while the most active tenth of seats run 2,000 tasks and cost $40 each. For those seats, the full cost to serve is $48.50 against a $50 price: a gross margin of about 3 percent. The blended number looks healthy because the light users are subsidising the heavy ones.

Flat pricing on a product with a marginal cost is a bet that your best customers will not use it very much. Growth is the process of losing that bet.

That matters for two reasons. First, the heavy users are often the most valuable accounts, the ones that expand, renew and refer, so growth tends to come disproportionately from the segment with the worst margin. Second, the mix shifts with time. As a feature gets better and users trust it more, they use it more, and the average slides towards the expensive tail. A margin that was 71 percent at launch can be 60 percent a year later without anyone having changed a price or a model.

The habit that prevents the surprise is to report gross margin by usage decile, not only in aggregate. If the top decile is below zero or close to it, the pricing needs work before growth makes it urgent. This is the same instinct as separating bookings, billings, revenue and cash: one blended figure is comfortable, and the decision lives in the breakdown.

Pricing has to know the cost curve

There are only a few ways to price a product with a real marginal cost, and each one puts the risk somewhere different. A flat seat price keeps the risk with the company; it is simple to sell and works when usage per seat is predictable or cheap. Pure usage pricing moves the risk to the customer; margins are protected, but buyers dislike bills they cannot forecast, and finance teams on the customer side will push back. Between them sit the arrangements most products end up with: a seat price that includes a generous allowance with a published fair-use limit, credit packs on top, or a higher tier for customers who use the feature heavily.

The choice is commercial, but it should be made with the cost curve on the table. A practical test: price the plan so that the 90th-percentile user is still above your target gross margin, and let the allowance absorb everyone below them without friction. If the 90th percentile cannot be served profitably at any price customers will accept, the issue is the cost per task, not the price. When I priced subscriptions for my own iOS apps, the same logic applied in a smaller form: whether costs recur decides whether the price should recur, and a cost that scales with use eventually needs a price that notices.

One trap is worth naming. Model prices have fallen sharply and will likely keep falling, and founders sometimes treat that as the plan: the margin will fix itself. Sometimes it does. More often, cheaper inference is spent on longer context, more calls per task and more ambitious features, so the cost per task stays flat while the product improves. Falling prices are a tailwind to take when it comes, not an assumption to build the runway on.

The levers, roughly in order

Once cost per task is measured, the ways to reduce it are well understood, and the order matters because the early ones are cheap and the later ones carry quality risk.

Trim the context. Input tokens usually dominate the bill, and much of what is sent is there because it was easy to include rather than because the task needs it. Sending the three relevant passages instead of the whole document is often the single largest saving, and it frequently improves quality as well.

Cache what repeats. Many requests share a long, identical prefix: the instructions, the schema, the reference material. Most providers now discount repeated prefixes heavily, and identical questions can be answered from a stored result without calling the model at all.

Route by difficulty. Not every task needs the largest model. Classification, extraction and short rewrites often run perfectly well on a smaller, much cheaper one, with the larger model reserved for the tasks that need it. This only works with an evaluation set that proves the smaller model is good enough for each route, which is one more reason the golden test set is production infrastructure rather than a nice-to-have.

Move work off the critical path. Tasks the user does not need answered in seconds, such as overnight summaries or bulk categorisation, can run through batch interfaces that are priced well below interactive calls.

Move work onto the device. For some features, especially on mobile, the cheapest inference is the one the customer's own hardware runs. That turns a variable cost into roughly zero, with its own constraints, which I have written about in latency, memory and heat terms.

Each lever should be judged on the same sheet: cost per completed task before and after, and the evaluation score before and after. A saving that costs quality is a price cut nobody approved.

What a board and investors will ask

Investors in AI products have learned to ask about margin early, and the questions are predictable. What is gross margin with inference included, and how has it moved over the last six quarters? What does it cost to serve your heaviest customers, and are they profitable? How much of your margin improvement came from your own work and how much from provider price cuts?

A company that tracks the sheet above can answer each of those in a sentence. A company that does not will answer them from a weekend of reconstruction, and the reconstruction will be less convincing than the numbers it replaces. The same figures belong in the monthly investor update: gross margin and cost per task, with the same definitions every month, so the trend is visible before anyone has to ask for it.

Margin is a design decision

The underlying shift is that in an AI product, gross margin is decided partly in the product and engineering meetings, not only in the pricing meeting. How much context a feature sends, how many calls a workflow makes, whether a cheaper model is good enough, whether a task runs interactively or overnight: each of these is a margin decision made by someone who may never look at the P&L.

The finance job is to make those decisions visible to the people making them. Put inference in the right line, measure it per completed task, look at the tail rather than the average, price for the 90th percentile, and read one sheet a month. That costs a few hours to set up and an hour a month to keep, and it means the runway model is built on a margin that will still be true when usage doubles.

If you are building a product with AI inside it and want help classifying the costs, setting up cost-per-task reporting or testing your pricing against your usage data, that is the kind of work I do as a fractional CFO, and I am glad to help.