What an LLM feature actually costs to run

Short answer

Model tokens are usually the smallest line. The costs that decide viability are retries and failures, the context you resend on every call, the evaluation you need to change anything safely, and the human review a feature requires when it is wrong. Estimate all five before shipping, and instrument the feature so the real numbers replace the estimate quickly.

Every estimate for a language-model feature starts from the published price per million tokens. That number is real, knowable, and rarely the reason a feature is too expensive.

Here is what else is on the bill.

1. Tokens, and why the estimate is always low

The published price is per token, and the count is not what you think:

The system prompt is sent every call. A 2,000-token instruction set on a feature called ten thousand times a day is twenty million input tokens a day, before any user says anything.
Conversation history is resent. Turn ten costs roughly ten times turn one unless you truncate or summarise.
Retrieved context is usually the largest input. Five documents at a thousand tokens each dwarf the question.
Output is dearer than input, typically several times over, and output length is the thing your prompt controls least reliably.

Prompt caching matters more than any other optimisation here, because the invariant prefix is usually most of the input. Where a provider offers it, structuring the prompt so the stable part comes first is the single highest-leverage change available.

2. Retries, failures and the requests you pay for twice

Every failure that occurs after tokens have been generated is a paid failure:

Timeouts on long generations, where you pay and receive nothing.
Schema violations when the output must be structured and is not, so you retry.
Rate limits and the backoff behind them, which turn one logical request into several.
Guardrail rejections, where a second model call classifies the first.

A feature with a 10% retry rate costs 10% more before anything else, and retry rates on structured output are frequently higher than that until the schema is tightened.

3. Evaluation, which is the cost people forget entirely

You cannot change a prompt safely without a way to tell whether the change helped. That means an evaluation set, and running it is itself model calls — often thousands per change.

This is not optional overhead. Without it, every prompt change is a deployment with no test suite, and the first person to notice a regression is a user.

Budget for the evaluation runs, and for the engineering time to build the set. Both are ongoing, because the set has to grow as you find new failure modes.

4. The infrastructure around it

Vector storage and embedding, if you retrieve. Embedding is cheap per document and not cheap across a corpus you re-embed whenever the chunking changes.
Observability. You need the prompt, the response, the latency, the token counts and the outcome for every call, or you cannot debug anything. That is a lot of storage, and it contains user data with all the retention obligations that implies.
Streaming. Holding a connection open per user changes your server's concurrency profile, which is an infrastructure cost that does not appear on any AI invoice.

5. The human cost of being wrong

The one that decides viability more often than tokens.

If a feature is right 95% of the time and being wrong is expensive, someone reviews the output — and that review is the real cost of the feature. A summary nobody checks is cheap. A classification that routes a customer's money is not, and 95% means one in twenty needs a person.

Ask what happens when it is wrong before asking what it costs when it is right. The first answer usually determines the second.

Estimating before you build

Count the tokens by hand for one realistic call — system prompt, retrieved context, history, expected output.
Multiply by realistic volume, not the demo's volume, and add a retry factor of at least 1.15.
Add the evaluation runs you expect per month.
Cost the review, in people, at your actual error rate rather than the one you hope for.
Then compare against the alternative, which is often a much simpler system: a rules engine, a search index, or a person doing it now.

Instrument from the first day

Log token counts and cost per request alongside the outcome, and aggregate by feature, by tenant and by prompt version. Two things fall out of that immediately:

Which prompt version costs what, which turns prompt engineering into something you can evaluate on both axes rather than one.
Which users cost what, which is how you find the one integration sending your whole knowledge base on every call.

Both are far cheaper to have on day one than to add after the invoice arrives.

Frequently asked questions

Why is my token usage higher than estimated?

Usually the system prompt and retrieved context, both sent on every call, plus conversation history that grows each turn. The user's actual question is normally the smallest part of the input.

What is the highest-leverage cost optimisation?

Prompt caching where the provider offers it, because the invariant prefix is usually most of the input. Structuring the prompt so the stable part comes first is a small change with a large effect.

Why budget for evaluation?

Because without an evaluation set you cannot tell whether a prompt change helped, which makes every change a deployment with no tests. Running the set is itself model calls, often thousands per change.

What most often makes a feature unviable?

Human review. If being wrong is expensive and the feature is right 95% of the time, someone checks one in twenty outputs, and that labour usually exceeds every other line on the bill.