Putting an LLM feature into production: retrieval, evaluation and cost
Most LLM features fail on retrieval, not on model choice. What decides whether one ships: passages good enough to answer, an evaluation set you own, a token and latency budget, and a fallback path.
Contents
- Key takeaways
- Retrieval decides the answer, not the model
- Where retrieval actually fails
- Build the evaluation set before you build the feature
- What public leaderboards can and cannot tell you
- Budgeting tokens before budgeting money
- Latency is a product decision, not a side effect
- The fallback path is part of the feature
- What changed in this update
- FAQ
Updated 8 July 2025 — added the 2024 and 2025 price and caching changes that reset the cost model.
Most LLM features fail on retrieval, not on model choice. Shipping one takes four things: passages good enough to answer the question, an evaluation set you own, and a token and latency budget written before launch. The fourth is a defined path for when the model has nothing useful to say.
Key takeaways
- Patrick Lewis and co-authors introduced retrieval-augmented generation in a paper submitted on 22 May 2020. It pairs a pre-trained generator with a dense vector index, so knowledge updates without retraining.
- Nelson Liu and co-authors reported in July 2023 that accuracy peaks when the relevant passage sits at the start or the end of the context. It falls when the passage sits in the middle.
- Jiawei Chen and co-authors benchmarked four retrieval abilities in September 2023. Models struggled with negative rejection, information integration and false information.
- OpenAI priced GPT-4 Turbo at $0.01 USD per 1,000 input tokens on 6 November 2023 and gpt-3.5-turbo-0125 at $0.0005 USD per 1,000 input tokens on 25 January 2024. That is a twentyfold gap on the same workload.
- MTEB spans 8 embedding tasks, 58 datasets and 112 languages, and its authors found that no single embedding method dominates across all of them.
Retrieval decides the answer, not the model
Model selection takes most of the meeting time and changes least of the output. If the passage holding the answer never reaches the prompt, no model recovers it. The prompt is a container; retrieval decides what goes inside.
Patrick Lewis and co-authors introduced retrieval-augmented generation in a paper submitted to arXiv on 22 May 2020. It combines a pre-trained sequence-to-sequence model with a dense vector index, and produced more specific and more factual output than a parametric-only baseline. (arXiv:2005.11401)
So the real work sits in the corpus: how documents are chunked, what metadata travels with each chunk, how stale content is retired, and which permissions apply. That is a content architecture problem before it is a machine learning one, which is why a content model built around meaning pays for itself twice.
Where retrieval actually fails
Four failure modes account for most bad answers, and each has a different fix.
- Chunk size. Chunks too small lose the context that makes them interpretable; chunks too large dilute the embedding and waste tokens.
- Vocabulary mismatch. The user writes “invoice not received” and the document says “billing dispatch failure”. Pure vector search misses it, so a hybrid of keyword and vector search recovers the case.
- Position. The right chunk is retrieved and then buried in the middle of a long prompt.
- Absence. Nothing relevant exists, and the system answers anyway.
Nelson Liu and co-authors reported on 6 July 2023 that model accuracy is highest when relevant information sits at the beginning or the end of the input context. It degrades significantly when the model must find that information in the middle. (arXiv:2307.03172)
The last mode is the dangerous one. Jiawei Chen and co-authors benchmarked models on 4 September 2023 across noise robustness, negative rejection, information integration and counterfactual robustness. Models handled noise reasonably. They struggled to refuse, to combine facts across passages, and to resist false information already in the context.
Build the evaluation set before you build the feature
An evaluation set is 100 to 200 real questions. Each one carries the expected answer, the passage that supports it, and a label for cases where refusal is the correct behavior. It takes about two days to assemble and it outlives every model you will use.
Run it on every change: new chunking strategy, new embedding model, new prompt, new provider. Report retrieval quality and answer quality separately, because a good answer from a bad passage is luck.
Shahul Es and co-authors published Ragas on 26 September 2023, a framework of reference-free metrics for retrieval-augmented pipelines. It scores whether retrieved context is relevant and focused, and whether the generated answer stays faithful to that context, without ground-truth annotations. (arXiv:2309.15217)
Automated metrics narrow the field; they do not settle it. Keep a human review of a fixed sample every release, and keep the sample stable so results stay comparable.
What public leaderboards can and cannot tell you
LMSYS published Chatbot Arena on 3 May 2023, ranking models through anonymous, randomized pairwise battles scored with Elo ratings. It measures aggregate human preference on open-ended prompts. It does not measure your task, on your corpus, under your latency budget, which is the gap most of the first wave of generative AI deployments ran into.
Vendor evaluations have the same limit. Anthropic stated on 21 November 2023 that Claude 2.1 showed a twofold decrease in false statements against Claude 2.0, alongside a 200,000-token context window. That is a useful directional signal from the vendor's own testing, not a result on your data.
The embedding side is clearer still. The MTEB authors, writing in October 2022, found no single method dominating 8 tasks across 58 datasets and 112 languages. Pick the embedding model by task and language, then confirm it on your own retrieval set.
Budgeting tokens before budgeting money
Cost is arithmetic: input tokens times input price, plus output tokens times output price, times requests. Retrieved context dominates the input side, so retrieval design is also cost design.
OpenAI priced GPT-4 Turbo on 6 November 2023 at $0.01 USD per 1,000 input tokens and $0.03 USD per 1,000 output tokens, with a 128,000-token context window. On 25 January 2024, gpt-3.5-turbo-0125 arrived at $0.0005 USD and $0.0015 USD. (OpenAI DevDay, OpenAI)
Take a support assistant that retrieves six passages of 500 tokens, adds a 400-token system prompt and returns a 300-token answer. That is 3,400 input tokens and 300 output tokens per request. At the GPT-4 Turbo prices above it costs $0.043 USD per request, or $430 USD across 10,000 requests. At gpt-3.5-turbo-0125 prices the same traffic costs $21.50 USD.
Indexing barely registers by comparison. OpenAI priced text-embedding-3-small at $0.00002 USD per 1,000 tokens on 25 January 2024, so embedding a ten-million-token corpus costs about $0.20 USD. The expensive part is answering, and the lever is how many tokens each answer carries.
Latency is a product decision, not a side effect
Every request pays for retrieval, then generation, and generation scales with output length. Set the budget as a percentile, not an average. A p95 under three seconds is a target a team can engineer against; a mean of 1.8 seconds hides the requests people complain about.
Three levers move it. Stream the response so the first token arrives early. Cut the retrieved context to what evaluation shows you need. Route the cheap deterministic steps, such as classification or routing, to a smaller model or to code.
The fallback path is part of the feature
Because models are weak at refusing, refusal has to be engineered around them. Set a retrieval score threshold below which the feature does not call the model at all. Write the refusal copy as a product decision. Route the user to a human with the conversation attached.
Then log every refusal and every human handoff, and feed them back into the evaluation set. Those logs are the only honest measure of coverage, and they belong in the same review as the rest of the metrics a board actually reads.
This is also the point where an LLM feature stops being a model problem and becomes an operations problem with a named owner. Someone has to own the corpus, the threshold and the escalation queue after launch.
What changed in this update
Prices moved and caching arrived. OpenAI introduced prompt caching on 1 October 2024, with a 50% discount on cached input tokens. It then priced GPT-4.1 on 14 April 2025 at $2.00 USD per million input tokens and $8.00 USD per million output tokens. Cached input costs $0.50 USD per million, and the context window reaches one million tokens. Against GPT-4 Turbo's $10.00 USD per million input tokens in November 2023, input cost fell by 80% in seventeen months. The engineering order is unchanged, and the build, buy or wrap decision still turns on retrieval quality rather than price.
FAQ
What is retrieval-augmented generation?
It is a pattern that retrieves relevant passages from your own corpus and places them in the prompt before the model answers. Patrick Lewis and co-authors described it in May 2020. The point is that knowledge can be updated by editing documents rather than by retraining a model.
Which model should we pick for a RAG feature?
Pick the cheapest model that passes your evaluation set, then re-run the set when a new one appears. Model choice moves output quality less than chunking, hybrid search and passage ordering. Leaderboards rank general preference, not performance on your corpus.
How big should an evaluation set be?
Between 100 and 200 real questions is enough to detect regressions in a first production feature. Each entry needs the expected answer, the supporting passage and a label for questions that should be refused. Grow it from production logs rather than from imagination.
How do we estimate the monthly cost of an LLM feature?
Multiply average input tokens by the input price, add average output tokens times the output price, then multiply by expected monthly requests. Retrieved context usually dominates. Cutting from eight passages to four halves most of the bill without changing the model.
What should happen when the model does not know the answer?
The feature should refuse and hand over to a person. Set a retrieval score threshold below which no model call happens, write the refusal text deliberately, and attach the conversation to the handoff. Every refusal then becomes a new test case.
Build in this order: corpus, evaluation set, retrieval pipeline, model, prompt. Write the token and latency budgets before launch, because both become political once the feature is live. Six months from now the number worth reporting is not a benchmark position. It is the share of real production questions the system answers correctly on the first attempt, with the refusal rate beside it.
