Retries and re-indexes: where LLM cost control stops working
Spring AI 2.0 moves schema enforcement to the provider, which trims input tokens on every healthy call. The retry path still bills the whole context again, in full.
The Engineer · Build desk
What happened
- A four-part cost-control series ends on the two spend paths that survive prompt trimming: failed responses and RAG indexing.
- The series puts a three-retry policy at up to four times the cost of a clean request.
- Spring AI 2.0 adds two per-call controls on entity mapping: provider-native structured output and schema validation.
Why it matters
- cost Schema text in the prompt is charged on healthy traffic, so the endpoints overpaying most are the highest-volume ones rather than the flakiest ones.
- constraint Provider enforcement stops at the type signature: a method returning a list gets no API-level guarantee on OpenAI, so those handlers stay on the retry bill.
- decision Teams now choose per provider: prevent at the API where the model supports it, and budget for four billed calls where it does not.
- exposure Embedding spend is set per token but paid per pass, so whoever can trigger a re-index - a parser change, a chunking change - controls the invoice.
The line item worth pricing first is not the failed call. Under the prompt-based route, the format instructions and the JSON schema are generated and appended to the prompt [7], which means they are billed on every call that works as well as on the ones that do not. Sending the schema to the provider as an API-level rule takes that block out of the input [9], and the input token count falls on every call [10]. That saving tracks request volume. The retry saving tracks defect rate, and the series' own example puts a 5% one-retry rate at roughly 5% extra input tokens with nothing to show for them [2]. On a busy endpoint with a low failure rate, the dull always-on saving is the larger of the two.
The failure path is expensive because of what it resends. A retry ships the system prompt, the conversation history, the RAG context and the tool schemas again, and the answer that could not be parsed is billed as output in full whether or not it was cut off [1]. With validation switched on, Spring AI appends the validation error to the user message and calls the model again, up to three attempts by default [13], so a single usable answer can reach four model calls [19] and up to four times the cost [3].
Prevention does not cover every method signature. Spring AI 2.0 supports provider-native structured output for OpenAI, Anthropic, Google GenAI, Mistral AI and, depending on the model, Ollama [11], but OpenAI rejects top-level array schemas, so asking for a List<T> fails, and Ollama models in reasoning mode can still return plain text [12]. Those endpoints keep the retry mechanism, which is why the series treats validation as a recovery step rather than the primary control [14].
The embedding side inverts the problem. At OpenAI's August 2026 list price, text-embedding-3-small is $0.02 per million input tokens [4], one fiftieth of the $1 per million input rate the series uses as its example rate elsewhere [5][18]. Per token, that is not a number anyone needs to manage. Per pass it is $20 for every billion tokens of corpus [17], and the pipeline pays it again whenever documents are added or updated, or a full re-index is required [15]. Dimensionality is the driver that keeps charging after the indexing run finishes, through storage and retrieval [16].
One gap to flag: the series says a fourth source of embedding calls has nothing to do with documents at all, and the text supplied to us stops in the pricing section before it is named [16][20].
What to watch
- Whether OpenAI lifts the rejection of top-level array schemas, which would close the main hole in provider-side enforcement.
- Whether Spring AI keeps three validation retries as the default once teams price the worst case.
- The fourth source of embedding calls the series names but the supplied text cuts off before identifying.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence36
- Adoption
- Insufficient
- Hype gap+12
- Incentives32
- Confidence44
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Each retry resends the request context - system prompt, conversation history, RAG context and tool schemas - so the input tokens are billed again, and the failed answer is billed as output in full, cut off or not.
- [2]
Example given: if 5% of requests require one retry, input token usage increases by roughly 5% without producing additional business value.
- [3]
A retry policy allowing three retries can make worst-case requests cost up to four times more.
- [4]
According to OpenAI's pricing as of August 2026, text-embedding-3-small costs $0.02 per million input tokens.
- [5]
The series uses list prices where a ratio matters and an example rate of $1 per million input tokens elsewhere.
- [6]
The fourth and final part of a dev.to series on LLM cost control covers controls for failed responses and for the indexing side of RAG, and closes with a map of all ten cost drivers.
- [7]
Spring AI's .entity() convenience method on ChatClient's CallResponseSpec traditionally used a prompt-based structured output approach: the StructuredOutputConverter generated format instructions including the JSON schema and added them to the prompt, and Spring AI parsed the returned JSON into the target Java type.
- [8]
Spring AI 2.0 adds two per-call controls through EntityParamSpec: .validateSchema() and .useProviderStructuredOutput().
- [9]
.useProviderStructuredOutput() sends the schema to the provider as an API-level rule instead of as prompt text, and the provider runtime enforces schema conformance for supported models, preventing invalid structured responses during generation.
- [10]
With provider-native structured output, the schema instructions disappear from the prompt, reducing input token usage on every call.
- [11]
Spring AI 2.0 supports provider-native structured output for providers with native capabilities including OpenAI, Anthropic, Google GenAI, Mistral AI, and Ollama depending on the model.
- [12]
Provider limitations apply: OpenAI rejects top-level array schemas, so requesting a List<T> fails, and Ollama models with reasoning mode may still return plain text.
- [13]
.validateSchema() adds StructuredOutputValidationAdvisor: the response is checked against the schema and, if validation fails, the validation error is appended to the user message and the model is called again, up to 3 retry attempts by default, each consuming tokens.
- [14]
The series advises using validation as a recovery mechanism rather than the primary strategy, combined with provider-native structured output where available to prevent failures before they happen.
- [15]
Embedding calls look cheap per request, but a RAG pipeline runs them across the entire document collection and may run them again whenever documents are added or updated, or a full re-index is required.
- [16]
Three drivers of embedding cost are named: the embedding model's price, vector dimensionality (which affects storage and retrieval costs), and how much text is embedded during each indexing run; a fourth source of embedding calls has nothing to do with documents.
- [17]
At $0.02 per million input tokens, one full embedding pass over a billion tokens of text costs $20.
- [18]
The text-embedding-3-small input rate is one fiftieth of the $1 per million input token example rate used elsewhere in the series.
- [19]
With the default of three validation retry attempts, one answer can involve up to four model calls, each billing the full input context.
- [20]
The supplied text of the article breaks off mid-sentence in the embedding pricing section, before the fourth source of embedding calls is named.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toSpring AI Retries and Embeddings: Failed Answers and Full Re-indexes — LLM Cost Control 4/4
1 article · August 25, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.