Build1 publisher3 min readPublished
A boolean stream flag leaves callers guessing whether they get a string or a generator
A dev.to post lists six LLM integration mistakes with a code fix for each. Read the snippets and almost all of the failure sits in contracts the developer wrote, including a retry ceiling four attempts never reach.
The Engineer · Build desk

What happened
- A dev.to post lists six LLM integration mistakes that survive a notebook demo and break under real traffic, and gives a code fix for each one.
- Its first fix replaces a single function carrying an if-stream branch with two names, ask() returning the full text and ask_streaming() yielding chunks as they arrive.
- The post says a naive loop over a streaming response often leaves the underlying HTTP connection open when it throws mid-iteration, and prescribes a with-block around the stream.
- It warns that FAISS's IndexFlatIP approximates cosine similarity only on unit-normalized vectors, and that skipping normalization mis-ranks results with no error raised.
- The retry snippet scopes retries to RateLimitError, APITimeoutError and APIError across four attempts, then hands off to a secondary model on failure.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Keeping the boolean makes every new call site a place where a wrong guess about the return type fails silently, so the cheapest time to split the function is before it has callers.
- cost A single rate-limit event can spend eight requests against the quota you are already over, and the user waits through both backoff sequences before the error arrives.
- exposure An unnormalized index degrades ranking with nothing to alert on, so the first detector is a person reporting answers that look close but wrong.
The normalization item is the only one of the six that returns a wrong answer without raising anything [10]. Cosine similarity is the dot product of two vectors divided by their norms, so an inner-product index equals cosine only when both vectors already have length 1 [1]. FAISS's IndexFlatIP computes the inner product, and the post treats it as a cosine approximation that is valid only after you normalize [10]. The helper it ships divides by `np.clip(norms, 1e-10, None)` [11]. That clip matters: a zero-norm embedding divides by 1e-10 and comes out as zeros, so it scores zero against every query instead of turning the column into NaN [2].
The retry snippet is easy to check against its own numbers. `stop_after_attempt(4)` permits three sleeps, and `wait_exponential` doubling from the one-second minimum makes them 1, 2 and 4 seconds, or 7 seconds of waiting in total [12][3]. The `max=20` on the same line never engages inside four attempts [3]. `reraise=True` is the parameter that makes the next snippet work at all: with it, the final `RateLimitError` propagates unwrapped and the fallback's `except` clause catches it, while tenacity's default would wrap it in `RetryError` and the fallback would never run [5].
Both calls go through the decorated function [14]. A rate-limit event that hits the primary and the fallback therefore costs up to eight requests and about 14 seconds of sleep before the caller sees an error [4]. The post says pairing retries with a fallback is the same pattern most production LLM gateways use under the hood [15].
The history helper slices `self.history[-(self.max_history_turns * 2):]` [9] to avoid the orphaned assistant message that the post says some APIs reject outright [8]. Two times N is even, so the slice keeps pairs intact only while every entry in the list is exactly one user or one assistant message; a tool-result message in the same list puts the boundary back in the middle of a pair [6]. The cookbook these snippets come from includes a tool-calling template [16].
The first item is about a signature. Hiding streaming behind a flag means the call site cannot see whether it gets a string or a generator, and the post's complaint is that a wrong guess either fails silently or crashes deep in UI code [3]. Its fix is two names: `ask()` returns the full text, `ask_streaming()` yields chunks [4]. "Now the function signature is the documentation," the post says [2].
None of the six items is about the model's output [7]. Four of them are contracts in code the integrator writes. One is a numerical precondition and one is an availability policy [7]. The post does not report any measurements [8]. One claim does have to be checked locally before the fix is worth the refactor: whether an abandoned stream iteration leaks a socket depends on your SDK and its version, and the post says only that a naive implementation often does not close the connection [5].
What to watch
- Whether the cookbook's tool-calling template trims history with the same times-two slice, since a tool-result message breaks the pair assumption.
- Tenacity's wait_exponential base differs across versions, so the sleep sequence can change, though the 20-second cap still does not bind at four attempts.
- A socket count taken after an aborted stream iteration on your own SDK version would settle whether the leak the post describes still happens.