Skip to content

Build1 publisher3 min readPublished

A boolean stream flag leaves callers guessing whether they get a string or a generator

A dev.to post lists six LLM integration mistakes with a code fix for each. Read the snippets and almost all of the failure sits in contracts the developer wrote, including a retry ceiling four attempts never reach.

The Engineer · Build desk

Illustration accompanying A boolean stream flag leaves callers guessing whether they get a string or a generator

What happened

  • A dev.to post lists six LLM integration mistakes that survive a notebook demo and break under real traffic, and gives a code fix for each one.
  • Its first fix replaces a single function carrying an if-stream branch with two names, ask() returning the full text and ask_streaming() yielding chunks as they arrive.
  • The post says a naive loop over a streaming response often leaves the underlying HTTP connection open when it throws mid-iteration, and prescribes a with-block around the stream.
  • It warns that FAISS's IndexFlatIP approximates cosine similarity only on unit-normalized vectors, and that skipping normalization mis-ranks results with no error raised.
  • The retry snippet scopes retries to RateLimitError, APITimeoutError and APIError across four attempts, then hands off to a secondary model on failure.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Keeping the boolean makes every new call site a place where a wrong guess about the return type fails silently, so the cheapest time to split the function is before it has callers.
  • cost A single rate-limit event can spend eight requests against the quota you are already over, and the user waits through both backoff sequences before the error arrives.
  • exposure An unnormalized index degrades ranking with nothing to alert on, so the first detector is a person reporting answers that look close but wrong.

The normalization item is the only one of the six that returns a wrong answer without raising anything [10]. Cosine similarity is the dot product of two vectors divided by their norms, so an inner-product index equals cosine only when both vectors already have length 1 [1]. FAISS's IndexFlatIP computes the inner product, and the post treats it as a cosine approximation that is valid only after you normalize [10]. The helper it ships divides by `np.clip(norms, 1e-10, None)` [11]. That clip matters: a zero-norm embedding divides by 1e-10 and comes out as zeros, so it scores zero against every query instead of turning the column into NaN [2].

The retry snippet is easy to check against its own numbers. `stop_after_attempt(4)` permits three sleeps, and `wait_exponential` doubling from the one-second minimum makes them 1, 2 and 4 seconds, or 7 seconds of waiting in total [12][3]. The `max=20` on the same line never engages inside four attempts [3]. `reraise=True` is the parameter that makes the next snippet work at all: with it, the final `RateLimitError` propagates unwrapped and the fallback's `except` clause catches it, while tenacity's default would wrap it in `RetryError` and the fallback would never run [5].

Both calls go through the decorated function [14]. A rate-limit event that hits the primary and the fallback therefore costs up to eight requests and about 14 seconds of sleep before the caller sees an error [4]. The post says pairing retries with a fallback is the same pattern most production LLM gateways use under the hood [15].

The history helper slices `self.history[-(self.max_history_turns * 2):]` [9] to avoid the orphaned assistant message that the post says some APIs reject outright [8]. Two times N is even, so the slice keeps pairs intact only while every entry in the list is exactly one user or one assistant message; a tool-result message in the same list puts the boundary back in the middle of a pair [6]. The cookbook these snippets come from includes a tool-calling template [16].

The first item is about a signature. Hiding streaming behind a flag means the call site cannot see whether it gets a string or a generator, and the post's complaint is that a wrong guess either fails silently or crashes deep in UI code [3]. Its fix is two names: `ask()` returns the full text, `ask_streaming()` yields chunks [4]. "Now the function signature is the documentation," the post says [2].

None of the six items is about the model's output [7]. Four of them are contracts in code the integrator writes. One is a numerical precondition and one is an availability policy [7]. The post does not report any measurements [8]. One claim does have to be checked locally before the fix is worth the refactor: whether an abandoned stream iteration leaks a socket depends on your SDK and its version, and the post says only that a naive implementation often does not close the connection [5].

What to watch

  • Whether the cookbook's tool-calling template trims history with the same times-two slice, since a tool-result message breaks the pair assumption.
  • Tenacity's wait_exponential base differs across versions, so the sleep sequence can change, though the 20-second cap still does not bind at four attempts.
  • A socket count taken after an aborted stream iteration on your own SDK version would settle whether the leak the post describes still happens.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories