Swapping LLMs barely moved Orient Software's document extraction quality

The evidence here is one practitioner's account of one project. Nguyen is CTO of Orient Software and co-founded Neurond AI, firms that sell software development, AI and data science services ^[1]. Orient Software sells the integration work this argument locates the hard part in, and integration work is billable. His own experiment cuts the other way on cost: the model swap was the cheap thing to try, and he reports it bought almost nothing ^[6]. The column does not publish the extraction scores, so "surprisingly small" is his characterisation ^[20].

Same prompt, same use case, different LLMs ^[6]. That design varies the model and holds everything else fixed, so it tells you what a model change is worth inside one existing pipeline, and leaves the worth of the rebuild unmeasured. The preprocessing, context and validation changes came afterwards, with no controlled comparison against the old configuration ^[8][18].

"Give the same model different tools, memory, context and guardrails, and you get dramatically different results," Nguyen wrote ^[5]. The document project tested the first clause and reported the answer. The second clause rests on the client work he describes.

Nguyen lists seven layers around the model: context management, memory, tools and integrations, orchestration, business data, verification, and security and permissions ^[10]. Three of those are defined by the customer. Business data means your pricing rules, policies and customer records, which he says no general-purpose model knows ^[13]. Verification means checking the work instead of trusting the first answer ^[10]. Permissions decide what an agent may do alone, and, as he puts it, "Reading a document is very different from deleting a file" ^[12]. Business data, verification and permissions all sit outside a model vendor's pricing sheet ^[19].

His forward case is about swap cost. If an application is tightly coupled to one model, he writes, every release cycle may become an expensive reengineering project, while a well-designed harness lets business logic, data pipelines, integrations and evaluations stay intact as the model underneath changes ^[15]. He describes a Claude Fable 5 generation built for long-horizon autonomous work, Gemini 3.7 Flash with adjustable reasoning levels and a 1-million-token context window, and DeepSeek as an open-weight family of V4 Pro, Flash and a vision variant ^[16]. "Once the system was designed properly, the LLM became a component we could swap, not the foundation everything depended on," he wrote ^[17].

The six-month stall is the line most likely to be quoted from this column: "I've seen teams pick the best model on paper and stall for six months," Nguyen wrote ^[3], with no team count and no period attached ^[20]. A buyer can run the cheaper half of his experiment in-house in days, holding the current pipeline and prompt and putting a second model behind them ^[6]. If output quality barely moves, the benchmark comparison has answered itself, and the argument moves to who is going to own the OCR, the validation and the approval rules ^[7].