Loading today’s stories
Build1 distinct publisher3 min readUpdated
A developer's design note on OpenThesis argues that unreliable AI company research is an architecture problem, not a prompting one, and splits it into failure modes with different fixes.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The developer behind OpenThesis has published a design note on dev.to explaining why company-research workflows built around one large model call keep producing numbers that read well and do not hold, and has released the system as an Apache-2.0 desktop tool for evidence-first research [1][2]. The useful contribution is not the tool, it is the taxonomy: the post insists that several distinct failure modes are being treated as a single prompting problem [3].
The modes it names are genuinely separate. A model can mix fiscal periods, accounting scopes, or currencies; missing values can quietly become zeros; a calculation that is deterministic can be performed probabilistically; and a citation can point at a real filing that does not actually support the claim attached to it [4]. None of those is fixed by better instructions, because each one has a different mechanism.
The diagnosis is structural. In the naive pipeline, a single model call is implicitly responsible for remembering reported values, selecting the right fiscal period, recognising the accounting scope, finding sources, performing calculations, comparing scenarios, identifying risks, and writing the conclusion [5]. Per the author, qualitative reasoning, connecting evidence, forming scenarios, and challenging assumptions are reasonable uses of a language model, while recalling an exact reported value, deciding whether a value is missing, and computing a margin or valuation are poor places to accept probabilistic behaviour [6]. The stated rule: deterministic work stays deterministic, and the model is not the database or the calculator underneath the reasoning [7].
Implementation follows the rule. Ingestion starts from official filings rather than model memory or general web search, currently SEC EDGAR for the US, CNInfo for SSE, SZSE and BSE companies, and HKEXnews for Main Board and GEM issuers [8]. The pipeline is staged from filings through evidence extraction, validated financial facts, deterministic finance, specialist agents, synthesis, verification, and a traceable thesis [9]. Extracted evidence is assigned IDs, agents receive a bounded evidence set and are expected to attach those IDs to factual claims, so unknown references and unsupported claims can be flagged rather than accepted because the prose sounds plausible [10]. A final verification step can mark a run partial when required evidence coverage is absent [11].
The arithmetic gets the same treatment. Financial summaries and reverse-DCF calculations are written in ordinary code, missing values are not silently converted to zero, and the pipeline fails closed when core coverage is insufficient instead of handing an incomplete numeric picture to a model [12]. The example given is revenue growth with one period missing: a fluent model will still emit a percentage because completing patterns is what it does, while a deterministic function returns "not available", preserves the reason, and keeps the gap from contaminating downstream work [13].
There are eight declared roles, covering financial analysis, business analysis, accounting risk, growth, skepticism, forecasting, synthesis and verification [14][15]. The author is explicit that agent count is not the claim, since eight agents holding eight inconsistent versions of the facts would produce more confidence without more reliability; what matters is that the roles operate over the same research pack and evidence model [16][17]. The post also concedes that none of this removes extraction errors or resolves ambiguous filings, only that it makes the failure boundary inspectable [18], and it presents no measured error rate or benchmark [19]. It is not a stock picker or a trading bot [20].
Worth watching: whether partial runs stay labelled partial once someone wants a finished report, and whether extraction accuracy holds equally across three filing regimes with very different formats. Coverage and error statistics per jurisdiction would tell you more than the architecture diagram does.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The author of a dev.to post titled "I stopped letting LLMs guess financial facts" describes building OpenThesis after finding that LLM reasoning about companies could be useful while the financial facts underneath it were much harder to trust.
OpenThesis is described as an Apache-2.0 desktop system for evidence-first, AI-assisted company research.
The author states that the observed failures are different failure modes, and that treating all of them as one giant prompting problem did not feel like a reliable architecture.
The named failure modes are: mixing fiscal periods, accounting scopes or currencies; missing values quietly becoming zeros; a deterministic calculation being performed probabilistically; and a citation pointing to a real filing without actually supporting the claim.
In the common company-question-to-LLM-to-answer workflow, a single model call is implicitly responsible for remembering reported values, selecting the right fiscal period, recognizing the accounting scope, finding sources, performing calculations, comparing scenarios, identifying risks, and writing a conclusion.
The author argues qualitative reasoning, connecting evidence, forming scenarios and challenging an assumption are reasonable uses of a language model, while remembering an exact reported value, deciding whether a value is missing, and calculating a margin or valuation are poor places to accept probabilistic behavior.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but wholly self-reported
The architecture is described with unusual specificity — named ingestion sources, an eight-stage pipeline, evidence-ID binding, fail-closed deterministic finance, eight declared roles — but all of it comes from one post written by the project's author, and the supplied text contains no benchmark, error rate, external audit or reproduction. Evidence is sufficient to describe the design, not to substantiate the reliability claim.
No usage signal supplied
The only adoption-adjacent datapoint is the author's own mention of v1.3.0 behaviour and available model adapters. There are no downloads, stars, installs, users, contributors, deployments or third-party references in the supplied material, so adoption cannot be scored without inference.
Modestly ahead of verification
The framing is comparatively disciplined: the author explicitly denies that agent count is the architectural claim, concedes extraction errors and ambiguous filings persist, and disclaims stock-picking use. The residual overstatement is that trustworthiness gains from determinism, evidence IDs and verification are asserted architecturally with no measurement of whether flagged-unsupported-claim detection or fail-closed thresholds actually reduce error in practice.
Author promoting own open-source project
The post is a first-person build note by the developer of the software it describes, published on a developer platform where such posts attract users and contributors — a clear promotional incentive. It is tempered by the Apache-2.0 licence, absence of any disclosed paid tier, funding or commercial offering in the supplied text, and by self-imposed caveats about remaining failure modes.
Design well described, effects unproven
High confidence that the described architecture and design intent are reported accurately, since the author is the primary source for their own system. Low confidence in the reliability outcome: one publisher, one self-interested source, no adoption data and no evaluation, so nothing external constrains the claims.
invest
Touchmark opens a forwards market for tokens because finance cannot forecast them1 distinct publisher
invest
Korea's 720-billion-dollar AI plan meets its first critic: the man running its research hub1 distinct publisher
build
Count invalid JSON as a failed classification, and model choice becomes a reliability problem1 distinct publisher
product
Incogni ranks 13 AI assistants by privacy risk: bigger is worse, except ChatGPT1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 16, 2026