Build1 distinct publisher3 min readUpdated
A dev.to post argues that recurring natural-language analysis confounds the metric with the instrument. The remedy is a frozen query, reviewed as a diff, with the model on either side of it.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A dev.to post argues that recurring natural-language analysis confounds the metric with the instrument. The remedy is a frozen query, reviewed as a diff, with the model on either side of it.
Every major assistant now ships some version of the scheduled prompt: you write "every Monday at 9am, analyze last week's signup funnel and tell me what changed," pick a cadence, and it runs [1]. A post on dev.to argues that this is fine for a news briefing and wrong for analysis, and that the fault lies not with the scheduler but with where the LLM sits relative to it [2]. The case is about measurement, not taste. Nobody schedules an analysis because they want a report: a one-off analysis tells you what the state of things is, while a recurring one tells you whether it is moving and in which direction [3]. That second question only works if the instrument holds still [4]. Regenerate the query from English on every run and a move in the number has two candidate causes, the world or the measurement, with no way to separate them [5]. That is confounding installed in the foundation of your monitoring [5]. The obvious rebuttal is that models are good at SQL now. The post's answer is that accuracy is not the issue and consistency is, because a model that writes correct SQL every single time can still write differently correct SQL each run [6]. Its worked example: run one counts distinct users with a LEFT JOIN to subscriptions over created_at >= '2026-08-01' and < '2026-09-01'; two weeks later the same prompt yields an INNER JOIN, so users with no subscription silently vanish, plus > and <= boundaries that shift the window by a day [7]. Neither query throws an error, both are defensible readings of the same English sentence, and the number moves by a few percent [8]. The author's recurring suspects are LEFT JOIN quietly becoming INNER JOIN, boundary operators flipping, NOT IN versus NOT EXISTS and their NULL handling, COUNT(*) versus COUNT(DISTINCT) on a fanned-out join, and UTC versus local day boundaries [9]. This is worse than an outright failure because it is silent, plausible, and unreconstructable: six months on, the query that produced March's number no longer exists anywhere [10]. The proposed fix is positional. Move the LLM from runtime to build time and split the work into five phases: explore, freeze, review, run, interpret [11]. Explore stays interactive and LLM-heavy, and nondeterminism there is a feature [14]. Freeze means the model emits SQL or a script that goes into version control, so the metric definition becomes an artifact with a name and a hash [12]. Review is the step that looks bureaucratic and, per the author, is the single most valuable one: a human reads the diff, which converts "the metric definition changed" from an invisible accident into an explicit, attributable, reviewable event, something a scheduled prompt cannot offer because the change happens inside a sampling distribution [13]. The run itself is deterministic code on a deterministic schedule, with the model returning downstream to interpret [18]. Two of the five phases use the model, and neither of them is the run [1]. The BI vendors got here first. The post points to dbt's Semantic Layer, Omni, Dremio and Cortex Analyst as sharing one thesis, that metric definitions must be codified so the LLM chooses which metric rather than how to compute it [15], with dbt's own framing being that fixed logic stops a model producing correct-looking numbers that differ subtly between runs [16]. What the author finds odd is that this argument is conducted in data engineering vocabulary about text-to-SQL and governance, while the "schedule an AI task" conversation happens in productivity blogs, and the two have not met [17]. The test to run on any recurring AI task you already operate is whether you can produce the exact query that generated last month's number [10]. If the answer is no, the time series is not a time series.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Worked example in the post: run 1 is SELECT COUNT(DISTINCT u.user_id) FROM users u LEFT JOIN subscriptions s ON s.user_id = u.user_id WHERE u.created_at >= '2026-08-01' AND u.created_at < '2026-09-01'; run 2, from the same prompt two weeks later, uses INNER JOIN (users with no subscription silently vanish) and WHERE u.created_at > '2026-08-01' AND u.created_at <= '2026-09-01' (window shifted by a day).
Neither of the two example queries is broken and neither throws an error; both are defensible readings of the same English sentence, and the number just moves by a few percent.
The proposed fix is to move the LLM from runtime to build time, splitting the work into five phases: explore, freeze, review, run, interpret.
In the five-phase split, the run phase is cron or another orchestrator executing deterministic code on a deterministic schedule, and the LLM returns in phase 5 to interpret, downstream.
The dev.to post argues that scheduled prompts are good for a news briefing but a mistake for analysis, and that the problem is not the scheduler but where the LLM sits relative to it.
People schedule analysis because they want to know what changed, not because they want a report: a one-off analysis answers "what is the state of things" while a recurring analysis answers "is the state of things moving, and in which direction."
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single-source argument with an illustrative example
One dev.to opinion post supplies the entire cluster. Its strongest artifact is a constructed pair of SQL queries plus an experience-based list of drift patterns; there is no evaluation, log, benchmark or second publisher. The reasoning about confounding is internally coherent and the phase prescription is specific, which lifts this above pure assertion, but the empirical base — how often regenerated queries actually diverge, and by how much — is absent, and the vendor precedent is paraphrased without citation.
No adoption data in supplied sources
The cluster contains no release, deployment, usage disclosure, benchmark or pricing evidence. The only adoption-adjacent statement — that every major assistant now ships scheduled prompts — names no product and cites nothing, and the proposed five-phase pattern is a recommendation with no reported implementations. Nothing here supports a measured adoption value.
Framing slightly outruns the evidence
The argument is modest in ambition — it prescribes ordinary version control and diff review rather than a breakthrough — and the author openly concedes that freezing the query does not make the pipeline safe, which suppresses the gap. What pushes it positive is rhetorical reach beyond what is shown: a universal ubiquity claim with no named products, 'textbook confounding' asserted on the strength of a constructed example, 'the single most valuable' step declared without comparison, and a settled-industry consensus attributed to four vendors without citation.
Practitioner thought-leadership post, no disclosed stake
The observable incentive is reputational: a first-person technical argument published on a developer platform where authorship builds standing, staking out a quotable rule ('the LLM may not define how values are computed'). The post discloses no employer, product or vendor relationship, and it recommends a generic pattern rather than a paid tool, so commercial pull is limited. It does, however, favorably position four named commercial semantic-layer offerings as validating the thesis, which is the one place a promotional current is visible.
Argument well characterized, real-world magnitude unknown
Confidence in what the post says is high — the text is explicit, the phases are enumerated and the SQL is printed. Confidence in the underlying world claims is low: one publisher, no corroboration, no frequency data on drift, no reported implementations of the remedy, unverified vendor attributions and a truncated body. The mechanism is plausible enough to act on cautiously, but the cluster cannot establish how material it is in practice.
build
launchd Tells You Nothing When a Job Dies, So Your Revenue Reports It Instead1 distinct publisher
build
Cost per customer is an allocation policy, not a warehouse project1 distinct publisher
build
The date bug that only misfires when the day is 13 or higher1 distinct publisher
build
Green is four claims wearing one badge, and nobody instruments the fourth1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026