Loading today’s stories
Build1 distinct publisher3 min readPublished
Mithil Vakde's from-scratch transformer landed one point behind TRM on the public eval. His own ablations put 20 of those 44 points on two representation choices rather than on any amount of compute.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Start with the two pieces the ablation table singles out. 3D RoPE encodes where a token sits in a grid and which grid it belongs to, and per-task embeddings tell the model which puzzle it is currently on [7]. Remove either one and 44% becomes roughly 24% [10]. Each removal costs 20 points on its own [2]. The capability lives in how the puzzle is presented to an 8-layer transformer [7], not in how many GPU-hours went past it.
The change that actually moved the number is smaller and less explained. Earlier runs trained on input and output tokens; this one trains on output tokens only, which makes the method supervised, and the score went from 40% to 44% [9]. Training loss got worse while the eval score rose, and Vakde says he does not fully understand why, guessing finite model capacity [9]. Reporting an unexplained delta as unexplained is the part I would like to see in more benchmark posts.
The 67 cents covers lifetime compute: initialization through inference on every task, which is only coherent because all tasks train at once [15]. Divide it by 1.5 hours and the implied rental is about 45 cents per hour [1]. Vakde's own footnote is the load-bearing one. Dividing the total by task count would misprice it against systems billed per task, which is why he now compares only with TRM, HRM and CompressARC [15]. Published LLM cost figures leave out pretraining entirely [16], so a cross-family price ranking is not something this data can produce.
For the number to transfer, several things have to hold in your setting. The tasks have to arrive as a closed set you can train over simultaneously, because that is what makes the amortization legal [15]. The inputs have to be small discrete grids of the kind ARC ships, tokenized as sequences [5]. You have to tolerate 1.5 hours of training before the first answer [1]. And the eval has to be clean, which the write-up concedes is not true for LLM scores here, since ARC answers circulate online and end up in training corpora [20]. That caveat also softens the comparison band that put this run ahead of DeepSeek R1, o3-mini and Gemini 2.5 Pro when the TRM paper ran it [3].
Headroom is measurable rather than asserted. Restricted to ARC-1 plus ConceptARC the score is about 40%, so the extra training data is worth 4 points [13][4]. A CompressARC-style unsupervised variant scores about 18%, so the labels are worth 26 [13][3]. A union of solved tasks across his own runs already reaches 55%, and he guesses 65% is attainable, which leaves 10 points nobody has shown [14][5]. Against that, 56% of ARC-1 still fails, ARC-2 sits at 7% [18], and o1's 75% on ARC-1 came from massive post-training and remains 31 points ahead [19][6].
Provenance matters for how much weight to put on this. An earlier post in the series went viral and was picked apart in public by Lucas Beyer, Jeremy Howard and Rohan Anil [4]; the new appendix answers those objections, but independent replication is still thin and this is one researcher's unreviewed work [21]. The write-up also closes by promoting vecq, a training-free quantization library whose 4-bit and 5-bit codes come from fixed Lloyd-Max centroids [22]; the shared property is that neither pays a training pass, which is where the analogy stops. Vakde's parameter critique travels further than his score: he argues TRM's advertised 7M omits O(100M+) trained embedding weights and should read "7M active weights" [17]. At 45 cents an hour [1], rerunning his ablation table costs less than arguing about his headline.
Ranked by verification strength, evidence, and original report placement.
Mithil Vakde trained a small transformer from scratch in 1.5 hours on a rented RTX 5090, with compute cost of about 67 cents.
The model scored 44% on the ARC-AGI-1 public eval, one point behind TRM's 45%.
The 44% score sits in the same band that left DeepSeek R1, o3-mini and Gemini 2.5 Pro behind when the TRM paper (arXiv 2510.04871) ran the comparison.
This is the third post in Vakde's ARC series; the previous result went viral and drew public scrutiny from researchers including Lucas Beyer, Jeremy Howard and Rohan Anil.
Each puzzle becomes a sequence of tokenized grids, and the transformer trains on those sequences from scratch at test time using both the train puzzles and the eval puzzle inputs, with the eval labels hidden.
ARC is a meta-learning benchmark built around skill acquisition, and Chollet's original paper (arXiv 1911.01547) frames it as a measure of how efficiently a system gains new skills.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 1, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
product
Salesforce's researchers get better CRM agents by writing the procedure into the prompt1 distinct publisher
build
The judge went synthetic first, which tells you which part of your pipeline is next1 distinct publisher
invest
Google says frontier models already know the facts they get wrong. That is a budget decision.1 distinct publisher
leadership
Re-baseline AI procurement on cost per completed task, not dollars per million tokens1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One unreviewed account, unusually candid
Every figure in this story — 67 cents, 44%, the 24% ablation floors — traces to a single researcher's blog post as relayed by dev.to, which crossposts it from its own site. What lifts it above a press release is how much of it cuts against the author: the ablations expose how brittle the score is, the restricted-data run gives back four points, the unsupervised variant collapses to 18%, and ARC-2 sits at 7%. Detailed and self-incriminating is not the same as verified, and this reporting concedes nobody has rerun it.
Scoreboard entries, no takers
Adoption at this stage means benchmark runs and nothing else: one public-eval score, a set of ablations and stripped-down variants, plus the publisher's separate quantization numbers appended at the end. No third party reports running the recipe, no leaderboard submission is claimed, and the board that would settle it — ARC's private eval — still belongs to frontier systems. The author's request that contributors stop adding training data is the closest thing to community traction on offer.
Restrained body, promotional tail
The title says a 1.5-hour transformer beat many LLMs; the body then spends paragraphs explaining why that comparison barely holds — 56% of puzzles still fail, LLM cost figures omit pretraining, public evals are contaminated, o1 is 31 points ahead. That self-correction keeps the gap small. It widens where the ARC result stops being the subject and becomes a lead-in to vecq, and where a strong claim about someone else's parameter count is made with no reply in sight.
The efficiency argument sells something
Follow the last third of the piece: ARC gives way to vecq, the publisher's training-free quantization library, with recall@10 figures, a compression ratio and a 98.2% number measured against ChromaDB and Mem0 — all in-house. The researcher has his own position to advance, this being the third viral instalment of a series, capped by a proposal that ARC's organizers ban offline pretraining, a rule his method happens to survive and pretrained rivals would not. Both sets of interests are visible rather than hidden, which is the redeeming part.
Clear on what was claimed, not on what replicates
We can say with confidence what was asserted and how it was measured; the internal arithmetic holds and the caveats are stated in the author's own words. What we cannot say is that 44% survives a second implementation, or that a rented-5090 recipe generalizes past ARC-1's thousand-odd examples. A single voice, no external rerun, and one contested claim about another team's parameter accounting keep this well below the level where we would treat the number as settled.