Science1 publisher2 min readPublished
An AI newsletter traces the new singularity talk to thousands of agents at two labs
An essay on interconnects.ai argues that rising AI-risk expectations at OpenAI and Anthropic follow from watching agents work at scale, and its author says he holds high uncertainty about that reading.
The Scientist · Science desk
What happened
- The post says the only organizations now running thousands of concurrent agents to improve their own processes and output are the frontier AI labs, OpenAI and Anthropic in particular.
- It argues that lab culture and the competitive San Francisco AI scene amplify any AI concern, and recalls that the primary risks debated in 2023 and 2024 did not arrive on the forecast timelines.
- Richard Ngo, quoted in the post, expects short-timeline arguments to prove directionally correct and factually wrong, with no superintelligence arriving within the next eight years.
- The author expects scaled agents working productively to account for more of the current safety concern than any unpublished breakthrough, and says he holds high uncertainty about that split.
- He warns against reading large steps in inference-time scaling, which he calls fairly predictable, as the outputs of recursive self-improvement, which he calls highly uncertain.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- decision Anyone sizing a safety budget or an open-weight strategy off lab sentiment is pricing a signal nobody has measured, and the 2023-24 round of the same debate is the only base rate the post offers.
- exposure Open-source AI spent the last round of this argument under a cloud, so a second wave of short-timeline consensus puts open-weight releases back in the path of restriction proposals.
- contradiction The essay allows that the labs may have seen specific breakthroughs that are not public. Its calm explanation and its alarming one both fit the same visible evidence, and it settles between them with a stated expectation held under high uncertainty.
The essay names the condition that would change its author's mind. "Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable," the post says [13]. The condition is about a capability, and nobody outside the two labs can evaluate it until such a result is published or shipped.
The cultural reading rests on a memory and an inference. The post never says which 2023-24 forecasts missed, and its account of sentiment inside OpenAI and Anthropic comes with no survey, poll or employee count [16]. On the leap from workplace anxiety to catastrophe it is blunt: "The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious" [14].
Richard Ngo, whose summary the post quotes at length, frames the problem as bandwagoning. He wrote that the rush toward the idea of a near-term singularity had gotten wild [6]. The remark is undated. Counted from the start of 2026, which the post treats as the recent past, his eight-year no-superintelligence window runs to 2034 [15].
One part of the argument can eventually be checked from outside. The author doubts the labs can afford to spend a constant portion of their compute on internal research and development as total volume rises, especially with plans to IPO and the scrutiny of basic economics that follows [9]. He also credits mass inference capacity with a large short-term acceleration, with thousands of agents thrown at important, measurable problems while available compute keeps scaling [11].
His alternative to true recursive self-improvement, which he calls lossy self-improvement, has three parts. Automatable research is too narrow to produce a big net acceleration against the exponential costs of scaling laws. Diminishing returns from running more agents in parallel are real. And resource bottlenecks and politics matter in ways AI can do little to speed up [10]. The second part is measurable inside a lab, and the post cites no measurement of it. For public material on the question it points readers to Dwarkesh's podcast with Noam Brown and one with John Schulman, Beren Millidge and Charlie O'Neill [12].
What to watch
- An IPO filing from either lab that discloses how much compute goes to internal research, which would test the essay's doubt about holding that share constant.
- A published measurement of the returns from running more agents in parallel on the same research task, the second pillar of the lossy self-improvement case.
- Any specific dated forecast from the 2023-24 safety debates being scored against what happened; the post makes that case only in general terms.