Leadership1 publisher2 min readPublished
Ramez Naam argues AI's self-improvement loop is too weak for a fast takeoff
Futurist Ramez Naam estimates AI's self-improvement loop would need to be roughly 5 to 10 times stronger to sustain itself, let alone run away. His Noahpinion case supports planning for rapid AI gains that hit diminishing returns.
The Board Room · Leadership desk
What happened
- Naam wrote the post for Noahpinion after a lengthy private debate with the blog's author about recursive self-improvement.
- Many AI researchers, entrepreneurs and safety people believe a fast takeoff is close at hand, and many in the industry are racing toward it.
- Naam reports real progress on the first two of his five types: AI aiding researchers inside AI companies, and powerful models training smaller ones.
- Noahpinion's author calls himself agnostic and expects AI to become strongly superhuman in most or all dimensions, with or without a takeoff.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- cost Committing capital on a takeoff premise pays today for gains that, in Naam's model, arrive in increments that each cost more than the last.
- exposure A plan paced to measured gains responds late if runaway self-improvement does appear, and Naam concedes forecasters have repeatedly underestimated AI.
- constraint Because the 5 to 10 times estimate rests on current data, it can justify pacing AI commitments but cannot justify ruling a takeoff out.
- contradiction Naam and his host both expect very fast progress, and the host doubts the takeoff question changes practical decisions. Operators have to settle that point for themselves.
Naam and the blog that published him agree on speed. "I expect incredibly rapid AI progress by the standards of nearly any other technology," Naam wrote [6]. His host describes him as normally among the most wide-eyed of techno-optimists [18]. The dispute is about the shape of the curve. Naam's next line: "But the evidence we have doesn't suggest a sudden explosion to incomprehensible superintelligence anytime soon." [7]
The takeoff theory holds that each AI generation builds a better successor, faster than the last generation did [4]. Naam's taxonomy divides on returns. Types 2 to 4 add autonomy and still face diminishing returns. Type 5, the runaway loop to superintelligence, needs accelerating returns, "if we can ever find" them [11]. His 5 to 10 times figure is the gap between today's loop and one strong enough to keep itself going [5].
Naam is open about the limits of that estimate. He says the math is set out in section 8 of his post [10] and rests on "our best current data" [5]. "I could be wrong. Forecasters have repeatedly underestimated AI progress!" he wrote [8]. The top of his range is twice the bottom [1]. The range comes from a forecaster his host credits with predicting the solar and battery revolutions long before they were widely understood [1].
The strongest objection comes from the host. "Ramez's case necessarily rests on a lot of assumptions," he wrote [13], adding, "I don't know how much this debate matters in the practical sense" [14]. "The AI of 2040 is going to look godlike, whether or not it explodes into an actual god in 2027," he wrote [16]. I think the debate matters to an operator even if the 2040 endpoint is the same, because it decides sequencing. Under a takeoff, capability arrives as a step, and only commitments made before the step count. Under diminishing returns, each gain costs more than the last. A commitment that follows the evidence by a quarter then loses little.
The board-deck version of Naam's case is that no singularity is coming, so there is no rush. That version drops his own forecast of progress faster than nearly any other technology has delivered [6]. What his case supports this quarter is spending paced to measured gains. The consequence next quarter is dependence on measurement. A paced plan works only if someone can detect the loop strengthening in time. "One thing that's clear is that we need better data," Naam wrote [9].
What to watch
- Publication or outside review of the section 8 model and the data behind Naam's 5 to 10 times estimate.
- Any measured case of accelerating returns in AI self-improvement, the condition Naam sets for his Type 5 runaway loop.
- Whether measured AI progress again outruns forecasts, the error Naam says forecasters have repeatedly made.