Berkeley statisticians find METR's time-horizon axis nearly flat between 2 and 30 minutes

Every point on the METR plot is solved from a fitted curve ^[4]. Each task in the suite carries a human time, the geometric mean of a skilled engineer's successful timed attempts ^[3]. Each AI attempts every task several times ^[3]. For each AI, the standard recipe fits logit p(t) = alpha - beta * log2(t) and solves for the t where p reaches one half ^[4]. A 2025 METR research note moved to one slope shared across AIs, and the Berkeley paper takes that version as its baseline ^[5]. The fit never sees a release date. The exponential trend shows up only later, when the fitted horizons are plotted by date ^[6].

One covariate means one strong assumption. AI difficulty has to rise linearly with the log of human time, so every 10x step in human minutes costs the same success probability ^[7]. According to a dev.to summary of the paper, nothing in the data collection guarantees that ^[7].

Nguyen and Fithian kept METR's data, 228 tasks in 79 families across 26 AIs, and changed the statistics ^[8]. The authors call the statistical fit a "model" and the language model an "AI" ^[16]. I follow them here. It spares everyone the phrase "each model's model."

The first fit replaces the linear term with a monotone spline so the curve can bend ^[9]. The second is an explanatory item-response model, the method behind standardized test scoring ^[10]. Each task gets a latent difficulty. The model predicts that difficulty from human time through the spline, then adds a task-family effect and a per-task residual ^[10]. That family effect replaces the square-root reweighting METR used to stop large families dominating ^[12].

The item-response fit also handles correlated runs. If runs were independent given the fitted probabilities, about 60% of an AI's run sets on a task would be all successes or all failures; in the data, 83% are ^[11]. The gap is 23 percentage points ^[17]. A correlation parameter downweights task-AI pairs with many redundant runs ^[11].

I think the validation design is the part worth copying. Folds are split by task family, so held-out tasks come from families the fit never saw ^[2]. Scoring uses proper rules, including an elementary score at q = 0.5 and 0.8 that grades the horizon estimate itself ^[2]. The authors report their estimates beat the baselines across that suite ^[2].

The result turns on the shape of the spline. The conversion from human minutes to AI difficulty is close to linear outside roughly 2 to 30 minutes and nearly flat inside that band ^[13]. A horizon that moves from 3 to 30 minutes stays inside the flat region the whole way ^[19]. The authors say that jump is much easier than 30 minutes to 5 hours, though both are 10x ^[14]. Their conditional success plot suggests a 4-to-15-minute move, a 3.75x multiplier, may be quite small in difficulty terms ^[15][18].

The paper does not dispute the trend ^[1]. For a horizon figure to transfer to a team's own backlog, those tasks need to map human minutes to AI difficulty the way METR's suite does. The refit shows that map bends at about 2 and 30 minutes ^[13]. The dev.to author offers one explanation, labelled as interpretation and not as a finding of the paper. Short tasks mostly test whether an AI follows instructions and calls tools without slipping, while longer ones add planning, error recovery and context management ^[20].