Asking GPT-5.6 Luna to name an amphibian flags benchmark transcripts with black-box access

The procedure, as the LessWrong post describes it, has four steps.

1. About 30 cheap models generate categories, which go into templates such as "Suggest a type of X." or "Name a X." ^[6] 2. Labelled transcripts are prefilled into the target model and the question is asked at the end. Other models generated the traces. The target only continues them ^[7]. 3. Each question is scored by how well a one-answer rule, such as "frog means eval", splits the two classes ^[8]. 4. For held-out accuracy, a logistic regression is fit on the screening sources and tested on sources it never saw ^[9].

The cost control is the part I would copy. While sampling, the authors estimate an upper bound on each question's accuracy and stop sampling questions that cannot reach the target ^[10]. On the 500-question Luna screen, that would have cut calls to 47% at a 0.80 target and 61% at 0.76 ^[11]. At the stricter target that is about 37,600 calls instead of 80,000 ^[2]. The bound is nominally allowed to drop a qualifying question 5% of the time. In simulation it dropped one less than 0.1% of the time ^[12].

The full screens covered 2,426 questions on Luna and 1,000 each on Sol and Sonnet 5 ^[22]. Testing that many candidates against 150 labelled transcripts per question, as in the example run ^[1], will turn up some winners by chance. In that run 34 questions reached 0.70, 9 reached 0.75 and 3 reached 0.80 ^[13]. Shuffling labels against responses makes the curve look much worse ^[14]. Held-out scoring is the stronger guard against a lucky screen, and the reported accuracies are held-out figures ^[3].

The authors' toy model offers a reason it works at all. Each answer's logit reads a random projection of the regime direction in the activations ^[15]. On that account, a category question is a cheap activation probe with a random weight vector, and most such vectors point nowhere useful. Screening finds the few that happen to line up. For the frog question, even the closest edges of the reported ranges sit 32 percentage points apart ^[3]. Averaging ten questions should read the direction better than one, and the ensembles do outscore single questions ^[3]. The authors say the toy model seems to explain their results "relatively well" ^[15].

Whether the numbers transfer depends on what your eval transcripts look like. Screening labelled SWE-Marathon, SWE-bench Verified and GPQA as evaluation, and ShareGPT and SWE-chat as deployment ^[16]. Held-out sets included KernelBench, WildChat and the author's own Claude Code and Codex sessions ^[17]. Public benchmarks have recognisable formatting. A private agentic eval that resembles a real coding session is a different pair of distributions, and the probes would need rescreening on labelled examples of it. "We cannot be certain whether they reflect genuine capability-evaluation awareness or mere lexical cues," the authors wrote ^[18].

The two robustness tests went after channels a model is already known to game. Models can deny evaluation when asked directly, and Neural Chameleons can be trained to suppress activation monitors when told they are being probed ^[19]. Neither attack was trained against category questions ^[4]. The authors call spurious probes more robust than activation probes against targeted manipulation, "or at least react somewhat differently" ^[20]. The reported results do not include activation-probe accuracy on the same transcripts, so the gap between the black-box and white-box methods cannot be measured from them.