WorkspaceBench grades activation-to-text tools on 3,356 questions one 27B model can answer

The two single-token readers on the list cost different amounts to stand up. Logit lens is training-free: it applies the unembedding matrix directly to the residual stream ^[10]. J-lens needs a fit first, a linear map from the final layer to the readout layer learned by backprop, with the Jacobian averaged over contexts and positions ^[11]. R-lens is offered as a drop-in replacement for J-lens ^[14].

The design problem the post names first is that a tool can score well without reading the workspace at all ^[7]. Its example is text inversion: invert the activations back into the input text, and an agent can infer the intermediate from the prompt ^[7]. The authors say the benchmark does not fully rule out that shortcut ^[7]. A high score is evidence that the tool produced the right string, and separate evidence is needed to show it got that string from the activation.

There is also no ground truth for what a workspace holds, so the tasks are built such that the model has to use a certain intermediate to compute the answer, and questions are tailored to what the model can deliver consistently ^[8]. For association and directed modulation the authors cannot be sure the model is thinking about the concept at all. They wrote: "we do not necessarily insist that it is possible to saturate the benchmark on these tasks, but higher scores are still directly indicative of a tool's capabilities" ^[9].

The stated motivation is monitorability. "Astra can do a concerning amount with no chain of thought," the authors wrote, which they say makes interpretability essential to understanding what is going on ^[13]. That is also why the hallucination eval exists: J-lens is single token but reliable, NLAs are expressive and prone to confabulation, and the benchmark plots accuracy against hallucination rate to compare the two shapes of tool ^[12].

Transfer is the part to check before quoting a WorkspaceBench number about your own stack. The benchmark was developed for Qwen-3.6-27B, chosen because recent open source models increasingly hold strong workspace representations ^[4][16]. The authors expect it to work on larger models and say smaller or weaker ones may need adaptation so the models can do the tasks ^[4]. For a score to say anything about your model, your model has to compute the intermediate the question depends on. And since J-lens requires fitting a linear map by backprop, I would expect a winning tool to need a fresh fit on each model it is pointed at ^[11].

On scale: 3,356 questions over 27 families averages about 124 questions per family ^[15]. The post does not break out per-family sizes ^[17]. A tool could therefore do well on the safety families, badly on multihop computation, and still post a respectable overall number. Overall performance is what the authors put forward as a good proxy for practical utility in model auditing, and they ask that new interpretability tools be evaluated on WorkspaceBench when they are released ^[6].