The softmax bottleneck leaks a model's hidden size to anyone who can read its logits

Start with the shape of the last layer. One matrix maps a hidden state of width d onto scores for every token in the vocabulary, so any score vector the model can emit is a combination of d columns ^[3]. The paper calls that constrained set the model's image ^[5]. Its dimension is the embedding size ^[3].

Collect outputs until new ones stop adding directions and you have a basis for that image, and the authors report algorithms that do it cheaply and quickly against standard APIs ^[5][10]. Against gpt-3.5-turbo the estimate came out at about 4,096 ^[4]. That is 2 to the 12th ^[13], so it would also have been a decent guess; the difference is that the method produces it from an endpoint you are billed for, for under $1,000 ^[2].

For that figure to transfer to another endpoint, the endpoint has to have the same shape: a single low-rank output layer over the full vocabulary, and enough per-call score information to assemble vectors that are linearly independent. Its architectural assumption is described in the paper as conservative ^[2]. Where a served system does not match it, the dimension you recover describes the output path the API exposes and not necessarily the hidden state behind it.

The authors considered several mitigation proposals. They wrote that they find "no obvious fix to prevent obtaining LLM images without dramatically altering the LLM architecture" ^[8]. The remaining lever is the API itself, and the paper notes that providers may choose to hide this information there, while clients rely on features like logit bias that have valuable and safe uses ^[9].

The same measurement is a client's audit tool. Applications listed in the paper include finding hidden size, obtaining full-vocabulary outputs cheaply, detecting and disambiguating model updates, identifying which model produced a single output, estimating output layer parameters, and recovering hidden prompts ^[11]. Because the signature is sensitive to slight parameter changes, it can be used to infer granular information about updates ^[6]. Otherwise, as the introduction puts it, users take a provider's word that the model changed only when the provider announced a version update ^[7].

In my view a provider selling to enterprises keeps logit bias and treats the signature as a published property of the endpoint, because the feature it would have to withdraw is one paying clients build on ^[9]. The authors reached the same place from the research side: they wrote that they "prefer to view our findings as a potential feature that LLM providers may choose to keep in order to better maintain trust" ^[12].