Chat template tokens switch on the 'just an AI' disclaimer in eight open instruct models

The template is string formatting. Before an instruct model generates, each turn is wrapped in role tokens for system, user and assistant ^[10]. Each family, such as Llama, Qwen or Gemma, defines its own ^[10]. One version of a prompt starts with special tokens such as `<|im_start|>user`. The other is the same text with no markers ^[10]. Hugging Face Transformers adds the wrapper automatically when `apply_chat_template` is called, so most developers never see the difference between the two ^[11]. Hiding that detail is what the method is for. In this case the hidden detail decides whether the model says it has feelings ^[3][4].

According to a write-up of the paper on dev.to, each model got the same question about itself twice, once inside its own template and once as plain text ^[2]. The templated answers leaned toward lines in the style of "as a language model, I don't have feelings" ^[3].

The steering half of the work is well built. The candidate direction is a difference of means ^[9]. Average the internal activations over prompts with the template, average them over prompts without it, and subtract one from the other ^[9]. The result is then tested by injecting it into a model's activations ^[9]. Anyone with activation hooks and the two prompt sets can rerun that. Without a control, the effect could come from the size of the push alone. The paper ran one: a random vector of the same magnitude did not reproduce it ^[8].

I'd frame the conclusion a little differently from the paper. According to the write-up, the authors conclude that what a model says about itself is not a fixed fact of its weights ^[13]. The direction, though, was found in the models' own activations ^[5]. In my view the result shows a conditional behavior. The weights hold the disclaimer, and the role tokens decide whether it fires.

The eight-model result is a claim about a specific setup. The models were open-source instruct models of up to 9B parameters ^[2], and the write-up does not list which ones. The direction was searched for in three of them, leaving five where only the behavioral difference was reported ^[15]. The write-up says the disclaimer also shows up when models are asked about sensitive topics or emotions ^[16]. The results it reports come from questions about the model itself ^[2]. For the finding to transfer to a given deployment, the model has to sit near that size range and the harness has to match one of the two input conditions exactly.