Google Research steers frozen image models with a small add-on network

The lizard is a fair test ^[14]. Ask for "a lizard wearing sunglasses" and a base model can drop the sunglasses, or distort the face to fit them in ^[14]. Hsu and Ryu, the Google Research engineers behind the September 29 post ^[1], wrote that the framework "strikes a balance" between the new preference and the base model's image quality ^[13]. Balance is the accurate word. The guardrail that protects the lizard's scales ^[10] also limits how hard the controller can push for the sunglasses. Adherence and quality still pull against each other. The difference is that the framework optimizes the trajectory shifts with feedback ^[5]. The authors say the older, fragmented tools left engineers to balance the two by guesswork ^[12].

Both fine-tuning methods learn from one reward score on the final image ^[6]. The controller will push toward whatever that score rewards, including its mistakes. For Google's win rate to carry over to another team's prompts, two things have to hold. That team's reward has to score adherence the way its own reviewers would. Its base model also has to behave like the baseline Google compared against ^[2]. The results summary does not name the industry standard the add-on beat ^[8].

The closed-model claim ^[7] needs the most checking. The authors note that changing a model's behaviour usually takes white-box access to its internal settings, and that the best image models are often corporate secrets they call black boxes ^[9]. Diffusion Controller acts during generation, adjusting the trajectory as the image forms ^[5]. A product sold on the secrecy of its internals is an odd place to attach something that steers them. If an endpoint returns only a finished picture, a controller that adjusts the trajectory mid-generation has no intermediate state to act on.

The part I'd expect to last is the framing. Classifier-free guidance changes the prompt's influence at inference time, while LoRA adapters, reward-weighted regression and policy gradients change the model through fine-tuning ^[11]. The authors wrote that these were treated as unrelated fixes, and that the field lacked "a single, principled mathematical language" for them ^[12]. Google presents Diffusion Controller as that language, recasting the whole denoising process as one continuous control problem ^[3]. If the formulation holds up, a guidance scale and a LoRA run can be compared in the same terms.