Product1 publisher2 min readPublished
Gemini's product team turns prototype failures into evals researchers can aim at
Tulsee Doshi, who leads product for Google's Gemini models, says no customer hands her team a requirements document, so the team writes the target itself as scored examples of a win and a loss.
The Product Desk · Product desk
What happened
- Tulsee Doshi leads product for Google's Gemini models, which the Gemini app, Search and every API developer build on, along with Google's video, image, music and audio models.
- She told a live conversation hosted by The Skip that no customer hands her team a requirements document, so the team itself decides what the model should get better at and then describes it for researchers.
- Incoming requests get triaged by building a prototype on the model as it exists today. The failures then get sorted into ones a system instruction or better prompting can fix and ones that need research time.
- Failures in the second group become evals: a set of examples with a clear definition of what a win and a loss look like. Once a team has one, the researchers have a target.
- The model serves three kinds of customer: Google's own Antigravity coding tool, other Google teams such as YouTube and Gmail, and every developer on the API, each needing a different level of steadiness.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- constraint Priorities go to whoever can write a scored example. A team that can only describe a quality problem in prose competes for nothing, and its complaint stays in the prompting layer.
- decision The work of asking moves to the asker: you build on the current model, collect the failures and define the scoring before you file. That is a different job from writing a feature request.
- exposure Google's internal surfaces can absorb a live experiment and improve the model incrementally, so the cost of a model changing underneath a product falls hardest on customers who cannot run that way.
The eval is the gate, and it is narrow. A failure has to survive a prompting or system-instruction fix, and it has to be expressible as scored examples, before it can compete for research time [23]. "Nothing can replace a great eval," Doshi said [13].
Capability discovery is real on this team. Doshi describes her light-bulb moments as "Wait, I didn't know the models could do that" [3]. She got one from Nano Banana, the image editing her team was building: hand the model a photo, describe the change you want, and it makes the change [4]. Someone on the team tried it for the first time, and it worked. "That level of joy as a PM I haven't experienced before," she said [5]. Moments like that do not come every day, and she said they are what "fuel all of the other chaos" [6].
The team still decides in advance what to chase. Its job as a platform team is to define what capable means, for which customers and which uses, choosing among coding, video, long documents, and agents that take actions over hours. Those choices are made before the research starts [9].
The requirements document has a different author and format here [24]. Enterprise product management starts from a customer's stated problem and a doc the customer wrote [8]. Here the platform team writes the target itself and hands researchers a set of examples with a clear definition of a win and a loss [7][12].
The write-up asserts one claim without measuring it. Among the interviewer's takeaways: a model has a personality, someone on the product team has to own it, and the benchmarks cannot see the part that decides adoption [16]. The published account does not give usage, retention or adoption figures for Nano Banana or any other feature discussed [22]. Personality may well drive adoption, and on this record that is a practitioner's judgement about a live product.
For a team sitting on the other side of somebody else's platform team, the transferable part is the sort. Every complaint about the dependency lands in one of three places: fixable by changing your own prompt or instructions, fixable only by the people who train the model, or not a model problem at all. The middle one earns a research ask when it arrives with examples and a scoring rule attached [23]. Doshi's team also expects its best product managers to prototype features that do not work yet, on purpose [18]. Being "technical" there counts as being able to say what good looks like; a computer science background is not what it means [17].
What to watch
- Whether Google publishes any of the eval suites or win/loss definitions behind a Gemini release, which would let outside teams check the target they are being built toward.
- Whether API developers get change controls, such as pinned versions or advance notice, matching the steadiness Doshi says they need.
- Whether the full episode or later interviews put usage or retention numbers behind the claim that model personality decides adoption.