Product1 publisher3 min readPublished
One customer's 'clean up' request led to four new evals and sixteen variants to diagnose the problem
A Vistaly beta customer asked for a button to tidy up her AI-generated opportunity tree. The team went after the generator instead and spent three weeks, four new evals and sixteen experiment variants doing it.
The Product Desk · Product desk
What happened
- A beta customer reviewing her first AI-generated opportunity solution tree in Vistaly said she wished she could click on a card and clean it up, according to a Product Talk account of the session.
- The branch she was looking at held several flat opportunities with no hierarchy, and her first instinct was to tidy it herself.
- The post's author declined to build a cleanup button and went after why the agent had not added structure to that branch, an effort the post describes as a three-week adventure.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- cost Twenty pieces of eval and experiment work for one complaint is what a source fix bills at, and it lands on whoever funds three weeks of a synthesis team's calendar.
- decision Teams already running their own synthesis pipeline on Claude or ChatGPT have to decide whether they will fund evals at that rate or ship the cleanup control and maintain it.
- constraint Without a frequency measure, a team cannot tell whether a complaint describes one bad branch or every generated tree. The workaround becomes the only defensible thing to ship.
- capability Because tree-shape is arithmetic over the generated output, a team can watch its structural rules hold or break on every trace without paying a reviewer to read each tree.
A cleanup button and a fix are separate tickets. The button lets the user repair a tree after the agent flattened a branch. The fix goes after the flattening. The customer, a beta user of Vistaly's new AI service, said: "That is a lot of first layer opportunities there. I wish I could click on this card and be like, 'Clean this up.'" [1][10] Everything before that comment had been positive, according to the post [2].
One complaint is not a rate. The post's author had noticed hints of the same error in other traces without digging into them, and wanted to know how often a parent node ended up with too many children before changing anything in the generator [11][12].
Tree-shape, the first of the new evals, is a code assertion. It counts total opportunities, average children per parent, mid-level parents, the distribution of children across nodes, and how many interview opportunities never made it onto the tree at all [13]. The sample output in the post shows a tree with 7 nodes, a children distribution of 1, 1, 1, 2, 3, 4, 4, and an average of 2.29 children [14]. Add that distribution and you get 16, the opportunity count printed a few lines down, and 16 divided by 7 is 2.29 [18]. It is arithmetic over the output, cheap enough to run on every trace.
The counts encode opinions the author holds about trees: three to seven key moments, parents that branch into sub-groups as they collect children, depth that grows as interviews accumulate, and width that eventually stabilizes [15]. Tree-shape shows whether those rules of thumb hold across the trees the agent generates [16].
Four new evals plus sixteen experiment variants is twenty pieces of work for one complaint [6][19], and sixteen variants inside three weeks runs a bit over five a week [5][20]. The post offers that count to people who have already built interview synthesis and tree generation on Claude or ChatGPT and ask, "Why do I need another tool for this?" [8] "High quality synthesis takes sweating the details," the author wrote [7].
Two tests separate a control worth building from a workaround the team will own forever. The first is whether the user would still want the button if the first output had been right; if not, the request is a bug report with a UI attached. The second is whether the defect can be counted in code across outputs you already have. If it can, you find out whether it is one bad branch or every tree, and the fix gets a target. If it cannot, ship the button, because an unmeasurable defect is one you will be arguing about from anecdotes while the user still has a bad tree in front of her.
The post says the author tried two measurement approaches to start, and the text supplied here breaks off inside the description of the first, before any error rate is reported [17].
What to watch
- Whether a later installment publishes the error rate before and after the sixteen variants.
- Whether Vistaly ships a manual restructure control anyway once more beta users hit flat branches.
- Whether the tree-shape counts get exposed to customers as a quality readout on their own generated trees.