Swapping Backstage for an agent hands the platform team an index to keep current

Retrieval is where this design costs money, and Farcic put that plainly to InfoQ: "The quality of semantic search is not defined by the embedding model or the vector database. Those are commodities. It's defined by what you put in and how you slice it." ^[13] Documentation is one source and usually the worst one, because it is outdated, he said ^[14].

Chunking breaks first. "If you embed a forty-page document as a single vector, you get one blob that is vaguely about everything and precisely about nothing," Farcic said ^[15]. A blob like that does answer every question, just not differently. His recommendation is to split into pieces that stand on their own, a section or a function or a resource definition, and keep metadata for which repo, which team, and when ^[16].

Freshness is the second failure. "An index is not a migration you run once," Farcic said, adding that ingestion has to be continuous and deletions have to propagate ^[17]. Stale content fails quietly: "If the information is six months old, the agent will confidently give you six-month-old answers, and that is worse than no answer, because you trust it" ^[18]. The index also only holds decisions that were written down somewhere. A team that settles design arguments on calls nobody records has nothing to embed.

The interview draws one boundary worth copying into a design doc. Cluster state "is not knowledge; it's state, and it changes every second," according to Farcic, who said not to embed it and to give the agent a live tool to fetch it instead ^[19]. So the platform team runs two paths, an ingestion pipeline for knowledge and a set of tools for current state. When the answer comes back wrong, Farcic said, "the retrieval is usually the culprit, not the LLM" ^[20].

The guardrail argument follows from the shape of the interface. "With an LLM, the input can be anything that a user provides to an agent, and the output is whatever the LLM thinks it should do," Farcic said, and: "We have no idea what the input or output is." He also said "we cannot limit what will happen" ^[6]. Enforcement therefore sits at the tool boundary, which means classifying every tool the agent can reach as auto-run or approval-gated, and writing that list down.

The observability plumbing already exists. OpenTelemetry has semantic conventions for GenAI, with spans for model calls carrying the model name, token counts, tool invocations and their arguments ^[9]. Farcic said, "With OpenTelemetry you can send the data anywhere: Jaeger or Grafana Tempo for traces, Prometheus for metrics, Loki for logs," and that Datadog, Honeycomb, Dynatrace and Elastic ingest OpenTelemetry as well ^[10]. Eight named destinations, four open source and four commercial ^[11].

Lee kept the user-side claim narrow. "As a user, you (or more likely your coding agent) don't have to understand where the information is, and because the information is indexed by semantic similarity, you don't have to guess the exact words used in order to find what you're looking for," she said ^[21]. InfoQ's report of the talk does not include adoption figures or a cost comparison against a portal, so the case rests on how the pieces fit and on what these two practitioners have seen ^[22]. For teams that build it anyway, Farcic offered the cheap route to evaluation: "Real traces are your test data. The questions your developers actually ask, and the paths the agent actually took, make a far better evaluation set than anything you'll invent in a meeting." ^[12]