BuildNot yet confirmed elsewhere1 publisher2 min readPublished
NVIDIA Dynamo makes the agent session the unit of inference routing and caching
NVIDIA's Dynamo uses one shared session ID to drive routing, KV-cache indexing and cache movement for agent traffic on vLLM and SGLang. Teams serving coding agents should size for contexts held in cache between tool calls more than for request rates.
The Engineer · Build desk
What happened
- Most of an agent session's wall-clock time passes between tool calls, with its context held in KV cache while nothing generates, according to the Dynamo post.
- Dynamo maps the session and parent headers that Claude Code, Codex and OpenCode already send onto session_id and parent_session_id, with no configuration needed.
- Session-aware admission control counts each session's working set and applies backpressure at tool boundaries to reduce cache thrashing.
- An experimental shared-pool indexer lets the router count reusable KV held in external stores when it decides where to place a request.
Why it matters
- cost When the cache is full, backpressure at tool boundaries makes incoming sessions wait so resident ones keep their reuse, and new work pays for that protection in latency.
- constraint Teams on custom harnesses get none of these features until their harness sends a session ID, because each optimization is an opt-in consumer of that field.
- exposure Cross-tier cache policies written against KvHint depend on an interface the post still calls a proposal, so the engine-side contract can change under them.
According to the Dynamo post, most open-source serving stacks route, admit and cache each request on its own [7]. They have no notion of the harness or the session that ties a request to earlier turns [7]. A coding agent sends a different shape of traffic [1]. Its session opens with a prefill that is often tens of thousands of tokens of system prompt, tool definitions and user guidance, and every later turn resends the whole growing context [3]. One task fans out into dozens of those calls, alongside subagents running in parallel [4].
A request-level scheduler cannot know that an idle context belongs to a session that is coming back. If it evicts that context while a tool runs, the next turn resends the full context and the engine prefills it again [18]. Dynamo's answer is one field. A session_id is a stable identifier for one chain of reasoning and tool calls, and every request in that trajectory carries it [9]. Child sessions usually add a parent_session_id, so traces can link an agent to the subagents it spawns [9]. The team says it spent the last few months unifying its router, engine and KV cache manager work around that primitive [8].
The post promises four parts and then lists five [16]. The last item, the programmatic cache work, splits responsibility in a way I'd keep. Dynamo, which sees the whole session, supplies policy intent through a Session-Prefix Indexer and a KvHint interface, and the inference engines control execution across memory tiers [14]. The same session ID drives that cache movement on vLLM and SGLang alike [2].
The post wrote that serving these workloads well comes down to "how much of that context a system keeps resident and reuses, and how many sessions it holds at once while still meeting latency targets" [6]. I'd build a capacity model on that sentence. Its inputs are peak concurrent sessions, context length per session, and how long each context sits idle while tools run [19]. A requests-per-second forecast measures none of those [19].
The post's introduction and session-ID sections include no throughput, hit-rate or latency figures [17]. When results appear, they will come from the team's own session shapes. They transfer only to a deployment with similar prefill sizes, fan-out and tool-call durations [21]. A team can test that on its own traffic: session-linked traces tie harness activity to inference performance and feed both offline simulation and live benchmarking [15].
What to watch
- Whether the Dynamo team publishes hit-rate and latency results for session-aware routing, and on what prefill sizes and tool-call durations.
- Whether KvHint moves from proposal to an interface that vLLM and SGLang actually implement.
- Whether more harness providers emit session and parent headers by default, as the Dynamo team says it is working toward.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence35
- Adoption
- Insufficient
- Hype gap+20
- Incentives65
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Agent sessions can involve a large initial prefill, repeated model calls, and subagents running in parallel, with KV cache remaining resident while tools execute between turns.
- [2]
NVIDIA Dynamo uses a unified session-level identifier to enable session-aware routing, shared KV cache indexing, and programmatic cache movement across vLLM and SGLang.
- [3]
A coding session opens with a large prefill, often tens of thousands of tokens of system prompt, tool definitions and user-specific guidance, and every turn after that appends to a context the session resends in full.
- [4]
One agent task fans out into dozens of such calls, alongside short and long lived subagents running in parallel.
- [5]
Most of an agent session's wall-clock time passes between tool calls, with its context sitting resident in the KV cache while nothing generates.
- [6]
"Serving these well comes down to how much of that context a system keeps resident and reuses, and how many sessions it holds at once while still meeting latency targets"
ReportedSupportedSource: NVIDIA Dynamo post on pytorch.org (no named author in supplied text)View cited source - [7]
Most open-source serving stacks route, admit, and cache each request on its own, with no notion of the harness or the session that ties it to previous turns.
- [8]
Over the last few months the Dynamo team unified its work across the router, the inference engine and the KV cache manager around a single primitive: a common session-level identifier.
- [9]
A Dynamo session ID is a unique stable identifier for a single agentic chain of reasoning and tool calls; every LLM request in a trajectory shares the same session_id, and child sessions usually carry a parent_session_id so traces and replay tools can connect agents and the subagents they spawn.
- [10]
Every optimization described in the post is an opt-in consumer of the session information in session ID and parent session ID.
- [11]
Dynamo recognizes the identity headers popular coding agents already emit and maps them onto an internal agent_context: the session header becomes session_id and the parent header, where present, becomes parent_session_id. Claude Code, Codex and OpenCode work without configuration.
- [12]
Dynamo's session-aware routing and admission control account for session working sets and apply backpressure at tool boundaries to reduce cache thrashing.
- [13]
An experimental shared-pool indexer lets the Dynamo router account for reusable KV in external stores when placing requests.
- [14]
The Session-Prefix Indexer and a proposed KvHint interface support session-aware cache policies across memory tiers, with Dynamo supplying policy intent and inference engines controlling execution.
- [15]
Session-linked traces connect harness activity to inference performance and support offline simulation and live benchmarking.
- [16]
The post says it covers its progress "in four parts" and then lists five items: Session_ID, tracing and replay, session-aware scheduling, shared KV cache awareness, and programmatic KV cache.
- [17]
The introduction and session-ID sections of the post do not include throughput, cache hit-rate or latency figures.
- [18]
A request-level scheduler that evicts a session's idle context while a tool runs forces the engine to prefill the resent full context again on the next turn.
- [19]
Capacity for agent serving depends on peak concurrent sessions, context length per session, and idle residency time between tool calls; a requests-per-second forecast measures none of these.
- [20]
The Dynamo team says it is working with harness providers to make session identification the default everywhere, so support expands without users changing anything.
- [21]
Any published Dynamo results would transfer only to deployments whose prefill sizes, fan-out and tool-call durations resemble the test sessions.
Sources
1 independent publisher whose own reporting we read for this story.
- pytorch.orgSession-Aware Agentic Inference with NVIDIA Dynamo
1 article · October 8, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.