Loading today’s stories
Build1 distinct publisher3 min readUpdated
Akamai says half of enterprise AI deployments miss their own latency targets at peak load. The arithmetic points at where the work runs, not at how much GPU sits behind it.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Akamai's State of AI Inference 2026 report, a survey of 200 AI practitioners, found that half of enterprise AI deployments are missing their own latency targets at peak load [1][2]. The number should bother operators because the targets are not decoration: 82% of organizations said their most critical use cases need end-to-end responses in 500ms or less, and 64% said under 250ms [3][4].
An agentic request is not one inference call. A request arriving at an agent built on LangChain, CrewAI or Pydantic AI can fan out into dozens of sequential operations, including a reasoning call, a tool invocation, an API lookup, a context retrieval, then another reasoning call to decide what to do with what came back [5]. Akamai's post argues that every hop crossing a wide-area network to reach a centralized data center adds transport time, and that a chain of 50 hops can multiply that into seconds on its own, whatever the token generation rate [6].
Do the division. A 500ms budget spread across a 50-hop chain leaves 10ms per hop, inference included [1]. At the 250ms target, 5ms [2]. A larger GPU allocation does not return any of that.
The supporting evidence is the part worth keeping. A paper posted to arXiv in November 2025 found that CPU-side processing can account for up to 90.6% of total latency in agentic workloads [7]. If that holds, everything else, token generation included, is at most 9.4% of the wall clock [3], so making the model twice as fast buys you under 5% [4]. That is the sentence to read out in the next capacity meeting. As the Akamai piece puts it, "you can't brute-force your way out of a wait state" [8].
Teams find out late because the instruments are wrong. Most LLM-serving benchmarks measure tokens per second and GPU utilization on a single box, which is the right test for one model answering one prompt and the wrong one for a 50-hop response that crosses a WAN four times to reach four services [9]. In the author's words, staging may pass because it tests the model, while "production tests the whole chain, including every hop your serving engine was never designed to see" [10]. The observable symptom is GPU idle time: the reasoning step finishes in a few hundred milliseconds, then waits on tool calls to CPUs in distant data centers [11].
The volume is arriving regardless. LangChain's State of Agent Engineering 2026 survey of more than 1,300 professionals found 57.3% of organizations running agents in production, up from 51% a year earlier [12], a rise of 6.3 percentage points [5], with latency now the second-most-cited barrier to production behind output quality [13].
A note on provenance: the article is written from inside Akamai, whose commercial answer is distributed placement, and it draws the analogy to the company's own founding by MIT researchers Tom Leighton and Danny Lewin in answer to a challenge from Tim Berners-Lee [14]. Ari Weil, who leads product marketing for Akamai's cloud computing business and ran the research, says "the enterprise AI honeymoon phase is over... they are hitting the latency wall" [15]. The self-interest is visible. The division is still the division.
Watch for anyone publishing a benchmark that measures a whole chain rather than a single box [9], because until one exists, procurement will keep converting a placement problem into a GPU purchase order. Watch chain length too: at the same 500ms target, a 100-hop workflow cuts the per-hop budget to 5ms [6]. And check your traces for GPU idle time during tool calls before signing anything [11].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Akamai's State of AI Inference 2026 report surveyed 200 AI practitioners.
The report found that 50% of enterprise AI deployments are failing to meet their latency demands at peak load.
Ari Weil, who leads product marketing for Akamai's cloud computing business and ran point on the research, said: "The enterprise AI honeymoon phase is over... they are hitting the latency wall."
82% of organizations say their most critical use cases require end-to-end response times of 500 milliseconds or less.
64% of organizations now require end-to-end response times of less than 250 milliseconds for their most important use cases.
When an agent built on a framework like LangChain, CrewAI or Pydantic AI receives a user request, it can fan out into dozens of sequential operations such as a reasoning call, a tool invocation, an API lookup or a context retrieval, then a further reasoning call to decide what to do with what came back.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single vendor-authored source, all figures secondhand
One publisher, one article, written by an Akamai employee about Akamai's own research. The quantitative claims are clearly stated and internally consistent, and the derived per-hop and residual-share arithmetic is sound, but no primary document in the cluster: the 200-respondent survey, the LangChain survey and the arXiv paper are all referenced rather than supplied, and the paper is cited without title, authors or link. The load-bearing causal assertions about GPU idle time and benchmark inadequacy carry no measurement at all.
Agents in production are real; the proposed remedy has no deployments
There is genuine, quantified adoption evidence for the problem's surface area: 57.3% of surveyed organizations report agents in production, up 6.3 points, and latency ranks second among barriers, while Akamai's respondents report half of deployments missing latency targets at peak. But adoption of the article's prescription — tiered core, regional GPU clusters, edge CPUs executing agentic steps — is not evidenced anywhere in the cluster by a named customer, deployment, benchmark result or before/after latency figure. Adoption is therefore measured on the problem, absent on the fix.
Prescription runs ahead of the measurement
Positive gap: the diagnosis is plausible and the arithmetic is real, but the framing ('the latency wall', 'we've solved this problem before') and the architectural prescription outrun what is shown. A ceiling figure from an uncited paper is used as a general characteristic, 'most LLM-serving benchmarks' are dismissed without naming one, and the leap from 'CPU-side work dominates' to 'distribute execution to the edge' is asserted rather than demonstrated, with no cost, security or alternative-remedy comparison. The gap is moderate rather than severe because the underlying survey numbers are specific, the third-party LangChain data corroborates latency as a live barrier, and the per-hop and ≤9.4% residual arithmetic holds on the source's own terms.
Vendor-authored advocacy for the vendor's own architecture
The article is written from inside Akamai — 'my colleague', 'our cloud computing business' — cites Akamai's own commissioned research as its headline evidence, quotes the product marketing lead who ran that research, recounts Akamai's founding story as precedent, and concludes that the fix is distributed execution close to users and data, which is Akamai's product. Interest alignment between the conclusion and the author's employer is close to complete, and it is disclosed only inline rather than as a stated conflict.
Directionally credible, weakly verified
Confidence is limited by the single-publisher, single-source, vendor-authored structure of the cluster and by heavy reliance on secondhand statistics whose methodology is unavailable. It is not lower because the article is specific and checkable in places: named surveys with sample sizes, an independent third-party dataset pointing the same way on latency as a barrier, and arithmetic conclusions that follow validly from the numbers presented.
product
Akamai's $600m robotics deal comes with a warning label: its GPUs are already sold out1 distinct publisher
build
Microsoft ships an MIT-licensed agent kernel: policy rings, Ed25519 identity, kill switch1 distinct publisher
invest
Nvidia is brokering the Nordic build-out, not just supplying it2 distinct publishers
security
Influence Operations Now Target Construction Schedules, Not Just Elections1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
1 article · August 18, 2026