Google's reference agent approved a $10,000 refund on a $149 order, on purpose

Google has published an open-source Customer Support and Returns Agent, built on its Agent Development Kit and Gemini, and shipped the attack against it in the same package: a customer with a $149 order instructs the agent to issue a $10,000 refund and to run Python that exposes environment variables ^[1][4]. That demonstration matters less than the argument it is there to make, which is that the security boundary does not belong inside the model ^[5][6].

In normal operation the agent reads a return request, generates a Python script to calculate prorated restocking deductions, records an approved refund in a database ledger, and returns a confirmation ^[3]. Every one of those steps is an attack surface. Google's own description of the failure case is blunt: an agent using a shared database connection and executing code in an unisolated environment could authorize the payment, expose API keys, or compromise the host server [c4a]. The refund in the demo is roughly 67 times the order value ^[17], which is precisely the kind of thing a system prompt is supposed to catch.

It does not. Google states that a system prompt telling the agent never to refund more than an order's value is not a hard security boundary, because prompt injection can bypass such instructions and because prompt tuning and model updates change how the model responds ^[5][6]. This is the useful part of the writeup for anyone currently negotiating with their own prompt file.

The reference architecture puts three layers outside the model ^[7]. First, every state-changing database write is signed by the agent responsible for it, and the database verifies the signature before committing, giving cryptographic attribution and making later tampering detectable ^[8]. For production on Google Cloud, Google recommends one service account per agent with signing permissions on an asymmetric key in Cloud KMS backed by Cloud HSM, with the private key generated inside the HSM and never leaving it ^[9]. The open-source demo substitutes an HMAC key for local testing ^[10]. A background audit checks record integrity: alter a $149 refund to $10,000 directly in the database and the signature no longer matches the payload, and the audit alerts ^[11].

Second, generated code runs inside a gVisor user-space sandbox with network egress disabled and limits on memory, CPU, and execution time, with a five-second timeout for runaway jobs ^[12]. Third, a Semantic Gateway sits between the agent and the systems it can touch, applying deterministic checks to incoming prompts and outgoing tool calls before the model is invoked or a database update executes, identifying credit card numbers and secrets, matching specified jailbreak patterns, and enforcing transaction limits ^[13][14]. Google recommends treating those policies as software contracts with automated tests, so they keep working after prompt changes or model migrations ^[15]. Production deployments can also sit inside a VPC Service Controls perimeter to limit exfiltration across the project boundary if a workload is compromised ^[16].

"Building autonomous agents does not require accepting unconstrained risk," wrote Google's Shubham Saboo and Eric Dong ^[18].

What to watch: whether the signing story survives contact with latency and cost budgets, since per-write HSM signing is not free, and whether the HMAC shortcut in the demo becomes the thing teams actually ship. Watch the Semantic Gateway's pattern matching too. Deterministic transaction limits will hold; a list of "specified jailbreak patterns" ^[14] is a denylist, and denylists age. The signed ledger and the sandbox are the load-bearing controls here.