Skip to content

Build1 publisher3 min readPublished

Fleet runs its frontier-model security tests in temporary cloud accounts on synthetic data

Luke Heath's written answers to Lets Data Science put eight named controls around the model, among them scoped credentials, engineer approval and hand reproduction of every candidate finding. Heath has not disclosed results yet.

The Engineer · Build desk

Photograph accompanying Fleet runs its frontier-model security tests in temporary cloud accounts on synthetic data
Photo: letsdatascience.com

What happened

  • Fleet said on September 15 that it had joined Anthropic's defensive security program Project Glasswing, with the work aimed at its own software and device-management protocols.
  • When the model needs a running system, Fleet deploys a temporary copy of itself in a separate cloud account containing synthetic data, and removes that environment when the session ends.
  • Heath did not disclose an individual finding, a reproducible test result or a measured improvement, saying detail follows once fixes ship and disclosure review is done.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • capability Buyers evaluating a vendor's AI security testing now have eight specific things to ask about, each with a factual answer: the account, the data in it, the credentials, the supervision, the approval, the teardown, the model access, the reproduction.
  • constraint A security engineer must approve each attempt and reproduce each candidate finding by hand, so engineer hours set the throughput of the program.
  • exposure A model with no application credentials that works from outside a synthetic-data deployment can be wrong without reaching anything real. The permission boundary bounds the damage.
  • decision Anyone treating this account as procurement evidence is buying a described process with no published finding attached and no independent audit of the controls. That makes it a question list for other vendors.

The exercise starts from outside. Heath said the model approaches the temporary deployment without application credentials and proposes possible attack paths [15]. Inside that deployment it can execute test steps and probe the running system [10]. Credentials are limited to that environment, a Fleet security engineer supervises the session, and anything extending beyond the environment requires that engineer's approval [11]. The environment is removed afterward [12].

Then a person has to reproduce it. Lets Data Science described the workflow this way: "The approval step sits in the middle: a security engineer decides what the model may try, then reproduces each candidate finding by hand before anyone treats it as real" [16]. Confirmed findings enter Fleet's existing remediation and disclosure process, Heath said [17]. Sometimes the model identifies a problem the team already knows about. That can help check a capability without turning up a new vulnerability [18].

Heath states eight controls separately. There is the synthetic-data test environment [6] and the exclusion of customer endpoint data [7]. There are credentials scoped to that environment, engineer supervision of the session, engineer approval to reach past it [11], and teardown [12]. Model access is restricted to named security team members with phishing-resistant multifactor authentication [13], and manual reproduction is required before a finding counts [16][23]. Most of that is ordinary access control applied to a new kind of client. Heath described those safeguards as already in place, and said broader coverage across the release process remains planned [14].

The value for a buyer is that each item is a question with a checkable answer. Which cloud account did the test run in? What data was in it? Did the model hold application credentials? Who approved an action that reached outside the environment, and who reproduced the finding? The answers have limits. Heath told LDS that "Customer endpoint data is not used in this work" [7] and added that no data from customer environments is shared with the model or Anthropic [8]. LDS notes that this describes the research workflow and should not be generalized into a claim about every AI integration a Fleet customer might separately use [9]. Heath also said the September 15 announcement does not introduce a new Mythos-powered feature for customers [4]. He separately described the endpoint-management functions Fleet already provides, including checking device state, deploying changes and reporting patch status [24].

Heath did not disclose an individual Glasswing finding, a reproducible test result or a measured performance improvement. He said detailed findings would follow after fixes have shipped and the material has been reviewed for safe disclosure [20]. Fleet's published engineering handbook covers an engineer's initial review of security reports and staging fixes while limiting disclosure of unresolved vulnerabilities, and its security handbook covers severity-based remediation and exceptions. LDS says those documents are background; they do not report results from the Glasswing work [19]. LDS also says its account does not establish that it has independently audited Fleet's implementation of these controls [22].

What to watch

  • Detailed Glasswing findings after fixes ship, and whether any describes a bug the model found before a Fleet engineer did.
  • Whether the planned coverage across Fleet's release process arrives with the same scoped credentials, supervision and approval step.
  • Whether other Glasswing participants publish comparable descriptions of their test environments and data handling.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories