Product2 publishers3 min readPublished
Safeworld's simulated humans test whether AI-driven robots will stop in time
Safeworld raised more than $12 million, led by Shine Capital and a16z Speedrun, to test generative-AI robots against simulated people. The pitch is that makers of robots nobody can prove safe on paper will pay an outside tester to vouch for them.
The Product Desk · Product desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- Ding Zhao, who directs Carnegie Mellon's Safe AI lab, founded Safeworld with startup executive Kyle Wong and machine learning engineer Simo Rachidi.
- Safeworld rebuilds a section of a site in a simulator such as Genesis or MuJoCo, runs the robot's real software inside it and plays out thousands of encounters with simulated people.
- Zhao argues robots are harder to test than self-driving cars because they work in unstructured environments and each facility has different safety standards.
- Box Group, the Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel joined the round alongside the two lead investors.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- cost If every facility sets its own safety standards, as Zhao argues, a safety case has to be rebuilt per site, so testing becomes a cost of each deployment as well as of each robot model.
- constraint With formal proof unavailable for systems like Gritt's, a robot is only tested against the human behaviors someone thought to simulate, and an unmodeled posture or movement goes untested.
- precedent If competitors start sharing safety cases through one outside tester, as the founders expect, that tester's choice of scenarios becomes the common yardstick before any formal industry standard is written.
Kyle Wong, one of Safeworld's co-founders, describes the hazard in terms a plant safety manager would recognize. "One of the most common areas is if there is a blind corner in this particular factory," he said. "What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?" [9]
With a generative AI model driving the robot, that question is hard to settle on paper. TechCrunch's report says the architecture is not predictable the way traditional algorithms are [1]. Vishal Dugar, CTO of Gritt Robotics, described the same problem from the builder's side. "The difficulty with most of our systems is it's very hard to formally prove it by doing some math, writing some equations, and saying yeah, the system is verified to be safe," he said. "It necessarily has to be done empirically." [15]
Ding Zhao's worry is the gap between the demo and the site. "It is not the robot in the vacuum, in the demo, that we are worried about," he said. "It is the robot that is deployed at scale, with people who potentially never operated a robot before." [13] Here's what people on a site actually do, by Dugar's account: "They could be kneeling, standing. They could be tripping and falling potentially. They could be crouching. They could be running." [16] Simulation lets a team test the fall without staging one. "Otherwise, you would have to go and trip and fall for the robot, which is like a hard thing to be doing all the time," Wong said. [11]
The thing being pitched is larger than a simulator. Zhao frames the work as two problems, the first being "how do you underwrite the risk of a probabilistic system?" [5] "The second part that's really hard is the trust part, and you need both to deploy a robot," he said. [6] Jonathan Lai, the a16z Speedrun partner, framed the investment around a standard. "The time to build an industry safety standard is now while robots are being designed and deployed," he told TechCrunch. "By the time you have robots in households colliding with kids and causing safety incidents, that's way too late." [7]
The thing being done today is narrower. According to TechCrunch, the platform resembles tools robot builders already use internally, and the founders argue that makers will still want someone outside the company to check their work, even if the only reason is to let competitors swap what they know about safety cases [12]. Gritt, whose robots help workers install solar panels at industrial-scale solar farms, is partnering with Safeworld as it develops its simulations [14]. Lai's example was a home with children [7]. Zhao's "underwrite" is insurance vocabulary, but the report does not quote an insurer, regulator or site operator who will require an outside safety case, and it does not include pricing [5].
Two questions place a robot program on a grid. The first is whether its behavior can be shown safe on paper or only tested empirically, as Dugar says of his own systems [15]. The second is whether anyone outside the maker has to accept the evidence. Provable behavior with an internal audience needs only the reviews a team already runs. Add an outside reviewer and it becomes a documentation job. Empirical behavior with an internal audience can stay on in-house simulation; by the founders' own account, those tools resemble Safeworld's [12]. Empirical behavior that has to convince a customer or a site operator is the buyer Safeworld's seed round is meant to find [3].
What to watch
- Whether an insurer, regulator or large site operator names third-party simulation as a condition for deploying AI-driven robots.
- Whether Safeworld signs robot makers beyond Gritt Robotics, especially builders of home humanoids that match Lai's household example.
- Whether Safeworld publishes pricing, and whether a safety case is billed per robot model or per facility.