visionOS 27 adds held-object tracking, with an optional extended training mode that can take 18 hours

The first day of a held-object tracking project on Vision Pro is spent waiting on a Mac ^[13]. A developer hands a 3D model of the target object to the Create ML tool, the tool spends several hours training a neural network, and the result is baked into the app before compilation ^[13]. In visionOS 2 what came out was a tracker for static objects, meant mostly to anchor virtual content to them, and it responded very slowly when the object moved ^[14].

visionOS 27 adds a high-frame-rate mode that works on moving and held objects ^[15]. Apple has not published the update rate, and developers testing it report around 30Hz ^[15], which works out to a pose update roughly every 33 milliseconds ^[19]. For held objects Apple strongly recommends a new Extended Training mode that it says "significantly" increases training time ^[16]. Brad Lynch, an XR enthusiast, found that a fish toy took around 18 hours to train ^[17]. One Mac gets through about one object a day at that rate ^[20].

A pilot deck will say the headset recognises the tools on the bench. What it recognises is the objects someone trained beforehand, one Create ML run each, with the catalogue fixed at the build ^[13]. UploadVR calls the older static tracking genuinely useful for enterprise applications ^[18]; the new mode is what makes a tool in someone's hand trackable at all, and it also puts the training queue ahead of the code in the schedule.

The assistant comes with more gates. Siri AI is in beta on both M2 and M5 headsets, and access means joining a waitlist in visionOS Settings and waiting for a download that Apple says starts automatically once access is granted ^[5][6]. It runs in English only, with the device and Siri languages set to the same supported English option ^[7]. Apple says it is a "profoundly more capable and conversational assistant" than traditional Siri, with "personal context understanding, broad world knowledge, and more" ^[10]. The model behind it was co-developed with Google and is based on Gemini, reportedly under a $1 billion a year deal, and the harder requests are handed off to Apple's Private Cloud Compute ^[8][9]. In visionOS it appears as an orb you can pin to a physical spot in the room and speak to by looking at it ^[11].

The consumer feature here is the panorama: an iPhone panorama becomes a spatial scene, and it can sit behind your windows as the home environment ^[23]. The rest of the list needs somebody to approve it. Two things settle the tracking half for a deployment team, namely whether the workflow needs the headset to recognise an object a person is holding and moving, and whether the list of those objects is stable enough to freeze at build time. Both yes, and visionOS 27 does something visionOS 2 could not. Either one no, and the limit sits in Create ML on a Mac, not in the headset. The assistant is a separate approval, an English-only beta on a shared device that sends complex requests to Apple's servers ^[7][9].