Reading the simulator framebuffer on a damage signal turns an agent's look into a 6.7 ms read

Cheap capture does not buy much on its own. In the OSWorld-Human measurements the author cites, planning and reflection calls to the model take 75-94% of total task time ^[6]. That leaves between 6% and 25% for capture, taps and the rest of the harness ^[1]. Cutting one look from a 130 ms blocking screenshot to a 6.7 ms read removes about 123 ms ^[2], and that saving lands inside the small slice. The 31x only shortens a task where it deletes a model call: screen memory lets the agent act on an intent such as "the Settings tab" without re-reading the screen ^[15]. Each step carries the whole observation history, so a look that never happens is an image that is never sent in later requests ^[7].

The unit change is the better idea. The daemon has no frame rate; it captures when the simulator reports a damage signal, about 52 per second while something moves and nothing while the screen is still ^[13]. Fifty-two per second is one change roughly every 19 ms ^[3]. A 130 ms blocking screenshot yields about 7.7 captures a second, close to one in seven of those signals ^[4], so the first version could not have tracked an animation even in principle. The author's CI check asserted that a frame counter had advanced on an idle screen, and it failed three times before he understood why ^[14].

Inside the 6.7 ms, the capture primitive is about a thousand times faster than a screenshot ^[12]. Against the post's own 130 ms figure that is on the order of 0.13 ms ^[5], and almost all the remaining time is the downscale ^[11]. For anyone copying this, the next thing to optimise is the resize. The 31x for the same work with 6.7 ms left implies a baseline near 208 ms ^[6], above the 130 ms call the loop version replaced ^[10].

Two conditions have to hold before those numbers transfer. The environment has to let you read a framebuffer without going through a screenshot command ^[11], and it has to tell you when the screen changed ^[13]. simframe gets both from the iOS Simulator. The post does not say what hardware produced the timings.

The author is direct about what he gave up. "I am not as precise as the model. It never mis-taps a control it can see; I do," he wrote ^[16]. The design's answer is that the newest frame is always held, so checking a remembered control position against the screen is a read ^[9]. "But I am much faster, and after a while I realised the speed came from architecture, not ability," he wrote ^[17]. "I ended up with three parts, and I think the vocabulary is more reusable than the code," he wrote ^[18]. That lands against a field where almost every framework, including the computer-use loops the model labs ship, still treats seeing as a function the model calls ^[5].