4DCodeBench: top coding agent scores 0.91 on static metrics but 0.67 on dynamic ones when recreating scenes from video
4DCodeBench, a 200-scene benchmark from Stanford's Ziga Kovacic and collaborators, finds the top coding agent scores 0.91 on static metrics but 0.67 on motion. Teams using agents to turn video into simulation code should test the motion on its own.
The Engineer · Build desk
Ranking of how coding agents represented motion across 18 evaluated configurations: analytic motion 67% of solutions, custom simulation 19%, Blender physics 10%, keyframing 3%.
Share of solutions across 18 evaluated configurations In % of solutions
| Rank | Item | Value | Claim |
|---|---|---|---|
| 1 | Analytic motion | 67% of solutions | 5 |
| 2 | Custom simulation | 19% of solutions | 5 |
| 3 | Blender physics | 10% of solutions | 5 |
| 4 | Keyframing | 3% of solutions | 5 |
What happened
- Agents must write code that regenerates each scene without seeing the reference video, and may script the movement, write their own solver or call physics tools such as Blender's.
- Half the scenes are real videos and half are physics simulations, spanning rigid bodies, deformable objects, cloth and rods, grains, fluids, fracture and mixed-material interactions.
- Across 18 evaluated configurations, 67% of solutions described motion analytically, 19% used custom simulations, 10% used Blender physics and 3% used keyframes.
- In a human check, 76 participants made 3,587 pairwise comparisons, and people agreed with the vision-language-model judge on 89.3% of the 916 comparisons both evaluated.
- Kovacic and his coauthors report that 90.1% of submissions ran end to end and produced output.
Why it matters
- decision A render or shape that passes review says little about forces and contacts, so teams using agents to write simulation or video-to-3D code need a separate test of motion against per-frame ground truth.
- constraint Automated judgments were least dependable when two configurations were close, so choosing between two similar agents on in-house scenes still needs a human to look at the close calls.
- cost Reproducing or extending the evaluation requires a GPU environment plus the scorer's dependencies and model checkpoints before any in-house comparison can start.
We take an analytic representation to mean motion written as a function of time, with no solver stepping forces and contacts forward. With the keyframed solutions included, 70% of the agents' answers involved no physics simulation at all [16]. Custom solvers and Blender's physics together came to 29% [17].
The top-ranked configuration was GPT-6 Astra at Max reasoning effort [7]. Its simulation use rose from 13% at Low effort to 32% at Max [6]. About two-thirds of the leading configuration's solutions still did not simulate [18]. Claude Opus 5.5 went the other way, using custom or Blender simulation in 76% of its solutions [6]. Runtimewire's account does not give Claude's scores, so the published figures cannot settle whether simulating more often produces better motion.
The two scores are benchmark scores across metric families, not shares of scenes reconstructed correctly [7]. The gap between them is 0.24 [19]. For that gap to mean an agent is worse at motion than at shape, the static and dynamic families have to sit on comparable scales. In our view the direction holds up even before that is checked. A formula fitted to a clip can match the frames it was fitted to while encoding no contact or force. According to Runtimewire, a convincing rendering or plausible static shape does not guarantee the agent recovered the forces, contacts or deformations governing what happens next [8].
The dataset design is careful work. Real videos carry real visual complexity but no ground-truth 3D geometry or motion, so they are judged in image space [9]. The synthetic half supplies exact geometry and motion at every frame, so it can be compared directly in 3D and over time [9]. For a team checking an agent's motion, the synthetic half is the part that measures it directly. Runtimewire cautions that results across the two settings should not be read as one verdict on an agent's physical understanding [9].
For the numbers to transfer to a production pipeline, the work has to look like the benchmark's: one video, a fixed prompt, a graphics program and a defined scoring pipeline [13]. The benchmark does not show whether an agent could run a robot, produce a faithful digital twin or reliably work out physics beyond its own scenes [13]. The paper went up on arXiv on October 2 [2]. The authors' institutions are Stanford, MIT, Johns Hopkins, the Max Planck Institute for Intelligent Systems and Peking University [15].
What to watch
- Whether the paper's per-model tables show Claude Opus 5.5, which simulated in 76% of solutions, beating or trailing GPT-6 Astra at Max on the dynamic metrics.
- Whether outside teams run new agents through the public pipeline and report dynamic scores that narrow the 0.24 gap.
- Whether follow-up work scores agents on predicting frames beyond the observed clip, where analytic fits and real simulations would be expected to diverge.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap0
- Incentives
- Insufficient
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Stanford PhD student Ziga Kovacic and collaborators introduced 4DCodeBench, a benchmark testing whether coding agents can turn a video of a physical event into executable graphics code reproducing the scene's geometry, appearance and movement.
- [2]
The 4DCodeBench paper appeared on arXiv on October 2nd; Kovacic described the project in a thread on X on October 5th.
- [3]
The benchmark contains 200 scenes: 100 real-world videos and 100 synthetic scenes created with physics simulations, covering rigid bodies, deformable objects, cloth and rods, grains, fluids, fracture and interactions among materials.
- [4]
Agents must produce code that can regenerate the scene without access to the reference video; the implementation is left open, so a system can script movement, write its own solver or use physics tools such as Blender's.
- [5]
Across 18 evaluated configurations, 67% of solutions represented motion analytically, 19% used custom simulations, 10% used Blender physics and 3% used keyframing; the shares total 99% due to rounding.
- [6]
Claude Opus 5.5 used custom or Blender simulation in 76% of its solutions; GPT-6 Astra's simulation use rose from 13% at Low reasoning effort to 32% at Max.
- [7]
The top-ranked configuration, GPT-6 Astra at Max reasoning effort, scored 0.91 on the benchmark's static metric families and 0.67 on its dynamic families; these are benchmark scores, not percentages of scenes reconstructed correctly.
- [8]
A convincing rendering or a plausible static shape does not guarantee that an agent has recovered the forces, contacts or deformations governing what happens next.
- [9]
Real videos capture actual visual complexity but lack ground-truth 3D geometry and motion, so they are judged in image space; synthetic scenes provide exact geometry and motion at each frame, allowing direct 3D and temporal comparisons. Results across the settings should not be read as a single verdict on an agent's general physical understanding.
- [10]
In 3,587 pairwise comparisons from 76 participants, human and vision-language-model judgments agreed on 89.3% of 916 comparisons shared between the two evaluations.
- [11]
Agreement was stronger between rankings aggregated at the model level; individual automated judgments were less dependable when two configurations were close.
- [12]
Kovacic and coauthors report a 90.1% end-to-end execution rate, meaning most submissions produced runnable outputs.
- [13]
The benchmark measures a constrained task (one video, a fixed prompt, a graphics program and a defined scoring pipeline) and does not establish that an agent can control a robot, build an accurate digital twin, or infer physics reliably outside the benchmark.
- [14]
Running the supplied evaluation pipeline requires a GPU-equipped environment; evaluating new outputs also requires the scorer's dependencies and model checkpoints.
- [15]
The paper, code and benchmark materials are public; the work is an academic collaboration with authors affiliated with Stanford, Johns Hopkins, MIT, Peking University and the Max Planck Institute for Intelligent Systems.
- [16]
70% of solutions represented motion without physics simulation (analytic plus keyframing).
- [17]
29% of solutions used a physics simulation, custom or via Blender.
- [18]
About 68% (two-thirds) of GPT-6 Astra at Max's solutions did not use simulation.
- [19]
The top configuration's dynamic score trails its static score by 0.24.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- runtimewire.com4DCodeBench finds coding agents struggle to reconstruct motion from video
1 article · October 10, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI BenchmarksFollow
- Video-to-3D reconstructionFollow
- Physics simulationFollow
- Coding AgentsFollow
Entities
- 4DCodeBenchFollow
- Ziga KovacicFollow
- GPT-6 AstraFollow
- Claude Opus 5.5Follow
- BlenderFollow
- Stanford UniversityFollow