Build1 publisher2 min readPublished
Runway's WorldPrompt turns a world-model prompt into an event timeline
Runway's WorldPrompt specifies a GWM Worlds 2 scene as a fixed first frame plus timestamped events, played out live at 24 frames per second. Authors get a timeline, but Runway's own researcher says how well the model follows it depends on the action.
The Engineer · Build desk
What happened
- Runway released GWM Worlds 2 as a research preview, describing it as an autoregressive diffusion model that turns video and audio generation into real-time interactive simulation.
- WorldPrompt, its proposed input format, lets new timestamped events be prompted while the world is already running, in addition to those written in advance.
- Latent Space reports the format is a prompting mechanism with no scripting or state control, unlike the scriptable worlds of Minecraft or Roblox.
- Runway fine-tuned its foundational audio-video model to follow the WorldPrompt format, then post-trained it to generate autoregressively.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Game logic that branches on world state, such as a door that opens only after a key is picked up, cannot be expressed in a WorldPrompt timeline today.
- decision Teams building on the preview have to lean on movement events and test each harder action before a timeline depends on it.
- capability Frame-by-frame generation lets an event typed mid-session steer frames not yet rendered; a model that emits a whole clip at once has already committed them.
- cost Any richer event handling Runway adds has to fit inside an average budget of about 41.7 ms per frame while holding audio in sync.
The spec has two parts. The first is initial conditions: the aspects of the environment a user pins down, including the opening frame [3]. The second is a list of timestamped events, and the list stays open for additions while the world runs [3]. A game level file has the same shape, a start state plus a schedule.
The comparison breaks down at state [4]. A level file can check a flag before an event fires. A WorldPrompt event has no state to check, so it cannot be made conditional on whether the previous one happened [4].
Runway CTO Kamil Sindi pointed to what the prompting approach gets you [2]. "You can create promptable worlds on-demand with video and audio in sync, across all these different domains and environments. That's not a distant-future hypothetical thing," he said [13].
Each event is still a request, and the model follows some kinds more often than others. "It really depends on how difficult the action is. I would say movement works quite reliably," said Robin Kahlow, Runway's principal research scientist for generative video and multimodal AI [5][2]. Sindi said more training and scaling the data and models is producing "better following" [6].
The open timeline depends on how the model generates. Kahlow said one of the two problems is "making the model not generate a whole clip at once. So instead, you want it to generate frame by frame while you're looking at it" [9]. Runway taught the base model the format before it made the model autoregressive [8]. So the event list is something the model learned to read. Frame-by-frame output is what leaves unrendered frames for a live event to steer [8][9].
The second problem is speed [9]. GWM Worlds 2 streams 720p video at 24 frames per second with audio at 48,000 Hz [7]. That leaves about 41.7 milliseconds per frame on average, with 2,000 audio samples per frame to keep in sync [1][2].
Session length is the other limit for this class of model. Google says its Genie 3 "can currently support a few minutes of continuous interaction, rather than extended hours" [10].
In my view the format is the right call for a research preview. It gives authors a timeline without promising behaviour the model cannot deliver. Programming these worlds would take two more things: harder actions followed as reliably as movement, and some state an event could condition on. Kahlow names movement as the reliable case today [5].
What to watch
- Whether Runway turns WorldPrompt from a 'proposed' input format into a published schema that third-party tools can write events against.
- Any state readback or scripting hook in a later GWM Worlds release, the missing piece for conditional events.
- Whether training and scale extend reliable following from movement to harder actions, as Sindi says is already happening.