Stage

A tiled floor in perspective holds a column, a screen and coloured balls with shadows beneath a lamp.

The question

The world

The models

Results

Post hoc

The page

The Mathematics mode

What this adds to world-first

Checkpoint

Limits

Stage

world-first asked, on a lattice gas, whether models that learn only from film can hold a world. This piece asks the same question of a world anyone can read: a small stage in linear perspective, with a tiled floor receding to a vanishing point, a lamp, a column, a screen and rolling balls. Its laws have everyday names. Things keep going, bounce off walls and knock each other. They keep their colour, keep existing when hidden, and cast shadows away from the lamp.

The video-first models match the stage at first and lose it. None of the three can keep what it cannot see, light a scene the way its archive never did, draw a colour its archive never showed, or leave out a part of the set its archive always had:

Recall plays what came next in the most similar archive clip.
The mosaic pastes archive tiles whose named neighbourhood matched.
The diffusion model denoises each tile toward the archive tiles that fit.

The world from a picture keeps a guessed state and the laws. It is lawful but not the world.

At first, and then not. At frame 4 the mosaic and the diffusion model have 90% and 87% of the balls within 3 pixels of the world's. Over frames 4–29 they keep 15% and 19% on new films, and about the same (19% and 20%) on films from their own archive.
Out of sight, out of existence. A ball that rolls behind the screen came back where the world has it in 12 of 12 films for the world. The world from a picture brought it back in 7; recall, the mosaic and the diffusion model in 0, 1 and 2.
What the archive always showed. With the lamp moved to the right, the video models' shadows overlap the world's at 0.03–0.04. A green ball gets not one green pixel from any of them. With the column taken away, recall and the diffusion model paint it back over 99.6% and 99.7% of its spot. The mosaic, which ignores position, paints 19.9%.
Lawful, not the world. The world from a picture guesses each ball from four frames: 6.4 cm off in position and 28% in speed (medians), and 10 of 50 balls were unseen at frame 3 and so do not exist for it. It keeps 48% of the balls with the world on new films.
Receipts. Recall copies archive frames, from 2.1 films per run on average. When the world shows a ball in a tile, only 15.3% of the mosaic's pasted tiles came from an archive tile of that same ball. The diffusion model's attribution narrows to one archive tile where it draws a ball and stays spread over 32 identical copies for the set.

Status, 28 September 2026. Of nine predictions written before any model code, five held and four failed (S2, S5, S8, S9). The interactive page shows the stage and the smoke side by side. A Mathematics mode, added the same night, works every step of every model by hand with the frame's own numbers; three checks written before its code held (M1–M3).

The world

Each of the four visual properties has one mechanism in the renderer, and each mechanism belongs to the world, not to the picture.

Property	Where it comes from
Linear perspective	a pinhole camera at (0, 3.3, −2.9) m sends a ray through each pixel. Equal floor tiles subtend smaller angles with distance, so their edges converge (Alberti's construction, computed rather than drawn).
Foreground and background	each ray stops at the nearest surface it meets, so a ball in front of the screen hides it and a ball behind the screen is hidden.
Form	each surface point is shaded by the angle between its surface and the lamp, so a ball is bright on the lamp's side and dark on the other.
Shadow	from each point a second ray goes toward the lamp. If a ball, the column or the screen blocks it, the point gets only ambient light.


Stage	floor 6 × 6 m of 1 m tiles, walls left, right and back, a pit in front. A column (radius 0.5 m, height 2.2 m) and a screen (2.1 × 2.0 m) stand on it.
Actors	two or three balls of radius 0.5 m, red, blue or yellow, rolling at 0.10–0.22 m per frame. They bounce elastically off walls, column, screen and each other, with collision times solved exactly (event-driven).
Lamp	a point lamp at (−2.2, 4.6, 0.9) m, up and to the left
Picture	64 × 48 pixels, each the mean of 2 × 2 rays. A shortcut re-renders only the pixels a ball or its shadow can reach; across 1,600 frames and both lamps it equals the full render pixel for pixel.
Archive	200 films of 30 frames, lamp on the left, set unchanged

The hide-and-seek film run by five models

The models

Every model is given the true frames 0–3 and makes frames 4–29. The three video models are the optima of their objectives on the archive, not networks trained toward them, so every tile they make can be traced to the archive tiles it came from.

Model	How it makes a frame
World	runs the laws one frame from the true state, then renders
World from a picture	inverse graphics. It finds each ball by its colour in frames 0–3 and tries floor positions until a drawn ball covers the same pixels. A least-squares line through the four positions gives the velocity. It then runs the same laws with the same set and lamp.
Recall	the archive frame that followed the nearest pair of consecutive archive frames (exact search, pruned by a lower bound on tile means)
Mosaic	cuts frames into 4 × 4 tiles and names each by the nearest of 64 codewords (k-means over the archive's tiles). Each next tile is drawn from archive tiles that followed the same named context: the tile and its four neighbours now plus the tile one frame earlier, backing off to shorter contexts. It pastes the drawn archive tile's own pixels and ignores position. This is a tokenised next-frame model, like those behind interactive video world models, at its empirical optimum.
Diffusion	for each tile position, the 32 archive transitions at that position whose surroundings one frame earlier were nearest, with a prior on that distance. It then takes 24 denoising steps from pure noise, under z = (1 − t) x + t ε, with the optimal denoiser for those 32 tiles. This is the locally optimal denoiser, with the tile as its window.

Results

At first, and then not

Balls kept where the world has them, frame by frame

At frame 4, one step after the given frames, the mosaic and the diffusion model have 90% and 87% of the balls within 3 pixels of the world's (S2 predicted at least 80% for both). Within a few frames they lose them. Over frames 4–29 they keep 15% and 19% on new films, and 19% and 20% on films from their own archive (S3). Recall keeps 17% on new films and all of them on archive films, which it replays exactly.

A new film run by five models

The mosaic shatters the balls into tiles: a ball becomes fragments of other balls, pasted wherever a neighbourhood looked right. The diffusion model keeps the set perfectly and turns the balls into smears that drift and fade. Each is doing what its objective asks. Neither has anything that could keep a ball whole and moving.

Out of sight, out of existence

A ball that rolls behind the screen is hidden for 5 to 23 frames. The world brought it back in all 12 films. The world from a picture kept a guessed state, so it had the ball to bring back, and did so at the right place in 7 of 12. Recall, the mosaic and the diffusion model brought it back in 0, 1 and 2 (S4 held). A hidden ball has no pixels, and a model whose state is pixels has nothing left to keep.

What the archive always showed

Three changes no archive film had

Change	World from a picture	Recall	Mosaic	Diffusion
Lamp moved right: shadow overlap with the world	0.28	0.03	0.04	0.04
Green ball: its pixels kept green	54.4%	0%	0%	0%
Column removed: its spot painted back	1.5%	99.6%	19.9%	99.7%

Recall's frames are archive frames, so they carry the archive's lamp, colours and column. The diffusion model chooses tiles per position, and at the column's position every archive film had a column, so it paints the column back. The mosaic ignores position; at the column's old spot it sees floor and wall and continues floor and wall, which is why it left the column out (19.9%, just under the registered fifth). None of the video models can make green: every tile they paste or blend came from a film without it.

Cause and effect

After a ball–ball collision the world sends both balls off along new lines. Of 24 struck balls in the new films, the world from a picture sent 6 the world's way, recall 2, the mosaic 5 and the diffusion model 0 (S8 failed on the world from a picture, predicted over 60%). A few centimetres of error in a guessed position change the angle at which balls meet, and the angle decides where they go. The video models do not collide balls at all. When a ball in their frames changes course, it is because an archive tile of a changing course was pasted there.

Receipts

Model	What a frame's receipt says
World	its state and the chain of events that led to it, e.g. "started at (−0.45, 2.21) m; bounced off the column at frame 5.0"
World from a picture	its guess from frames 0–3, with that guess's error, then the laws
Recall	one archive frame, "frame 14 of archive film 18": exact, and a copy
Mosaic	one archive tile per tile, with how many archive tiles shared its context and its codeword. The median tile shares both with 5,600 others, all of them one piece of set.
Diffusion	weights over 32 archive tiles at every step. Where its frame shows a ball, the weights narrow to one source (median 1.0 at the last step, post hoc); a tile of set stays spread over 32 identical copies (median 32 at the first step and at the last).

Concept by concept, the mosaic's receipts show what its names are. Its 64 codewords group the archive's tiles by colour, not by what they show. In the registered archive, codeword 17 names 1,887 tiles: 18% floor in the red ball's shadow, 17% red ball, 13% floor in the yellow ball's shadow and 12% floor in the blue ball's shadow. One name, four things the world keeps apart. When the world shows a ball in a tile, only 15.3% of the mosaic's pasted tiles came from an archive tile of that same ball.

Scored predictions

These were fixed before any model code existed.

ID	Prediction	Verdict	Evidence
S1	energy kept to 10⁻¹², reversal within 10⁻⁶ m, recall replays archive films exactly	held	7 × 10⁻¹⁶; 7 × 10⁻¹³ m; 20 of 20
S2	frame 4: mosaic and diffusion track ≥ 80% of balls, recall < 40%	failed	90%, 87%; recall 53%
S3	archive films: mosaic and diffusion track < 50%, recall all	held	19%, 20%; recall 100%
S4	hidden ball back: world 12 of 12, world from a picture ≥ half, video models ≤ a third	held	12, 7; recall 0, mosaic 1, diffusion 2 of 12
S5	lamp moved: video models' shadow overlap < 0.3, world from a picture > 0.5	failed	0.03, 0.04, 0.04; world from a picture 0.28
S6	green kept: video models < 5%, world from a picture ≥ 50%	held	0, 0, 0; 54.4%
S7	column painted back: recall and diffusion > half, mosaic < a fifth	held	99.6%, 99.7%; mosaic 19.9% (narrowly); world from a picture 1.5%, not none
S8	struck balls launched the world's way: mosaic and diffusion < 30%, world from a picture > 60%	failed	21%, 0%; world from a picture 25%
S9	recall one film in ≥ 18 of 20 runs; mosaic ≥ 50 films per frame; diffusion median sources ≥ 8 first, ≤ 1.5 last	failed	6 of 20; at least 110; 32 and 32

S2 failed the same way world-first's P3 did: among thousands of archive pairs, recall usually finds one with balls nearby, so its first frame is better than predicted (53%). S5 failed on its instrument. The shadow measure compares each frame with the empty stage under the same lamp, so it counts only the balls' shadows, and the world from a picture's shadows are only as good as its ball positions. S8 failed because the world from a picture's small errors decide collisions. S9 failed twice. Recall hops between films (2.1 per run) instead of latching onto one as it did on the gas. The diffusion model's set tiles have 32 identical candidates, so its attribution cannot narrow; that was seen after the predictions were written and before the run.

Post hoc

Written after scoring; none of this was predicted.

The mosaic found its full context (the tile, its four neighbours and its past) for 85.3% of tiles on new films, fell back to the tile now and before for 14.5%, and almost never further. Its failures are not from missing data: it has matching archive contexts for nearly every tile.
Pixel error is the wrong yardstick here. The set fills most of the frame, so the diffusion model, which keeps the set perfectly, has about the world from a picture's pixel error (3.6–4.5 per channel) while keeping far fewer balls (19% against 48%).
The world from a picture's guesses: median 6.4 cm in position and 28% in speed, from four frames of 64 × 48 pixels.
Over five new films, the diffusion model's tiles that show a ball end with a median of 1.0 effective sources (79% at 1.5 or fewer); its set tiles end at 32. Three of the mosaic's 64 codewords name tiles of at least two ball colours; its largest, codeword 42, names 101,086 tiles, 88% right wall and 12% back wall.

The page

The interactive page has two tabs, Stage and Smoke, in one viewing room.

Part	What it shows
The line	Drag it across the picture: the world on the left, the chosen model on the right, pixel for pixel.
How this frame was made	A slider through each model's steps for the current frame. For the world: state, depth, what each ray hits, shadows, picture. Recall: its last frame, then the closest archive pair, then what came next there. The mosaic: named tiles, then pasting row by row. Diffusion: static, then its guess at each of 24 steps. The smoke world also shows its hidden particle directions, collision by collision.
Notes and subtitles	In plain language, frame by frame: what the world did and why ("Red hits blue. Each changes course because the other hit it"). Where each model departs from the world, with the reason in its mechanism ("blue is gone, because at this spot the archive tiles that fit best showed no ball"). Also where a model keeps up, and why it can.
Notes on the picture	Rings around the world's balls, dashed where a ball is hidden, with arrows for the next five frames. On the model side, marks where a ball is gone, off, or shown when it should be hidden.
Where did this come from?	Click any pixel. For the world: what is there, what shades it, and the ball's history of collisions. For each model: its source archive frame or tile with a thumbnail, what that source showed, how many archive tiles were equally good, and for diffusion the weights over its 32 candidates at the current step, grouped by what they showed.
What each model keeps from the archive	Attribution in training: what each model stores. The mosaic's 64 codewords are clickable, each with the archive tiles it groups and what they showed.

The page trains its video models in the browser on the first 30, 60, 100 or 200 archive films; with 200 they are the registered ones.

The Mathematics mode

The request: show the matrix multiplication, linear and algebraic geometry and calculus that people recognise from school, since at the end of the day that is all these models are, as a second experience for the mathematically inclined.

The switch at the top of the page, Pictures | Mathematics, turns the same room into that second experience; an address ending in #math opens it directly. The step slider still walks through how the chosen model makes the frame. For each step, the caption and subtitle give one equation, and a chalkboard under the picture works it with this frame's numbers. The side panel works one pixel by hand, ray by ray, and compares the result with the rendered bytes. The picture carries the constructions: the horizon, the vanishing point and the floor's receding lines, each ball's outline as the ellipse it is, rays from the lamp with each cast shadow's outline, the edge of each form shadow and each highlight.

Model	Mathematics on the chalkboard
World	position as a straight line in time (dp/dτ = v); the next collision as the earliest root of a linear or quadratic equation; energy and momentum kept, with the last collision's exchange of velocities checked against the world's; the camera as a rotation by 27.3° and a division by depth, and K M as one matrix; Pythagoras for the depth picture; vanishing points as K M v, including Alberti's distance points 48.7 px either side of the centre and the verticals meeting 60 px below the picture; each ray as linear equations for the set and quadratics for the balls and the column, the nearest root winning; each ball's outline as a conic, an ellipse because B² − 4AC = 4q(R² − w_f²) < 0; shadows as the same projection from the lamp (similar triangles); the form shadow's edge as a circle (Thales); Lambert's cosine law and gamma
World from a picture	overlap maximised by a grid of 794 trial drawings, since the score has no slope to follow; the least-squares line where the derivative is zero; its error growing along a straight line between bounces
Recall	a frame as a point in 18,432 dimensions; Pythagoras; the bound 16 d̄² ≤ Σ d_i² that lets it skip most distances
Mosaic	12 numbers per tile; the nearest of 64 codewords; k-means, the mean being where the derivative vanishes; P(y \| context) = N(context, y) / N(context); one uniform random number
Diffusion	noise and picture as the two ends of a line; the prior; Bayes' rule with a bell curve; the weighted average x̂ = Xᵀw; Euler's method; Tweedie's formula, ε̂ = −t ∇ ln p_t(z)
Smoke	the state as 65,536 bits; collision as a 16 × 16 permutation with C² = I; motion as a permutation matrix with Sᵀ = S⁻¹; the camera as a 1,024 × 65,536 matrix of rank 1,024, which by rank–nullity cannot see 64,512 directions; the number of states behind one picture, Π C(64, y_k): 10^5,423 for frame 0 of the word WORLD, and from 10^4,939 to 10^10,997 (median 10^8,267) for frame 0 of the 20 new films, out of 2^65,536 ≈ 10^19,728 states in all; the least-squares inverse Πᵀy/64; turning time around, T F T = F⁻¹ with T = RC; the world from a picture as a random point among those states, with Π F n′ ≠ Π F n; the learned rule as a table of 2¹⁶ rows keyed in base 16; the mosaic's context as a number in base 8

Checked against the programs

The chalkboard's arithmetic is separate code. It never calls the renderer: every surface a ray can meet is its own equation, solved in the renderer's order with the same floating-point steps, so the bytes can be compared exactly. M1–M3 were written before that code existed, and all three held (the check runs in 3.5 s):

ID	Prediction	Verdict	Evidence
M1	every pixel of 40 frames, worked ray by ray, equals the renderer's bytes and labels	held	368,640 of 368,640 bytes; 245,760 of 245,760 labels
M2	every ball's outline is an ellipse; largest axis ratio 1.10–1.25, median 1.01–1.06; the centre displaced outward by under half a pixel	held	15,120 of 15,120 ellipses; largest 1.177 (sec θ = 1.175) for a ball at x = 2.50, z = 0.51, the front right corner; median 1.025; all outward, by at most 0.36 px
M3	the recomputed weights and Euler steps equal the model's to 10⁻¹²	held	weights identical; Euler steps within 9 × 10⁻¹⁶; the last step equals the last guess exactly

Not registered: the next collision, solved as the earliest root from the current state, matches the simulator's first event in 6,000 of 6,000 archive frames, to 1.3 × 10⁻¹³ frames. The page repeats the comparisons live: every card that can be compared with the program carries ✓ or ✕, and a pass through every step of every model in both worlds found no ✕. The hand estimates written with M2 before any code, 1.17 to 1 at the front corners and 0.36 px of displacement, matched to two places.

What it shows

The camera is the whole of linear perspective. One 3 × 3 matrix and one division place every point, and the vanishing points are the same matrix applied to directions. Alberti's tiled floor, its horizon and its distance points follow from M. The shadows follow from the same division, with the lamp as the eye.
Spheres are drawn as ellipses, by less than a pixel. At 64 × 48 the largest departure from a circle is 1.18 to 1 on a radius of about 4 px; the overlay draws the exact conic at the screen's scale. Strict perspective stretches an off-centre sphere away from the picture's centre, and painters often draw a circle instead.
Same arithmetic, different objects. Recall is Pythagoras over 1,680 points, the mosaic is counting, and the diffusion model is a weighted average whose weights come from a bell curve. The world's quadratics are about balls; the video models' sums are over archive pictures. The mathematics is not what separates them. What it is applied to does.
School pieces, composed. Each collision is one quadratic, but which collision comes next depends on every earlier root, and a few centimetres of error grow at each bounce off a curved surface (S8). To know where a ball is at frame 29, the arithmetic has to be done event by event. The chalkboard shows one step; a film is thousands of them.

Checkpoint

Check	Answer
What did the subject force to change?	The cast-shadow outline first followed each grazing ray from the lamp and drew a ball's cone onto the column standing between them; it now starts where the ray touches the ball. The least-squares card had to admit bounces: in one film the blue ball bounced off the front edge during frames 0–3, its fitted speed was 133% off, and the card now says when no straight line fits.
What does it leave unestablished?	That anyone learns from it. The checks establish that the numbers are the programs' numbers, not that the chalkboard explains them.
Can a stranger enter?	Pictures stays the default. Mathematics assumes school algebra and says each step in words as well as symbols.
Can a participant change the terms?	Any pixel, step, model, frame or scenario; the chalkboard and the worked pixel follow.
Global verdicts?	None: three checks held, and the limits below stand.

Limits

The chalkboard works one example per step: the clicked pixel, or the middle of the first ball the world shows, and for the tile models the tile under it. The programs do the same arithmetic for every pixel and tile.
Displayed numbers are rounded to two to four digits; the checks compare the unrounded ones.
The mode makes no new claim about the models. Its one claim is that its numbers are the programs', which M1 and M3 test on 40 frames and 5 diffusion frames and the page re-checks on the frame shown.

What this adds to world-first

The same failures, with everyday names. On the gas the failures were conservation, steering and reversal. On the stage they are a ball that vanishes behind a screen and never returns, and shadows cast by a lamp that has moved. They also include a colour that cannot be drawn, a column that returns after it was taken away, and a collision with no consequence. All come from one cause. A model whose state is the picture can carry forward only what the picture shows and what its archive showed, and the laws live in what the picture does not show: the hidden ball's position, the lamp's place, the ball's velocity.

Emergence is a collage of instances. When a video model does something that looks lawful, such as a ball rolling straight or bouncing, the receipts show where it came from: archive tiles in which a ball rolled or bounced. Nothing in the model computes the roll. This is why the lawful-looking stretches end: the next context has no matching instance, or the drawn one belongs to a different ball.

Attribution is exact and still not causal. Every tile the mosaic pastes names its archive source, yet that source is a reason only in the sense that its neighbours looked alike. When the world shows a ball in a tile, the source showed that same ball 15% of the time. The world's receipt is a chain of causes: this ball is here because it bounced off the column at frame 5.0. The diffusion model's weights are the optimal denoiser's attribution made visible, and they show its limit. Where the archive holds identical copies, and a set is mostly identical copies, no weight can pick one.

The world from a picture is the honest middle. It is what "world first from video" can mean: perceive, guess a state, run laws. It keeps hidden balls and obeys the lamp. Its errors come from perception (a few centimetres, a third of the speed) and grow at every collision, and a ball it never saw does not exist for it. That is pictorial-form's result in miniature: a picture does not fix the state, even for a model with the right laws.

Checkpoint

Check	Answer
What did the subject force to change?	The camera had to be moved and the screen raised before anything could be hidden (a high camera sees over a low screen). The world from a picture needed four frames, not two, because one pixel spans 0.1–0.3 m of floor. The shadow measure turned out to count only ball shadows, which is why S5 failed on the world from a picture.
What does the compelling result leave unestablished?	The shattered mosaic and the returning column are properties of these optimal models on a 64 × 48 stage. Trained networks sit between recall and the mosaic with biases of their own; training one is the open test.
Can a stranger enter?	The page opens on hide and seek with the world and the diffusion model side by side, and the subtitles say what is happening in plain language.
Can a participant change the terms?	The viewer picks the scenario and the archive size, drags the line, steps through any model's frame, and clicks any pixel. The smoke tab adds your own word, emptying gas, and turning time around.
Global verdicts?	5 of 9 held, 4 failed. The failure that bears on the claim is S8, which shows the world from a picture's lawfulness has limits: right laws, wrong state.

Stage?

The question?

The world?

The models?

Results?

Post hoc?

The page?

The Mathematics mode?

What this adds to world-first?

Checkpoint?

Limits?

Stage

The question

The world

The models

Results

Post hoc

The page

The Mathematics mode

What this adds to world-first

Checkpoint

Limits