Rishabh Sharma, Gijs Hogervorst, Wayne E. Mackey, David J. Heeger, Stefano Martiniani
New York University · Statespace
Cross-View World Models (XVWM) predict future frames from any viewpoint given context
from another — egocentric, bird’s-eye, over-the-shoulder, or front-facing —
and in doing so develop a view-invariant spatial code: an internal “GPS” that
localizes and orients from visual cues alone. This poster was presented at the
ICLR 2026 Workshop on World Models (Understanding, Modelling and Scaling) in
Rio de Janeiro, April 2026, and extends the
arXiv preprint
with layer-by-layer analyses of the model’s internal representations.
Probing the internal representation: a ring manifold for heading
The two animations linked by the QR codes on the poster probe how the model encodes
heading. A linear probe trained to read heading from the hidden state defines a
two-dimensional readout plane (a cos ψ axis and a sin ψ axis); each
panel shows hidden states projected onto that plane, layer by layer, colored by the
agent’s original heading, while an injected heading change Δψ (yaw)
sweeps back and forth (dial on the right).
Matched view — heading is stored on a ring, and injected yaw rotates it.
Egocentric hidden states in the ego probe plane. Across the early layers a ring
slowly begins to take shape, but it is indifferent to the injected yaw. In the
middle transformer layers the states organize onto a clean ring manifold ordered
by heading — a circular code. Only there, after the view-invariant code
has formed (next animation), does sweeping Δψ rigidly rotate it: the
model updates heading by rotating this ring — angular-velocity
integration, carried out in representation space. The poster’s QR code
links to the GIF version
of this animation.
Cross-view — the same ring in the same basis: the code is view-invariant.
Bird’s-eye-view hidden states projected onto the same readout
plane: the probe was trained only on egocentric activity, and is applied to the
BEV states unchanged — no re-fitting. In the early layers nothing
transfers — the features are view-specific. In the middle layers —
the view-invariant section of the network — the same ring reappears,
ordered the same way, and the injected yaw rotates it just as rigidly: one
linear readout captures heading in both views. The ring is a shared code, not a
view-specific artifact. The poster’s QR code links to the
GIF version of this
animation.
Videos from the paper
The model in action — one context, predictions in both views.
Each row is one input view: egocentric on top, bird’s-eye (BEV) below. The
left column is the ground-truth context; from it, the model predicts the scene
0.2 s ahead at each instant, rendered egocentrically (middle) and in
bird’s-eye view (right). Watch the sky in the bottom row: the BEV context
carries no information about the sky, so the model samples one — a
one-to-many prediction, and evidence that it has learned spatial structure
rather than memorized input–output pairs.
Following the path on the map — the internal “GPS.”
The agent walks a path the model sees only egocentrically (left). Middle: the
true bird’s-eye trajectory; right: the trajectory the model imagines from
the egocentric input, traced over time. Egocentric movements transfer
consistently to the imagined map — two example episodes.
Egocentric
Bird’s-eye (BEV)
Over-the-shoulder
Front-facing
The four viewpoints of the dataset.
One minute of gameplay recorded simultaneously from the four cameras, along
with the agent’s actions. 4,186 such minute-long episodes make up the
training data.
Citation
@inproceedings{
sharma2026crossview,
title={Cross-View World Models},
author={Rishabh Sharma and Gijs Hogervorst and Wayne Mackey and David Heeger and Stefano Martiniani},
booktitle={ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling},
year={2026},
url={https://openreview.net/forum?id=0g7De0sXWM}
}
