HumanWorldModels.com

From world models to models of people

The modern sense of world model comes from a 2018 paper by David Ha and Jurgen Schmidhuber: a network learns "a compressed spatial and temporal representation of the environment", an agent trains inside it, and the policy transfers back to the real one.1 The Dreamer line scaled it — behaviour improved by imagining futures inside a learned model, across more than 150 tasks with one configuration, in Nature in 2025.2 A 2022 position paper by Yann LeCun puts a configurable predictive world model at the centre of an architecture for machines that plan at several horizons.3 Joint-embedding predictive architectures predict in representation space, not pixels;4 their video successor reports human action anticipation among its headline results.5

What changes when the environment is people

A world model of a concourse or a hallway is mostly a model of the people in it, and they are the hardest part of the scene. Their motion is multimodal: given the same history, "there are many socially plausible ways that people could move in the future".11 They act on goals nobody can observe — theory-of-mind work infers an actor's beliefs, desires and percepts from how they move through a partly seen space,14 and machine theory-of-mind models build models of other agents from behaviour.15 And people react to the machine, so a prediction is useful only when conditioned on what the machine is about to do.12

How people are captured, and forecast

Prediction needs a shared way to describe a body. A learned, skinned body model from 2015 became the parameterisation that lets incompatible capture sets be compared;7 AMASS used it to unify 15 marker-based datasets into over 40 hours of motion from more than 300 subjects.8 Forecasting has its own lineage: the 1995 social force model treats walking as acceleration towards a desired velocity plus repulsion from people and obstacles,9 and learned methods grew out of bird's-eye video annotated for 2009 tracking work, the origin of the field's long-running pedestrian benchmark.10 Later models predict a distribution of futures rather than one line,11 add dynamics and map context,12 and separate intent from position: one traffic dataset labels 1,842 pedestrians over 300,000 frames and reports that crossing intention improves the predicted path.13

Machines that have to predict people

Social navigation is where these models are spent. One dataset records it from the machine's own viewpoint: a mobile robot's sensors in everyday buildings, 2.3 million boxes and over 3,500 consistent human tracks.16 A 2023 survey in ACM Transactions on Human-Robot Interaction argues that crowded public space draws on psychology, design research and sociology as much as on engineering.17 Simulators now treat people as residents, with humanoid avatars, human-in-the-loop control and tasks named Social Navigation and Social Rearrangement.18 Generative video world models are built for other safety-critical domains6 and interactive generated environments learn controllable dynamics from unlabelled video.19

What is still unsolved

Long-horizon consistency: in a controlled 2D physics setting, video models generalised well inside their training distribution but failed outside it, imitating the nearest training example rather than learning a rule; the authors conclude that "scaling alone is insufficient".20 Causality: an ICLR 2024 result shows that any agent staying within a regret bound across many distribution shifts must have learned an approximate causal model.21 Evaluation: the social-navigation survey treats evaluation methodology as an open category of its own.17 Provenance and consent: human data is people's data. One 1,286-hour, 740-participant capture states it was recorded with formal consent under a privacy and ethics policy, and is released under a signed licence.22 Under the EU AI Act, in force since 1 August 2024, some biometric and emotion-recognition uses sit in the prohibited tier that applied from February 2025.23

Who builds around human world models

The same list, read as buyers:

• Research groups in world models, human motion, theory of mind or social navigation.17
• Robotics companies whose machines move where people are.16
• Simulation and synthetic-data teams, where simulated people are benchmark material.18
• Game and interactive-media studios, where believable crowds are the human half of a world.19
• Dataset and benchmark efforts, the shared infrastructure of the field.8
• A publication or conference, where the plural reads as a subject, not a product.

Papers say world models, human motion prediction, pedestrian trajectory forecasting and social navigation for pieces of one problem. Human world models covers all of it in three words.

1. Ha D, Schmidhuber J. World Models.arXiv:1803.10122, 2018
2. Hafner D et al. Mastering diverse control tasks through world models.Nature 640, 2025
3. LeCun Y. A Path Towards Autonomous Machine Intelligence.OpenReview, 2022
4. Assran M et al. Self-Supervised Learning with a Joint-Embedding Predictive Architecture.arXiv:2301.08243; CVPR 2023
5. Assran M et al. V-JEPA 2: Video Models for Prediction and Planning.arXiv:2506.09985, 2025
6. Hu A et al. GAIA-1: A Generative World Model.arXiv:2309.17080, 2023
7. Loper M et al. SMPL: A Skinned Multi-Person Linear Model.ACM Trans. Graphics, 2015
8. Mahmood N et al. AMASS: Motion Capture as Surface Shapes.arXiv:1904.03278; ICCV 2019
9. Helbing D, Molnar P. Social force model for pedestrian dynamics.Physical Review E 51, 1995
10. Pellegrini S et al. You'll Never Walk Alone.ICCV 2009, ETH Zurich
11. Gupta A et al. Social GAN: Socially Acceptable Trajectories.arXiv:1803.10892; CVPR 2018
12. Salzmann T et al. Trajectron++: Trajectory Forecasting.arXiv:2001.03093; ECCV 2020
13. Rasouli A et al. PIE: Pedestrian Intention Estimation.ICCV 2019
14. Baker CL et al. Rational attribution of beliefs, desires and percepts.Nature Human Behaviour 1, 2017
15. Rabinowitz NC et al. Machine Theory of Mind.arXiv:1802.07740; ICML 2018
16. Martin-Martin R et al. JRDB: Robot Perception of Humans.arXiv:1910.11792; IEEE TPAMI
17. Mavrogiannis C et al. Core Challenges of Social Robot Navigation.ACM Trans. Human-Robot Interaction, 2023
18. Puig X et al. Habitat 3.0: Humans, Avatars and Robots.arXiv:2310.13724; ICLR 2024
19. Bruce J et al. Genie: Generative Interactive Environments.arXiv:2402.15391; ICML 2024
20. Kang B et al. How Far is Video Generation from World Model?arXiv:2411.02385, 2024
21. Richens J, Everitt T. Robust agents learn causal world models.arXiv:2402.10877; ICLR 2024
22. Grauman K et al. Ego-Exo4D: Skilled Human Activity.arXiv:2311.18259; CVPR 2024
23. Regulation (EU) 2024/1689, the AI Act.European Commission