A coffee chat
Last week the phrase "world model" came up over coffee with a friend. I'd started explaining it with great confidence when I realized I was still figuring it out myself. So this post is a continuation of that chat. Coffee in hand, you across the table.
First, a disclaimer: I'm no expert. Just someone who's been following this for a few months and occasionally catches himself staring at the screen.
So what is it, really?
The language models we use today are built on text. They've read billions of pages and learned to write, but they've never seen a glass fall off a table. They don't know physics; they know what's been written about physics. There's a difference.
World models aim at exactly that gap. They're systems that build an internal picture of how the world works and can answer "what happens if I do this" in their own head. Let go of the ball, it drops. Push the door, it opens. Turn right, and the table is now on your left. Ordinary stuff for us, nothing ordinary about teaching it to a machine.
My friend asked, "so, like a game engine?" No. In a game engine we write the rules. Here the model works the rules out by itself, by watching.
Who's chasing it?
The name behind the idea for years is Yann LeCun. While everyone at Meta ran toward language models, he kept insisting this wasn't the road, that machines should learn like babies, by watching. Their V-JEPA work does just that: It watches video and figures out how objects move on its own. Like a baby learning a ball doesn't vanish when it rolls behind the couch. Nobody teaches it; it learns by looking.
Then Google DeepMind's Genie came out, and I sat staring at it for a while. You give it one photo or one sentence and it generates a world you can walk through. There's no pre-drawn map. When you turn right, the model decides on the spot what's on the right. A bit dreamlike, a bit unsettling.
Fei-Fei Li's World Labs is focused on persistent 3D worlds. NVIDIA's Cosmos is mostly for robots: Fall a thousand times in your head instead of in the real world. Wayve is on the self-driving side, Runway on video. Everyone's looking through a different window at the same door.
Why should we care?
Teaching a robot to walk in the real world is expensive. Every attempt breaks an arm or burns out a motor. A robot that can rehearse in its head has already practiced by the time it wakes up. Same math for self-driving cars: The crash you never meet on the road, you can live through a thousand times in simulation.
Maybe more important is this: Perhaps "understanding" is exactly that. Understanding something means being able to predict its next step. That's how children learn. That's how we learned.
Of course there's criticism on the table, and my friend went straight for it. These models can't hold things together for long; wander around for a few minutes and the world "melts," the door you just saw is gone. And the real question is still open: Do they actually understand, or are they just very good video predictors? Honestly, nobody has a clear answer yet.
The coffee's finished, the conversation isn't
Where I've landed: Language models gave us something that talks. World models might give us something that looks. We'll know in a few years which one was bigger.
The chat ended there, and so did the coffee. But the question is still open for me. What do you think, do they really understand?
Computers Have Started Imagining the World
Language models learned by reading text; world models try to learn by watching the world. Meta's V-JEPA pulls physics out of video, DeepMind's Genie turns a single image into a world you can walk around in, and World Labs and NVIDIA Cosmos are chasing the same goal for 3D spaces and robots. The promise is big: Robots and self-driving cars could run their experiments in their heads instead of on the street. But these models still can't stay consistent for long, and the question of whether they truly understand anything or are just very good predictors is still wide open.