A small replication of the JEPA "good views" paper

Last month Yann LeCun and Randall Balestriero argued that models trained to match two "good views" of the same thing learn representations that line up with how other models see the world. A good view is a different, honest description of the same event. Their claim is that this works better than next-word prediction. We wanted to see if it holds on a small budget, on one 4 GB GPU.

The test. Flickr30k, where every photo has five captions written by different people. A small text model was trained on captions only, three ways:

• Good views: pull together two different captions of the same photo.
• Weak views: pull together two masked copies of one caption.
• Next-token: plain next-word prediction on captions.

The text model never sees an image. To score it, we check how often a caption's embedding and a frozen image model's features for the same photo pick each other as nearest neighbours (mutual kNN@10). Three data sizes, three seeds each, plus a shuffled control as the floor.

The shuffled control stayed around 0.004 throughout.

What we saw. Good views won at every size, and the gap grew with data. At 800k samples they were about 3× weak views and 2.4× next-token. A text model that only ever read captions moved closer to how a vision model sees the same photos. A small replication, but the direction matches the paper.

What this was not. Plain English captions only. No Thinking DNA and no structured data were involved. This was a check of the paper's claim, nothing more.

Why we care. At Spatial Thinking we're building structured training data where every fact can be shown more than one exact way, which is the kind of good view the paper describes. We generate it with code rather than distilling it from a larger model, currently around 5 million examples a night. We haven't trained on it this way yet. More when we have results we trust.