Human-like Object Grouping in Self-supervised Vision Transformers

[Submitted on 14 Mar 2026 (v1), last revised 9 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects’ reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3’s feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.

Submission history

From: Hossein Adeli [view email]
[v1] Sat, 14 Mar 2026 15:43:10 UTC (4,189 KB)
[v2] Sat, 4 Jul 2026 16:49:42 UTC (3,713 KB)
[v3] Thu, 9 Jul 2026 15:26:35 UTC (3,713 KB)