Description: CroCo paper note — a self-supervised pretext task tailored to 3D/geometry: mask ~90% of a first image and reconstruct it using the visible patches plus a second image of the same scene from another viewpoint, forcing the model to reason about cross-view geometry. Siamese ViT encoder + cross-attention decoder (MAE-style); pre-trained on 1.8M Habitat image pairs; transfers to monocular depth and binocular tasks (flow, relative pose). The direct ancestor of DUSt3R/VGGT
My Notion Note ID: K2F-C-1
Created: 2026-08-02
Updated: 2026-08-02
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion Authors: P. Weinzaepfel, V. Leroy, T. Lucas, R. Brégier, Y. Cabon, V. Arora, L. Antsfeld, B. Chidlovskii, G. Csurka, J. Revaud (NAVER LABS Europe) Paper: arXiv:2210.10716 (NeurIPS 2022) Github: naver/croco

CroCo (Cross-View Completion) — a self-supervised pre-training pretext task designed specifically for 3D/geometric downstream tasks, rather than semantic ones.

Core problem: masked image modeling (MAE, BEiT) pre-trains great vision backbones by masking patches of one image and reconstructing them — but that task rewards semantic/appearance priors ("a dog usually has two eyes"), not geometry. For 3D tasks (depth, flow, pose) you want a backbone that has learned about space and viewpoint, which single-image completion doesn't force.

Key idea: complete a masked image using a second view of the same scene. Mask ~90% of image 1; to fill it back in, the model must find the corresponding content in image 2 and reason about the geometric relationship between the two viewpoints. Cross-view completion turns masked modeling into an implicitly geometric pretext task.

Architecture (MAE-like, two-tower):

A Siamese ViT encoder (shared weights) encodes the visible patches of image 1 and all patches of image 2 independently.
A cross-attention decoder reconstructs image 1's masked patches, letting image-1 tokens attend to image-2 tokens — this cross-attention is exactly where cross-view geometry gets used.

Main results: pre-trained on 1.82M synthetic indoor image pairs (Habitat), CroCo beats other masked-image-modeling pre-trainings on monocular 3D tasks (NYUv2 depth, Taskonomy) and transfers directly to binocular tasks (optical flow, relative pose) — while being weaker on ImageNet classification, exactly because it pre-trains on geometry, not semantics.

CroCo pre-training: patches of the first (masked) image and the second reference image are encoded independently by a shared Siamese ViT encoder; the decoder receives the visible first-image tokens plus mask tokens and, via cross-attention to the reference-image tokens, reconstructs the masked patches of the first image.

2. Key Contributions

Cross-view completion — a new self-supervised pretext task that conditions masked image modeling on a second view, making the objective geometric rather than semantic.
A 3D-oriented pre-trained model — transfers strongly to monocular (depth, Taskonomy) and binocular (flow, relative pose) 3D tasks, unlike ImageNet-style MIM pre-training.
The very-high-mask-ratio finding — 90% masking is optimal (higher than MAE's own ~75%), because the reference image supplies the missing information.

3. Method

3.1 The Cross-View Completion Task

Given two images $x_1, x_2$ of the same scene from different viewpoints, split each into $N$ non-overlapping patches. Randomly mask $n = \lfloor rN \rfloor$ patches of $x_1$ (masking ratio $r$ , typically $r=0.9$ ), leaving visible set $\tilde p_1$ . An encoder $E_\theta$ processes $\tilde p_1$ and $p_2$ ; a decoder $D_\phi$ reconstructs $x_1$ using both:

\hat p_1 = D_\phi\big(E_\theta(\tilde p_1);\ E_\theta(p_2)\big)

$\tilde p_1$ = visible patches of image 1, $p_2$ = all patches of the reference image 2, $\hat p_1$ = reconstructed image 1. The reference image is the only way to recover most of $x_1$ (90% is hidden), so solving the task requires cross-view reasoning.

3.2 Siamese Encoder

$E_\theta$ is a Siamese ViT (shared weights) encoding the two patch sets independently. Images are $224\times224$ , patch size $16\times16$ ; a linear projection of flattened RGB patches + sinusoidal position embeddings, then transformer blocks (MHSA + MLP). Backbone: ViT-Base/16 (12 blocks, 768-dim, 12 heads).

3.3 Cross-Attention Decoder

$D_\phi$ takes the encoded visible first-image tokens $E_\theta(\tilde p_1)$ concatenated with a learned mask embedding $e_\text{mask}$ (repeated $n$ times for the masked slots), plus sinusoidal positions, and reconstructs the hidden patches while attending to $E_\theta(p_2)$ . Two decoder-block designs (Fig. 4):

CrossBlock (default) — (a) self-attention over image-1 tokens, (b) cross-attention from those tokens to the reference tokens $E_\theta(p_2)$ , (c) MLP. More parameters (the cross-attention module) but cheaper compute — it avoids self-attention over the joint $2N$ token set.
CatBlock — concatenate both images' tokens (+ a per-image learnable embedding) and run standard self-attention blocks; only image-1 tokens are used for prediction. Quadratic in $2N$ .

Decoder: 8 blocks, 512-dim, 16 heads; a linear layer bridges the encoder (768) and decoder (512) dimensions.

3.4 Loss & Pre-training Data

The decoder outputs $16\times16\times3 = 768$ RGB values per patch. Loss = MSE between predicted and ground-truth pixels, averaged over masked patches only (MAE-style):

\mathcal{L}(x_1, x_2) = \frac{1}{|p_1 \setminus \tilde p_1|} \sum_{p_1^i \in p_1 \setminus \tilde p_1} \lVert \hat p_1^i - p_1^i \rVert^2

(a variant normalizes each target patch by its own pixel mean/std, as in MAE).

Pre-training: 400 epochs, AdamW, cosine schedule, base lr $1.5\times10^{-4}$ , effective batch 256, 40-epoch warmup. Data = "Habitat": 1,821,391 synthetic image pairs rendered with the Habitat simulator from indoor 3D scenes (HM3D, ScanNet, Replica, ReplicaCAD) — up to 1000 viewpoint pairs per scene with co-visibility > 50%.

4. Experiments & Results

Downstream setup: for monocular tasks the ViT encoder is used alone (with a DPT head for dense outputs); for binocular tasks both encoder and decoder are kept. Tasks:

Monocular: ImageNet-1K linear-probe classification; ADE20K segmentation; NYUv2 depth ( $\delta_1$ = % pixels with error ratio < 1.25); Taskonomy (8 dense tasks, L1 loss).
Binocular: optical flow (regress 2 values/pixel from the decoder); relative pose (append a pose-regression head).

Findings:

Masking ratio — best at 90% (Fig. 5), notably higher than MAE's ~75% optimum (a figure from the MAE paper, not stated in CroCo), because the reference view supplies the missing content (validating the whole premise). Reconstructions are blurry (MSE), which as in MAE doesn't hurt pre-training.
3D tasks — CroCo significantly outperforms existing MIM pre-trainings (same data or ImageNet) on NYUv2 depth and Taskonomy.
Semantic tasks — CroCo is worse on ImageNet classification than ImageNet-pretrained MIM models — a direct consequence of pre-training on indoor geometry instead of semantic imagery, and evidence the objective genuinely learns geometry over semantics.
Binocular transfer — flow and relative pose work "without bells and whistles," showing the pre-trained encoder+decoder already encode cross-view correspondence.

5. Strengths / Limitations / Legacy

Strengths

Reframes self-supervision toward geometry with a one-line change (condition on a second view), and shows the semantic-vs-geometric trade-off cleanly (wins on 3D, loses on ImageNet — a feature, not a bug, for the intended use).
The 90%-masking result is a satisfying confirmation that the second view is doing real work.
The pre-trained encoder + cross-attention decoder is directly reusable for binocular tasks — this exact two-image encoder/decoder is the template DUSt3R scales up.

Limitations

Pre-training is synthetic + indoor (Habitat) → domain gap to real/outdoor scenes; poorer semantic features.
Needs posed/co-visible image pairs to render — not arbitrary unlabeled images like single-image MAE.
ViT-Base scale, $224\times224$ , fixed patch — modest by later standards.

Legacy (why it's the pivot of this reading arc)

CroCo is the direct ancestor of DUSt3R (and thus VGGT): DUSt3R takes CroCo's Siamese-ViT-encoder + cross-attention-decoder over two images and, instead of reconstructing pixels, regresses pointmaps — turning the cross-view pretext task into a full two-view 3D reconstruction model. The "attend across views" mechanism VGGT relies on starts here.
Followed by CroCo v2 (next in the arc), which scales the data to real image pairs and improves the architecture, providing the weights DUSt3R initializes from.

References

Weinzaepfel, P., Leroy, V., Lucas, T., Brégier, R., Cabon, Y., Arora, V., Antsfeld, L., Chidlovskii, B., Csurka, G., & Revaud, J. (2022). CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion. NeurIPS. arXiv:2210.10716, code. — source paper (Fig. 3 overview above)
He, K., et al. (2022). Masked Autoencoders Are Scalable Vision Learners (MAE). CVPR. — the single-image masked-modeling method CroCo extends to two views (mask-token decoder, high mask ratio, normalized-pixel target)
ViT: the Siamese encoder backbone
DPT: the dense-prediction head used for CroCo's monocular 3D downstream tasks
Savva, M., et al. (2019). Habitat. ICCV. — the simulator rendering the 1.8M pre-training pairs
DUSt3R (next in the arc): scales CroCo's two-view encoder/decoder to direct pointmap regression

CroCo: Cross-View Completion Pre-training for 3D Vision

Table of Contents