Description: DUSt3R paper note — the pointmap-regression paradigm: from two uncalibrated images, directly regress two pointmaps (per-pixel 3D coords) both expressed in the first image's frame, with no camera poses or intrinsics as input. A Siamese ViT encoder + two cross-attention decoders + DPT heads (initialized from CroCo); a lightweight global alignment fuses many pairs into one scene. Recovers depth, matches, intrinsics, and poses from the pointmaps — replacing the classical SfM/MVS pipeline. The direct precursor of VGGT
My Notion Note ID: K2F-C-3
Created: 2026-08-02
Updated: 2026-08-02
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: DUSt3R: Geometric 3D Vision Made Easy Authors: S. Wang (Aalto University); V. Leroy, Y. Cabon, B. Chidlovskii, J. Revaud (NAVER LABS Europe) Paper: arXiv:2312.14132 (CVPR 2024) Github: naver/dust3r

DUSt3R (Dense Unconstrained Stereo 3D Reconstruction) — turn a pair of images into 3D by directly regressing pointmaps, with no camera calibration and no known poses. It is the paradigm shift the whole VGGT line descends from.

Core problem: classical 3D reconstruction (SfM + MVS) is a brittle pipeline of many separately-solved subproblems — feature matching, triangulation, bundle adjustment, pose estimation, dense stereo — where errors in one stage cascade. And each stage needs known/estimated camera parameters. Learning-based methods still mostly slot into this pipeline rather than replace it.

Key idea — the pointmap: instead of predicting depth (which needs intrinsics to become 3D) or matches (which need triangulation), predict a pointmap $X \in \mathbb{R}^{W\times H\times 3}$ : a dense per-pixel map of 3D coordinates, one 3D point per pixel. Regress two pointmaps from two images — and crucially express both in the first image's coordinate frame. That single choice folds relative pose, depth, and correspondence into one representation: two images' pointmaps in a shared frame are already a registered two-view reconstruction.

Radical simplification: DUSt3R takes the "generalized stereo" problem — two images, unknown everything — and solves it by direct regression, learning all geometric priors from data rather than enforcing a camera model. Everything downstream (depth, matches, focal length, relative/absolute pose) is then read off the pointmaps by simple post-processing.

Main results: from uncalibrated, unposed images DUSt3R does monocular & multi-view depth, camera intrinsics, relative & absolute pose, and dense reconstruction — often beating task-specific methods — with a single network + a fast global-alignment step (pair inference ≈ 40 ms on an H100).

$DUSt3R architecture: two views are encoded by a shared (Siamese) ViT encoder into token sets F¹, F²; two transformer decoders process them while constantly exchanging information via cross-attention; two regression heads output two pointmaps X^{1,1}, X^{2,1} — both in image 1's coordinate frame — plus confidence maps.$

2. Key Contributions

Pointmap regression as the interface — recast two-view 3D reconstruction as directly regressing per-pixel 3D pointmaps, sidestepping explicit camera models and the SfM/MVS stage pipeline.
Both pointmaps in one frame — expressing the two views' pointmaps in the first image's frame makes the output an already-aligned two-view reconstruction, from which pose/depth/matches/intrinsics all follow.
Global alignment — a fast optimization that stitches many pairwise pointmaps into one globally consistent scene (a lightweight, differentiable stand-in for bundle adjustment).
A generic architecture with strong pretraining — a CroCo-style ViT encoder + cross-attention decoders that inherits CroCo pre-training, letting a task-agnostic net beat task-specific ones.

3. Method

3.1 Pointmaps

A pointmap $X\in\mathbb{R}^{W\times H\times 3}$ associates each pixel $(i,j)$ with a 3D point. Given intrinsics $K$ and a depthmap $D$ , the pointmap in the camera frame is $X_{i,j} = K^{-1}[\,i\,D_{i,j},\ j\,D_{i,j},\ D_{i,j}\,]^\top$ . Write $X^{n,m}$ for camera $n$ 's pointmap expressed in camera $m$ 's frame:

X^{n,m} = P_m\,P_n^{-1}\,h(X^n)

$P_n, P_m$ = world-to-camera poses, $h$ = homogeneous mapping $(x,y,z)\mapsto(x,y,z,1)$ . (This is only how ground-truth pointmaps are built for training — at inference DUSt3R regresses pointmaps directly, without $K$ or $P$ .)

3.2 Network: Encoder + Twin Decoders + Heads

Input: two RGB images $I^1, I^2$ . Output: two pointmaps $X^{1,1}, X^{2,1}\in\mathbb{R}^{W\times H\times 3}$ (both in $I^1$ 's frame) + confidence maps $C^{1,1}, C^{2,1}$ . Architecture (inspired by, and initialized from, CroCo):

Siamese ViT encoder (shared weights): $F^1 = \text{Encoder}(I^1)$ , $F^2 = \text{Encoder}(I^2)$ .
Two intertwined transformer decoders that constantly exchange information via cross-attention. Each decoder block does: self-attention (tokens attend within their own view) → cross-attention (tokens attend to the other view's tokens) → MLP:

G^1_i = \text{DecoderBlock}^1_i(G^1_{i-1}, G^2_{i-1}), \qquad G^2_i = \text{DecoderBlock}^2_i(G^2_{i-1}, G^1_{i-1})

initialized $G^1_0 = F^1$ , $G^2_0 = F^2$ . This constant cross-view sharing is what makes the two output pointmaps come out aligned in one frame.

Two DPT regression heads output the pointmaps + confidence from the decoder tokens.

Backbone: ViT-Large encoder, ViT-Base decoder, DPT head. The architecture enforces no geometric constraints — pointmaps needn't correspond to any physical camera model; the network learns geometric priors purely from geometrically-consistent training data.

3.3 Training Objective

3D regression loss — Euclidean distance between predicted and ground-truth pointmaps, each normalized by a scale factor (average distance of valid points to the origin) to handle scale ambiguity:

\ell_\text{regr}(v, i) = \left\lVert \tfrac{1}{z}X^{v,1}_i - \tfrac{1}{\bar z}\bar X^{v,1}_i \right\rVert

$v\in\{1,2\}$ = view, $z, \bar z$ = predicted / GT scale normalizers. Confidence-aware loss — the net also predicts a per-pixel confidence $C$ and is trained with:

\mathcal{L}_\text{conf} = \sum_{v}\sum_{i\in\mathcal{D}^v} C^{v,1}_i\,\ell_\text{regr}(v,i) - \alpha\log C^{v,1}_i

The $-\alpha\log C$ term rewards confidence, so the net down-weights ill-defined pixels (sky, translucent, single-view regions) without explicit supervision for confidence.

3.4 What Pointmaps Give You

Because both pointmaps live in $I^1$ 's frame, standard quantities are read off by simple post-processing:

Point matching — nearest-neighbor (reciprocal / mutual) search in 3D pointmap space.
Intrinsics — solve for focal $f^*$ from $X^{1,1}$ (assuming centered principal point, square pixels) via a fast iterative (Weiszfeld) solver.
Relative pose — compare $X^{1,1}\leftrightarrow X^{1,2}$ by Procrustes alignment (closed form), or more robustly via PnP-RANSAC on 2D-3D correspondences.
Absolute pose (localization) — estimate query intrinsics from $X^{Q,Q}$ , get 2D-3D correspondences to a reference image, run PnP-RANSAC.

3.5 Global Alignment for Many Views

$F$ only handles a pair; a fast post-processing optimization fuses $N$ images into one frame. Build a connectivity graph $\mathcal{G}(V,E)$ (vertices = images, edges = overlapping pairs, found by retrieval or by running $F$ and thresholding pair confidence). For each edge predict pairwise pointmaps, then optimize global pointmaps $\{\chi^n\}$ with a per-pair pose $P_e$ and scale $\sigma_e$ :

\chi^* = \arg\min_{\chi, P, \sigma} \sum_{e\in E}\sum_{v\in e}\sum_i C^{v,e}_i\,\lVert \chi^v_i - \sigma_e P_e X^{v,e}_i \rVert

with $\prod_e \sigma_e = 1$ to avoid the trivial $\sigma=0$ solution. Unlike bundle adjustment this optimizes in 3D pointmap space (not 2D reprojection), so it is fast and simple; camera parameters fall out of the aligned $\chi$ .

4. Experiments & Results

Training data: 8.5M pairs from many datasets (Habitat, MegaDepth, ARKitScenes, ScanNet++, Waymo, …) spanning indoor/outdoor/synthetic/object-centric; pairs extracted via image retrieval + point matching when not provided. Progressive resolution 224×224 → 512-px (largest dim), randomized aspect ratios so the net sees varied shapes at test time. ViT-Large encoder + ViT-Base decoder + DPT head.

Results: across monocular & multi-view depth, camera intrinsics, relative & absolute pose (visual localization), point matching, and dense multi-view reconstruction, DUSt3R is competitive with or beats task-specific methods — all from uncalibrated, unposed input, with a single network + global alignment. Pair inference ≈ 40 ms on an H100.

5. Strengths / Limitations / Legacy

Strengths

A genuine paradigm shift: one regression target (pointmaps in a shared frame) collapses matching + triangulation + pose + depth into one learned step, killing the brittle SfM/MVS cascade.
No calibration/poses required at input — works on casual, in-the-wild image pairs.
Everything downstream is derived from the pointmaps, so one model serves many tasks.
Rides CroCo pre-training — generic architecture + strong pretext beats specialized designs.

Limitations

Fundamentally pairwise; scenes need a separate global alignment optimization, whose cost grows with the number of pairs (and it's still an optimization, not pure feed-forward).
Pointmaps are scale-ambiguous and unconstrained by any camera model (can be geometrically implausible where data priors are weak).
ViT-Large at ≤512 px — resolution and memory bounded; many-view scenes are a graph of pairs, not a joint model.

Legacy (why it's the immediate precursor to VGGT)

DUSt3R establishes pointmaps as the output representation and proves feed-forward regression can replace SfM. Its main remaining seam is that it's two-view + a post-hoc global-alignment optimization.
VGGT removes exactly that seam: a single feed-forward transformer ingests N images at once (via alternating frame-wise / global attention) and predicts cameras, depth, pointmaps, and tracks jointly — no pairwise decomposition, no global-alignment optimization. DUSt3R is the "why pointmaps / why no COLMAP" note; VGGT is "now do it for all views in one pass". (MASt3R and MASt3R-SfM are the intermediate steps adding metric pointmaps + matching + full SfM.)

References

Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., & Revaud, J. (2024). DUSt3R: Geometric 3D Vision Made Easy. CVPR. arXiv:2312.14132, code. — source paper (Fig. 2 architecture above)
CroCo / CroCo v2: the pre-training and encoder/decoder DUSt3R initializes from and repurposes — read first
DPT: the regression head producing the dense pointmaps
Leroy, V., Cabon, Y., & Revaud, J. (2024). Grounding Image Matching in 3D with MASt3R. ECCV. — follow-up adding a metric pointmap + dense matching head (the "M" = matching)
VGGT: the successor — one feed-forward pass over all views, no pairwise global alignment

Table of Contents