Description: DPT (Dense Prediction Transformer) paper note — use a ViT encoder as the backbone for dense per-pixel prediction: reassemble tokens from several transformer stages into image-like feature maps at multiple resolutions, then RefineNet-style fusion upsamples them to a fine-grained output. Global receptive field at every stage → big gains in monocular depth (zero-shot) and semantic segmentation (ADE20K 49.02% mIoU); the dense-prediction head DUSt3R/CroCo later reuse
My Notion Note ID: K2F-B-3
Created: 2026-08-02
Updated: 2026-08-02
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: Vision Transformers for Dense Prediction Authors: R. Ranftl, A. Bochkovskiy, V. Koltun (Intel Labs) Paper: arXiv:2103.13413 (ICCV 2021) Github: isl-org/DPT

DPT (Dense Prediction Transformer) — an architecture for dense per-pixel prediction (depth, segmentation) that swaps the convolutional backbone for a Vision Transformer encoder, keeping an encoder–decoder shape.

Core problem: fully-convolutional dense-prediction nets progressively downsample the image to build up receptive field, so deep features lose resolution and fine detail is hard to recover through upsampling. The bottleneck is structural — once downsampled, information is gone.

Key insight: a Transformer keeps the number of tokens constant through all layers and has a global receptive field at every stage (each token can attend to all others, no downsampling). So representations never lose their granularity, and every stage has full-image context — ideal for dense prediction.

Approach: use ViT as the encoder; then a convolutional decoder turns tokens back into images:

Reassemble — take tokens from several transformer stages and reshape them into image-like feature maps at multiple resolutions (deeper layers → coarser, earlier layers → finer).
Fusion — RefineNet-style blocks progressively fuse and ×2-upsample these maps into one fine-grained prediction, then a task head.

Main results: on monocular depth, trained on a 1.4M-image meta-dataset, DPT beats the CNN SOTA (MiDaS) in zero-shot cross-dataset transfer by >28% (DPT-Large) / >23% (DPT-Hybrid) average relative improvement; on semantic segmentation sets a new ADE20K SOTA at 49.02% mIoU.

2. Key Contributions

Transformer backbone for dense prediction — first to show a ViT encoder + convolutional decoder substantially beats convolutional dense-prediction nets of similar capacity, especially with large training data.
The Reassemble operation — a clean way to recover image-like features from the tokens of arbitrary transformer layers, at a chosen resolution (Read → Concatenate → Resample).
New SOTA — monocular depth (zero-shot transfer) and ADE20K / Pascal Context / NYUv2 / KITTI segmentation & depth.

3. Method

3.1 Transformer Encoder

The image is split into non-overlapping $p\times p$ patches, each linearly embedded into a token (ViT-style; DPT-Hybrid instead extracts patches from a ResNet-50 feature map). Add position embeddings and one patch-independent readout token (the ViT [class]-token analog, not grounded in any image location). For an $H\times W$ image this gives $N_p = HW/p^2$ tokens (+1 readout); $L$ transformer layers map them to new representations $t^l$ . Crucially, the token count is constant and each layer is global — no resolution loss. All experiments use patch size $p=16$ .

3.2 Reassemble: Tokens → Feature Maps

Recover an image-like representation from the tokens of a chosen layer at output size ratio $s$ :

\text{Reassemble}_s^{\hat D}(t) = (\text{Resample}_s \circ \text{Concatenate} \circ \text{Read})(t)

Read ( $\mathbb{R}^{(N_p+1)\times D} \to \mathbb{R}^{N_p\times D}$ ) — handle the readout token. Three variants: ignore (drop it), add (add it to every token), or proj (default: concatenate readout to each token, MLP + GELU back to $D$ ).
Concatenate — place each token at its patch's spatial position → feature map $\tfrac{H}{p}\times\tfrac{W}{p}$ with $D$ channels.
Resample — $1\times1$ conv to $\hat D$ channels, then strided $3\times3$ conv (downsample) or transpose conv (upsample) to size $\tfrac{H}{s}\times\tfrac{W}{s}$ .

Reassemble runs at four stages / four resolutions: deeper layers reassembled coarser, early layers finer. Default head width $\hat D = 256$ .

3.3 Fusion & Head

Feature maps from consecutive stages are combined by RefineNet-based fusion blocks (residual convolutional units), each upsampling ×2. The final representation is at half the input resolution; a task-specific head produces the dense output (the head further upsamples to full resolution). Batch norm is disabled in the decoder (found to hurt regression).

3.4 Variants & Varying Image Sizes

Variant	Encoder	Reassembled layers
DPT-Base	ViT-Base (12 layers)	$l = \{3, 6, 9, 12\}$
DPT-Large	ViT-Large (24 layers)	$l = \{5, 12, 18, 24\}$
DPT-Hybrid	ResNet-50 + ViT (12 layers)	ResNet blocks 1–2 + $l = \{9, 12\}$

Like an FCN, DPT handles varying image sizes (as long as divisible by $p$ ): the transformer is set-to-set so it takes any token count; position embeddings are linearly interpolated on the fly; reassemble/fusion handle variable sizes as long as the input aligns to the decoder stride (32 px).

4. Experiments & Results

Monocular depth — trained on MIX 6, a meta-dataset of ~1.4M images (largest ever for the task), with a scale/shift-invariant trimmed loss on inverse depth + gradient-matching loss. Encoder initialized from ImageNet, decoder random; Adam, lr 1e-5 (backbone) / 1e-4 (decoder), 60 epochs.

Zero-shot cross-dataset transfer (Table 1, relative improvement over MiDaS; lower error is better):

	DIW (WHDR)	ETH3D (AbsRel)	Sintel	KITTI	NYU	TUM
DPT-Large	10.82 (−13.2%)	0.089 (−31.2%)	0.270 (−17.5%)	8.46 (−64.6%)	8.32 (−12.9%)	9.97 (−30.3%)
DPT-Hybrid	11.06 (−11.2%)	0.093 (−27.6%)	0.274 (−16.2%)	11.56 (−51.6%)	8.69 (−9.0%)	10.89 (−23.2%)
MiDaS (retrained on MIX 6)	12.95	0.116	0.329	16.08	8.71	12.51

Average relative improvement over MiDaS: >28% (DPT-Large), >23% (DPT-Hybrid). Retraining the CNN MiDaS on the same larger dataset still leaves DPT ahead — DPT benefits more from data, matching the transformer-scaling story from other fields. DPT-Hybrid gets this at comparable capacity/latency; DPT-Large is ~3× larger.

Semantic segmentation — new SOTA on ADE20K (49.02% mIoU), plus strong results on Pascal Context, NYUv2, KITTI. Ablations confirm the default proj readout and $\hat D=256$ .

5. Strengths / Limitations / Legacy

Strengths

Turns "keep resolution + global context" into a concrete recipe (Reassemble + Fusion) that drops onto any ViT encoder.
Clean empirical case: beats CNNs at equal capacity, and scales better with data.
General — depth and segmentation from the same backbone with only a head swap.

Limitations

Inherits ViT's cost: global $O(N^2)$ attention → expensive at high resolution / many patches.
Data-hungry, like ViT — the big wins need large training sets.
Reassemble/fusion add convolutional decoder machinery and hyperparameters (which layers, $\hat D$ , stride) on top of the "pure transformer" idea.

Legacy (why it's in this reading arc)

The DPT dense-prediction head is reused downstream: CroCo puts a DPT head on its ViT encoder for dense 3D tasks, and the DUSt3R/VGGT line regresses dense pointmaps with DPT-style heads. DPT is the bridge from "ViT gives good features" to "ViT gives dense per-pixel geometry".

References

Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021). Vision Transformers for Dense Prediction. ICCV. arXiv:2103.13413, code. — source paper (Fig. 1 architecture above)
ViT: the encoder backbone DPT builds on — read first
Ranftl, R., et al. (2020). Towards Robust Monocular Depth Estimation (MiDaS). IEEE TPAMI. — the CNN baseline and the mixed-dataset / scale-invariant-loss training protocol DPT adopts
Lin, G., et al. (2017). RefineNet. CVPR. — the fusion-block design DPT's decoder is based on
CroCo: next in the arc — reuses a DPT head on a ViT encoder for dense 3D downstream tasks

DPT: Vision Transformers for Dense Prediction

Table of Contents