Description: FunRec paper note — training-free, optimization-based pipeline that turns a single egocentric RGB-D interaction video into a functional 3D digital twin: auto-discovers articulated parts, estimates kinematics (prismatic/revolute joints), tracks part motion, and reconstructs static + moving geometry in canonical space → simulation-ready meshes (URDF/USD)
My Notion Note ID: K2E-B-N3-1
Created: 2026-07-18
Updated: 2026-07-18
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos Authors: A. Delitzas, C. Zhang*, A. Gavryushin*, T. Di Mario, B. Sun, R. Dabral, L. Guibas, C. Theobalt, M. Pollefeys, F. Engelmann, D. Barath (ETH Zurich / MPI Informatics / Stanford / Microsoft / USI Lugano) Paper: arXiv:2604.05621 (v2, Apr 2026) Github: code + datasets "coming soon" — project page (no public release yet)

FunRec — reconstruct a functional 3D digital twin of an indoor scene directly from a single egocentric RGB-D interaction video. "Functional" = capture not just static geometry but how parts move and articulate (doors swinging, drawers sliding, fridges revealing interiors). Output is a simulation-compatible scene: static background mesh + per-part canonical meshes + inferred joint models, so any articulated part can be continuously manipulated along its estimated motion axis.

Core problem: Prior articulated-reconstruction methods need controlled setups, multi-state captures (scan the room twice, open/closed), or CAD priors. They cannot handle casual, in-the-wild human interaction. Meanwhile 4D-reconstruction pipelines recover dynamic point maps but do not understand scene semantics or joint parameters, and flicker under interaction-induced occlusion.

Key insight: Human interaction is the richest supervision for functionality. As a person manipulates the scene, egocentric observations directly reveal which parts articulate, about which joints, what interiors are exposed, and the associated affordances. So recover kinematics from observed motion, not from static geometry.

Approach: A training-free, optimization-based pipeline that stitches together off-the-shelf foundation models (VLM, dense matcher, 3D point tracker, segmentation, registration) with classical geometric estimation (pose-graph optimization, TSDF fusion, loop closure). It splits the video into fragments, uses a VLM to flag interactions and their joint type, clusters 3D motion trajectories into articulated parts, segments the moving part pixel-accurately, jointly optimizes part poses + articulation parameters, and fuses everything into one globally aligned twin.

Main results: Across three datasets (new RealFun4D real, new OmniFun4D simulated, adapted HOI4D), FunRec beats SOTA by a large margin: up to +50 mIoU in moving-part segmentation, 5–10× lower articulation and pose errors, and much lower reconstruction Chamfer Distance — with near-zero failure rate. Demonstrated downstream: URDF/USD export for physics simulation, hand-guided affordance mapping, and robot-scene interaction (Boston Dynamics Spot reproducing a human-demonstrated cabinet opening).

FunRec takes a single egocentric RGB-D interaction video (top: four interactions, revolute/prismatic) and reconstructs a functional 3D digital twin (center) with articulated components, their kinematic axes, per-timestep poses, and interiors.

2. Key Contributions

First scene-scale functional twin from in-the-wild egocentric video: no controlled rig, no multi-state capture, no CAD retrieval — operates on a single casually captured RGB-D interaction sequence and recovers a physically grounded, interactable scene.
Training-free pipeline fusing foundation-model priors with geometric optimization: combines semantic/motion priors (VLM, dense matching, 3D point tracking, promptable segmentation) with pose-graph optimization, TSDF fusion, and registration — no task-specific training.
Articulation-aware motion clustering: fits a per-track joint hypothesis (3D line for prismatic, circle for revolute) and clusters trajectories by joint parameters to discover independently moving parts, then picks the interacted one via 2D interaction evidence.
Joint part-pose + articulation optimization: recovers globally consistent per-frame part poses and joint parameters together on a pose graph with loop closures, keeping parameters on their motion manifolds.
Two new datasets: RealFun4D (351 real egocentric interactions, 60 apartments, 4 countries) and OmniFun4D (127 photorealistic simulated sequences, 12 OmniGibson scenes), both with articulation/pose/mask annotations for benchmarking functional scene reconstruction.

3. Method

3.1 Problem Formulation & Articulation Models

Input: RGB-D video $V=\{(I_i, D_i)\}_{i=1}^N$ ( $I_i$ = RGB frame, $D_i$ = depth map, known intrinsics). The scene = static background geometry $P^s$ + a set of $K$ articulated parts $\{P^{m_k}\}_{k=1}^K$ . Each part $m_k$ has articulation parameters $\phi_k$ and a sequence of rigid transforms $\{T_i^{m_k}\}_{i=1}^N$ , where $T_i^{m_k}\in SE(3)$ maps the part's canonical frame to the world frame at time $i$ . $SE(3)$ = the group of 3D rigid transforms (rotation + translation).

Two joint types are modeled:

Prismatic (sliding parts, e.g. drawers): unit translation axis $\mathbf{a}\in S^2$ , scalar displacement $\lambda_i\in\mathbb{R}$ at frame $i$ ( $S^2$ = unit sphere of directions):

\mathcal{T}_\text{pris}(\mathbf{a}, \lambda_i) = \begin{bmatrix} I & \lambda_i \mathbf{a} \\ \mathbf{0}^\top & 1 \end{bmatrix}

$I$ = $3\times 3$ identity.

Revolute (rotating parts, e.g. doors): unit rotation axis $\mathbf{a}\in S^2$ , pivot point $\mathbf{p}\in\mathbb{R}^3$ on the axis nearest the origin ( $\mathbf{a}^\top\mathbf{p}=0$ ), rotation angle $\theta_i$ at frame $i$ :

\mathcal{T}_\text{rev}(\mathbf{a}, \mathbf{p}, \theta_i) = \begin{bmatrix} R(\mathbf{a}, \theta_i) & (I - R(\mathbf{a}, \theta_i))\,\mathbf{p} \\ \mathbf{0}^\top & 1 \end{bmatrix}

$R(\mathbf{a},\theta)$ = $3\times 3$ rotation about axis $\mathbf{a}$ by angle $\theta$ . The part's pose at time $i$ is $T_i^{m_k}=\mathcal{T}_\text{pris}$ or $\mathcal{T}_\text{rev}$ per its joint type.

3.2 Fragment Construction

Split $V$ into temporally contiguous fragments $V=\bigcup_{k=1}^K V_k$ . A video-language model (VLM) (Gemini 2.5 — a large multimodal model that reads the clip and answers in text) classifies each fragment as static (no interaction) or dynamic (interaction with an articulated part), and for dynamic fragments also predicts the joint type $\sigma_k\in\{\text{prismatic, revolute}\}$ . Each fragment is processed independently.

3.3 Dynamic Fragment Reconstruction

Camera pose estimation. Dense correspondences between consecutive RGB-D frames come from RoMa (a robust dense feature-matching network producing pixel-level correspondences). Matches are filtered with per-frame hand and interacted-object masks from VISOR (the EPIC-KITCHENS egocentric hand/object segmentation annotations): points inside the hand mask $\{M_i^h\}$ are discarded; points inside the interacted-object mask $\{M_i^\text{obj}\}$ are down-weighted (scaling their RoMa confidence). Surviving 2D matches are lifted to 3D; relative camera motion is estimated by SupeRANSAC (a unified RANSAC variant for robust geometric model fitting). A fragment-level pose-graph optimization yields globally consistent camera poses $\{T_i^c\}$ .

Sparse 3D trajectories. TAPIP3D (a 3D point tracker that tracks points directly in the world frame over time) gives track positions $\tau\in\mathbb{R}^{T\times N\times 3}$ and per-frame visibility scores $o\in\mathbb{R}^{T\times N}$ ( $T$ = number of tracked points). New tracks are periodically seeded on a uniform 2D grid to capture surfaces revealed mid-interaction (e.g. a drawer interior).

Articulation-aware motion clustering. For track $\tau_l$ , discard it if its max displacement is below $\epsilon_s$ . Otherwise fit a per-track joint hypothesis $\hat\phi_l$ consistent with the fragment joint type $\sigma_k$ : a 3D line for prismatic, a 3D circle for revolute. Predicted positions $\hat\tau_{l,i}$ come from transforming the initial point $\tau_{l,1}$ by the fitted motion; a track is kept if its mean fit error $\frac{1}{|\tau_l|}\sum_i\|\tau_{l,i}-\hat\tau_{l,i}\|_2 < \epsilon_f$ . Surviving tracks are clustered by HDBSCAN (hierarchical density-based clustering) on their fitted joint parameters (axis, pivot, motion pattern) → motion clusters, one per independently moving part.

Pick the interacted cluster by scoring each cluster $\gamma$ against 2D interaction evidence:

s_\gamma = \sum_{l\in\gamma}\sum_i o_{l,i}\cdot C_i^\text{obj}\cdot \mathbb{I}\!\left[\pi(\tau_{l,i})\in M_i^\text{obj}\right]

$o_{l,i}$ = track visibility, $C_i^\text{obj}$ = confidence of the interacted-object mask at frame $i$ , $\pi(\cdot)$ = projection to the image plane, $\mathbb{I}[\cdot]$ = indicator. Highest- $s_\gamma$ cluster → moving track set $\tau^m$ ; the rest → static set $\tau^s$ .

Pixel-aligned part segmentation. $\tau^m$ is sparse and noisy, so directly projecting tracks or prompting a segmenter is unreliable. Instead, combine track geometry with image semantics: on uniformly sampled keyframes, run SAM's automatic mask generator (Segment Anything Model — a promptable segmentation foundation model) to over-segment into regions $G_q$ . Project moving/static tracks into each keyframe and, per region $r$ , count contained moving ( $n_r^m$ ) vs static ( $n_r^s$ ) track projections. The motion ratio $\gamma_r = n_r^m/(n_r^m + n_r^s + \epsilon)$ (with $\epsilon$ guarding division by zero); regions with $\gamma_r>\eta_m$ form the keyframe motion mask. These masks prompt SAM2's video-propagation module (SAM2 = the video extension of SAM, propagating masks across frames) → a temporally consistent sequence of articulated-part masks $\{M_i^m\}_{i=1}^N$ .

3.4 Part Pose & Articulation Estimation

Given moving tracks $\tau^m$ and visibilities $o^m$ , recover globally consistent part poses $\{T_i^m\}$ and articulation parameters $\phi^m$ ( $=(\mathbf{a},\{\lambda_i\})$ prismatic or $(\mathbf{a},\mathbf{p},\{\theta_i\})$ revolute).

For each frame pair $(i,j)$ , build 3D-3D correspondences between visible points in $\tau^m$ and estimate the relative part transform $T_{i\to j}^m\in SE(3)$ via SupeRANSAC, weighting each correspondence by $o_{l,i}^m\cdot o_{l,j}^m$ . This forms a pose graph over $\{T_i^m\}$ . Jointly recover absolute poses + articulation by minimizing:

L(T^m, L^m, \phi^m) = \sum_i f(T_i^m, T_{i+1}^m, T_{i\to i+1}^m) + \sum_{i,j} l_{ij}^m\, f(T_i^m, T_j^m, T_{i\to j}^m) + \mu\sum_{i,j}(l_{ij}^m - 1)^2

with pose-graph residual

f(T_i^m, T_j^m, T_{i\to j}^m) = e_{ij}^\top \Omega_{ij} e_{ij}, \qquad e_{ij} = \log_{SE(3)}\!\left((T_i^m)^{-1} T_j^m\, T_{i\to j}^m\right)

$l_{ij}^m\in[0,1]$ = optimized loop-closure confidences, $\mu$ = their regularization weight, $e_{ij}$ = discrepancy between estimated relative pose $(T_i^m)^{-1}T_j^m$ and observed $T_{i\to j}^m$ , $\Omega_{ij}$ = information (weight) matrix, $\log_{SE(3)}(\cdot)$ = map to the $SE(3)$ tangent space (twist coordinates). First and second sums are sequential and loop-closure edges; the third keeps loop confidences near 1. $\phi^m$ is initialized by least-squares fitting to $\tau^m$ , then $\{T_i^m\}$ and $\phi^m$ are refined together in Ceres Solver (a nonlinear least-squares library) with manifold optimization so parameters stay on their motion manifolds ( $S^2$ axis, $SE(3)$ pose).

3.5 Reconstruction & Global Alignment

Two-volume TSDF reconstruction. A TSDF (truncated signed distance function — a volumetric surface representation fusing depth into a distance field) is built separately for the static background and the moving part:

Static TSDF: integrated in the world frame using $\{T_i^c\}$ , excluding hand pixels $M_i^h$ and moving-part pixels $M_i^m$ .
Part TSDF: integrated in the part's canonical frame — each frame's depth is mapped by $(T_i^m)^{-1}T_i^c$ and only moving-part pixels are fused → a clean canonical model $P^m$ free of camera motion and occlusion.

Meshes are extracted for $P^s$ and $P^m$ ; the complete scene at time $i$ is $P_i = P^s \cup T_i^m(P^m)$ . The feasible motion range $[\lambda_\text{min},\lambda_\text{max}]$ or $[\theta_\text{min},\theta_\text{max}]$ is read from the tracked trajectory, so the twin can be re-simulated at any intermediate state.

Global fragment alignment. Each fragment yields a local submap $S_k$ (static fragments run camera-pose estimation only). Align all $\{S_k\}$ by extracting geometric correspondences per submap pair and estimating a rigid transform via PREDATOR (a point-cloud registration network designed for low-overlap pairs). Loop closures are accepted only if the alignment RMSE is below a threshold; a scene-level pose graph over submaps is optimized, then all submaps are fused into one unified TSDF → the complete static scene $P^s$ and the full set of articulated parts $\{P^{m_k}\}$ .

Method overview: split the egocentric video into static/dynamic fragments; per dynamic fragment estimate camera poses, sparse 3D trajectories, articulation-aware motion clusters, pixel-aligned part masks, joint part-pose/articulation optimization, and two-volume 3D reconstruction; finally align all fragments into one functional 3D scene.

4. Datasets

HOI4D was the only prior dataset for the task but offers low-motion, single-object, lab interactions with no full scenes. FunRec introduces two datasets and adapts HOI4D:

RealFun4D (real): 351 in-the-wild human-scene interactions across 60 apartments in 4 countries. Head-mounted Azure Kinect DK, $1920\times1080$ @ 15 FPS, synchronized RGB + depth. Annotations: interaction intervals + text descriptions, per-frame 2D hand and part masks (for dynamic-region filtering and static-scene reconstruction), articulation joints, and per-frame 2D part tracks.
OmniFun4D (simulated): 127 photorealistic sequences in 12 OmniGibson scenes (derived from iGibson). A human operator triggers scripted interactions; camera poses + events are logged and replayed offline, rendered with NVIDIA RTX path tracing. Stochastic Gaussian perturbations are added to camera poses to emulate natural head motion.
HOI4D (adapted): 30 interactions across 4 articulated categories (laptop, cabinet, safe, trash can); the provided 6D part poses are post-processed into ground-truth articulation parameters.

A benchmark evaluation set of 60 interaction sequences is drawn from RealFun4D + OmniFun4D using the annotated interaction intervals.

5. Experiments & Results

Tasks / metrics: articulated motion estimation (axis-direction error °, axis-position error m [revolute only], per-timestep joint-state error [° revolute / m prismatic], failure rate %); 6D part pose (ADD and its symmetric variant ADD-S — average 3D distance between predicted and GT part points, reported as accuracy % ↑); reconstruction (Chamfer Distance, CD cm ↓ — bidirectional nearest-neighbor surface distance); moving-part segmentation (mIoU ↑).

Baselines (three families): Type 1 — 4D reconstruction pipelines (MonST3R, SpatialTrackerV2); Type 2 — 6D object-pose tracking (BundleSDF, given GT camera poses + masks); Type 3 — articulated-object reconstruction (ArtGS, needs two static open/closed states, cannot track motion). Type 1 baselines are augmented (MonST3R + ICP + CoTracker3; SpatialTrackerV2 + SAM2) and optionally fed GT depth for fairness.

Articulated motion estimation (Table 1, revolute / prismatic where split):

Method	OmniFun4D Axis° · State · Fail	HOI4D Axis° · State · Fail	RealFun4D Axis° · State · Fail
MonST3R (GT depth+CoTr3)	57.6/63.5 · 46.5°/0.18m · 11.7%	48.6/30.1 · 37.6°/0.06m · 0%	51.4/56.2 · 54.3°/0.21m · 0%
SpatialTrackerV2 (GT depth)	48.7/69.3 · 46.8°/0.19m · 30%	54.3/38.8 · 38.7°/0.07m · 0%	60.8/42.6 · 55.6°/0.22m · 20%
BundleSDF (GT mask)	38.2/55.9 · 23.4°/0.20m · 55%	26.3/24.5 · 9.1°/0.07m · 43.3%	32.0/52.1 · 15.6°/0.22m · 36.7%
ArtGS	64.3/23.4 · — · 95%	60.5/53.2 · — · 66.7%	70.1/18.2 · — · 66.7%
FunRec	5.3/5.4 · 5.0°/0.02m · 1.7%	12.4/1.3 · 9.1°/0.02m · 0%	5.6/5.5 · 8.4°/0.03m · 0%

FunRec estimates axis direction within $5.3^\circ$ and position within $0.03$ m on OmniFun4D — beating BundleSDF by $>30^\circ$ and an order of magnitude in distance — with a $0\%$ failure rate on both HOI4D and RealFun4D (axis-position error is defined for revolute joints only).

Moving-part segmentation (Table 3, mIoU): FunRec 77.9 / 76.4 / 74.8 (Omni / HOI4D / Real) vs MonST3R 23.6–26.8 and SpatialTrackerV2 5.8–13.4 → up to +50 mIoU.

6D part pose & reconstruction (Table 2, ADD-S / ADD ↑ · CD cm ↓):

Method	OmniFun4D	HOI4D	RealFun4D
BundleSDF (GT mask)	22.8 / 12.4 · 17.1	53.1 / 35.2 · 1.4	20.9 / 12.9 · 10.6
FunRec	79.0 / 71.3 · 3.2	79.4 / 69.9 · 0.7	75.6 / 68.1 · 6.1

Part-pose accuracy is more than 2× BundleSDF and far above the dynamic trackers; CD drops to 3.2 / 0.7 / 6.1 cm. The gains confirm that jointly optimizing part pose + articulation yields stable, temporally consistent motion, and the training-free design generalizes across synthetic, controlled, and real egocentric recordings.

6. Applications

Simulation-ready export: articulation parameters define revolute/prismatic joints between static structure and movable parts; export to URDF/USD. Physical properties (mass, inertia) are inferred from RGB by querying a VLM (GPT-5). Loaded directly into physics simulators (e.g. Isaac Sim), where a simulated mobile manipulator interacts with a drawer reconstructed from a real scan.
Hand-guided affordance mapping: 3D hand meshes from off-the-shelf estimators are aligned into the digital-twin space → localize the hand in 3D, recover contact regions on the object, and reason jointly over hand motion and part motion.
Robot-scene interaction from human demonstration: the functional model transfers to a mobile manipulator (Boston Dynamics Spot with Arm). From the inferred contact points, articulation parameters, and interaction trajectory, the robot reliably reproduces a human-demonstrated cabinet opening in the real world.

7. Strengths / Limitations / Future Work

Strengths

Only method to produce scene-scale, physically grounded functional twins from a single casual egocentric RGB-D video — no rig, multi-state capture, or CAD priors.
Training-free: composes foundation-model priors with classical geometric optimization → strong cross-domain generalization (synthetic / lab / real) with near-zero failure rate.
Large, consistent margins on every metric and dataset; canonical-frame part reconstruction recovers occluded interiors.
Ships two reusable benchmarks (RealFun4D, OmniFun4D) plus concrete simulation / affordance / robotics demos.

Limitations

Only two joint types (prismatic, revolute) — no multi-DoF, screw, or deformable articulation; one interacted part per dynamic fragment.
Heavy dependence on many foundation models (VLM fragment/joint-type classification, RoMa, VISOR masks, TAPIP3D, SAM/SAM2, PREDATOR); errors (e.g. wrong VLM joint type) propagate, and inference cost is non-trivial.
Needs RGB-D input with known intrinsics; parts must actually be manipulated in-video to be reconstructed (no interaction → no articulation recovered).
Physical properties (mass/inertia) are VLM-guessed, not measured.

Future work (note-level extrapolation, not all stated by the paper): richer joint models and multi-part interactions; reducing the foundation-model stack / cost; monocular RGB-only functional reconstruction; online / on-robot operation for embodied agents.

References

Delitzas, A., Zhang, C., Gavryushin, A., Di Mario, T., Sun, B., Dabral, R., Guibas, L., Theobalt, C., Pollefeys, M., Engelmann, F., & Barath, D. (2026). FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos. arXiv:2604.05621, project page.
Liu, Y., et al. (2022). HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction. CVPR. — source dataset adapted for evaluation.
Edstedt, J., et al. (2024). RoMa: Robust Dense Feature Matching. CVPR. — dense correspondence.
Darkhalil, A., et al. (2022). EPIC-KITCHENS VISOR Benchmark. NeurIPS. — hand/object masks.
Barath, D. (2025). SupeRANSAC: One RANSAC to Rule Them All. arXiv:2506.04803. — robust estimation.
Huang, S., et al. (2021). PREDATOR: Registration of 3D Point Clouds with Low Overlap. CVPR. — submap registration.
Liu, Y., et al. (2025). ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting. ICLR. — articulated-reconstruction baseline.
Wen, B., et al. (2023). BundleSDF. CVPR. — 6D pose-tracking baseline.
Zhang, J., et al. (2025). MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR. — 4D-reconstruction baseline.
Xiao, Y., et al. (2025). SpatialTrackerV2: Advancing 3D Point Tracking with Explicit Camera Motion. ICCV. — 4D-reconstruction baseline.

Table of Contents