Completing Dynamic Human Reconstruction with I2V-Synthesized Views
Zhewen Zheng, Changwei Yao, Aman Goel
Carnegie Mellon University
Abstract
Reconstructing dynamic human avatars from a single monocular video is fundamentally ill-posed due to depth ambiguity, self-occlusion, and unstable 2D point tracks under fast motion. We present a practical framework that combines generative view augmentation with human-aware 3D tracking to improve monocular 4D reconstruction quality within the Shape-of-Motion (SoM) pipeline.
A commercial image-to-video diffusion model synthesizes a smooth turnaround sequence from a pseudo-canonical frame, exposing geometry that is never observed in the original capture. We further incorporate structured correspondences from the Momentum Human Rig (MHR), which provides stable body-aware 3D tracks and reduces drift and fragmentation compared to standard 2D tracking. These improved priors supervise SoM optimization through photometric, depth, and motion objectives, yielding more consistent human avatars over time.
On DNA-Rendering sequences, our method produces measurably better foreground PSNR and qualitatively more stable reconstructions than baselines, while also reducing preprocessing time by an order of magnitude.
Shape-of-Motion Pipeline
Method Overview
Our pipeline combines generative view augmentation, SAM-3D-Body based 3D tracking, and a depth prior to supervise Shape-of-Motion (SoM) optimization of canonical 3D Gaussians and SE(3) motion bases, producing high-fidelity 4D human avatars from a single monocular video.
Method
Our method is built on the Shape-of-Motion (SoM) framework, but augments it with stronger supervision from both generative models and human-aware tracking. Given a monocular input video, we first select a pseudo-canonical frame and feed it to a commercial image-to-video diffusion model to synthesize a smooth, palindromic turnaround sequence of the subject.
In parallel, we run SAM-3D-Body and the Momentum Human Rig (MHR) to extract temporally-stable, body-anchored 3D correspondences, which serve as robust motion anchors even under fast articulation and self-occlusion. A depth network provides smooth depth maps that act as a geometric prior. These augmented views, tracks, and depths jointly supervise SoM optimization of canonical 3D Gaussians and low-rank SE(3) motion bases.
By constraining SoM with generative coverage of unseen views and structured human priors, our approach reduces drift and fragmentation, leading to more consistent 4D human avatars and improved foreground reconstruction quality.
Qualitative Comparisons
Baseline vs. Ours
We compare our method against a baseline Shape-of-Motion configuration without generative view augmentation or MHR-based tracking. Each row shows a specific modality rendered over time; videos on the left are from the baseline, while videos on the right are produced by our full model.
RGB Frames
Baseline (left) vs. Ours (right)
Key Contributions
- Generative view augmentation using an image-to-video diffusion model to synthesize smooth turnaround sequences from monocular input.
- Human-aware 3D tracking via Momentum Human Rig (MHR), providing stable, body-anchored correspondences that reduce drift compared to generic 2D trackers.
- Integration of these priors into the Shape-of-Motion pipeline through photometric, depth, and motion losses for high-fidelity 4D human avatars.