Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars
Abstract
Video diffusion models (VDMs) trained on internet-scale data encode a rich prior on appearance, motion, and physics that is difficult to replicate with explicit 3D rendering and costly 3D data capture. We introduce Video Diffusion Avatars (VDAs), a method that creates photo-realistic, multi-view consistent avatars from a single image by fine-tuning a VDM for precise control over camera viewpoint, facial expression, and upper-body pose. Departing from 3D morphable models (3DMMs), VDAs use a learned expression space that lifts the expressiveness ceiling of 3DMMs, e.g., for the tongue and mouth interior. However, learned expression codes exhibit heavy viewpoint bias, as our DISTRACT benchmark reveals. We show that this bias can be removed through disentanglement, enabling their use as driving signals for multi-view consistent avatars. Camera and upper-body pose are controlled jointly via proxy-mesh renderings, giving independent control over expression, pose, and viewpoint. A camera-triplet training strategy makes the approach efficient: our models train for 34 hours on a single A100 using only public multi-view data. VDAs outperform state-of-the-art single-image 3D avatars on novel-view reenactment, reducing face LPIPS by 29% and improving perceptual quality (JOD) by 66% on BecomingLit. A static 3DGS reconstruction probe further confirms that our generations are the most multi-view consistent. Built on a video prior, VDAs produce effects beyond the reach of explicit renderers, including physically plausible hair dynamics, shadows, reflections, and refractions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Video Diffusion Avatars (VDAs), a method that fine-tunes video diffusion models to create photo-realistic, multi-view consistent 3D avatars from a single image with precise control over expression, pose, and viewpoint. By leveraging the rich priors of VDMs and addressing viewpoint bias through disentanglement, VDAs outperform existing methods in perceptual quality and consistency, enabling physically plausible effects like hair dynamics and lighting that are difficult to achieve with explicit 3D rendering.
Methodology
The paper proposes Video Diffusion Avatars (VDAs), a method that leverages the rich priors of video diffusion models (VDMs) to generate controllable, multi-view consistent 3D avatars from a single image. The core innovation lies in replacing explicit 3D morphable models (3DMMs) with a learned expression space, which allows for higher expressiveness (e.g., tongue and mouth interior) but introduces viewpoint bias. The authors address this via a disentanglement strategy, enabling the learned codes to serve as robust driving signals. Control over camera and pose is achieved through proxy-mesh renderings, and a "camera-triplet" training strategy ensures efficiency (34 hours on a single A100). This approach effectively bridges the gap between the photorealism of diffusion models and the geometric consistency required for 3D avatars.
Results & evaluation
The experiments demonstrate significant improvements over state-of-the-art single-image 3D avatar methods. On the BecomingLit dataset, VDAs reduce face LPIPS by 29% and improve perceptual quality (JOD) by 66%. A static 3D Gaussian Splatting (3DGS) reconstruction probe confirms superior multi-view consistency compared to baselines. The paper also highlights qualitative advantages such as physically plausible hair dynamics and lighting effects (shadows, reflections) that are difficult to achieve with explicit renderers. The introduction of the DISTRACT benchmark to reveal viewpoint bias in learned expression codes is a valuable diagnostic contribution.
Reproducibility
The paper provides specific training details (34 hours on A100, public multi-view data), which aids reproducibility. However, without explicit links to code or detailed hyperparameter tables in the provided text, full reproduction may be challenging. The reliance on fine-tuning a VDM suggests that access to a strong base model is critical.
Limitations
The method requires fine-tuning a video diffusion model, which may be computationally intensive for non-experts despite the claimed efficiency. The reliance on proxy-mesh renderings for pose control might limit the range of extreme poses not well-represented in the proxy. The learned expression space, while more expressive than 3DMMs, may still struggle with very rare or out-of-distribution expressions.
Broader impact
This work has significant implications for the field of 3D human modeling, potentially shifting the paradigm from explicit geometric modeling to implicit diffusion-based generation. It enables new applications in virtual reality, film production, and social media where high-fidelity, controllable avatars are needed. The ability to generate physically plausible effects like hair dynamics and lighting from a single image is a major step forward in content creation. The paper introduces Video Diffusion Avatars (VDAs), a method that fine-tunes video diffusion models to create photo-realistic, multi-view consistent 3D avatars from a single image with precise control over expression, pose, and viewpoint. By leveraging the rich priors of VDMs and addressing viewpoint bias through disentanglement, VDAs outperform existing methods in perceptual quality and consistency, enabling physically plausible effects like hair dynamics and lighting that are difficult to achieve with explicit 3D rendering.
Author of this paper? Send a correction.