DynOLAT: OLAT Video Prediction for Controllable Dynamic Full-Body Relighting
Abstract
Accurate lighting response is essential for realistic dynamic human relighting, because shadows, highlights, and color changes should follow the target illumination in both direction and relative strength. Current learning-based relighting methods can generate visually plausible videos, but they often exhibit imprecise lighting responses: shadows and highlights may drift away from the target direction, and the output brightness may not faithfully follow the relative intensities in the target environment, making the result closer to appearance harmonization than controllable relighting. To address this issue, we propose DynOLAT, a generalizable OLAT video prediction framework for dynamic full-body humans. Instead of directly synthesizing a single relit appearance, our method predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, and then linearly combines the predicted bases according to the target environment map. This OLAT-based formulation makes the directional response explicit in the predicted bases, while the relative lighting strength is determined by the linear weights derived from the environment map. We further introduce a dynamic capture pipeline and the first large-scale dynamic full-body OLAT dataset, comprising over 24 million frames from 224 identity-outfit combinations, where different outfits of the same person define separate combinations. Experiments show that DynOLAT responds more faithfully to variations in target lighting direction and relative intensity than the tested baselines, enabling controllable video relighting while preserving fine appearance details and temporal coherence. The dataset and processing tools will be publicly released to foster future research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DynOLAT introduces a novel OLAT video prediction framework and a large-scale dynamic full-body dataset that enables controllable, physically consistent relighting of humans by explicitly modeling directional lighting responses and intensity scaling, significantly improving upon existing methods that suffer from lighting drift and intensity inaccuracies.
Methodology
The paper proposes DynOLAT, a framework that shifts the paradigm of video relighting from direct appearance synthesis to the prediction of temporally consistent One-Light-At-a-Time (OLAT) basis videos. By predicting these bases from flat-lit inputs and linearly combining them using environment map weights, the method explicitly decouples directional lighting response from intensity scaling. This formulation addresses the common failure mode of diffusion-based relighting models where shadows and highlights drift or intensity does not match the target environment. The use of a latent bridge matching schedule (LBM) with LoRA fine-tuning suggests a robust approach to maintaining temporal coherence in video generation, which is a significant technical hurdle in dynamic relighting.
Results & evaluation
The authors introduce a dynamic capture pipeline and claim the first large-scale dynamic full-body OLAT dataset with over 24 million frames from 224 identity-outfit combinations. This dataset contribution is substantial and likely to be a key resource for the community. The experiments demonstrate improved fidelity in lighting direction and relative intensity compared to baselines like UniRelight and DiffusionRenderer. The qualitative results described in the abstract and teaser figures show clear improvements in shadow orientation and color accuracy, particularly in complex lighting scenarios.
Reproducibility
The paper promises to release the dataset and processing tools, which is a strong positive for reproducibility. The specific architectural details (LoRA, 16-step LBM schedule) are mentioned, but the full technical report would be needed to verify the exact implementation details of the OLAT prediction network and the capture pipeline. The scale of the dataset (24M frames) makes independent reproduction of the training process difficult without access to the released data.
Limitations
The method relies on a specific capture pipeline for the dataset, which may limit generalizability to in-the-wild videos with varying camera motion or occlusions. The linear combination of OLAT bases assumes a certain level of linearity in the lighting response, which may not hold for highly non-linear materials or extreme lighting conditions. Additionally, the computational cost of generating and combining multiple basis videos for each frame could be a bottleneck for real-time applications.
Broader impact
This work has significant potential impact on virtual production, gaming, and AR/VR applications where realistic dynamic relighting of human characters is crucial. The release of a large-scale OLAT dataset will likely accelerate research in controllable video generation and relighting. The explicit modeling of lighting direction and intensity provides a more physically grounded approach to relighting, which could influence future architectures in the field. DynOLAT introduces a novel OLAT video prediction framework and a large-scale dynamic full-body dataset that enables controllable, physically consistent relighting of humans by explicitly modeling directional lighting responses and intensity scaling, significantly improving upon existing methods that suffer from lighting drift and intensity inaccuracies.
Author of this paper? Send a correction.