4D Point Splatting for mmWave Radar Micro-Doppler Differentiable Rendering
Abstract
Millimeter-wave radar resolves the micrometer displacements of a moving body as micro-Doppler, enabling gait, gesture, activity, pose and vital-sign sensing. Each task relies on a downstream model trained on real data, yet radar datasets lag orders of magnitude behind vision and language. Brute-force collection will not close that gap, and existing synthesis methods (forward-only simulators, learned generators and static-scene inverse renderers) cannot fit a real capture of a moving body, leaving a sim2real gap. We present 4D Point Splatting (4DPS), a physics-based differentiable renderer that fits an explicit moving scene to a real capture. Using the synchronized camera to initialize the room and body, we represent the scene as oriented points carrying material, normal and velocity, and splat each point into range and Doppler bins. Our output is the raw complex return a real radar records, so we apply the same signal processing on the rendered data to obtain any representation for training any SOTA downstream model. We fit 235 sequences on HuPR and 100 on RT-Pose, two datasets with different radars. Our reconstructions set the state of the art, with median correlations of 0.940 and 0.830 on range-azimuth, 0.829 and 0.695 on range-Doppler, and 0.988 and 0.966 on micro-Doppler. Our renders also set the state of the art in downstream sim2real gap. On RT-Pose, a pose model trained on them reaches 21.9 cm MPJPE on real data, against 16.7 cm for real training and 127.6 cm for the RF-Genesis baseline, where both the rendered- and real-trained models learn nearly the same features. Our custom CUDA kernels fit a sequence in 5.7 minutes on HuPR and 12.9 on RT-Pose on one H100, making dataset-scale inverse rendering of mmWave radar possible for the first time. This work can enable synthetic data generation for any human-centered downstream task and model, and scene editing for data augmentation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a physics-based differentiable renderer for mmWave radar that enables dataset-scale inverse rendering, significantly reducing the sim2real gap for downstream tasks. The combination of a novel point-based representation, efficient CUDA implementation, and rigorous evaluation on multiple datasets demonstrates a substantial technical contribution with high potential for adoption in radar sensing research.
Methodology
The paper proposes 4D Point Splatting (4DPS), a physics-based differentiable renderer for mmWave radar. The core innovation lies in representing the scene as oriented points with material, normal, and velocity attributes, which are splatted into range and Doppler bins to simulate the raw complex radar return. This approach bridges the gap between static scene rendering and dynamic micro-Doppler simulation, allowing for the fitting of explicit moving scenes to real captures. The method leverages synchronized camera data for initialization, ensuring geometric consistency. The development of custom CUDA kernels is a significant technical contribution, enabling efficient fitting of sequences (5.7-12.9 minutes on H100), which makes dataset-scale inverse rendering feasible for the first time in this domain.
Results & evaluation
The experiments are conducted on two distinct datasets, HuPR and RT-Pose, which utilize different radar hardware, demonstrating the method's generalizability. The results show state-of-the-art reconstruction quality with high median correlations (e.g., 0.988 on micro-Doppler for HuPR). The downstream evaluation on RT-Pose is particularly strong, showing that a pose model trained on rendered data achieves 21.9 cm MPJPE on real data, significantly outperforming the RF-Genesis baseline (127.6 cm) and approaching the performance of models trained on real data (16.7 cm). This effectively demonstrates a reduction in the sim2real gap.
Reproducibility
The paper provides specific details on the hardware used (H100) and the time taken for fitting, which aids in understanding the computational requirements. However, the full text provided does not explicitly list a code repository or detailed hyperparameter settings in the main body, though the mention of custom CUDA kernels suggests a significant engineering effort that would likely be shared in a full release. The reliance on synchronized camera data for initialization is a clear procedural detail.
Limitations
The method requires synchronized camera data for initialization, which may not always be available in all sensing scenarios. The fitting time, while fast for the domain, is still non-trivial (minutes per sequence), which could limit real-time application. The evaluation is limited to human-centric tasks (gait, pose); performance on other object types or complex multi-agent scenes is not explored.
Broader impact
This work has significant implications for the radar sensing community by providing a scalable method for synthetic data generation. By enabling the creation of large-scale synthetic datasets that closely mimic real radar returns, it can accelerate the training of downstream models for various human-centered tasks, reducing the reliance on expensive and time-consuming real-world data collection. It also opens avenues for scene editing and data augmentation in radar-based perception systems. The paper introduces a physics-based differentiable renderer for mmWave radar that enables dataset-scale inverse rendering, significantly reducing the sim2real gap for downstream tasks. The combination of a novel point-based representation, efficient CUDA implementation, and rigorous evaluation on multiple datasets demonstrates a substantial technical contribution with high potential for adoption in radar sensing research.
Author of this paper? Send a correction.