Motion Equivariant Convolutional LSTM: When Motion Becomes a Gauge Symmetry
Abstract
Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper proposes a Motion Equivariant Convolutional LSTM that treats motion as a gauge symmetry, enabling equivariance to time-varying velocities. By using a velocity model to transport hidden states, the method outperforms flow-equivariant baselines in accuracy and generalization, offering a robust framework for sequential data with dynamic symmetries.
Methodology
The paper introduces a rigorous theoretical framework extending equivariance from static spatial symmetries to dynamic, time-dependent motions. By framing motion as a gauge symmetry over the time axis, the authors derive a "Motion Equivariant Convolutional LSTM" (MELSTM). The core innovation is the use of a velocity model (acting as a gauge connection) to transport hidden and cell states between timesteps, ensuring equivariance to arbitrary motions rather than just constant-velocity flows. The mathematical derivation is sound, proving that standard ConvLSTMs are not flow-equivariant for generic kernels (measure-zero set) and establishing the necessary transformation laws for the velocity model to maintain equivariance. The extension to local motions via slot tracking is a clever architectural choice that addresses the correspondence problem in multi-object scenes.
Results & evaluation
The experiments are well-designed to isolate the impact of the transport mechanism. Comparisons are made against a standard ConvLSTM and a Flow Equivariant ConvLSTM (FERNN) with identical parameter counts. The tasks (Moving MNIST prediction, Motion-Only MNIST classification, KTH action recognition under camera shake) effectively test the model's ability to handle time-varying velocities. The results show that MELSTM outperforms baselines, particularly in generalization to unseen motions and longer prediction horizons. The finding that MELSTM is less sensitive to recent velocity changes than the enumerating FERNN is a significant empirical insight.
Reproducibility
The code is available on GitHub. The paper provides detailed descriptions of the velocity estimation methods (phase correlation) and the architectural modifications. The use of parameter-free velocity models in the main experiments simplifies reproduction, though the paper notes that trainable velocity models are a future direction. The experimental setup is clearly defined, allowing for fair comparison.
Limitations
The primary limitation is the reliance on parameter-free velocity estimation (phase correlation) in the main experiments, which may not generalize to all types of motion or complex scenes without learning. The model assumes a specific structure for local motions (slots), which may not capture all possible scene decompositions. The theoretical framework is complex and may be difficult to implement for non-abelian groups or more complex Lie algebras.
Broader impact
This work bridges the gap between geometric deep learning and sequential modeling, offering a principled way to incorporate dynamic symmetries into recurrent architectures. It has potential applications in video prediction, robotics, and any domain where objects move with time-varying velocities. The gauge theory perspective provides a new lens for understanding equivariance in time-series data, which could inspire further research in this area. The paper proposes a Motion Equivariant Convolutional LSTM that treats motion as a gauge symmetry, enabling equivariance to time-varying velocities. By using a velocity model to transport hidden states, the method outperforms flow-equivariant baselines in accuracy and generalization, offering a robust framework for sequential data with dynamic symmetries.
Author of this paper? Send a correction.