LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Abstract
As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight $γ$: with depth, $γ$ grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise $γ$ vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and $<0.02\%$ change in FLOPs. Across a model ladder scaled up to $100$B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm and Layer-Norm Scaling, reaching Pre-Norm's 1.3B loss with $3.4\times$ less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by $3$-$10\times$, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LayerRoPE reinterprets residual stream norm growth as an emergent depth-positional encoding, replacing per-layer normalization weights with a shared, depth-conditioned RoPE-like mechanism that improves depth scaling, learning rate robustness, and compute efficiency across LLMs and ViTs. The paper provides a compelling theoretical and empirical case for widening rather than damping the residual stream, demonstrating state-of-the-art results in depth scaling up to 512 layers and significant compute savings in model training.
Methodology
The paper proposes LayerRoPE, a method that reinterprets the growth of hidden state norms in deep Transformers not as a pathology to be suppressed, but as an emergent depth-positional encoding. The core insight is that the RMSNorm weights ($\gamma$) across layers exhibit systematic magnitude growth and directional rotation, effectively encoding the layer index. LayerRoPE makes this explicit by replacing the $L$ independent per-layer $\gamma$ vectors with a single shared vector modulated by depth-conditioned scalars (magnitude and rotation) derived from a RoPE-like mechanism along the depth axis. This reduces parameter count and introduces a structured, learnable depth schedule. The methodology is sound, leveraging the analogy between positional encoding in sequence space and depth encoding in network space. The ablation studies confirm that both magnitude and rotation components are necessary and complementary.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors train a model ladder from 58M to 1.3B parameters on C4, comparing LayerRoPE against Pre-Norm, Post-Norm, Peri-Norm, and Layer-Norm Scaling. LayerRoPE consistently outperforms baselines, reaching Pre-Norm's 1.3B loss with 3.4x less compute. Crucially, the paper demonstrates superior depth scaling, with LayerRoPE being the only method to show monotonic loss improvement up to 512 layers, whereas baselines diverge or plateau. The paper also shows improved learning rate robustness (wider basin, delayed divergence) and successful transfer to looped latent models (Parcae) and Vision Transformers (ViT) without tuning. The analysis of the learned depth schedule reveals that LayerRoPE widens the residual stream rather than damping it, which is a counter-intuitive and significant finding.
Reproducibility
The paper provides detailed hyperparameters, training recipes, and compute accounting. It specifies the initialization of depth slopes, rotation base frequency, and optimizer settings. The use of standard datasets (C4, ImageNet) and architectures (LLaMA-style, ViT) enhances reproducibility. However, the specific code implementation is not linked in the text provided, which is a minor gap for immediate reproduction, though the description is sufficiently detailed for re-implementation.
Limitations
The primary limitation is the scale of the experiments. While 1.3B parameters and 512 layers are significant, the method's behavior at frontier scales (100B+) is extrapolated or tested at reduced budgets (6.7B). The paper relies on C4 for language modeling, which may not fully capture the benefits on more diverse or multilingual data. Additionally, the "widening" of the residual stream requires careful monitoring to ensure it doesn't lead to numerical instability in mixed-precision training, although the paper notes stability improvements.
Broader impact
This paper has high potential impact by challenging a fundamental assumption in Transformer architecture design: that activation norm growth is a defect. By providing a principled way to exploit this growth for depth encoding, it offers a path to training deeper, more stable, and more compute-efficient models. The transferability to looped models and ViTs suggests broad applicability across modalities. The reduction in learning rate sensitivity is particularly valuable for practitioners, as it simplifies hyperparameter tuning for deep networks. LayerRoPE reinterprets residual stream norm growth as an emergent depth-positional encoding, replacing per-layer normalization weights with a shared, depth-conditioned RoPE-like mechanism that improves depth scaling, learning rate robustness, and compute efficiency across LLMs and ViTs. The paper provides a compelling theoretical and empirical case for widening rather than damping the residual stream, demonstrating state-of-the-art results in depth scaling up to 512 layers and significant compute savings in model training.
Author of this paper? Send a correction.