Communicate the Change: Efficient Expert Parallelism via Activation Compression
Abstract
Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
EchoMoE introduces a cross-layer activation compression technique that significantly reduces communication overhead in MoE inference. The paper presents a rigorous system-level evaluation demonstrating substantial throughput gains and RDMA reductions, offering a practical path to scaling MoE models on heterogeneous and bandwidth-limited hardware.
Methodology
The paper proposes EchoMoE, a method to reduce communication overhead in Mixture-of-Experts (MoE) models by exploiting the high similarity between hidden states of adjacent transformer layers. The core idea is to cache a reference activation from an earlier layer and transmit only the compressed difference (residual) to remote experts during dispatch, while also compressing the expert outputs during combine. The method is designed as a retrofit, requiring only a small amount of fine-tuning (approx. 4.3B tokens) rather than full pre-training. The technical approach involves learning linear compressors/decompressors and folding these projections into the expert weights to avoid runtime overhead. The mathematical formulation is sound, leveraging the linearity of the expert MLPs to absorb the reconstruction steps. The observation that residual stream updates are incremental and predictable is well-motivated and empirically validated.
Results & evaluation
The experiments are extensive, evaluating Qwen3-30B-A3B across 18 benchmarks and 97 trained variants. The paper demonstrates a 58% reduction in inter-node RDMA traffic with a 22.6% throughput improvement. It also validates the approach on four other MoE models (Gemma 4, Kanana-2, Nemotron-3.5, Moonlight). The ablation studies are thorough, comparing against plain compression, LSH-MoE, and analyzing the impact of stride and code width. The throughput benchmarks on DGX H100 nodes provide concrete evidence of the system-level benefits. The performance drop is kept within 3.8 percentage points, which is a reasonable trade-off for the significant communication savings.
Reproducibility
The authors release code, models, and a vLLM integration. The paper provides detailed hyperparameters, training recipes, and evaluation protocols. The use of standard benchmarks and open-source models (Qwen3, etc.) enhances reproducibility. The specific hardware setup (DGX H100) is clearly defined, although results may vary on different interconnects.
Limitations
The method requires a non-trivial amount of fine-tuning (4.3B tokens) to retrofit existing models, which may be prohibitive for some users. The communication savings are topology-dependent; at very high EP degrees (e.g., EP128), the cost of broadcasting reference states to all nodes reduces the net savings. The method assumes a specific hardware setup where RDMA is the bottleneck; on systems with faster interconnects or different parallelism strategies, the benefits may be less pronounced.
Broader impact
This work addresses a critical bottleneck in scaling MoE models, which are increasingly used for frontier LLMs. By enabling efficient distributed inference over bandwidth-constrained networks (like Ethernet), it could democratize access to large MoE models and enable new deployment scenarios such as wide-area distributed inference. The technique is complementary to existing systems like DeepEP and can be combined with quantization. EchoMoE introduces a cross-layer activation compression technique that significantly reduces communication overhead in MoE inference. The paper presents a rigorous system-level evaluation demonstrating substantial throughput gains and RDMA reductions, offering a practical path to scaling MoE models on heterogeneous and bandwidth-limited hardware.
Author of this paper? Send a correction.