Attention-Aware Routing: Coupling Routing and Attention in MoEs

Attention-Aware Routing (AAR) has been introduced as a new mechanism for Mixture-of-Experts (MoE) language models, augmenting the traditional router—which normally relies solely on a token’s hidden state—with temporal and spectral features derived from a sliding window of attention weights. By extracting a summary of the model’s contextual state that is separate from the hidden representation, AAR enhances routing decisions while keeping the underlying transformer frozen; only the routing parameters are trained. This approach yields a notable performance boost on the GSM8K benchmark, improving results by 3.37 percentage points over a routing‑only supervised fine‑tuning baseline on the OLMoE architecture.

The study reveals that routing and attention operate as a coupled circuit: modifications to routing at a given layer propagate through the residual stream and amplify “attention sinks” in the subsequent layer, effectively reshaping attention patterns without any direct changes to the attention mechanism itself. This coupling explains why AAR reduces instances of long, divergent generations—incorrect answers become shorter while correct answers retain their length—highlighting a more efficient alignment between routing decisions and the model’s attention dynamics. Moreover, the researchers found that the benefits of AAR are highly depth‑sensitive; indiscriminate application across all layers can harm factual retrieval, whereas introducing AAR deeper in the network preserves gains in mathematical reasoning, exposing a tension between retrieval and reasoning capabilities at different depths.

These findings position AAR as a controlled probe for investigating the information carried by attention at various layers, offering a tool to selectively enhance routing where it most benefits reasoning tasks. The depth‑specific behavior suggests that future MoE designs could leverage layer‑wise routing adjustments to balance retrieval accuracy and complex problem solving. By isolating routing as the sole variable and demonstrating its interaction with attention, the work provides a clear pathway for further research into modular improvements of large language models without altering their core transformer components.

⚡ Effects Interpreter