One-sentence Summary

Jingdi Lei, Di Zhang, and Soujanya Poria from Nanyang Technological University and Fudan University propose Error-Free Linear Attention (EFLA), a theoretically exact, numerically stable linear-time attention mechanism derived from solving the continuous-time ODE governing linear attention via an infinite-order Runge–Kutta method. By exploiting the rank-1 structure of the dynamics matrix, EFLA achieves closed-form, error-free updates with full parallelism and linear complexity, outperforming DeltaNet in robustness to noise and downstream tasks while eliminating discretization errors inherent in prior Euler-based approximations.

Key Contributions

• Existing linear attention methods suffer from inherent numerical instability and truncation errors due to their reliance on first-order Euler discretization of continuous-time dynamics, which limits their accuracy and robustness in long-context scenarios despite their computational efficiency.

• The paper reformulates linear attention as a continuous-time dynamical system governed by a first-order ODE, revealing that the standard approach corresponds to a low-order numerical integration scheme that fails to capture the true evolution of the state.

• By exploiting the rank-1 structure of the dynamics matrix, the authors derive an exact closed-form solution equivalent to the infinite-order Runge–Kutta limit, achieving error-free integration with linear time complexity, full parallelism, and consistent performance gains over DeltaNet and other baselines.

Introduction

The authors leverage the growing role of large language models as autonomous agents in complex, long-context tasks—such as reasoning and tool use—where standard attention mechanisms become computationally prohibitive due to their quadratic time complexity. Prior linear attention methods, while efficient, rely on low-order numerical approximations like Euler integration to solve the underlying continuous-time dynamics, introducing truncation errors and instability, especially under long sequences or high decay rates. These approximations are inherently limited, with heuristic fixes like gating or adaptive coefficients only mitigating symptoms rather than eliminating the root cause. The authors’ main contribution is EFLA, a principled reformulation of linear attention as a continuous-time dynamical system governed by a first-order ODE. By exploiting the rank-1 structure of the system, they derive an exact closed-form solution equivalent to the infinite-order Runge–Kutta limit, achieving error-free integration while preserving linear time complexity. This approach not only ensures numerical stability and robustness in noisy settings but also outperforms existing methods like DeltaNet across benchmarks, offering a theoretically sound and practically scalable foundation for high-fidelity attention.

Method

The authors leverage a continuous-time dynamical systems perspective to derive an exact, error-free solution for linear attention, addressing the numerical instability and error accumulation inherent in low-order discretization schemes. The core insight is to model the online learning update of the associative memory state $\mathbf{S}_t$St​ as a first-order ordinary differential equation (ODE). This ODE is defined as $\frac{d\mathbf{S}(t)}{dt} = -\mathbf{A}_t\mathbf{S}(t) + \mathbf{b}_t$dtdS(t)​=−At​S(t)+bt​, where the dynamics matrix $\mathbf{A}_t = \mathbf{k}_t\mathbf{k}_t^\top$At​=kt​kt⊤​ and the forcing term $\mathbf{b}_t = \mathbf{k}_t\mathbf{v}_t^\top$bt​=kt​vt⊤​ are derived from the key and value vectors at time $t$t. This formulation generalizes the delta rule update, which corresponds to a first-order explicit Euler discretization of this ODE. By recognizing that the dynamics matrix $\mathbf{A}_t$At​ is rank-1, the authors exploit its algebraic properties to compute the exact analytical solution of the ODE. This solution, which corresponds to the infinite-order limit of the Runge-Kutta family of methods, is given by $\mathbf{S}_t = e^{-\beta_t \mathbf{A}_t} \mathbf{S}_{t-1} + \int_0^{\beta_t} e^{-(\beta_t - \tau)\mathbf{A}_t} \mathbf{b}_t \, d\tau$St​=e−βt​At​St−1​+∫0βt​​e−(βt​−τ)At​bt​dτ. The rank-1 structure allows the matrix exponential $e^{-\beta_t \mathbf{A}_t}$e−βt​At​ to be computed in closed form as $\mathbf{I} - \frac{1 - e^{-\beta_t \lambda_t}}{\lambda_t} \mathbf{A}_t$I−λt​1−e−βt​λt​​At​, where $\lambda_t = \mathbf{k}_t^\top \mathbf{k}_t$λt​=kt⊤​kt​. Similarly, the integral term simplifies to $\frac{1 - e^{-\beta_t \lambda_t}}{\lambda_t} \mathbf{b}_t$λt​1−e−βt​λt​​bt​. Substituting these closed-form expressions yields the final update rule for the Error-Free Linear Attention (EFLA) mechanism. This update maintains linear time complexity with respect to sequence length, enabling efficient computation while capturing the exact continuous dynamics.

[[IMG:|The framework diagram illustrates the continuous-time dynamical system formulation of linear attention, showing the state $\mathbf{S}(t)$S(t) evolving according to the ODE $\frac{d\mathbf{S}(t)}{dt} = -\mathbf{A}_t\mathbf{S}(t) + \mathbf{b}_t$dtdS(t)​=−At​S(t)+bt​. The figure highlights the transition from the discrete delta rule update to the continuous-time model, emphasizing the role of the dynamics matrix $\mathbf{A}_t$At​ and the forcing term $\mathbf{b}_t$bt​. The exact solution of this ODE, derived using the rank-1 structure of $\mathbf{A}_t$At​, is the foundation of the EFLA mechanism.]]

Experiment

• Numerical Stability and Robustness Verification: On sMNIST with pixel dropout, OOD intensity scaling, and additive Gaussian noise, EFLA outperforms DeltaNet in convergence speed and robustness, maintaining high accuracy under severe interference. EFLA achieves significantly better performance at high input scales and with larger learning rates, validating its exact saturation mechanism mitigates error accumulation and state explosion.
• Language Modeling: On Wikitext and zero-shot reasoning tasks (LAMBADA, PiQA, HellaSwag, WinoGrande, ARC-e, ARC-c, BoolQ, OpenBookQA, SciQ), EFLA with 340M parameters achieves lower perplexity (37.01 vs. 38.09) and higher accuracy (23.9% vs. 22.5% on LAMBADA), with a +7.4% absolute improvement on BoolQ. At 1.3B parameters, EFLA maintains a performance lead even at 16B tokens, indicating superior long-sequence fidelity and scalability.

The authors use a 340M and 1.3B parameter model to compare EFLA against DeltaNet on language modeling and reasoning tasks, with results shown in Table 1. Results show that EFLA consistently outperforms DeltaNet across most metrics, achieving lower perplexity on Wikitext and LAMBADA and higher accuracy on multiple reasoning benchmarks, with the performance gap widening at larger model sizes.