Fast Weight Attention for Continual Learning

Fast Weight Attention for Continual Learning treats a recurrent memory matrix as an online learner rather than a passive cache. Its central correction is temporal alignment: under read-after-write autoregression, each newly observed value v_t is written using the prefix feature ϕ(k_{t−1}), not the commonly used same-step feature ϕ(k_t). From this causal pairing, the paper derives Falcon-1, Falcon-2, and Falcon-3 regression updates using normalized least-mean-squares steps, explicit ridge-based forgetting, per-channel plasticity, and sliding-window rehearsal; Falcon-1A, Falcon-2A, and Falcon-3A provide corresponding inner-product writes. WY, triangular-solve, and ParallelFlow formulations enable chunk-parallel training while retaining fixed-size recurrent inference, complementing architectures such as Mamba-2, DeltaNet, Gated DeltaNet, and Transformer models. On FineWeb-Edu, 124M–130M-parameter models trained with a 50B-token budget achieved perplexity 17.10 for Falcon-1.3, compared with 17.32 for Gated DeltaNet, while Falcon-1A.2 reached a 49.30 zero-shot average and Falcon-1.3 reached 49.54 one-shot accuracy. On variable-digit addition trained through width 32 and evaluated across 33–48 digits, Falcon-3A.3 achieved 87.2 mean accuracy, exceeding Falcon-1A.3 at 85.9 and demonstrating improved length extrapolation. The implementation uses RMSNorm-based feature scaling and was evaluated on NVIDIA H100 or H200 GPUs; the work includes a ByteDance Seed contribution.