The Key to Going Linear: Analysis-Driven Transformer Linearization

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference.

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design…