Prefix Sliding for efficient test-time scaling

Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem.

Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive.…