REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation

As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck.

As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follows a context but leaves the intermediate reasoning behind that continuation implicit. We introduce…