Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Measuring training data influence consistently across language model pretraining is challenging.

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints…