Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training.

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that…