Takashi Kodama

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

Average loss curves during language model pretraining obscure diverse training dynamics across individual training data instances. Gaining a more granular understanding of these dynamics currently requires tracking intermediate checkpoint performance on predefined tasks, or computing higher-order gradients with respect to each data instance. To instead enable a scalable bottom-up decomposition of language model training dynamics, we propose identifying data-driven learning domains using non-negative matrix factorization (NMF) to group training data based solely on their per-instance losses, without any gradient computation. Applied to 3.2k checkpoints across 21 configurations of the Pythia model suite, our method automatically recovers distinctive training dynamics for different training data clusters. The resulting learning domains are consistent across model sizes and random seeds, while being interpretable as, e.g., code, mathematics, repetitive patterns, and non-English texts. The corresponding domain-level loss curves further reveal potential causes for low-performing training runs that do not surface in the average loss alone.