Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Sean Michael McLeish · Ang Li · John Kirchenbauer · Dayal Singh Kalra · Brian R. Bartoldson · Bhavya Kailkhura · Avi Schwarzschild · Jonas Geiping · Tom Goldstein · Micah Goldblum
We retrofit depth recurrence into pretrained transformer models.
摘要
Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to retrofit existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that retrofitting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.