Why Fine-Tuning Encourages Hallucinations and How to Fix It
Guy Kaplan · Zorik Gekhman · Zhen Zhu · Yuval Reif · Lotem Rozner · Swabha Swayamdipta · Derek Hoiem · Roy Schwartz
SFT-induced hallucinations are factual forgetting caused by interference between new and stored entity representations, and self-distillation prevents it.
摘要
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t. knowledge acquired during pre-training. Since these errors arise as a by-product of knowledge degradation, we explore whether established continual learning tools can mitigate them. We propose a *self-distillation*–based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t. pre-existing knowledge by regularizing output-distribution drift. We also show that when new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups preserves task performance while reducing hallucinations. Lastly, we investigate the mechanism, contrasting capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, which self-distillation mitigates and an associative-memory model explains: forgetting grows with the overlap between new and stored facts.