Quality of Rejections Matters: Preference Data Construction for Faithful Summarization
Jihwan Oh · Taewon Yun · Minkyu Che · Sundong Kim · Hwanjun Song
Summarization alignment depends more on the quality of rejected summaries than their quantity, and a simple prompt can generate optimal rejections for improving factuality.
摘要
Factual reliability is a key challenge in abstractive summarization, as LLMs continue to generate hallucinations. A widely adopted solution is preference optimization, training models to prefer faithful over unfaithful summaries, but prior work emphasized the quantity of rejections over their alignment quality. In this paper, we show that effective alignment arises when rejected summaries achieve high alignment potential, characterized by a small preference margin that keeps rejections non-trivial and a large factuality margin that enforces clear factual contrast. Through both theoretical analysis and controlled prompting, we show that three factors, hallucination level, summary length, and prompt complexity, critically shape alignment potential. Building on these insights, we propose SPICE, a simple prompting strategy that produces rejected summaries with strong alignment value. Across diverse model scales and alignment algorithms, SPICE achieves superior factuality without sacrificing coherence, relevance, or abstractiveness, outperforming existing rejection strategies.