Beyond Logit Adjustment: A Residual Decomposition Framework for Long-Tailed Reranking
Zhanliang Wang · Hongzhuo Chen · Quan Minh Nguyen · Mian Umair Ahsan · Kai Wang
We propose REPAIR, a post-hoc reranking framework that decomposes residual ranking bias into classwise and pairwise corrections, improving long-tailed ranking on both standard benchmarks and rare disease diagnosis.
摘要
Long-tailed classification, where a small number of frequent classes dominate many rare ones, remains challenging because models systematically favor frequent classes at inference time. Existing post-hoc methods such as logit adjustment address this by adding a fixed classwise offset to the base-model logits. However, the correction required to restore the relative ranking of two classes need not be constant across inputs, and a fixed offset cannot adapt to such variation. We study this problem through Bayes-optimal reranking on a base-model top-$k$ shortlist. The gap between the optimal score and the base score, the *residual correction*, decomposes into a *classwise* component that is constant within each class, and a *pairwise* component that depends on the input and competing labels. When the residual is purely classwise, a fixed offset suffices to recover the Bayes-optimal ordering. We further show that when the same label pair induces incompatible ordering constraints across contexts, no fixed offset can achieve this recovery. This decomposition leads to testable predictions regarding when pairwise correction can improve performance and when cannot. We develop **REPAIR** (**Re**ranking via **Pair**wise residual correction), a lightweight post-hoc reranker that combines a shrinkage-stabilized classwise term with a linear pairwise term driven by competition features on the shortlist. Experiments across nine benchmarks spanning text classification, visual recognition, and multimodal rare-disease prediction confirm that the decomposition explains when pairwise correction is beneficial and when classwise correction alone suffices. Together, these results establish residual decomposition as a principled framework for understanding and improving post-hoc correction in long-tailed classification.