Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Gyuseok Lee · Wonbin Kweon · Zhenrui Yue · SeongKu Kang · Jiawei Han · Dong Wang
VRF represents each user's preferences as a variational distribution over shared probabilistic preference bases for uncertainty-aware personalized reward modeling.
摘要
Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. Our code is available at https://github.com/Gyu-Seok-Lee/VRF_COLM26.