Lior Wolf

Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination. We propose a training-free, post-hoc representation editing technique that operates in the representation space of the language backbone. The method performs a lightweight, one-time offline calibration on a modest paired dataset to estimate the required covariance structures, using only forward passes and empirical second-order statistics with no gradient updates or fine-tuning, after which the correction is absorbed directly into the model's existing weights. By modeling hidden states as a superposition of truthful and hallucination-associated components, we derive a Wiener-type estimator whose optimal gains are given in closed form from the covariances of paired truthful and hallucinated representations. An eigendecomposition yields mode-wise attenuation that respects a stability criterion, i.e., the filter responds continuously to estimation noise. The correction is applied once to the feed-forward output projections of selected deeper layers, at inference time, the model runs unchanged and at the same speed. Experiments on LLaVA-1.5, MiniGPT-4, Gemma3, and mPLUG-Owl2 demonstrate consistent reductions in object hallucination on CHAIR, POPE, and MME while maintaining caption fluency and overall response quality. We further demonstrate the generality of our approach on the TempCompass video understanding benchmark and on discrete diffusion language models for grounded dialogue, showing that representation filtering reduces hallucinations even in temporal video reasoning and multi-step, sequence-wide denoising settings.

Transformer-based large language models (LLMs), trained via backpropagation on massive textual corpora, encode substantial factual knowledge, yet may also retain false or outdated associations. Prior work on factual editing in LLMs shows that fine-tuning (FT) often induces a repelling effect, degrading generalization and overwriting non-targeted knowledge, motivating alternative approaches such as rank-1 optimizations. In this work, we revisit knowledge editing in LLMs through FT by analyzing the activations and Vector-Jacobian Products (VJPs), the error signals produced during backpropagation when learning new facts. We show that each layer and token induces its own error signals, and that attempts to edit a specific association can inadvertently encourage the model to memorize the entire prompt in a diffuse or inefficient manner. Building on this analysis, we introduce Precise-FT (PFT), a method that constructs gradient updates from a small set of projected error signals most strongly associated with the learned fact, achieving strong performance across multi-knowledge editing benchmarks and LLMs.