MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
Haohang Huang · Xuan Lu · Mingyi Su · Xuan Zhang · Ziyan Jiang · Ping Nie · Kai Zou · Tomas Pfister · Wenhu Chen · Wei Zhang · Xiaoyu Shen · Rui Meng
We present MMEB-V3 and OmniSET for full-modality evaluation, revealing that existing embeddings fail to enforce modality constraints due to asymmetric retrieval, modality bias, and misaligned instruction-induced shifts.
摘要
Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.