Cat- M o D : Accelerating Multimodal Alignment via Caption Token Guided Asymmetric Mixture-of-Depths
YiJie Huang · Xiaocui Yang · Shi Feng · Wen Zhang · Kaisong Song · Yifei Zhang · Daling Wang
摘要
Efficiently aligning visual features with Large Language Models (LLMs) remains a critical bottleneck in Multimodal LLMs. Existing query-based alignment modules (e.g., Q-Former) rely on randomly initialized queries, resulting in an inefficient cold start exploration process. Furthermore, they enfo...