TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action
Jen-Hao Cheng · Yi-Hao Peng · Huapeng Zhou · Vivian Wang · Huayu Wang · Hsiang-Wei Huang · Wenhao Chai · Hou-I Liu · Kuang-Ming Chen · Cheng-Yen Yang · Yi-Ling Chen · Vibhav Vineet · Qin Cai · Jenq-Neng Hwang
TEMPURA improves fine-grained temporal grounding and causal reasoning in videos through masked event prediction and dense event segmentation, trained on our video dataset (VER) with structured temporal annotations.
摘要
Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLMs). We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances the video temporal understanding of VLMs. Inspired by infilling techniques in language modeling, TEMPURA first performs masked event prediction, learning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations. It then learns video segmentation and dense captioning, decomposing videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, our large-scale dataset of 500K videos annotated with temporally aligned event descriptions and structured reasoning steps. Experiments on video temporal grounding and highlight detection benchmarks show that TEMPURA substantially improves strong base VLMs across model families and scales, confirming that combining event-level reasoning with fine-grained temporal segmentation is an effective recipe for video temporal understanding.