Verb-ICL: Rethinking In-Context Learning for Structured Prediction
Fan Bai · Hengshuo Miao · Sanjit Singh Batra · Hamid Hassanzadeh · Ardavan Saeedi · Mark Dredze
Under a fixed annotation budget, Verb-ICL improves in-context structured prediction along two axes: selecting which examples to annotate by token-level coverage, and enriching what those annotations contain with LLM-generated error feedback.
摘要
Structured prediction tasks pose two challenges: outputs composed of sub-sentence units that require fine-grained local context understanding, and task-specific conventions that pretraining alone cannot supply. Both are especially difficult under few-shot in-context learning (ICL), where the model must infer them from a small demonstration set selectively annotated under a fixed budget. We propose Verb-ICL, which addresses them along two axes: not only which examples to annotate, but also what those annotations should contain. Verb-ICL selects examples with a token-level coverage strategy that captures the local patterns these tasks predict over, then generates error feedback that codifies these conventions and attaches it to the corresponding demonstrations. Across six datasets spanning information extraction and semantic parsing, and with multiple LLMs, Verb-ICL consistently outperforms strong selective annotation baselines and continues to gain as the annotation budget increases. Analyses show that our feedback design beats globally aggregated and label-free alternatives, and that both components reduce structural violations. The feedback also helps regardless of the selection strategy.