Eugene Yang

We introduce the task of grounded article generation with the goal of creating a Wikipedia-style article from multiple diverse videos about real-world events—from natural disasters to political elections—where all the information in the article is supported by video evidence. Videos are intuitive...

The unjudged document problem, where systems that did not contribute to the original judgement pool may retrieve documents without a relevance judgement, is a key obstacle to the reuseability of test collections in information retrieval. This is commonly dealt with by treating unjudged documents...

Long-form Retrieval-Augmented Generation (RAG) brings the challenge of coverage-based ranking, because ranking methods must ensure the inclusion of comprehensive relevant nuggets (i.e., facts), which can thereby be synthesized into a comprehensive output. In this work, we propose CoveR, a dense r...

We present the Auto-Judge resource for the meta-evaluation of automated LLM judges, especially judges that evaluate Retrieval-Augmented Generation (RAG) systems that ground their response with citations. The resource couples (i) a data release of topics, pooled RAG responses, and human judgments,...

Prior work has shown that vector-based pseudo relevance feedback (PRF) is an effective technique for query expansion for improving retrieval results in dense information retrieval. In dense retrieval, ColBERT-PRF has emerged as a novel mechanism, using cluster centroids built from feedback docume...

While ColBERT is an effective neural retrieval architecture, it requires a heavy index structure to support candidate set retrieval based on approximated token embeddings, gathering and decompressing document token embeddings, and applying the MaxSim operation. Indexes in PLAID and similar ColBER...

We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results.Information coverage is a central aspect for retrieval, especially when the retrieval system is integrated with generativ...

Nugget-based LLM judges evaluate Retrieval-Augmented Generation (RAG) systems using a bank of questions that capture the key facts and criteria an answer should address. These nugget banks are typically constructed through a combination of human input and LLM generation. System outputs are graded...

Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools designed for report generation are lacking. Accordingly, we introduce Auto-ARGUE, a robust LLM-based implementation o...

Advancements in retrieval models necessitate re-indexing, a computationally expensive process for large-scale production environments. While updating only the query encoder and continuing to use the old index could be a promising middle ground, dense retrieval systems suffer severe performance dr...