QualAlign: Benchmarking Automated Qualitative Coding Against Human Schemas
Taggert Smith · Chuong Nguyen · Qisen Yang · Oishani Bandopadhyay · Yaru Su · Nadia Polikarpova · Xinyu Pi
QualAlign benchmarks how well LLM-based qualitative coding systems align with expert human coders, showing that high semantic similarity often masks low agreement in actual code assignments.
摘要
Large language models are increasingly used for qualitative analysis, but existing inductive coding systems are difficult to compare because they use different datasets and system-specific evaluation metrics. We introduce \textbf{QualAlign}, a human-grounded benchmark for automated qualitative coding spanning 8 public datasets, each with research questions, inductively-derived codebooks, and code-excerpt assignments produced by expert researchers. The benchmark measures alignment with human expert coding via these artifacts using lexical, semantic, and assignment-based scores. Under a shared setup, we compare four recent systems for inductive codebook generation: LLooM, ThematicLM, HICode, and LOGOS. We find that while human and LLM-produced codebooks show moderate semantic resemblance given the same datasets and research questions, lexical agreement scores remains low, indicating different vocabulary used to describe similar concepts. Moreover, code-excerpt assignments diverge dramatically: given semantically similar conceptual labels, LLMs still partition the data differently than human researchers. These patterns suggest that current systems may be useful for proposing candidate codes or surfacing broad conceptual patterns for analyst review and revision—similar to prior topic modeling methods; however, they cannot mimic human researchers' excerpt-level coding decisions. Based on these findings, we caution that LLM-generated codebooks that appear semantically similar to human work cannot necessarily be taken to represent a similar qualitative interpretation of the data, and thus should not replace the interpretive work of human coders.