Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Xinyu Wang · Sai Koneru · Wenbo Zhang · Wenliang Zheng · Saksham Ranjan · Sarah Rajtmajer
This paper introduces a benchmark that shows state-of-the-art fake news detectors perform well on fully false stories but fail when small false claims are embedded in otherwise accurate news.
摘要
Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and persuasive news-like content. While prior work has often treated fake news detection as a binary classification problem, today's fake news increasingly arises through multi-stage human–AI collaboration, where strategically placed inaccuracies are embedded within otherwise credible narratives. These mixed-truth cases are both more complex and more difficult to detect, yet remain underrepresented in existing benchmarks. To address this gap, we introduce MANYFAKE, a synthetic benchmark containing 6,798 news articles generated through a strategy-driven pipeline that explicitly captures the many ways fake news is constructed and refined. Using this benchmark, we evaluate a range of state-of-the-art fake news detectors, including LLMs with and without explicit reasoning. We find that models achieve high fake-news detection rates on fully fabricated stories, but remain brittle when falsehoods are subtle, optimized, and interwoven with accurate information. These results highlight substantial variation in detector robustness across different forms and degrees of falsity. The dataset is available at https://github.com/XinyuWang1998/MANYFAKE_Benchmark.