StoryScope: Investigating idiosyncrasies in AI fiction
Jenna Russell · Rishanth Rajendhran · Mohit Iyyer · John Frederick Wieting
We introduce StoryScope, a pipeline that automatically extracts 304 narrative features from 61K+ stories, and show that AI fiction can be distinguished from human writing by narrative structure alone, without relying on stylistic signals.
摘要
As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing (e.g., word choice, syntactic structure), we ask instead whether AI-generated stories can be distinguished from human ones *without relying on stylistic signals*, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions (e.g., plot, agents, temporal structure). We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2\% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97\% of the performance of models that include stylistic cues. A compact set of 30 *core* narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity (e.g., flashbacks, nonlinear structure). Per-model *fingerprint* features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction. We release the StoryScope code, 10,272 writing prompts, and 51,336 AI-generated narratives to support future work on narrative analysis and AI authorship.