Human Masterworks vs AI Fiction
Our creative writing benchmark measures how much language models vary when they retell the same story, from each other and from themselves. It answers "how different are the models." It does not answer the question underneath it: how does any of this compare to real, acknowledged good writing?
Scoring machine prose is hard without a yardstick. A number like "lexical diversity 91" or "cliche density 0.05 per thousand words" only means something once you know the range that skilled, published fiction actually occupies. So we scored a corpus of 26 canonical, public-domain masterworks (Austen, Dickens, Conrad, Collins, Eliot, Tolstoy, Dumas, Stoker, Wells, Haggard, Buchan, Sabatini, and others) on the same tooling we point at model output. The result is a reference band: the values real literature falls in, so AI-generated text can be measured against books rather than against a guess about what "good" should look like.
Two benchmarks
- Surface craft (deterministic, no LLM). Lexical diversity (MTLD), cliche and "slop" density, self-duplication, dialogue ratio, sentence-length rhythm (burstiness), and a cast census. Cheap, reproducible, and a fast screen for gross departures from published prose: does a text repeat itself, drown in stock phrasing, or collapse in vocabulary?
- Narrative dynamics (LLM-judged). The things a word-frequency count structurally cannot see: whether dramatic tension actually rises and falls across a book, whether the prose alternates between action, dialogue, and interiority in a shape that reads like a novel, and whether a multi-character cast's plotlines run in parallel and converge the way real narratives do.
Headline findings
- The masters are a clean quality baseline. Across all 26 books: zero duplication flagged (the longest verbatim overlap anywhere is 79 characters, against a 500-character threshold), cliche density 0.00 to 0.15 per thousand words (median 0.05), slop density 0.01 to 0.41 (median 0.05), and lexical diversity 70 to 106 (median 91). This is the band good prose lives in, and it is what gives the benchmark teeth: model output that duplicates its own passages or collapses in vocabulary is measurably outside the range real fiction occupies.
- Tension discriminates by genre, cleanly. Mean dramatic tension spans 2.96 to 7.00 across the corpus. The three Austen domestic novels are the three lowest; the adventure and peril books fill the top. The judge was never told what kind of book it was reading; it recovered the genre structure from the prose alone.
- Structure follows a rule: hot books peak early. Mean tension and the position of a book's most dramatic chapter anti-correlate (r = -0.64). Books that run hot place their climax early and sustain the heat; books that run calm withhold it until near the end. The two numbers should be read together, not in isolation.
- For grading AI output: the reusable product is the reference band itself. The surface metrics give hard pass/fail ranges; the narrative-dynamics gauges are better read as coordinates relative to the masters' own distribution, since even acknowledged masters sit outside the inherited "expected" bands on several structural measures.
Method and validation
Every book was extracted from raw Project Gutenberg text to a verified, canonical Markdown file and then frozen, so a later extractor change cannot silently move already-published scores. The narrative-dynamics judge (DeepSeek) was validated against a second model (Claude Haiku 4.5) on Dracula's 28-chapter tension curve: the two judges agree at Pearson r = 0.86, differing by less than a point on average on the 0 to 10 scale. Every claim links back to the source texts and the analysis code, the same standard as the rest of our research.
Honest caveats
- Public-domain skew. "Public masterworks" means largely pre-1929 texts, so the human baseline leans historical. Victorian prose against a model prompted for modern science fiction is not a clean apples-to-apples comparison, and we control for era and genre rather than claim a single "human versus AI" number.
- Curation bias. Masterworks are, by definition, selected for quality. They are a ceiling, not an average human writer, and we say so wherever it matters to the reading.
- Single judge for narrative dynamics. The tension metric is validated against a second model (above); block rhythm and thread architecture were spot-checked rather than fully cross-validated, so agreement there is assumed, not yet measured.
- Length outliers. The five longest works (War and Peace, Bleak House, Monte Cristo, Middlemarch, The Woman in White) dominate the raw plot-thread counts through sheer size and cast, so those counts are read as length-and-cast signals, not as a uniquely dense plot.
GitHub repo Masters corpus report Related: LLM Creative Writing Benchmark Benchmark findings video (2026) Benchmark findings video (2025)