LLM Creative Writing Benchmark
When you ask a model for the same story ten times, how differently does it write? The benchmark sends one science-fiction prompt to each model ten times (1,500-word target) and measures verbatim similarity, semantic similarity, vocabulary, named entities, and text structure. The 2025 study covered GPT-4o, o1, Gemini 2.0/2.5, and Claude 3.5/3.7; the 2026 re-run covers Gemini 3 Flash and Pro, Claude Opus 4.8, Sonnet, and Fable 5, and GPT-5.5.
Headline findings (June 2026 run)
- Models vary their wording far more than their ideas. Verbatim overlap between runs is near zero, yet semantic similarity sits around 0.48 to 0.61: the sentences change, the stories rhyme.
- The "Elara Phenomenon" has split by vendor. In 2025 the name "Elara" recurred across providers (both OpenAI and Anthropic leaned on it), while "Kaelen" was specifically a Gemini favourite. In 2026 the vendors have diverged. Google has not budged: Gemini 3 Pro still draws "Kaelen" in six of ten runs, echoing the seven of ten it favoured back in Gemini 2.0. OpenAI rotated its favourites without breaking the habit, and GPT-5.5 now shows the strongest name-pull of the cohort, with the full name "Mara Venn" recurring in four of ten runs. Claude Sonnet keeps its own surname pull, favouring "Voss" in seven of ten.
- Anthropic's flagship is the clean break: Claude Opus 4.8 never repeats a character name across the ten runs (a true count of zero). Fable 5, Anthropic's new top tier, sits between Opus and the strong-pull models: a faint residual habit an order of magnitude weaker than the 2025 norm, with its protagonist "Yara Venn" recurring in just two of ten runs.
- The bias survives at the level of sound. All three vendors over-produce short V-initial surnames (Vance, Voss, Venn, Vale). Even Opus, which never repeats a character name, coined a fresh V-surname in eight of ten runs: the shared bias is phonetic rather than lexical, and most plausibly inherited from the science-fiction corpus itself.
- Thematic diversity and naming diversity are orthogonal. Codex (GPT-5.5) is the extreme case: it records the lowest semantic similarity (0.4761) and lowest verbatim similarity (0.0097) of any model in either study year, yet still locks onto favourite names. It diversifies everything except the names.
- For builders: model choice is now a first-order lever for naming diversity, and automated repetition checks must filter named-entity-recognition false positives or the metrics mislead. This is not a generic tip; the headline Opus result rests on it. The raw metric flagged Claude Opus 4.8 with two repeated name components, "Minds" and "Kepler", but both are NER false positives (an AI "Minds" and a probe or place, not character names). Corrected, its true count of repeated character names is zero.
Method and caveats
The 2026 runs were generated through local agent CLIs (claude,
gemini, codex) in headless mode, not the provider
APIs used in 2025. The CLIs do not forward sampling parameters, so the 2025
temperature of 0.7 no longer applies and each CLI used its own default
sampling. The 2026 figures are therefore same-prompt, same-model-family
comparisons rather than strictly controlled replications; the natural next
step is an API re-run at a controlled temperature.
This work feeds directly into the narrative engines behind DungeonGPT and our writing tools. It also sets up a natural follow-up question: how do these numbers compare against fiction written by humans? We built a human baseline for exactly that. See Human Masterworks vs AI Fiction.
GitHub repo Full report 2026 findings video 2025 findings video