LLM Creative Writing Analyzer
A research tool that measures how repetitive and varied large language models are as creative writers. It sends the same prompt to each model ten times and quantifies how much the wording, meaning, vocabulary, and character names vary, turning "models reuse names" into hard numbers across GPT, Gemini, and Claude.
View source on GitHub →Tech Stack
Features & Highlights
Repeatability Benchmark
Sends one prompt to each model ten times and compares the outputs, exposing how much a model actually varies when asked for the same story.
Multi-Metric Analysis
Combines verbatim similarity (difflib), semantic similarity (sentence-embedding cosine), vocabulary diversity, named-entity recognition, and a custom name-component analysis.
Naming-Bias Detection
Surfaces the "Elara Phenomenon": the small pool of character names and phonetic patterns (like short V-initial surnames) that models reach for, broken out by vendor.
Multi-Provider
Runs OpenAI, Google, and Anthropic models via their APIs or local agent CLIs (codex, claude, gemini).
Human Masters Baseline
Scores 26 public-domain literary masterworks on the same tooling to build a reference band, so model output can be graded against real literature rather than a guess.
Dated, Reproducible Reports
Longitudinal study runs (2025 and 2026) with full written reports in the repo, so every claim links back to reproducible data.