LitHarness: findings

What two months of coding agents writing LitRPG serials, judged by one human reader, found about measuring fiction.

Artem SkulimovskiyAs of 14 October 2026Code on GitHubProject page

LitHarness is a solo project (12 August to 14 October 2026) in which coding agents write LitRPG web-serial chapters from a short brief, and one human reader, the operator, is the only judge of quality. What it found is mostly about measurement and specification, not prose:

• Model judges and simulated readers did not predict the human reader. A 4B-parameter judge answered "A" on all 106 passes, and all 4 simulated readers kept reading a sentence-shuffled chapter, as many as read the ordered one.
• One machine reader replicated: a reader that pays to keep reading. It gave shuffled books a smaller share of its reads, +0.164 and then +0.189 in a redrawn replication (90% intervals above zero).
• Prompt words become story. Swapping a permission clause for a prohibition cut the numbers in a listing 6.7x, a list in the prompt was recited back as plot, and the model brought money and debt into stories that never asked for them.
• Cost lived in the call graph and the code size. A 16-call, 129,661-token chapter lost both blinded model reads to a 2-call, 15,904-token one, and the code grew from 6k to 66k lines in 47 days before a rebuild of under 1,600 lines replaced it.
• A 1% to 52% climb measured reliability, not quality. An audit found the pitch climb's gain was mostly format, and in a separate chapter climb, passing chapters looked the same before and after.
• Agent CLIs leak their surroundings. Run inside a repo, claude -p put an untracked filename into its answer in 1 to 2 of 5 calls.

The project

The goal is fiction that a defined audience would choose to keep reading and recommend, with no human in the production loop. The audience is RoyalRoad readers of LitRPG, aged 20 to 30: progression fantasy in which the hero's ranks and abilities are printed as numbers on the page.

The rebuild runs under two rules these findings produced. The operator (the project's author) is the only quality gate, and no model judges or selects text. No human data is collected beyond household reads the operator relays: no paid readers, panels or labels.

Terms used below:

• Brief: a few lines of premise from the operator.
• Pitch and bible: one model call per serial writes the bible (premise, hero, the System that grants ranks, threat, allies and limits) in at most 900 words.
• Draw: one model call for one stage. A stage that fails a code check is drawn again, up to 3 times.
• Hard checks: code checks that force a redraw, such as money words, leaked schema words, a missing printed rank-up, shape and length.
• Tells: counted patterns of machine prose, such as chained "and", compared with published chapters.
• Sheet: the hero's stats, which code reads from [Label: value] lines the hero prints on the page.

The evidence base is 20 operator reads (18 Aug to 23 Sep), each naming defects in a chapter, and a 267-entry decision ledger kept at the tag legacy/incumbent-2026-09-27. Writers were Codex (gpt-6-astra, then gpt-6.1-sol), Claude Code (Opus 5.5) and Haiku 5.5 through the API.

The rebuild's first serial has 3 chapters (4,831 words) awaiting the operator's 21st read; with its pitch it took 15 calls and 118,679 tokens. The code is public at github.com/skulitom/LitHarness.

How a chapter is made

Models write and code checks. Only the operator judges, and each defect they name is located, in our text or the model's, and fixed with the cheapest change before a redraw.

Finding 1: model judges and simulated readers did not predict the human reader

The model judges and simulated readers the incumbent built tracked position, saturated, or preferred surface to story. The one gate scored against the operator's verdicts passed chapters the operator then failed.

One format did carry signal. Asked to name the single most salient difference between two texts, the same Haiku found it 40 of 40, 30 of 32 and 18 of 36 times against nulls of 0.21, 0.36 and 0.26, and called the placebo pair identical. A report carried signal where a verdict did not.

So the rebuild has no model judge. Its instruments are code-graded and describe; only the operator's read decides.

Finding 2: the one machine reader that replicated pays to keep reading

A simulated reader that spends from an attention budget to continue gave a book a smaller share of its reads when the book's paragraphs were shuffled. Free readers never separated these (Finding 1); this one did, in a registered run and a rerun.

Setup: 20 of our drafted books, four books offered per session, claude-haiku-4-5 through claude -p, 180 sessions per run, 180 of 180 scorable.

Our reading is that the reader had to give something up to keep going, so it could not say yes for free. The limits are real: a whole-book shuffle is the loudest damage there is, a lean toward the first slot ate three quarters of an earlier version's signal, 20 books cap its power, and it needs about 3,900 words a book. The rerun reused the same 20 books and model with only the shuffle redrawn, and the claim holds only for the slot the reader attends to. Both runs predate the transport fix in Finding 6. It stays a research result, not a gate.

Finding 3: prompt words become story

When a failure was traced to its cause, the cause was often words we sent, not the model's taste. The rest came from a lean the model brings on its own, and that lean moved around rather than going away.

• Clause form sets the output. Number tokens per 1,000 words of a listing: 47.2 with a permission clause, 29.4 with none, 7.0 with a prohibition. Ten market listings carry 7.2.
• Lists are recited. A six-item list of pressures was recited back: across three concepts "water" drew 66, 16 and 8 mentions, and "fight" none. A checklist in the listing task produced a list-shaped listing.
• Fact packets become minutiae. Facts were 65% of a scene prompt, and 23 of 37 facts shared a content-word pair with the drafted page.
• Our words, then the model's. 18 of 30 generated worlds put a register, debt, court, deed or clerk in the premise, traced to four lines of our rules. After those were deleted, an institutional lean remained. Later, 4 of 5 failed draws had a located cause in our text, yet 3 of the 5 held money words no request contained.
• Naming a banned word supplies it. Prohibiting a behaviour cut numbers (above), but a "not currency" gloss put the word into every cost record that fed the page.

In the rebuild's pitch climb, each field we re-asked moved the money and office frame into the next field:

On Haiku the same words behaved like a partly conserved budget: blocking one section moved them to another, while blocking two sections together reduced them. The working rule became: delete prompt text first, then ask a positive field, then add a code check that redraws. A prose clause comes last, and never one about register or craft.

Finding 4: cost lived in the call graph and the code size

The incumbent's slowness came from its call graph, not from Python, and its code regrew after cuts.

• Calls. One chapter took 16 calls and 129,661 tokens. 8 of them were "OK" probes that used 27% of the wall time, one outline planned 24 scenes to draft 4, and Python itself took about 11 s.
• A 2-call prototype. A 264-line rewrite made 2 calls (15,904 tokens) and won both orders of a blinded read by a same-family model judge, on one pair. By Finding 1 that read decides little, and the heavy pipeline also did continuity work the prototype skipped.
• Machinery. A stat-sheet engine of over 6,000 lines in nine modules mostly fixed its own defects. One 24-scene arc (2 h 20 min, $38) spent the hero's only currency twice and brought back a man who had been culled.
• Growth. Runtime code went from 5,964 to 65,891 lines in 47 days. A 34% cut on 24 August regrew in 5.2 days, and CI took 10 to 14 minutes a push.

Tests and research code grew alongside the runtime: on 27 September the incumbent's three trees held about 263,000 lines.

The rebuild enforces budgets in its own test suite: runtime at most 1,564 lines, any file at most 350, prompts at most 701 words, each doc capped by lines. Raising a cap is its own commit naming the case. A chapter is designed for 2 or 3 calls and 21,000 to 27,000 tokens; the first serial's took 2 to 4 calls and 20,000 to 41,000 tokens, since failed checks add draws.

Finding 5: a 1% to 52% climb measured reliability, not quality

Thirteen rounds of prompt edits lifted Haiku 5.5's first-draw pass rate on the pitch checks from 1% to 52%, and four more found nothing better. Most rounds changed prompt text, round 6 raised the model's effort setting, and every variant was re-graded after three parser fixes. A later audit found the number tracked format and retries, not whether the stories got better.

The pitch is a bible of at most 900 words that code checks for money words, schema leaks and shape. Between rounds, an Opus analyzer read failing transcripts and proposed edits; each round tried one to three, under a keep rule declared before drawing: pass up on the current best with its 95% interval above zero, and no guardrail worse. Draws ran on 11 past briefs, up to 128 a brief, for $65 of a $150 budget.

Two grey dots sit above the line at 54%: each gained 2 points with an interval that included zero, so neither was kept, and stacking them gave 51%.

The best prompt also passed 66% on Codex against 47% for the shipped prompt, +19 points (95% interval +7 to +31). The audit, run on stored rows with no new calls, then found:

• Retries hide the metric. The harness draws a failing stage up to 3 times, so everything the operator reads has passed. Pass within 3 draws went from 73% to 91% on the Codex pitch; first-draw pass mostly measures cost.
• Most of Haiku's gain was its own format quirks. Shape failures fell from 94% to 4%; Haiku had been reading [Label: value] as a placeholder and dropping the brackets.
• Passing chapters looked the same. In the follow-on chapter climb (28% to 52%), passing chapters before and after had 1,421 and 1,368 words and 10% and 9% dialogue. Scene breaks went from 0.6 to 2.1 a chapter.
• The money gains were real. Unlisted commerce words fell with the listed ones (4.0 to 2.6 per 1,000), so the model was not swapping in euphemisms.
• The sample was thin. About 40 variants ran on the same 11 briefs, which count as about 7 independent ones, with no holdout.

The audit proposes keeping first-draw pass as a guardrail, not a target, and judging content edits by the operator's blind read of stored same-input pairs. A 36-pair packet was built on 11 October and has not been unblinded.

Finding 6: agent CLIs leak their surroundings into answers

A coding-agent CLI used as a plain text model still sees where it runs. Run from inside a repo, claude -p put an untracked filename into its answer in 1 to 2 of 5 calls, and the path in about 2 of 5. Without the right flags a project's CLAUDE.md reached the model, and the --bare mode answered "Not logged in". The incumbent's hardening drifted across 23 files.

For a writer this contaminates the story. For a judge or a simulated reader it means the instrument can read the experiment around it.

The rebuild sends every call through one module, the only file allowed to import subprocess. Each call runs from a fresh empty directory outside any git tree, with tools, MCP servers and project instruction files switched off. A canary checks this against each CLI version:

1. Build a temporary git repo with a random nonce in AGENTS.md, AGENTS.override.md, CLAUDE.md, a commit message and an untracked filename.
2. Run the agent from inside that repo on purpose, and ask it to echo only a control nonce.
3. Fail if any planted nonce appears in the answer, the event stream or stderr, or if the control line is wrong.
4. On a clean pass, record the CLI's version and binary hash. Paid commands refuse to run on a version that has not passed.

Limits

These findings come from one project, one reader and small samples. They point to what is worth testing at scale; they do not establish it.

• One reader. The quality gate is one person. No audience retention or recommendation data is reported here.
• The rebuild is unproven on its own goal. Its first chapters await the operator's 21st read, so there is no evidence yet that it writes better chapters than the incumbent.
• Small samples. The climbs used 11 briefs (about 7 independent) with no holdout. The 2-call prototype won on one pair.
• Draft variance is large. Five drafts of one prompt kept 3, 4, 1, 4 and 4 of 4 simulated readers reading, so single-draft comparisons are noise.
• Checks find named defects, not quality. A passing bible or chapter is free of what the checks look for, nothing more.
• Versions move. Results are tied to the models and CLI versions of their day; the incumbent, the first serial and the Haiku climb each ran on different ones.

Sources

All in this repo unless marked. Incumbent-era numbers cite entries of its decision ledger, kept at the tag.

• LEARNINGS.md: Findings 1 to 4 and 6, with ledger references
• Decision ledger at the incumbent tag (267 entries, 2 MB; search by entry number)
• Costed reader findings: Finding 2
• Pitch draws report: the Codex rounds in Finding 3
• Opening comparison report: the 2-call prototype in Finding 4
• transport.py: the isolation and canary in Finding 6
• Code-growth chart: line counts from git history, at each day's last commit on the incumbent tag and on main
• Haiku climb, chapter climb and metric audit: run records kept outside the repo with the generated text (not public)