Benchmarks

Every number on this page is reproducible from the repository (python -m cypherquill.judge.train, python -m benchmarks.loop_bench, python -m benchmarks.speed_bench, python -m benchmarks.report). All authors and models labelled "held-out" or "unseen" were excluded from training.

Headline: Your Style on the live product

3 held-out human writers (Reddit/blog). Voice imported from 20 of their own posts. 3 genuine ChatGPT/GPT-4 drafts rewritten per round. Feedback between rounds: accept the best round-1 output, thumbs-down the worst. Test accounts deleted afterwards. Run on https://cypherquill-mvp.fly.dev (production config, commercial-safe judge, trial plan -> claude-sonnet-5-5 writes, claude-haiku-5-5 checks), 2026-11-09.

n=9 drafts per round. Directional evidence that feedback helps, not a significance test. For reference, held-out authors' own unseen writing scores a median of 95 against their own voice, and different authors score a median of 7.

Judge v0 — authorship verification (held-out authors)

Training data: 14,107 passages by 406 authors across blogs, Reddit, IMDb reviews, Reuters news and Project Gutenberg books. Authors are split into encoder-fit (55%), head-fit (15%) and held-out test (30%, 125 authors). Negatives are drawn from the same domain, so the judge can't win on genre.

Task: is query passage q (~250 words) by the author of a profile built from k other passages?

By domain (k=10):

Ablation (k=10): learned-encoder cosine alone gives EER 0.327 (LDA view) and 0.295 (SVD view). The authorship head, which fuses four embedding views with grouped stylometric feature distances, brings it to 0.179. The product thresholds come from the head-fit split: "elevated" link risk at p ≥ 0.52 (EER point) and "high" at p ≥ 0.89 (1% false-positive rate).

Judge v0 — model-likeness (LLM-ness) head

Protocol. Training uses human text plus generations from: cohere, cohere-chat, deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro, google/gemini-3.6-flash, google/gemini-3.7-flash, mistral, mpt, qwen/qwen3.6-35b, qwen/qwen3.8-27b, qwen/qwen3.8-max. Every other model is unseen, including whole vendors that never appear in training: OpenAI (GPT-3/3.5/4 and the 2026 luna/sol/gpt-5.6 models), Anthropic, Meta, Moonshot, xAI, Zhipu and NVIDIA. Each model is scored against matched human text from the same dataset or domain, so the head can't win on genre. The head ensembles an interpretable 155-feature model with a topic-masked char 2-5grams model. Sparse c and blend weight chosen on the calibration split of seen models only.

Mean AUC across unseen models: 0.928 (0.975 for models with n ≥ 30). Seen models on held-out prompts: pooled AUC 0.971.

Seen models (held-out prompts):

False positives on genuine human writing from held-out authors. These authors are never seen by any head. This is the number that matters for a steering signal:

Human false-positive rate at 0.5: 0.080 overall (authorship-human: 0.05, raid: 0.15, rasbt/human-vs-ai-50k: 0.04, textsightai/ai-text-detection-benchmark-2026: 0.11).

Strongest interpretable features (dense-model coefficients; positive = more model-like):

para_len_mean (-1.93), lower_start_rate (-1.07), word_len_mean (+1.05), semicolon_rate (-0.85), heading_rate (+0.74), sent_len_mean (-0.69), p3_rate (+0.64), fw_a (+0.63), opener_diversity (-0.61), sent_bin_1_5 (-0.59), sent_bin_6_10 (-0.58), hapax_rate (-0.51)

> How to read this. Model-likeness is a style signal that we use to steer writing toward a human register. It isn't a forensic AI detector, and we don't sell it as one. The newest models in the strongest families (for example Claude Opus 5) are the hardest to tell from human prose. Human writing sometimes reads as model-like too. That's fine for steering a rewrite, and it would be wrong for accusing anyone.

Commercial-safe judge (production default)

models/judge-v0-cs is trained only on sources whose terms allow commercial use: Reddit (webis/tldr-17, CC BY 4.0), Project Gutenberg (public domain), and the RAID and 2026 LLM sets minus IMDb-derived rows. It drops the Blog Authorship Corpus, IMDb62 and Reuters C50, which are research or non-commercial only. To make up for them it adds 209 extra Reddit and Gutenberg training authors (scripts/data/fetch_commercial_extra.py) and 1200 passages of formal human prose from the Wikipedia 2023-11 dump (CC BY-SA, scripts/data/fetch_formal_human.py). The Docker image uses it by default (CQ_JUDGE_DIR).

Head-to-head on the same held-out authors, all 5 domains. Every variant uses one author-to-split assignment, so no test author is ever trained on. Blog, news and reviews are entirely out-of-domain for the commercial-safe judge (python -m benchmarks.compare_judges).

The commercial-safe judge spots AI-style writing slightly better. It gives up some authorship precision, and it still over-flags formal news-wire prose, because it has no commercially licensed newswire to learn from. Licensed news and blog corpora, or design-partner data, are the first data task after launch.

Rewrite loop — claude-cli (sonnet / checks: haiku) · steered and scored by the research judge (judge-v0)

Held-out authors only. 28 documents, generated 2026-11-08T15:44:03. Each document is rendered four ways: unchanged, a plain "rewrite this" LLM call, our prompt with no judge feedback (1 pass), and the full judge-guided loop (≤3 passes).

Voice transfer between two human writers (author A's text → B's voice; stress test) — n=12

Loop: 2.76 passes per paragraph on average (cap 3), 7.8 model calls per document, meaning check passed on 75% of documents, 23.7 s per document.

Anonymize (author A's text) — n=8

Loop: 2.82 passes per paragraph on average (cap 3), 12 model calls per document, meaning check passed on 88% of documents, 21.1 s per document.

Your Style: AI-written draft → the writer's own voice (Lanes A & B) — n=8

Loop: 2.62 passes per paragraph on average (cap 3), 5.2 model calls per document, meaning check passed on 100% of documents, 17.9 s per document.

Unit economics (measured tokens, list prices)

Per 1,000 source words the loop made 38 model calls and sent about 89,503 system-prompt tokens (mostly cache reads), 19,699 uncached input tokens and 5,725 output tokens. Estimated from prompt sizes; system prompt cached within a document; judge scoring is local cpu (~$0).

Judge scoring runs on our own CPU and is effectively free (see speed below), so model spend is the only variable cost.

Rewrite loop — claude-cli (sonnet / checks: haiku) · steered and scored by the commercial-safe judge (judge-v0-cs)

Held-out authors only. 20 documents, generated 2026-11-08T16:47:46. Each document is rendered four ways: unchanged, a plain "rewrite this" LLM call, our prompt with no judge feedback (1 pass), and the full judge-guided loop (≤3 passes).

Voice transfer between two human writers (author A's text → B's voice; stress test) — n=8

Loop: 2.91 passes per paragraph on average (cap 3), 8 model calls per document, meaning check passed on 75% of documents, 23.7 s per document.

Anonymize (author A's text) — n=6

Loop: 2.50 passes per paragraph on average (cap 3), 6.7 model calls per document, meaning check passed on 100% of documents, 16.4 s per document.

Your Style: AI-written draft → the writer's own voice (Lanes A & B) — n=6

Loop: 2.67 passes per paragraph on average (cap 3), 5.3 model calls per document, meaning check passed on 100% of documents, 18.1 s per document.

Unit economics (measured tokens, list prices)

Per 1,000 source words the loop made 34 model calls and sent about 84,668 system-prompt tokens (mostly cache reads), 18,570 uncached input tokens and 5,656 output tokens. Estimated from prompt sizes; system prompt cached within a document; judge scoring is local cpu (~$0).

Judge scoring runs on our own CPU and is effectively free (see speed below), so model spend is the only variable cost.

Loop design iterations (same 28 held-out cases)

Voice stability over time — claude-cli:sonnet

Automated stand-in for the spec's "week-1 vs week-8" reader test. Week 1 renders fresh sources and publishes them to history; week 8 renders different sources against the grown history. A cross-validated classifier (judge embeddings + 155 features) then tries to tell the batches apart. ≈0.5 means indistinguishable.

Persona consistency gauge: week 1 43, week 8 42, against 10 for the unrendered source passages. 2 personas. Small samples, so treat ±0.15 as noise.

Judge speed (CPU only)

Machine: Apple M1, Darwin, single process. Artifact size: 38.4 MB. Load: 0.12 s.

• Full score report (3 gauges + tells + portrait, ~238 words): p50 11.5 ms, p95 15.7 ms
• Fingerprint from 200 samples: 1.43 s
• Batch model-likeness throughput: 147.3 texts/s on one process

Known limitations

• Short text is hard. Single ~250-word passage vs. single passage gives EER ≈0.3. Profiles of 10+ samples are much better, which is why Real Voice asks for 30–200 samples. Gauges show a reliability label below 150 words.
• English only (spec §5 cut list).
• Model coverage. The unseen-model set includes 2025–26 frontier models from public datasets. The newest vendors change quickly, so retrain monthly (python -m cypherquill.judge.train).
• Human blind tests. The spec's "week-1 vs week-8" human readers test runs in production through /lineup and the in-app Blind check. The live accuracy is reported at /api/admin/metrics (blind_lineup.accuracy against chance).
• Remote large encoder (docs/REMOTE_MODELS.md) is the expected path to EER ≈0.1. It is a placeholder in this build.