Ground Truth

The problem

Anyone choosing a model for a document-extraction task faces vendor benchmarks that are self-defined, unreproducible, and silent about sample size. “97% accuracy” with no n, no interval, no definition of “accurate,” and no held-out set is marketing, not measurement. The buyer’s real question — “do I need a frontier model for this, or will the cheap one do?” — is one no vendor is incentivised to answer.

Why the obvious approach fails

• Vendor benchmarks are self-graded: the metric definition, the held-out set and the sample size are all chosen by the party being measured.
• Generic “eval platforms” fail structurally: the buyer must be technical enough to want APIs but not technical enough to run their own evals — and the category keeps getting absorbed into observability tooling.
• Point estimates without intervals invite over-reading: two models 0.4% apart on different days are usually indistinguishable, but a bare number hides that.

Architecture

Ground Truth deliberately does one task — field-extraction accuracy on pay-application and invoice documents — and refuses to generalise, enforcing that refusal in CI. It produces a public benchmark with a leaderboard where every figure is a bootstrap interval, calibration curves exported byte-stable for Ledger Sense’s confidence routing, and a paid triage deliverable that outputs automate / route-to-human / never-automate per intent with expected-value maths.

1. Corpus
2. Grader
3. Replicates (n ≥ 5)
4. Statistics
5. Calibration
6. Leaderboard

ground_truth/
  golden/       synthetic corpus generator, seeded public/private splits,
                manifests, rubric versioning
  graders/      deterministic/ (exact, numeric_tolerance, normalised, schema,
                arithmetic) · judge/ (majority vote + order/length effects;
                agreement: α + rejection)
  stats/        bootstrap (cluster-BCa interval), agreement (Cohen κ,
                Krippendorff α), correction (Benjamini–Hochberg),
                sequential (O'Brien-Fleming α-spending), mde
  matrix/       list-price cost model with cited source URL + retrieval date
  calibrate/    isotonic / Platt on replicate agreement; fit public,
                validate private; export
  regression/   fingerprint observation, drift detection gated on sequential
                boundary AND BH correction
  triage/       intake CSV → expected value → sensitivity sweep → flip point
                → print report
  publish/      leaderboard payload, methodology renderer, hash-keyed
                private verdicts
web/            leaderboard · items · judge · calibration · regression · triage

Threat model

• Self-serving measurement. The tool itself could become a vendor — so the methodology opens by stating the population is synthetic and external validity is unmeasured by design.
• Private-split leakage. Published hashes must not let outsiders reconstruct private items: salted content hashes with a 128-bit secret salt mean a public generator plus a public seed enumerates nothing.
• Judge manipulation. LLM-as-judge is only used where a deterministic grader cannot reach, with pre-registered rejection thresholds and mandatory bias controls.
• Data publication accidents. A triage HTML was briefly published to Pages (caught 29 Aug 2026); the recorded incident now gates every publish step behind a mandatory security review.
• Vendor relations. Results are versioned and dated; corrections are new entries with the old value struck through; a right-of-reply process with 5/15-business-day windows and 10-day advance notice before naming a provider in a regression finding.

Decisions

Every rule has a test

Private-split items never appear in published output; every number carries n, interval and method; temperature zero is not determinism (n ≥ 5 replicates, mean and variance reported); cost is published list price labelled as such, never measured spend. Each rule is enforced by a test, not a comment.

Statistics code is the highest-risk directory

Bootstrap CIs are validated against known distributions; Benjamini–Hochberg against the 1995 worked example; Cohen’s κ and Krippendorff’s α against published examples; O’Brien–Fleming boundaries validated under null simulation so 13 quarterly “looks” don’t inflate false positives.

Two independent seeded generations, never one split pool

The public and private splits are generated independently from separate seeds. The private hash is sha256(salt:seed:n:payload) with a 128-bit secret salt — a public generator plus a public seed gives an outsider no way to enumerate private items.

Regression alerting as a statistics problem

Drift is reported only when both the sequential boundary (O’Brien–Fleming α-spending) and the multiple-comparison correction (Benjamini–Hochberg) agree, with the test and p-value or interval stated — never a bare red arrow.

The triage report shows the flip point

A 92%-accurate intent with a $4 cost of error is automatable; a 97%-accurate one with a $40,000 cost of error is not. The report quotes the client’s own cost-of-error assumption and marks the recommendation that would flip under a plausible alternative as flipping.

Judge verdicts are counted per judge, and only once

The one judged field (LINE_ITEM_DESCRIPTION) is scored from the judge’s majority vote on each slot’s modal output. Agreement is decided per judge; an unassessed judge never scores; a judged field counts toward the headline only when every current target is judged; and a re-grade supersedes the old verdict instead of being counted twice. A deterministic re-grade that disagrees with a stored verdict raises GradingConflict rather than being ignored. The rule change was cut as analysis plan v4, not an edit to the merged v3.

Rehearse the pipeline before spending a real call

A simulated sweep (including a “contested” scenario with overlapping models and a rejected weak judge) runs through the real pipeline and views behind a SIMULATED banner. A Phase 0 probe then makes one synthetic call per provider slot, never a golden item, to replace desk-research rate limits with measured ones before the sweep is scheduled.

Admit when a version can’t be pinned

On free tiers there is often no model fingerprint; the observation record says so rather than implying a version was tracked.

Implementation

• Python 3.12, Pydantic v2, DuckDB (embedded, no server), NumPy/SciPy/scikit-learn, Jinja2. Next.js 16.3 / React 19.2 / TypeScript strict frontend with six screens: leaderboard, items, judge, calibration, regression, triage.
• ~4,900 LOC of non-test Python across 122 files; ~1,700 LOC TypeScript with 34 Playwright specs.
• 5 deterministic graders (exact, numeric tolerance, normalised, schema, arithmetic); LLM-as-judge only where deterministic graders cannot reach.
• scripts/reproduce.py — stdlib-only, no repo checkout needed — recomputes every leaderboard row from the published per-item verdicts and exits 0 on zero differences.
• Docker Compose ‘up’ runs the full test suite with coverage as the smoke test; weekly regression-watch cron that skips placeholder data and never opens the same issue twice; ruff, mypy --strict, gitleaks, no-float gate on the money path.
• Fail-closed publish gate: fixture routes return 404 unless explicitly enabled, and a publish fails if a fixture page is exported. A two-stage release gate lets a tagged release pass only after the reproducibility and contamination checks.
• Six accessible screens with a WCAG 2.2 AA gate, keyboard paths and print stylesheets; the latency free-tier caveat is exposed to assistive technology, not only on hover.

Evidence

• A benchmark designed to be distrusted: hash-committed private split, stdlib reproduction script, pre-registered analysis plan, right-of-reply process, corrections log.
• A documented 46-defect self-audit and a recorded data-leak incident (a triage HTML briefly published to Pages, caught 29 Aug 2026) that now gates every publish step behind a mandatory security review.
• Byte-stable calibration exports — isotonic/Platt curves per field type, fit on public data and validated on private, exported for Ledger Sense’s confidence routing.
• Reproducibility test: published numbers regenerate from published data alone; the private split never appears in published output.

Security controls

• Private-split discipline: only salted content hashes of private items are published, proving existence at a date without revealing content.
• Publish-gate security review: every publish step is gated behind a mandatory security review since the recorded 29 Aug 2026 incident.
• No vendor characterisation: results are versioned and dated; corrections are new entries with struck-through old values; right-of-reply with defined windows.
• Cost transparency: cost is published list price with cited source URL and retrieval date per slot — labelled as list price, never presented as measured spend.

Trade-offs

• One task, refusing to generalise — enforced by AST-checked non-goals. Costs market breadth; buys methodological integrity.
• Synthetic corpus — fully reproducible and license-clean, at the price of unmeasured external validity until a real-invoice corpus exists.
• Bootstrap intervals everywhere make every number heavier to compute and read — and honest in a way point estimates are not.

Caveats & status

No leaderboard figure comes from a real model run yet: the provider sweep (9,000 generation and 5,400 judge calls) is scheduled but waits on provider accounts, and the views currently show a simulated run behind a SIMULATED banner. The golden set is synthetic and single-annotator; the methodology page states both.

• The golden set is synthetic — there is no real-invoice corpus to validate against.
• Single-annotator ground truth: the rubric has one author and the judged field’s human labels come from one annotator, so inter- and intra-rater agreement are unmeasured. Stated in the methodology’s Scope section since 28 Oct 2026 and guarded by a test.
• The real provider sweep has not run: no provider keys are provisioned, so published views show a simulated run, labelled as such.
• The provider audit (terms, training-on-input, list prices with dated sources) is re-run whenever a provider changes its catalogue.
• External validity is unmeasured by design and stated as such on the methodology page.

Repository

Public repository: github.com/Berakhah/ground-truth.