How often do week-one verification verdicts hold up against a later verdict built from cited evidence?

VerifiedNews gives every news claim a verification tier in its first week and never revisits it. I re-checked 600 of those verdicts months later against a verdict built from cited sources, to settle 3 product rules.

A Contested verdict held up when the later verdict found the claim FALSE or PARTLY. The other 76.1% were found TRUE, so about 3 in 4 were true (69.5% to 82.7%).

Findings

• About 3 in 4 claims marked Contested in week one were true.

  V4 hold-up 23.9% [17.3%, 30.5%], FPC; 117 resolved claims; H3 supported

• Supported tiers held up between 94.8% and 98.6% of the time.

  V1 98.6% [96.3%, 100%], n=40 · V2 Sonar-checked 94.8% [87.2%, 99.4%], n=37 · V2 mechanical 96.4% [89.9%, 100%], n=41

• The mechanical V2 rule showed no sign of being worse than an individual check, and the study cannot claim the 2 are equal.

  H2 difference −1.7 points [−10.4, 7.2]; inconclusive at this n

• The failures were knowable in week one, because for most verdicts that did not hold up, the earliest source against them was published within 7 days of the claim.

  Wrong at T0 85.0% to 94.8%, depending on whether search-index page ages count as dates; superseded by later events 0.9%

• Hold-up did not change with a claim's age at re-verification.

  Supported 92.8% to 98.6% across 0 to 30, 31 to 90 and over 90 days (29, 59, 30 resolved) · V4 17.9% to 29.8% (17, 42, 58 resolved)

• Source reporting quality as a predictor of hold-up (H4) could not be tested in practice, because the mechanical V2 stratum had only 2 failures.

  AUC difference −0.122 [−0.261, 0.246]; rests on 2 cases

What it changed

How week-one verdicts are made

VerifiedNews assigns every extracted news claim one of 5 tiers within its first week: V1 Verified, V2 Corroborated, V3 Reported, V4 Contested and V5 Unverified. A tier is assigned from the evidence available then and is not revisited, even though much more is known months later. Tiers come from 2 routes.

• The mechanical rule. A claim reaches V2 when its story cluster has 3 or more outlets across 2 or more bias categories with mean factuality at least 0.6. The pipeline links every article in a cluster to every claim in it, so the rule measures how widely the story was covered and does not check whether those outlets stated that particular claim. 11,246 of the 13,347 V2 claims in the study population got V2 this way.
• A Perplexity Sonar web check, usually within a day. It runs on V3 and V5 claims and moves them. A confirmation with an official source moves the claim to V1, a confirmation with 2 or more sources moves it to V2, and a denial moves it to V4. Every V4 claim in the population was set by a single Sonar denial.

The scoring and tier code has been static since May 2026, while ingest has run daily throughout.

Design

• Population. 16,181 factual and attributional claims from the main pipeline, extracted on or before 2026-10-06.
• Sample. 600 claims, seeded and stratified, with 150 each from V1, V2 Sonar-checked, V2 mechanical and V4, and at most 3 claims per story cluster. V3 and V5 were dropped to fit the budget.
• T1, the later verdict. Every sample claim was first re-checked with the product’s own Sonar request. Claude Opus 5.5 with web search then adjudicated every claim where Sonar now disagreed with the week-one tier’s stance (167) and a seeded 25% of the agreements (108), which makes 275 adjudications.
• Blinding. The adjudicator saw only the claim text and its report date, and returned TRUE, PARTLY, FALSE or UNRESOLVABLE with dated citations.
• Estimate. Hold-up rates are 2-phase inverse-probability-weighted estimates with cluster-bootstrap 95% intervals. V4 also gets a finite-population correction, because the sample covers 62% of that stratum.
• Pre-registration. The sample, the procedure, the estimator, the decision rules, the audit gate and a $40 spend ceiling were committed and tagged before the first post-freeze paid call.
• The audit gate. A second model, Claude Fable 5.1, re-judged 50 randomly drawn adjudicated claims. It saw the same cited sources but not the adjudicator’s label or reasoning. The gate required 45 of 50 outcomes to agree, and 47 agreed.

For V1 and V2, “held up” means the later verdict was TRUE. For V4, it means the claim was in fact FALSE or PARTLY, so the denial had substance. UNRESOLVABLE claims are excluded from the rates, between 11% and 21% of adjudicated claims per stratum.

H3 is supported, with V4 hold-up at 23.9% [17.3%, 30.5%] against a 50% threshold. H1 and H2 are inconclusive at this n, since the study detects differences of about 13 points or more.

Examples

These are illustrative and sit outside the pre-registration. Both are week-one V4 claims the adjudicator found TRUE, shown with the week-one Sonar explanation.

• “The New Hampshire Senate election is to replace Jeanne Shaheen.” The week-one explanation read: “Shaheen is the incumbent and is not seeking reelection, so the 2026 NH Senate race is to replace her, not reelect her.” The explanation supports the claim, yet the verdict was a denial.
• “An August CPI report was released in 2026.” The week-one explanation read: “BLS shows July 2026 CPI was released Aug. 12.” That too is consistent with the claim.

In both, the week-one Sonar verdict contradicts its own explanation. This pattern was observed in examples and its rate was not measured, and it points to a fix that adds no API calls: a denial whose own explanation supports the claim does not set V4.

The supported-tier failures were few (8 of 118 resolved claims) and mostly wrong details: an interview length (45 minutes against “hour-long”), a date (“today” for a policy published the day before), and a claim that Saudi Arabia is the top oil producer, against EIA data showing the United States.

How the study got here

Probes changed the design 3 times before any result existed.

1. A capture of later coverage through GDELT was abandoned when it turned out GDELT had been removed from the product pipeline and was never the instrument for 95% of clusters.
2. Linking later articles to each claim’s story with the product’s own clustering rule reached only 14 of 50 same-event links (28%) against a 90% bar. After a month, linked coverage was nearly always a different event in the same storyline, so in-corpus linking was retired.
3. A 20-claim pilot showed that re-running Sonar alone could not serve as a reference, because it reversed itself in both directions and sometimes adopted widely repeated figures over primary ones. That is why T1 uses an independent adjudicator that must cite sources.

Those findings are recorded in docs/PROBES.md, docs/LINKING_PROBE.md, docs/VERIFY_PILOT.md, docs/ADJUDICATOR_PROBE.md, docs/ADJUDICATOR_FOLLOWUP.md and docs/SMOKE_PILOT.md.

Reproducing it

The inputs are sample/sample.csv and the t1/ code at tag freeze-v1, with the module specs specs/05 to specs/08. The adjudicator’s model, tool limits and prompt were pinned in the freeze. Raw API responses are kept locally under data/ and are not published.

Limits

• T1 is a model's verdict, and no human ground truth anchors it. The adjudicator (Claude Opus 5.5) and the auditor (Claude Fable 5.1) are both Claude models, so their errors may be correlated. A human audit was planned, and I replaced it with the model audit to save time. A Gemini check was planned and could not run, because the key had no quota.
• Unresolved claims are excluded. 40 of 275 adjudicated claims were UNRESOLVABLE, and if those differ systematically from the resolved ones, the rates move.
• The adjudicator flagged 29 of 275 claims as having no checkable content on their own, for example "Putin made this statement on the…" with no referent. That is a claim-extraction problem in the product, separate from verification.
• The sample covers 4 tiers. V3 and V5 were dropped for cost, and their behavior is unmeasured.
• Up to 15% of failures are dated only by a search index's page age, which can be a last-modified date, so the routing was checked under both dating rules and holds under each.
• Week-one Sonar and T1 Sonar are the same request a few months apart, whereas the adjudicator is a different system, so the instrument changed between T0 and T1 for some claims.
• Engine spend was $36.83, measured, against a $50 piece cap and a $40 study ceiling, both frozen at e391352.

# run 2026-10-11 against a local clone of the study repo
$ git rev-parse freeze-v1^{commit}
e39135297cf8db0a0a59b804995de6e49ae48187
$ git log -1 --format=%cI freeze-v1
2026-10-08T23:14:59-07:00
PASS freeze-v1 resolves to e391352, committed 2026-10-09 06:14:59 UTC