← All studies

Understudy · a pre-registered seeded detection study

Can findings from synthetic user populations be trusted?

47.4%
of sessions that reached the fully dead chip were caught, in activation mode, in the first population.[26.3%, 68.4%] · 95% interval, persona bootstrap · 9 of 19 sessions; all 19 reached the chip · decided 2026-10-11

Understudy sends synthetic personas through a web app and turns what they do into findings. I seeded 5 known defects into Sidestreet's web app at controlled doses, ran Understudy's standing intents unchanged, and reran every hit under 3 resampled persona populations, to decide whether its findings can back product decisions.

The interval does not account for the decision cache coupling sessions within an archetype, so it is too narrow. Each of the 3 resampled populations caught 18 of 19, and every session that produced a qualifying run was caught, 64 of 64 across both doses and all 4 populations.

Figure 1

Dead-chip detection by dose and interaction mode, against the predicted ceiling

Dead-chip detection by dose and interaction mode, against the predicted ceiling Dot chart, module 7 population. Filled dots: observed share caught of sessions that reached the chip. Open circles: the frozen per-session ceiling. 0.50 activation: 1 of 19, ceiling 0.629; 0.50 retention: 0 of 17, ceiling 0.512; 1.00 activation: 9 of 19, ceiling 1.000; 1.00 retention: 0 of 10, ceiling 1.000 0.50, activation 1 of 19 ceiling 0.629 0.50, activation: 1 of 19 that reached the chip caught, predicted ceiling 0.629 predicted ceiling 0.629 observed 1 of 19 that reached the chip 0.50, retention 0 of 17 ceiling 0.512 0.50, retention: 0 of 17 that reached the chip caught, predicted ceiling 0.512 predicted ceiling 0.512 observed 0 of 17 that reached the chip 1.00, activation 9 of 19 1.00, activation: 9 of 19 that reached the chip caught, predicted ceiling 1.000 predicted ceiling 1.000 observed 9 of 19 that reached the chip 1.00, retention 0 of 10 1.00, retention: 0 of 10 that reached the chip caught, predicted ceiling 1.000 predicted ceiling 1.000 observed 0 of 10 that reached the chip 0.00 0.25 0.50 0.75 1.00 share of sessions that reached the chip and were caught
Dead-chip detection by dose and interaction mode, against the predicted ceiling Dot chart, module 7 population. Filled dots: observed share caught of sessions that reached the chip. Open circles: the frozen per-session ceiling. 0.50 activation: 1 of 19, ceiling 0.629; 0.50 retention: 0 of 17, ceiling 0.512; 1.00 activation: 9 of 19, ceiling 1.000; 1.00 retention: 0 of 10, ceiling 1.000 0.50, activation 1 of 19 ceiling 0.629 0.50, activation: 1 of 19 that reached the chip caught, predicted ceiling 0.629 predicted ceiling 0.629 observed 1 of 19 that reached the chip 0.50, retention 0 of 17 ceiling 0.512 0.50, retention: 0 of 17 that reached the chip caught, predicted ceiling 0.512 predicted ceiling 0.512 observed 0 of 17 that reached the chip 1.00, activation 9 of 19 1.00, activation: 9 of 19 that reached the chip caught, predicted ceiling 1.000 predicted ceiling 1.000 observed 9 of 19 that reached the chip 1.00, retention 0 of 10 1.00, retention: 0 of 10 that reached the chip caught, predicted ceiling 1.000 predicted ceiling 1.000 observed 0 of 10 that reached the chip 0.00 0.25 0.50 0.75 1.00 share of sessions that reached the chip and were caught

Scroll sideways to see the whole figure.

For each dose and mode in the module 7 population, the filled dot is the share caught among sessions that reached the chip, and the open circle is the frozen per-session ceiling.

The emphasized gap is half dose in activation mode, where 1 of 19 sessions was caught against a predicted 0.629. The ceiling assumed independent clicks, and the click events show personas spacing their retries, an order of magnitude below that assumption.

At full dose in activation mode, 9 of 19 were caught. In retention mode none were caught at either dose, 0 of 17 at half dose and 0 of the 10 that reached the chip at full dose. Detection given a qualifying run is a separate fact, 64 of 64 across both doses and all 4 populations.

Source: understudy-detection-study/analysis/house_figures.py. Data in report/figures/figure_data.json, recomputed by analysis/verify_report_figures.py.

Findings

  • The rage-click signal caught every session that produced the behavior it looks for, and no session without it.

    64 of 64 sessions with a qualifying run caught, none caught without one, across both doses and all 4 populations

  • Personas produced that behavior reliably only in activation mode at full dose.

    9 of 19 sessions at full dose in activation mode, 1 of 19 at half dose, 0 in retention mode at either dose

  • The pre-computed ceiling for the half-dead chip was off by an order of magnitude, because personas spaced their retries.

    Observed 0.05 per session (1 of 19) against a frozen ceiling of 0.629; 1 qualifying run across 93 sessions on that chip

  • Both page-level signals fired without their defect.

    findability_gap at /saved with 0 of 17 sessions arriving and every entry point intact; confusion_topology at /saved in an arm with no defect there

  • 6 of 9 resampling survival tests passed, and all 3 failures were the half-dead chip's single-session hit.

    Per defect, 2 of 3 survived; only the fully dead chip's survivals rest on an element-level signal

Figure 2

Resampling survival by defect, activation-default

Resampling survival by defect, activation-default Grid. Dark cells: hits on the element-level rage-click signal. Light cells: hits on a page-level signal that also fired without its defect. Empty cells: misses. The emphasized cell is the half-dead chip's module 7 hit, resting on 1 session, which no resampled population reproduced. module 7 reseed 1 reseed 2 archetype mix dead chip 0.50 dead chip 0.50, module 7: hit, element hit, element dead chip 0.50, reseed 1: miss miss dead chip 0.50, reseed 2: miss miss dead chip 0.50, archetype mix: miss miss dead chip 1.00 dead chip 1.00, module 7: hit, element hit, element dead chip 1.00, reseed 1: hit, element hit, element dead chip 1.00, reseed 2: hit, element hit, element dead chip 1.00, archetype mix: hit, element hit, element server error /saved server error /saved, module 7: miss miss server error /saved, reseed 1: miss miss server error /saved, reseed 2: miss miss server error /saved, archetype mix: hit, page signal hit, page signal entry removal, total entry removal, total, module 7: hit, page signal hit, page signal entry removal, total, reseed 1: hit, page signal hit, page signal entry removal, total, reseed 2: hit, page signal hit, page signal entry removal, total, archetype mix: hit, page signal hit, page signal
Resampling survival by defect, activation-default Grid. Dark cells: hits on the element-level rage-click signal. Light cells: hits on a page-level signal that also fired without its defect. Empty cells: misses. The emphasized cell is the half-dead chip's module 7 hit, resting on 1 session, which no resampled population reproduced. dead chip 0.50 module 7 dead chip 0.50, module 7: hit, element hit, element reseed 1 dead chip 0.50, reseed 1: miss miss reseed 2 dead chip 0.50, reseed 2: miss miss archetype mix dead chip 0.50, archetype mix: miss miss dead chip 1.00 module 7 dead chip 1.00, module 7: hit, element hit, element reseed 1 dead chip 1.00, reseed 1: hit, element hit, element reseed 2 dead chip 1.00, reseed 2: hit, element hit, element archetype mix dead chip 1.00, archetype mix: hit, element hit, element server error /saved module 7 server error /saved, module 7: miss miss reseed 1 server error /saved, reseed 1: miss miss reseed 2 server error /saved, reseed 2: miss miss archetype mix server error /saved, archetype mix: hit, page signal hit, page signal entry removal, total module 7 entry removal, total, module 7: hit, page signal hit, page signal reseed 1 entry removal, total, reseed 1: hit, page signal hit, page signal reseed 2 entry removal, total, reseed 2: hit, page signal hit, page signal archetype mix entry removal, total, archetype mix: hit, page signal hit, page signal

Scroll sideways to see the whole figure.

Each cell is the frozen classifier's verdict for one defect in one population. Dark cells are hits on the element-level rage-click signal, light cells are hits on a page-level signal that also fired without its defect, and empty cells are misses.

6 of 9 survival tests passed; per defect, 2 of 3. The emphasized cell, also an element-level hit, is the half-dead chip's module 7 hit, which rested on 1 session and was reproduced in none of the 3 resampled populations.

Only the fully dead chip's survivals rest on an element-level signal.

Source: understudy-detection-study/analysis/house_figures.py. Data in report/figures/figure_data.json, recomputed by analysis/verify_report_figures.py.

  • Sessions within an archetype are coupled through the decision cache, so a per-session count overstates the independent evidence.

    19 sessions across 3 archetypes carry closer to 3 independent units; confident_power_user caught 1 of 7 in module 7 and 7 of 7 under reseed 1

  • The server error on the Saved page was missed in both pairings with reach intact.

    11 of 19 and 11 of 17 sessions arrived at /saved

What it changed

Can the causal piece and the customer and UX branch be built on this instrument?

The decision is to proceed with the causal piece's design now and to gate its paid runs on 2 instrument conditions, because the study certified detection within a boundary rather than everywhere. I therefore trust element-level detections in activation mode at full dose and any finding reproduced across reseeds; I do not trust page-level signals on their own, any finding resting on a single session, or the absence of detection in retention mode, which is a blind spot of the instrument and never evidence that nothing is wrong.

Decided · what happened after: not yet recorded

Informed by: 64 of 64 sessions with a qualifying run caught; 9 of 19 at full dose in activation mode, 1 of 19 at half dose, none in retention mode; page-level signals fired without their defect

What happens to page-level signals that fire when nobody arrived?

The nobody-arrived behavior of page-level signals gets the cheap fix, an arrival precondition in finding evaluation, or page signals are demoted to advisory permanently if that fix is not small.

Decided · what happened after: not yet recorded

Informed by: findability_gap fired at /saved with 0 of 17 sessions arriving and every entry point intact; confusion_topology fired at /saved in an arm with no defect there

What happens to the decision cache that couples sessions within an archetype?

The per-archetype decision cache coupling must be resolved before the causal piece's arms are paid for, either fixed or bounded by extending that piece's planned A/A comparison, because the paired design and the ICC-based power analysis both assume session independence within arms and this coupling is a known channel against it.

Decided · what happened after: not yet recorded

Informed by: confident_power_user caught 1 of 7 in the first population and 7 of 7 under a new seed; 19 sessions across 3 archetypes carry closer to 3 independent units

Should the rage-click recorder keep every incident in a session?

The rage-click recorder keeping only each session's first incident is deferred, since presence-based outcomes are unaffected.

Decided · what happened after: not yet recorded

Informed by: In 1 session the first incident fell on another control, so a later rage on the chip went uncounted

What happens to the intermittent 1-node crawl, escalation 001?

Escalation 001, the intermittent 1-node crawl with an unconfirmed cause, gets a halting assertion now so that an unexplained instrument failure stops a run loudly instead of corrupting it quietly, and its diagnosis is deferred until it next fires behind that guard.

Decided · what happened after: not yet recorded

Informed by: A crawl intermittently maps 1 node and 0 actions; cause unconfirmed

The design

The target is Sidestreet’s own web app at commit 24926fd. Each of 5 defects was injected by a patch through one harness and reverted between arms, and each dose is a failure probability or a scope, defined without severity labels.

Defect What is broken Dose
dead_control_home_chip_05 the “Coffee & work” chip on the home page does nothing fails on 50% of clicks
dead_control_home_chip_10 the same chip fails on 100% of clicks
server_error_saved_route the Saved page returns a server error every request
entry_removal_saved_total entry points to the Saved page removed 4 of 4
entry_removal_saved_partial entry points to the Saved page removed 2 of 4

The defects sit only on paths the standing intent library already covered, and the coverage map was produced before any site was chosen. The commit history shows the order: the coverage run closed at 10ab52e on 2026-09-19, the candidate defect set was first committed at cf68b2a on 2026-09-23, and the freeze followed at 0cc3924 on 2026-10-06, all in UTC.

Every defect ran at 2 intent-and-mode pairings, activation-default at n 19 and retention-default at n 17, with 1 session per persona and each pairing run exactly once. A control arm ran the same pairings on the unpatched build, and all arms in a pairing share run seed 42, so the same synthetic individuals appear in the control and every seeded arm. Detection ran 216 sessions and resampling 285 more, with the defect set, criterion, n, analysis code, stopping rule and resampling plan frozen at commit 0cc3924 before any arm ran.

A defect is a hit in a run when a finding of an expected type lands at its location, and where a finding’s evidence can localize it, the evidence decides. The control arm’s findings were sealed as a per-pairing baseline before any seeded arm ran, and seeded-arm findings that match it are not counted as false positives. Each resampled population sealed its own baseline the same way.

Rage-click behavior clusters within a persona, at an ICC of 0.555 measured before the design was set, and at 1 session per persona the formal design effect within an arm is 1.00. The clustering that mattered came through the decision cache. Each run keeps its own cache keyed by archetype, so a session replays decisions earlier sessions of its archetype already made, and with 3 archetypes per arm a count of 19 sessions carries closer to 3 independent units on the paths the cache serves.

Detection

Of the 3 signal types that localize, 2 localize only to a page, and both fired without their defect. findability_gap at /saved fired in an arm with every entry point to /saved intact, where 0 of 17 sessions reached the page, so it fires when nobody arrived and cannot distinguish that from entry points that are gone. confusion_topology at /saved fired in an arm with no defect there. The verdicts that survive this intact come from rage_click on the dead chip, which resolves to the element through the evidence of the sessions involved, and no rage click on the chip appeared in any arm without the dead chip.

Defect Activation-default Retention-default
dead_control_home_chip_05 hit, element signal miss
dead_control_home_chip_10 hit, element signal miss
server_error_saved_route miss miss
entry_removal_saved_total hit, page signal that also fired without its defect hit, page signal that also fired without its defect
entry_removal_saved_partial not scoreable not scoreable

4 of the 8 scoreable runs were hits, and 2 of those stand on the element-level signal. The 2 partial-scope runs cannot be hits by construction, because that defect’s only expected type carries no location, and the instrument’s output for them was the same findability_gap at /saved it produced for the total removal.

At retention-default, 10 of 17 sessions reached the fully dead chip and none of the 10 rage-clicked it. Those personas clicked the chip a median of 1 time per session and moved on, where activation-default personas clicked it a median of 82 times in module 7, so the instrument’s sensitivity depends on the mode personas run in, in a way the ceiling arithmetic cannot see.

The rage-click rule needs 3 or more consecutive clicks on one element in an unchanged state, which let the half-dead chip’s ceiling be computed in advance at 0.629 per session in activation mode and 0.512 in retention mode. The observed rate was 0.05 in module 7, 1 of 19 sessions, and 0 in every resampled population. Personas clicked the half-dead chip a median of 18 to 31 times per session in activation mode, so they kept trying, and they spaced their retries with other actions: across all 93 sessions on that chip, 1 produced a qualifying run.

At the realized rates, detecting the fully dead chip with 80% power takes 12 sessions per arm, and the frozen n gives 0.99 power. For the half-dead chip the point estimate is 145 sessions, resting on 1 detected session, and the figure is 45 at the top of its interval and undefined at the bottom, where the realized rate equals the control’s.

The server error on the Saved page was missed in both pairings with reach intact, since 11 of 19 and 11 of 17 sessions arrived at /saved.

A false positive here is a seeded-arm finding absent from its population’s single control run, so it counts differences from 1 run, with no adjudication of whether either run was wrong. There were 9 across the 10 module 7 seeded runs and 1, 5 and 3 in the resampled populations, and findability_gap at /itinerary was counted as one in 5 runs while appearing in 2 of the 3 resampled baselines, a recurring finding the module 7 control happened not to emit.

Resampling survival

Each module 7 hit at activation-default was rerun under 3 resampled populations: 2 reseeds of the same archetypes with new jitter seeds, and 1 perturbed archetype mix. A hit survives a population when the frozen classifier scores the same defect a hit there. None of the 3 resampled baselines carried a finding at any seeded location, which rules out on this data the explanation that a new population produces the finding with or without the defect.

The archetype mix also scored a hit on the Saved page server error that the original population missed. That hit rests on confusion_topology at /saved, which fired as a false positive in an arm with no defect there, on 1 run, so it is not evidence for the resampling axis.

2 constraints bound how far this generalizes. The decision cache is keyed by archetype and the model-tier prompts see only the archetype-level persona, so jitter reseeding perturbs only rule-level decisions, think time and input dynamics. And the mix arm kept run seed 42, so 12 of its 19 personas are the same individuals as module 7’s.

Deviations from the pre-registration

The DIFF-5 assertion failed, and the spec’s rule for a failed assertion was to halt and return to design. After seeing the DIFF-5 figures and before module 8 started, I ruled to record the failure as a finding and proceed. No verdict changed, and the direction was conservative: the instrument’s output at partial scope matched its output at total scope, and the frozen type list scores it a near miss.

Amendment 1 fixed defects that the cold dry reads of the analysis and write-up specs found before any arm ran. It pinned the sealed baseline’s shape, made a wrong shape raise an error instead of loading empty, and froze the bootstrap seed at 20261006, leaving classification logic unchanged. Amendment 2 dropped the 3 resampling arms of the one defect that cannot be a hit by construction, during module 7, before any verdict was computed and before any resampling data existed.

The probes that measured the ICC and the click budgets behind the dose ceilings ran on engine commit 9e47dcc, before the decision-cache change, and the arms ran on 591f985.

The engine defects the study raised

The study raised 5 engine spec-changes, 105 to 109. Spec-change 105 added within-persona replication at af2756f, a missing capability without which persona-level ICC could not be estimated. Spec-change 106 fixed sessions that died on a navigating target, where a run of dead sessions reported the status a healthy run gets, at 9e47dcc. Spec-change 107 fixed a decision cache that latched at a per-step cost superlinear in session length, at b7bc7e8. Spec-change 108 recrawled the 1-node state graph to test a hypothesis, which was refuted, and found the real cause, which spec-change 109 fixed at 591f985: the crawler read the destination before navigation committed, and a truncated crawl printed a clean result.

Reproducing it

analysis/verify_report_figures.py recomputes every figure in the study report from its source and checks that the report is exactly its template rendered from results/figure_registry.json. 3 of those figures read a gitignored run log and the article’s instance table reads paths in other repos, so a public clone cannot rerun all of it until those sources are published or moved onto committed files.

Limits

  • Every verdict rests on 1 run per arm and pairing. Run-to-run variance is unmeasured and confounded with the defect, and the historical corpus says it is large, with the completion rate's spread per run at 0.319 against 0.090 per persona.
  • Page-level signals fire without their defect, so the 2 entry-removal hits and the archetype mix's server-error hit rest on signals that also fired where nothing was wrong.
  • Sessions are coupled by archetype through the decision cache, so per-session rates and their intervals overstate the independent evidence.
  • There is 1 target, Sidestreet's web app, which I own and studied under a clean-room rule.
  • The personas are synthetic and no human ground truth anchors the findings. Each baseline finding is either a real pre-existing defect or a false positive of the instrument.
  • The thresholds are the engine's, pinned at engine commit 591f985 and checked before every run, and the probes behind the dose ceilings ran on the earlier engine commit 9e47dcc.
  • On one page the engine's model calls return invalid JSON most of the time. The study's sources do not name the page, so whether it bears on either defect site is unknown, and it can suppress confusion_topology.
  • The rage-click recorder keeps only each session's first incident, and in 1 session that hid a later rage on the chip.
  • The realized ICC cannot be estimated at 1 session per persona.
  • Between 84% and 100% of sessions per run ended at a step or wall-clock limit, which barely discriminates arms and is why findability_gap fires when nobody arrived.
  • Removing 2 of 4 entry points to the Saved page cut reach to 0, the same as removing all 4, so reach falls in a single step along the scope axis.
  • The DIFF-5 assertion failed and its halt was overridden by ruling before module 8, to record the failure as a finding. The same ruling added a cost stop the frozen stopping rule does not contain, and it did not fire.
  • Engine spend was $149.51, measured, across modules 0, 5, 6, 7 and 8.
  • The target app's own model calls were billed to the same account and recorded by nothing in the study.
  • The paired contrasts, achieved power and covariates are reported outside the frozen metrics registry.
Check this study yourself
Freeze
0cc3924
Pre-registered
2026-10-06, UTC date of commit 0cc3924, made 2026-10-05 19:40 -07:00
Timestamp
git-dated only; the date rests on author-controlled git metadata, and no third-party timestamp attestation exists for this freeze.
# run 2026-10-13 against a local clone of the study repo at dc7b92a
$ git log -1 --format='%H %cI' 0cc3924
0cc392433bb471d577b6feaa5fa79d0200091eaa 2026-10-05T19:40:44-07:00
$ jq -r .freeze_commit.hash freeze/MANIFEST.json
0cc392433bb471d577b6feaa5fa79d0200091eaa
$ .venv/bin/python -B analysis/verify_report_figures.py | tail -1
VERIFY PASS
PASS freeze/MANIFEST.json names 0cc3924, committed 2026-10-06 02:40:44 UTC

Evidence sources

  • Report figure verifier understudy-detection-study/analysis/verify_report_figures.py154 checks OK, VERIFY PASS, run 2026-10-13 at dc7b92a; repo private until release; links activate then.Recomputes every figure in the report from its source and checks the report is exactly its template rendered from the registry. 3 figures read a gitignored run log, so a public clone cannot rerun all of it yet.
  • Freeze manifest understudy-detection-study/freeze/MANIFEST.jsonCommit 0cc3924; 2 dated amendments; repo private until release; links activate then.The sha256 of every frozen item, from the defect set and criterion to n per arm, the analysis code, the stopping rule and the resampling plan.
  • Run verdicts and session counts understudy-detection-study/results/f1_qualified_verdicts.csvWith sessions.csv, per_archetype_detection.csv, chip_click_behavior.csv and tpr_by_dose.csv; repo private until release; links activate then.Run verdicts with the page-signal qualification, per-session and per-archetype detection, and the click counts behind the ceiling shortfall.
  • Study report and decision understudy-detection-study/report/study.mdRendered from its template; decision section 2026-10-11; repo private until release; links activate then.Full design, results and limits, and the decision section with the dated calls on each open engine item.
  • DIFF-5 ruling understudy-detection-study/docs/rulings/2026-10-09-diff5-and-module-8.md2026-10-09, before module 8; repo private until release; links activate then.The override of a pre-registered halt, recorded as a finding, with the cost stop it added.
  • Outside audit of the write-up understudy-detection-study/docs/audits/2026-10-09-fable-writeup-audit.md16 findings, all applied; repo private until release; links activate then.The cold, read-only audit that found 3 of the 4 analysis-layer instances after the study had passed its own checks.
  • Engine defect notes understudy-detection-study/docs/engine-notes/109-crawler-navigation-commit-race.mdWith notes 106 and 108, reproduced from the private engine repository; repo private until release; links activate then.The 3 engine defects the study found, each with the commit that fixed it.