What does each retrieval layer buy, and what does strictness cost a discovery engine?

Sidestreet's retrieval engine once required a venue to carry an occasion label before it could be returned, and that filter was removed in July 2026. I ran a pre-registered ablation to measure what the filter had cost and what each other retrieval layer contributes.

Restoring the filter moved judged list quality from 3.93 to 3.97 on a 5-point scale, inside the run-to-run spread. A good answer is a venue in a list the engine's judge scored 4 or 5 in most of 10 baseline runs, a classification checked against 30 human-graded items.

Findings

• Restoring the strict occasion filter puts about a sixth of the engine's good answers out of reach.

  16.7% [8.0, 26.6]; 31 of 186 good-answer pairs; the only strict arm that cleared the stability gate

• Restoring the filter moved judged list quality by 0.04 points, inside the run-to-run spread.

  3.93 to 3.97 on a 5-point scale; per-run judge scores are not published, so this does not recompute from the repo

• A single run of the engine captures 55.5% of the venues that 10 identical runs reach for a query.

  8.47 venues per query at 1 run, 15.28 at 10 and still rising; 611 venue-query pairs on 40 queries

• Removing query interpretation costs the most good answers of any layer, and removing the cluster layer costs the least.

  69.4% [52.8, 84.5] against 11.3% [4.5, 20.0]; both arms ungated, so both figures are directional

• The headline metric covers only the 28 of 40 queries that have a good answer, so it says nothing about the 12 the engine answers badly.

  186 pairs from 28 queries; the metric and the answer-set rule were frozen separately and together drop the other 12

• The arm built to measure the expansion stage's coverage failed the pre-registered stability gate and was not compared.

  Median split-half Jaccard 0.867 against a 0.90 bar; the stability table needs per-run data that is not published

• The engine's judge returned a passing score whenever its own call failed.

  Fallback signature in 13 of 240 query-runs (5.4%) before the fix and 0 of 280 after; the patch is published, the run counts are not

• A 30-item blind human check agreed with the judge's good-answer classification on 26 items, and the disagreements did not lean one way.

  Judge higher on 5 items, human higher on 6; on 19 items where 2 model graders disagreed, the human sided with the judge 17 times; I graded it, so it was not blind to intent

What it changed

What was measured

The engine is layered. Query interpretation runs as its own model stage, routing then sends the query to name matching or contextual matching under a loose hard filter, results are re-sorted, and a final stage scores them and labels the match logic. Each of the 9 arms removes 1 stage or restores the old filter, and every arm ran against 40 held-out benchmark queries.

The metric is recall of good answers lost. Of the venues the engine’s own judge grades as good under the full architecture, it counts the share an arm can no longer return. A good answer is a venue that appeared in a list the judge scored 4 or 5 in a majority of 10 baseline runs, which gives 186 venue-query pairs from 28 queries. Intervals resample those 28 queries 10,000 times under a fixed seed, because pairs within a query are not independent.

Answer sets are majority-rule sets over repeated runs. The engine sets no temperature or seed at any model call site, so the same query returns different venues from run to run, and a single run proved too variable to stand for the engine. A single run captures 55.5% of the venues 10 identical runs reach, so across the roughly 30 engine versions benchmarked on single runs, movement inside that spread was never signal.

A stability gate was fixed before the arms ran: an arm had to reproduce its own answer set across independent halves of its runs, at a mean split-half Jaccard of 0.70 and a median of 0.90 on a 5-vs-5 split, before it could be compared. It admitted the baseline and 1 strict arm, and the other 6 arms ran 5 times where the gate needs 10.

The strict arm reconstructs the July filter by running the original July prompt over the benchmark to recover the filter’s own routing decisions, then applying its database predicates unchanged. The routing decisions were resolved in advance by majority vote over repeated parses and frozen as a table.

What the other layers appear to buy

The 6 ungated arms ran 5 times where the gate needs 10, so their figures are directional. Each row removes 1 layer with the others intact, so the rows cannot be summed into a decomposition of where quality lives. Removing a layer can also raise the average score of what survives while shrinking what survives: A3 loses 39.8% of good answers and has the highest judged quality of any arm, at 4.04. Only A5 degrades on both measures, by a margin no other arm approaches.

The cluster layer is the most elaborate machinery in the system, with 144 emergent clusters over the catalog, 32,535 membership rows, a similarity table of 3.9 million rows and a boost applied at query time. Removing both of its mechanisms cost 11.3% [4.5, 20.0], about a sixth of what removing the query parse cost. Ablating it took a code change at 2 sites, because the system’s own feature flag disabled 1 mechanism and swapped a legacy scorer in for the other. The metric also cannot see what the layer might do for result diversity, which this study did not measure.

Why the architecture is layered at all

The corpus the engine retrieves over is 4,682,278 tokens, 23.4 times a 200,000-token context window, so a single model call over the whole corpus cannot be run. The single-pass arm therefore runs on the largest representation that fits, a 70,003-token structured index of name, type, neighborhood, rating and review count, and loses 87.1% of good answers. That loss is a floor on what layering buys, since the index is a weak stand-in for the reviews and editorial content behind it. Given that index and asked only to return matching venues, the model returned venues that are not in it for 3.3% of the names it returned.

What the checks caught

The engine’s judge returned a passing score of 3 whenever its own call failed, through 3 layers that each preserved the behavior, and its output truncated on about half of large result sets. The fix cut that fallback from 5.4% of query-runs to 0 and moved the mean score from 3.95 to 3.83.

The arm built to measure what the content-native expansion stage buys in coverage failed the stability gate on its median, 0.867 against 0.90, and was excluded from comparison. Its provisional coverage figure, 8.5 points of recovered good answers, was the number this study most wanted, and it is not in the results. What the pairing showed instead, directionally, is that removing the stage made the arm less able to reproduce its own answer set.

2 arms built exactly as frozen would not have measured what they claimed. The strict arm leaked 15.8% of venues that failed its filter, through the expansion stage, which was introduced 1 commit after the filter reached peak strictness. The SQL filter arm changed nothing, because 23 further conditions are appended after the line it was specified to change. Verifying each arm’s output before running it caught both.

The standing process requires a smoke pilot before the freeze, and this piece’s module list left it out. Written afterward as assertions on properties of the output and pointed at data that already existed, it found 5 defects in 1 pass, 2 of which the build phase had missed and none of which crashed anything.

The benchmark harness truncated venue names at 33 characters before printing them, and the analysis matched on those names, so 144 of 5,751 venues could not be matched. Long names cluster among libraries, botanic gardens, park visitor centers and restaurants with descriptive names, so the defect would have read as the strict filter removing civic and cultural venues.

The judge’s call seeds its reply with an opening brace to get bare JSON, and the model tier used to cross-check it rejects that technique outright. The first cross-check run failed on all 60 items for that reason, and the workaround gave the cross-check a different prompt shape, which weakens the comparison in exchange for making it possible.

That cross-check, a different model tier given only the query and the returned venues, disagreed with the judge’s good-answer classification on 19 of its 60 items (32%) and rated systematically lower. Within the 30-item human-check sample the 2 models disagreed on 19 items (63%), and the human sided with the judge 17 times and with the cross-check twice. The cross-check had been deprived of the judge’s context to make it independent, and that made it the weaker grader.

How layer isolation was arrived at

The layered architecture accumulated one separation at a time, as each part of what began as a single retrieval call failed in a way that could not be diagnosed from outside. The first design labeled every venue at ingest from a fixed list of 11 occasion labels and 3 sequence roles, the labels built the clusters, and the clusters built the relationship edges, so one wrong label could reshape the graph around it. Labels also carried 70% to 78% of the ranking weight, which buried venues the list described badly and kept new venues with no collaborative-filtering history out of results.

I spent several rounds repairing this inside the same framework, with better lists, prompts and filters, and each round made the framework more expensive to abandon. The repairs improved the labels without improving retrieval: a request for cheap late-night food after midnight came back with museums, and the public library ranked ahead of cafes for espresso and pastry. One pass was parsing the query, matching names, filtering on meaning and pulling related results, and a wrong answer gave no way to tell which of the 4 had failed.

The repair rounds ended when the premise turned out to be wrong: the query has the intent and the venue has attributes, and the first design had fused the 2 at ingest. Each layer that followed had to emit an intermediate result that could be inspected and blamed on its own, which is why the evaluation harness this study runs on scores intent, retrieval and result quality separately. Query interpretation took 7 weeks to separate, from its own model stage on June 20 to the label leaving its output schema on August 8. By the time the strict filter came out, on July 13 and 14, nobody could say which layer was carrying quality, because quality had only been measured through all of the layers at once.

Reproducing it

The arm percentages, their intervals and the convergence figures recompute from 3 CSVs in derived/ with python3 specs/verify_published.py, which reads nothing else and exits non-zero on any disagreement, and specs/make_figures.py draws figures 2 and 3 from the same files. The split-half stability table and the judge quality column need per-run data that is not published. The engine is not published either, which limits a reader to re-checking the published figures. The verifier’s first run caught the second wrong version of the single-run figure, which had already passed an independent audit.

Limits

• The judge scores lists and the metric counts venues. A venue counts as a good answer when it appeared in a list scored 4 or 5, no venue was judged on its own, and the direction of the resulting error is not established. The human check graded the same unit, so it does not bound this.
• The judge shares a model snapshot with the stages it grades, including the labeling pass the strict arm filters on. The 30-item human check narrows this. I graded it myself, so it was not blind to what the engine is meant to do, and a second independent grader has not been run.
• 6 of 9 arms ran 5 times where the stability gate needs 10, so everything attributed to them is directional. Measured at the 2-vs-2 split available to them, the baseline itself fails the gate at 0.708 mean and 0.800 median.
• The arm table does not add. Each row removes 1 layer with the others intact, and by my account of the system's development, which no committed record documents, the layers interact, so the cost of removing 2 layers is unmeasured.
• The strict arm runs the byte-identical July prompt on a different model from the one that ran it in July, and its routing input is a modal vote over a nondeterministic parse, frozen in advance as a table that records the vote counts.
• Engine and judge spend was $63.26, measured, against a ceiling frozen at $40 with a $6 per-arm cap and raised to $110 on 2026-09-28.
• To afford 10 runs of 3 arms, the study cached the query-interpretation system prompt in the study copy, which modifies the system under study. Every arm ran with it, so no arm comparison is affected, and the single-run figure is a floor on the uncached engine's variability.
• 3 pre-registered commitments are not met. Unscored queries were measured and not reported as a covariate (section 1.6), the metric was never crossed against review count, content-unit count or cluster depth (section 1.7), and every figure is pooled where per-query pairing was promised (section 1.8).
• The pre-registration's counts of 2,302 unlabeled and 81 never-evaluated venues do not reproduce under any predicate tried. The freeze is published with the discrepancy intact, and the write-up reports the recomputed 2,245 and 99 with their predicate stated.
• 1 of the original filter's 5 branches is reachable from no routing rule and never executed.
• The split-half stability table and the judge quality column need per-run data that is not published, so they cannot be recomputed from the repo.

# run 2026-10-13 against a local clone of the study repo at b1ef6ef
$ git rev-parse freeze-piece-two^{commit}
373c881f3c0fbcc93d2ad96b60367ad5509a14f9
$ git log -1 --format=%cI freeze-piece-two
2026-09-25T00:41:09-07:00
$ python3 specs/verify_published.py | tail -1
  all checks passed
PASS freeze-piece-two resolves to 373c881, committed 2026-09-25 07:41:09 UTC