Artifacts · published store
Evaluation evidence
Ten adversarial fixtures, three frozen corpora, and recorded story traces. Pick a run to inspect missed stories.
The sieve
Fixture Ray, 1:1 with the corpusStill in the pipeline
1,362
of 1,362100.0%
Stage 01 / 10 · pool
Corpus pool
Nothing removed yet.
Every article in the frozen snapshot, plus the needles planted for this persona.
Where these counts come from
Recorded survivorship for fixture Ray against corpus 2026-09-02, reconstructed from the scorecard’s stage tallies and asserted against three independently recorded facts in funnel.test.ts. Which particular cell a stage took is not recorded anywhere, so cells are scattered by a fixed hash. The quantities are evidence, the positions are not.
Funnel for Ray
Candidates surviving each stage
The harness records each article once, at the furthest stage it reached, so survivors at a stage are that tally plus every later one. Reading the raw tallies as a funnel would show it growing.
| Stage | Survivors | Lost here | Pass rate | Proportion of the pool remaining |
|---|---|---|---|---|
| Corpus poolEvery article in the frozen snapshot, plus the needles planted for this persona. | 1,362 | 0 | unknown | |
| Recency windowWhat the recency window actually returned from the database. | 300 | 1,062 | 22.0% | |
| LoadedRows that survived loading and had usable text. | 298 | 2 | 99.3% | |
| Prefilter(not reported)Survivors of the cheap keyword and exclusion pass that runs before any model call. | 100 | 198 | 33.6% | |
| ScoredCandidates the relevance scorer actually saw. Nothing beyond this point can rescue a story dropped earlier. | 100 | 0 | 100.0% | |
| Blended(not reported)Candidates left after the score blend. | 55 | 45 | 55.0% | |
| Deduplicated(not reported)Candidates left after near-duplicate removal. | 55 | 0 | 100.0% | |
| Diversity pass(not reported)Candidates left after the source-diversity pass. | 55 | 0 | 100.0% | |
| Role assignmentCandidates left after hero and row roles were assigned. | 55 | 0 | 100.0% | |
| Delivered feedWhat the reader would have opened. | 50 | 5 | 90.9% |
Where wanted stories were lost
Only must-see stories that never reached the reader.
| Stage or mechanism | Must-see stories lost |
|---|---|
| lookback | 6 |
| prefilter:cap | 1 |
prod-llm · 2026-09-02 · k=12
compared with proto-hybrid-judge-events · 2026-09-02 · k=12
Across 1 fixture: 5 unwanted placements delivered (5 distinct articles), and 7 wanted placements lost before scoring (7 distinct articles).
- delivered-wanted7 pairs / 7 articles
- delivered-unwanted5 pairs / 5 articles
- delivered-unlabelled0 pairs / 0 articles
- lost-before-scorer7 pairs / 7 articles
- lost-at-or-after-scorer0 pairs / 0 articles
Inspect run details
- Estimated model cost, all personas
- $0.265
- Model calls, all personas
- 44
- Peak model calls for one persona
- 7
- Cache misses
- 0
Published scorecards and corpus inventory
9 published scorecards, including retained baselines. Snapshots share articles, so their counts overlap. 11,413 provisional model and agent label rows across these snapshots.
2026-08-31
1,328 corpus articles
SHA-256 6950caf50c74b867208f7f962f612de5cb7711f70f7eb1218f4ce2bf64e66886
2026-08-31-quiet
1,234 corpus articles
SHA-256 e0f365f2a18bb852501262897c436333555d78b4a498c7d90353e284bd96d7e5
2026-09-02
1,358 corpus articles
SHA-256 73f93e5b8cc02c189823f0fc6bbf6b97c27e5bd5e44bcef9d79fa5a0045921b4
Provenance · prod-llm · 2026-09-024 caveats on this run
Three revisions, kept apart
- Executed the evaluation
- 47edb50not reachable from the default branch
- Stores the scorecard
- 3b11a3c
- Built this artifact
- 3b11a3c
This artifact imports a stored scorecard. It is not a run this pipeline executed, and no current revision is stamped onto its numbers.
Run identity
- Runner
- prod-llm
- CLI argument
- prod
- Protocol
- unknownabsent in the source scorecard
- k
- 12
- Execution mode
- unknownThe scorecard does not record an execution mode. A zero cache-miss total cannot stand in for one, because the harness only counts misses on the live network path.
- Recorded at
- 2026-09-02T07:14:29+00:00
- Needles planted
- true
- Quiet corpus
- false
- Cache keys touched
- 89
Models observed: gpt-4o-mini
Corpus
- Snapshot
- 2026-09-02
- Articles
- 1,358
- Frozen at
- 2026-09-02T05:06:49.958134+00:00
- Content hash
- 73f93e5b8cc02c18…
- Derived from
- original capture
- Clusters removed
- none
second live RSS snapshot, 50+ hours after 2026-08-31
Ground truth
- Label rows
- 3,918persona/article pairs
- Distinct articles
- 1,260labelled at least once
- Fixtures
- 10
- Contested
- 209
- Status
- provisional model and agent
- Label models
- gpt-4.1, gpt-4.1-mini
By source: agent 345 · model 3,573. Labels are model-written with an agent editorial pass; product-owner human review is outstanding, so absolute values are provisional and only run-to-run differences are gate-enforced.
Caveats
- Warning. The revision that executed this run (47edb50) is not reachable from the current default branch, so "this result came from that code" cannot be verified by checking out the revision. Everything under backend/evals/ reached the default branch in a single squash commit.git merge-base --is-ancestor 47edb50 HEAD
- Warning. This scorecard records no protocol identifier. The harness only began emitting one later (backend/evals/run.py writes meta.protocol), so the evaluation protocol cannot be read from the artifact and is reported as unknown rather than inferred from the runner name.backend/evals/run.py
- Caution. This scorecard is missing summary keys that the current harness emits unconditionally (reader_pipeline_calls_max_per_persona, event_slot_contamination_mean). It therefore could not have been produced by the code on the current default branch, and re-running today would not yield a document of the same shape.backend/evals/run.py, backend/evals/metrics.py
- Caution. A reported cache-miss total of zero does not prove offline replay: the harness increments its miss counter only on the live network path, so the counter is structurally zero whenever offline mode is active. Execution mode is therefore reported as unknown unless the scorecard states it.backend/evals/llm_cache.py
- Note. Stored in the repository tree at backend/evals/results/47edb50-prod-llm-2026-09-02.json.backend/evals/results/47edb50-prod-llm-2026-09-02.json
Provenance · proto-hybrid-judge-events · 2026-09-025 caveats on this run
Three revisions, kept apart
- Executed the evaluation
- 47edb50not reachable from the default branch
- Stores the scorecard
- 3b11a3c
- Built this artifact
- 3b11a3c
This artifact imports a stored scorecard. It is not a run this pipeline executed, and no current revision is stamped onto its numbers.
Run identity
- Runner
- proto-hybrid-judge-events
- CLI argument
- unknown
- Protocol
- unknownabsent in the source scorecard
- k
- 12
- Execution mode
- unknownThe scorecard does not record an execution mode. A zero cache-miss total cannot stand in for one, because the harness only counts misses on the live network path.
- Recorded at
- 2026-09-02T07:14:42+00:00
- Needles planted
- true
- Quiet corpus
- false
- Cache keys touched
- 282
Corpus
- Snapshot
- 2026-09-02
- Articles
- 1,358
- Frozen at
- 2026-09-02T05:06:49.958134+00:00
- Content hash
- 73f93e5b8cc02c18…
- Derived from
- original capture
- Clusters removed
- none
second live RSS snapshot, 50+ hours after 2026-08-31
Ground truth
- Label rows
- 3,918persona/article pairs
- Distinct articles
- 1,260labelled at least once
- Fixtures
- 10
- Contested
- 209
- Status
- provisional model and agent
- Label models
- gpt-4.1, gpt-4.1-mini
By source: agent 345 · model 3,573. Labels are model-written with an agent editorial pass; product-owner human review is outstanding, so absolute values are provisional and only run-to-run differences are gate-enforced.
Caveats
- Warning. The revision that executed this run (47edb50) is not reachable from the current default branch, so "this result came from that code" cannot be verified by checking out the revision. Everything under backend/evals/ reached the default branch in a single squash commit.git merge-base --is-ancestor 47edb50 HEAD
- Warning. This scorecard records no protocol identifier. The harness only began emitting one later (backend/evals/run.py writes meta.protocol), so the evaluation protocol cannot be read from the artifact and is reported as unknown rather than inferred from the runner name.backend/evals/run.py
- Caution. This scorecard is missing summary keys that the current harness emits unconditionally (reader_pipeline_calls_max_per_persona, event_slot_contamination_mean). It therefore could not have been produced by the code on the current default branch, and re-running today would not yield a document of the same shape.backend/evals/run.py, backend/evals/metrics.py
- Warning. This runner name belongs to the corrected default prototype pipeline, but the repository’s own regression gate maps this runner key to the historical proto-s0-legacy-v1 adapter, whose protocol reproduces known defects (per-reader event discovery, uncapped forced delivery that ignores reader exclusions). Treat this run as evidence about the historical S0 protocol, not as validation of the corrected default pipeline.backend/tests/test_eval_gate.py, backend/evals/legacy_s0.py
- Caution. A reported cache-miss total of zero does not prove offline replay: the harness increments its miss counter only on the live network path, so the counter is structurally zero whenever offline mode is active. Execution mode is therefore reported as unknown unless the scorecard states it.backend/evals/llm_cache.py
- Note. Stored in the repository tree at backend/evals/results/47edb50-proto-hybrid-judge-events-2026-09-02.json.backend/evals/results/47edb50-proto-hybrid-judge-events-2026-09-02.json