Daily makes news personal.

Algorithm comparison: prod-llm against proto-hybrid-judge-events, same corpus and same k. Protocol equality is unverified.

Inspect comparison details

protocol: Neither run recorded a protocol identifier. The harness only began emitting one after these scorecards were written. Protocol equality therefore cannot be verified from the artifacts, only assumed from the runner name.
eval_revision: Both runs executed at the same revision, so any difference reflects the two pipelines, not a code change over time.

01

Run comparison

Averaged over 10 fixtures

prod-llmproto-hybrid-judge-eventsworse

Capped recall@k+15.9 ppCapped recall@k: prod-llm 22.1 per cent, proto-hybrid-judge-events 38.0 per cent, a difference of +15.9 pp, past the fixed materiality cutoff.
Raw recall@k+15.0 ppRaw recall@k: prod-llm 20.3 per cent, proto-hybrid-judge-events 35.3 per cent, a difference of +15.0 pp, past the fixed materiality cutoff.
Reached the scorer+39.1 ppReached the scorer: prod-llm 32.1 per cent, proto-hybrid-judge-events 71.2 per cent, a difference of +39.1 pp, past the fixed materiality cutoff.
Unwanted rate−21.9 ppUnwanted rate: prod-llm 26.9 per cent, proto-hybrid-judge-events 5.0 per cent, a difference of −21.9 pp, past the fixed materiality cutoff.
Need-to-know recall+21.8 ppNeed-to-know recall: prod-llm 17.4 per cent, proto-hybrid-judge-events 39.1 per cent, a difference of +21.8 pp, past the fixed materiality cutoff.
World-critical delivery+75.0 ppWorld-critical delivery: prod-llm 25.0 per cent, proto-hybrid-judge-events 100.0 per cent, a difference of +75.0 pp, past the fixed materiality cutoff.
Planted-needle recallunchangedPlanted-needle recall: prod-llm 60.0 per cent, proto-hybrid-judge-events 60.0 per cent, a difference of unchanged.
Lookalike rate−5.0 ppLookalike rate: prod-llm 15.0 per cent, proto-hybrid-judge-events 10.0 per cent, a difference of −5.0 pp, past the fixed materiality cutoff.

Fraction metrics share a 0–100% scale, ticked at 0, 50 and 100.

Inspect all metrics

Evaluation metrics, averaged across the ten reader fixtures, with the comparison run’s difference
Metric	prod-llm	proto-hybrid-judge-events	Difference	Weakest fixture
Capped recall@kNeeded stories that made the feed, capped at the slots available.	22.1%	38.0%	Capped recall@k moved +15.9 pp: better, past the fixed ±0.02 materiality cutoff.	wei 0.0%
Raw recall@kSame, divided by every needed story. Always ≤ capped recall.	20.3%	35.3%	Raw recall@k moved +15.0 pp: better, past the fixed ±0.02 materiality cutoff.	wei 0.0%
Reached the scorerNeeded stories that survived long enough to be judged at all.	32.1%	71.2%	Reached the scorer moved +39.1 pp: better, past the fixed ±0.02 materiality cutoff.	wei 11.8%
Unwanted rateDelivered stories the reader had labelled never-wanted.	26.9%	5.0%	Unwanted rate moved −21.9 pp: better, past the fixed ±0.02 materiality cutoff.	dilshod 83.3%
Need-to-know recallRecall over stories the reader genuinely needed to know.	17.4%	39.1%	Need-to-know recall moved +21.8 pp: better, past the fixed ±0.02 materiality cutoff.	dilshod 0.0%
World-critical deliveryWorld-critical events that put at least one story in the feed.	25.0%	100.0%	World-critical delivery moved +75.0 pp: better, past the fixed ±0.02 materiality cutoff.	aisha 0.0%
Planted-needle recallRecall on planted articles whose right answer is known by construction.	60.0%	60.0%	Planted-needle recall is unchanged.	cold 0.0%
Lookalike rateDecoys that got through: right keyword, wrong thing.	15.0%	10.0%	Lookalike rate moved −5.0 pp: better, past the fixed ±0.02 materiality cutoff.	aisha 100.0%
Follow-up recall	11.1%	10.7%	Follow-up recall moved −0.4 pp: worse, within the fixed ±0.02 materiality cutoff.	farrukh 0.0%
Major-event delivery	22.0%	14.0%	Major-event delivery moved −8.0 pp: worse, past the fixed ±0.02 materiality cutoff.	dilshod 0.0%
Judge precision	45.6%	82.7%	Judge precision moved +37.1 pp: better, past the fixed ±0.02 materiality cutoff.	aisha 0.0%
Judge recall	35.0%	75.5%	Judge recall moved +40.5 pp: better, past the fixed ±0.02 materiality cutoff.	aisha 0.0%
Distinct sources	6.70	6.60	Distinct sources moved −0.10: worse, past the fixed ±0.02 materiality cutoff.	farrukh 5
Feed size at k	11.70	11	Feed size at k changed by −0.70; direction is neutral.	unknown
Build latency	2.581s	0.021s	Build latency moved −2.560s: better, past the fixed ±0.02 materiality cutoff.	ray 5.170s

Weakest fixture shows the worst result for each metric. A filled dot marks the fixed ±0.02 materiality cutoff, not statistical significance.

What each metric means, exactly

Capped recall@khigher is better: Of the stories this reader had to see, the share that made the feed, with the target capped at the number of slots available.
Raw recall@khigher is better: The same count divided by every must-see story, even when there are more must-see stories than feed slots. Always at or below capped recall.
Reached the scorerhigher is better: The share of must-see stories that survived long enough to be judged at all. A story never loaded cannot be ranked.
Unwanted ratelower is better: The share of the delivered feed that the reader had explicitly labelled as never wanted. Lower is better.
Need-to-know recallhigher is better: Recall restricted to stories labelled as ones the reader genuinely needed to know.
World-critical deliveryhigher is better: Of the events everyone should have been told about, the share that put at least one story in the feed.
Planted-needle recallhigher is better: Recall on stories injected at run time whose correct answer is known by construction rather than by opinion.
Lookalike ratelower is better: How often a deliberate decoy got through: the right keyword attached to the wrong thing.
Follow-up recallhigher is better: Recall restricted to stories that continue a thread the reader was already following.
Major-event deliveryhigher is better: The same measure for events rated major rather than world-critical.
False-major ratelower is better: How often the pipeline promoted a story as a major event when no major event existed. Measured on the quiet-day corpus, where the right answer is restraint.
Judge precisionhigher is better: When the model accepted a story, how often the labels agreed. Reported only for runners whose judge emits an explicit verdict.
Judge recallhigher is better: Of the stories the labels wanted, how many the model accepted once it saw them.
Distinct sourceshigher is better: How many different publications the feed drew from. More breadth is generally healthier, but this is a diversity proxy, not a correctness measure.
Feed size at kno direction: How many slots the feed actually filled. A size, not a quality score.
Build latencylower is better: Wall-clock time to assemble one reader’s feed during the run. Offline replay timings are not production latency.

Inspect fixture results

Delivered, wanted
Delivered, unlabelled
Delivered, unwanted
Lost at the scorer
Lost before the scorer

Bar length is labelled story placements; the right column is capped recall at k, mean 22.1%. A fixture at 0.0% received none of the stories its labels said it needed. Select one to filter this view.

prod-llm · 2026-09-02 · k=12

compared with proto-hybrid-judge-events · 2026-09-02 · k=12

Across 10 fixtures: 32 unwanted placements delivered (29 distinct articles), and 88 wanted placements lost before scoring (84 distinct articles).

delivered-wanted62 pairs / 55 articles
delivered-unwanted32 pairs / 29 articles
delivered-unlabelled23 pairs / 22 articles
lost-before-scorer88 pairs / 84 articles
lost-at-or-after-scorer9 pairs / 8 articles

Inspect run details

Estimated model cost, all personas: $0.265
Model calls, all personas: 44
Peak model calls for one persona: 7
Cache misses: 0

Published scorecards and corpus inventory

9 published scorecards, including retained baselines. Snapshots share articles, so their counts overlap. 11,413 provisional model and agent label rows across these snapshots.

2026-08-31
1,328 corpus articles
SHA-256 6950caf50c74b867208f7f962f612de5cb7711f70f7eb1218f4ce2bf64e66886
2026-08-31-quiet
1,234 corpus articles
SHA-256 e0f365f2a18bb852501262897c436333555d78b4a498c7d90353e284bd96d7e5
2026-09-02
1,358 corpus articles
SHA-256 73f93e5b8cc02c189823f0fc6bbf6b97c27e5bd5e44bcef9d79fa5a0045921b4

Provenance · prod-llm · 2026-09-024 caveats on this run

Three revisions, kept apart

Executed the evaluation: 47edb50not reachable from the default branch
Stores the scorecard: 3b11a3c
Built this artifact: 3b11a3c

This artifact imports a stored scorecard. It is not a run this pipeline executed, and no current revision is stamped onto its numbers.

Run identity

Runner: prod-llm
CLI argument: prod
Protocol: unknownabsent in the source scorecard
k: 12
Execution mode: unknownThe scorecard does not record an execution mode. A zero cache-miss total cannot stand in for one, because the harness only counts misses on the live network path.
Recorded at: 2026-09-02T07:14:29+00:00
Needles planted: true
Quiet corpus: false
Cache keys touched: 89

Models observed: gpt-4o-mini

Corpus

Snapshot: 2026-09-02
Articles: 1,358
Frozen at: 2026-09-02T05:06:49.958134+00:00
Content hash: 73f93e5b8cc02c18…
Derived from: original capture
Clusters removed: none

second live RSS snapshot, 50+ hours after 2026-08-31

Ground truth

Label rows: 3,918persona/article pairs
Distinct articles: 1,260labelled at least once
Fixtures: 10
Contested: 209
Status: provisional model and agent
Label models: gpt-4.1, gpt-4.1-mini

By source: agent 345 · model 3,573. Labels are model-written with an agent editorial pass; product-owner human review is outstanding, so absolute values are provisional and only run-to-run differences are gate-enforced.

Caveats

Warning. The revision that executed this run (47edb50) is not reachable from the current default branch, so "this result came from that code" cannot be verified by checking out the revision. Everything under backend/evals/ reached the default branch in a single squash commit.git merge-base --is-ancestor 47edb50 HEAD
Warning. This scorecard records no protocol identifier. The harness only began emitting one later (backend/evals/run.py writes meta.protocol), so the evaluation protocol cannot be read from the artifact and is reported as unknown rather than inferred from the runner name.backend/evals/run.py
Caution. This scorecard is missing summary keys that the current harness emits unconditionally (reader_pipeline_calls_max_per_persona, event_slot_contamination_mean). It therefore could not have been produced by the code on the current default branch, and re-running today would not yield a document of the same shape.backend/evals/run.py, backend/evals/metrics.py
Caution. A reported cache-miss total of zero does not prove offline replay: the harness increments its miss counter only on the live network path, so the counter is structurally zero whenever offline mode is active. Execution mode is therefore reported as unknown unless the scorecard states it.backend/evals/llm_cache.py
Note. Stored in the repository tree at backend/evals/results/47edb50-prod-llm-2026-09-02.json.backend/evals/results/47edb50-prod-llm-2026-09-02.json

Provenance · proto-hybrid-judge-events · 2026-09-025 caveats on this run