Daily Lab · recorded replay, not a live run
The scorer judged forty articles and never said which verdict belonged to which.
So every later verdict landed on the wrong article.This measures whether a fix actually fixes it — decided by independent checks, not by the candidate.
The response that started it
One real batch, read from the recordings- Articles sent
- 40one batch
- Verdicts returned
- 254the model kept going
- finish_reason
- stopnot even truncated
- Article ids sent
- 0position is the only link
What was sent · first 6 of 40
- 00Late-stage deals revive in Southeast Asia, but early-stage founders remain under pressuree27
- 01NASA-linked, MIT-trained founders’ nSWX raises US$2M for AI chip packaginge27
- 02Cloud Nine: Singapore's newest rooftop bar opens with infrastructure-themed cocktailsStraits Times
- 03Bringing tech and people together for Singapore’s AI futureBusiness Times SG
- 04The creator economy is distribution, not marketing. Most Asian businesses are still scaling it like a campaigne27
- 05Economists raise Singapore’s growth forecast, lower inflation projections in latest MAS surveyBusiness Times SG
What came back · first 6 of 254
- 00“This article discusses the current state of Southeast Asia's venture capital market, which is highly relevant for a senior backend engineer interested in the startup scene.”
- 01“The article covers a Malaysian startup focused on AI chip packaging, which intersects with AI engineering and cloud infrastructure interests.”
- 02“This article is about a rooftop bar opening in Singapore, which does not align with the user's interests.”
- 03“The article discusses the creator economy, which is not a primary focus for the user.”
- 04“This article focuses on economic forecasts and inflation, which do not align with the user's technical interests.”
- 05“This article discusses new capital rules for payment firms in Singapore, directly related to fintech regulation.”
Recorded during an offline replay of the production runner. No article identifier appears anywhere in the request or the response: position is the only thing associating a verdict with an article.
The recorded response · 44,411 bytes, first 600 shown
{
"results": [
{
"relevant": true,
"score": 0.8,
"reason": "This article discusses the current state of Southeast Asia's venture capital market, which is highly relevant for a senior backend engineer interested in the startup scene."
},
{
"relevant": true,
"score": 0.7,
"reason": "The article covers a Malaysian startup focused on AI chip packaging, which intersects with AI engineering and cloud infrastructure interests."
},
{
"relevant": false,
"score": 0.0,
"reason": "This article is about a rooftop bar opening in SinCase observed-2026-09-02-040, corpus 2026-09-02, read from backend/evals/.cache/llm.
The three walkthroughs
Every verdict below was computed, not written- Accepted for reviewA candidate that passes the stated checksEvery verdict names its article. The association survives all 720 orderings of the same response.keyed-v2
- RejectedA defective candidate the checks rejectA seeded control that keeps the last verdict when an article is judged twice. Rejected on a case with a named counterexample.Seeded control
- Accepted for reviewAn interrupted execution that resumesThe orchestrator was SIGKILLed with an attempt in flight. Recovery resolved it as unknown-outcome and finished the work.keyed-v2
What the experiment asks
spec f027762ab4d08b35Does a candidate parser associate every returned verdict with the article it was actually about, and refuse when that association cannot be recovered?
What it measures6
- association correctness on cases whose ground truth is known by construction
- refusal on responses that are truncated, malformed, or miscounted
- refusal on duplicate, unknown and missing article ids
- refusal on non-finite and out-of-range scores
- termination without crash or hang
- refusal of inputs outside the protocol the candidate declares
What it does not measure3
- relevance quality — no keyed recordings exist, so no protocol-v2 feed has ever been built
- latency and cost — reported separately and never traded against correctness
- generalisation — the cases are public and a candidate may be written against them
Why the criteria are hashed2
- The criteria were written down and hashed before any candidate ran.
- The hash travels with every verdict, so moving a threshold to get a green result changes the hash and invalidates the comparison.
- Cases
- 6442 recorded, 22 fault-injected
- Runs
- 83 versions, 3 seeded controls
- Accepted
- 2for human review only
- Rejected
- 6by independent checks
Every run in the set
Including the ones that are not walkthroughsWhat the Lab never claims
The boundaries of this resultAccepted is not shipped
It means eligible for human review under this spec hash. Merging and promotion stay a human decision.
The cases are public
A candidate may have been written against them, so passing does not establish generalisation. Fixture performance is reported as fixture performance.
Relevance quality is unmeasured
Sending article ids changes the request, which invalidates every recorded response. New budgeted recordings would be needed and none exist.
No live execution here
Every run replayed committed recordings offline. No inference call was made; provider spend for these runs is $0.
No agent has run
The investigator’s tools, budget and scope gate are implemented and tested, and no model has been called — there is no gateway key on this deployment. No agent behaviour is depicted anywhere on this site.
Egress is an upper bound, not a measurement
1 of 8 runs executed in an isolated microVM. The metered egress on those runs includes the bytes spent reading the record bundle back, so it is non-zero on a run that reached nothing. The negative controls are the direct evidence.