Daily

Daily Lab · recorded replay · run 008af826 · clean

control-zero-filling

Rejected

universal-refusal: 0/48 cases satisfied, threshold 1Under the current criteria, generation 2. This run faced generation 1 when it executed — both verdicts are below.

This is a seeded control. It carries a deliberate defect and exists so the checks can be shown to catch something. SEEDED DEFECT: turns every failure, including a missing recording, into a confident all-zero answer. This is the blanket except Exception on the production path.

01

Verdict

2 criteria generations

Criteria generation

spec f027762ab4d08b35

Rejected

universal-refusal: 0/48 cases satisfied, threshold 1

Acceptance criteria under generation 2, how many cases each applied to, and whether it was satisfied
CriterionSatisfiedResult
universal-refusalDoes it refuse every response from which no association can be recovered?0/48not met
association-exactOn its own protocol, does every article receive exactly the verdict it was given?0/3not met
protocol-violation-refusalDoes it refuse duplicate, unknown, missing ids and unusable scores?0/1not met
no-crashDoes it terminate on every case without crashing or hanging?52/52met
complete-evidenceIs there a prediction record for every applicable case?52/52met
protocol-exclusivityOn cases outside its declared protocol, does it refuse rather than associate anyway?0/12not met

Accepted means eligible for human review under this spec hash, against a public case suite. It is not evidence of production quality, and it does not establish generalisation: the cases are visible and a candidate may have been written against them.

03

The patch

against positional_v0.py
Unified diff · 89 lines · applies with git apply
diff --git a/backend/lab/contract/versions/positional_v0.py b/backend/lab/contract/controls/zero_filling.py--- a/backend/lab/contract/versions/positional_v0.py+++ b/backend/lab/contract/controls/zero_filling.py@@ -1,18 +1,12 @@-"""Historical behaviour, transcribed from `origin/main`.+"""CONTROL — turns every failure into a confident all-zero answer. -Source: backend/app/services/openai_service.py, score_articles_batch, the-`normalized` loop. Verbatim semantics:+Seeded defect: reproduces the blanket `except Exception` on the production+path, where `CacheMiss` (a RuntimeError) becomes a full set of 0.0 verdicts and+the run reports zero cache misses. An offline replay looks clean while having+scored nothing. -    if len(results_list) != len(articles):-        logger.warning("... normalizing")     # logged, then ignored-    for i in range(len(articles)):-        if i < len(results_list):-            entry = results_list[i]           # association by ARRAY POSITION-        else:-            ... {"relevant": False, "score": 0.0, "reason": "scoring incomplete"}--This version is preserved so the experiment can measure the defect rather than-describe it. It is not a control: it is what production does today.+Expected outcome: REJECTED on the no-recording and truncation cases, because+an unavailable answer must surface as incomplete and never as a verdict. """  from __future__ import annotations@@ -54,45 +48,23 @@ import json from typing import Any  -VERSION_ID = "positional-v0"+VERSION_ID = "control:zero-filling" PROTOCOL = "positional-v0"  -def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:-    if response.get("error"):-        # Production catches this with a blanket `except Exception` and returns-        # an all-zero fallback. Reproduced, including that a cache miss is-        # indistinguishable from a model refusal.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])+def _zeros(articles: list[dict[str, Any]]) -> dict[str, Any]:+    return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles]) -    content = response.get("content")-    if content is None:-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles]) +def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:     try:-        result = json.loads(content)+        if response.get("error"):+            return _zeros(articles)+        result = json.loads(response.get("content") or "")+        entries = result["results"]+        return ok([+            verdict(a["id"], bool(e.get("relevant")), float(e.get("score", 0.0)), str(e.get("reason", "")))+            for a, e in zip(articles, entries)+        ])     except Exception:-        # No finish_reason check: a truncated completion is indistinguishable-        # from a malformed one, and both become the all-zero fallback.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])--    results_list = result.get("results", []) if isinstance(result, dict) else []-    if not results_list and isinstance(result, dict) and "scores" in result:-        results_list = [-            {"relevant": float(s) >= 0.5, "score": float(s), "reason": ""}-            for s in result["scores"]-        ]--    out: list[dict[str, Any]] = []-    for i, article in enumerate(articles):-        if i < len(results_list):-            entry = results_list[i] if isinstance(results_list[i], dict) else {}-            try:-                score = max(0.0, min(1.0, float(entry.get("score", 0.5))))-            except Exception:-                score = 0.5-            relevant = bool(entry.get("relevant", score >= 0.5))-            out.append(verdict(article["id"], relevant, score, str(entry.get("reason", ""))))-        else:-            out.append(verdict(article["id"], False, 0.0, "scoring incomplete"))-    return ok(out)+        return _zeros(articles) 

Reproduce this run

cd backend
EVAL_OFFLINE=1 venv/bin/python -m lab.orchestrate \
  --candidate control-zero-filling --tag clean
cd ../web && npm run export:lab -- --check

Source under test

backend/lab/contract/controls/zero_filling.pysha256 32afb5fd6da82f90… · 2449 bytestranscribed from unknown

04

Timeline

1 attempt
  1. 01succeededcompleted 64 cases in 30.9mslocal-known · started 2026-09-22T19:47:57.813Z · ended 2026-09-22T19:47:57.845Z

Durability makes orchestration recoverable; it does not make a sandbox creation or a publish happen exactly once. An attempt the orchestrator never saw finish is recorded as unknown-outcome rather than assumed to have failed.

05

Cases

52 scored, 12 not applicable
Recorded cases
42real batches, replayed
Fault-injected
10labelled synthetic
Correct
0of the scored cases
Wrong
52see the table
Every case this candidate got wrong
CaseGroupWhat happenedWhy it is wrong
observed-2026-09-02-000observedshould-have-refused27 verdicts recorded for 40 articles
observed-2026-09-02-001observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-002observedshould-have-refused18 verdicts recorded for 20 articles
observed-2026-09-02-003observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-004observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-005observedwrong-associationno ground truth recorded
observed-2026-09-02-006observedshould-have-refused201 verdicts recorded for 40 articles
observed-2026-09-02-007observedshould-have-refused33 verdicts recorded for 40 articles
observed-2026-09-02-008observedshould-have-refused37 verdicts recorded for 40 articles
observed-2026-09-02-009observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-010observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-011observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-012observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-013observedshould-have-refused25 verdicts recorded for 40 articles
observed-2026-09-02-014observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-015observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-016observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-017observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-018observedshould-have-refused59 verdicts recorded for 40 articles
observed-2026-09-02-019observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-020observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-021observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-022observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-023observedshould-have-refused36 verdicts recorded for 40 articles
observed-2026-09-02-024observedshould-have-refused21 verdicts recorded for 20 articles
observed-2026-09-02-025observedshould-have-refused31 verdicts recorded for 40 articles
observed-2026-09-02-026observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-027observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-028observedshould-have-refused38 verdicts recorded for 40 articles
observed-2026-09-02-029observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-030observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-031observedshould-have-refused31 verdicts recorded for 40 articles
observed-2026-09-02-032observedshould-have-refused53 verdicts recorded for 40 articles
observed-2026-09-02-033observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-034observedshould-have-refused34 verdicts recorded for 40 articles
observed-2026-09-02-035observedwrong-associationno ground truth recorded
observed-2026-09-02-036observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-037observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-038observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-039observedshould-have-refused25 verdicts recorded for 40 articles

Showing the first 40 of 52. The full set is in the artifact below.

06

Unscored

Measured, and deliberately not graded

A criterion decides; a diagnostic reports. Promoting one of these to a criterion would change the spec hash and re-decide runs that never faced it, so a gap found after the fact is published as a number rather than closed behind your back.

12/12

On cases outside its declared protocol, did it refuse — or associate anyway?

It produced a complete association on 12 case(s) outside its declared protocol. Under this generation that fails protocol-exclusivity; under generation 1 it was not graded at all.

syn-keyed-in-order, syn-keyed-reversed, syn-keyed-rotated, syn-duplicate-id, syn-unknown-id, syn-missing-id, syn-score-nan, syn-score-out-of-range, syn-score-string, syn-score-null, syn-relevant-not-bool, syn-empty-results

07

Provenance

What can and cannot be established
Executed at revision
4e8bee7625820107e84184f8e24c6bd7125f2e86+dirtyrecorded when the harness ran, not re-derived at export
Inputs sha256
01df3df6f1fa00e3cases, records, candidates and event logs
Evaluator sha256
277ec81521ba14d8
Spec hash
f027762ab4d08b35
Execution mode
offline-replayThe harness reads committed responses from disk and makes no network call. The candidate imports nothing beyond the standard library.
Python
3.12.13
Model calls
0
Spend for this run
$0Offline replay of committed recordings: no inference call was made, so provider spend for this run is $0. What the original recordings cost is not attributed per batch anywhere in this repository, so it is left unknown rather than estimated.
Recording cost
unknown
Sandbox limits
python3.13, network disabled120s wall clock, none secrets. This candidate matched a committed implementation, so it ran locally and the boundary was not exercised here.

Case suites

backend/lab/cases/observed.json42 cases · sha256 3d7f4b4143d85ab3backend/evals/.cache/llm via offline replay

backend/lab/cases/synthetic.json22 cases · sha256 21c92da332a9837elab/build_synthetic.py — fault injection, ground truth by construction

  • Warning. This is a seeded control with a deliberate defect. SEEDED DEFECT: turns every failure, including a missing recording, into a confident all-zero answer. This is the blanket except Exception on the production path.backend/lab/contract/controls/zero_filling.py
  • Note. Every case ran offline against responses already committed to this repository. No inference call was made and no provider was charged.backend/evals/.cache/llm
  • Caution. The case suite is public. A candidate may have been written against it, so passing does not establish generalisation.backend/lab/cases/
The full artifact, as published

Validated against the schema in web/lib/lab/artifact.ts before it was written. Download the JSON.