Daily Lab · recorded replay · run agent-k2-generous-out-06 · sandbox

agent-k2-generous-out-06

Accepted for review

all 6 criteria satisfied over 60 applicable casesUnder the current criteria, generation 2. This run faced generation 1 when it executed. Both verdicts are below.

The defect is that positional association assumes results[i] corresponds to articles[i], which fails when the model returns results in a different order or omits some results. The fix is to use keyed association where each verdict explicitly contains the article_id it applies to. This allows us to verify that every article has exactly one verdict, detect duplicates, unknown IDs, and missing IDs. The parser expects each result entry to have an 'article_id' field and uses PROTOCOL_KEYED to signal this requirement.

01

Verdict

2 criteria generations

Criteria generation

spec f027762ab4d08b35

Accepted for review

all 6 criteria satisfied over 60 applicable cases

Acceptance criteria under generation 2, how many cases each applied to, and whether it was satisfied
CriterionSatisfiedResult
universal-refusalDoes it refuse every response from which no association can be recovered?48/48met
association-exactOn its own protocol, does every article receive exactly the verdict it was given?3/3met
protocol-violation-refusalDoes it refuse duplicate, unknown, missing ids and unusable scores?9/9met
no-crashDoes it terminate on every case without crashing or hanging?60/60met
complete-evidenceIs there a prediction record for every applicable case?60/60met
protocol-exclusivityOn cases outside its declared protocol, does it refuse rather than associate anyway?4/4met

Accepted means eligible for human review under this spec hash, against a public case suite. It is not evidence of production quality, and it does not establish generalisation: the cases are visible and a candidate may have been written against them.

03

The patch

against positional_v0.py
Unified diff · 180 lines · applies with git apply
diff --git a/backend/lab/contract/versions/positional_v0.py b/backend/lab/contract/candidates/agent_k2_generous_out_06.py--- a/backend/lab/contract/versions/positional_v0.py+++ b/backend/lab/contract/candidates/agent_k2_generous_out_06.py@@ -1,30 +1,13 @@-"""Historical behaviour, transcribed from `origin/main`.+"""Keyed association parser - fixes the positional association defect. -Source: backend/app/services/openai_service.py, score_articles_batch, the-`normalized` loop. Verbatim semantics:--    if len(results_list) != len(articles):-        logger.warning("... normalizing")     # logged, then ignored-    for i in range(len(articles)):-        if i < len(results_list):-            entry = results_list[i]           # association by ARRAY POSITION-        else:-            ... {"relevant": False, "score": 0.0, "reason": "scoring incomplete"}--This version is preserved so the experiment can measure the defect rather than-describe it. It is not a control: it is what production does today.+Each verdict explicitly identifies which article it applies to via article_id,+allowing unambiguous association and detection of duplicates, unknown IDs,+and missing IDs. """  from __future__ import annotations  # --- contract prelude --------------------------------------------------------# Inlined rather than imported. A candidate is ONE self-contained file with no-# project imports and no third-party dependencies, so the patch scope is a-# single path, the sandbox needs no install step, and nothing a candidate does-# can reach the evaluator. The canonical definitions live in-# lab/contract/types.py, and tests/test_contract_prelude.py asserts that every-# copy still agrees with it.- import math as _math  @@ -54,45 +37,108 @@ import json from typing import Any  -VERSION_ID = "positional-v0"-PROTOCOL = "positional-v0"+VERSION_ID = "keyed-v2"+PROTOCOL = "keyed-v2"   def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:-    if response.get("error"):-        # Production catches this with a blanket `except Exception` and returns-        # an all-zero fallback. Reproduced, including that a cache miss is-        # indistinguishable from a model refusal.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])-+    """Parse response using keyed association.+    +    Each result entry must contain an 'article_id' field that explicitly+    identifies which article the verdict applies to. This allows verification+    of complete, unambiguous association.+    """+    # Handle error responses+    error = response.get("error")+    if error:+        if error in {"no_recording", "budget_exceeded"}:+            return refuse("no_recording" if error == "no_recording" else "retries_exhausted", error)+        if error in {"timeout", "cancelled"}:+            return refuse(error, error)+        return refuse("retries_exhausted", str(error))++    # Get content     content = response.get("content")     if content is None:-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])+        return refuse("retries_exhausted", "no content returned") +    # Parse JSON     try:         result = json.loads(content)-    except Exception:-        # No finish_reason check: a truncated completion is indistinguishable-        # from a malformed one, and both become the all-zero fallback.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])--    results_list = result.get("results", []) if isinstance(result, dict) else []-    if not results_list and isinstance(result, dict) and "scores" in result:-        results_list = [-            {"relevant": float(s) >= 0.5, "score": float(s), "reason": ""}-            for s in result["scores"]-        ]--    out: list[dict[str, Any]] = []-    for i, article in enumerate(articles):-        if i < len(results_list):-            entry = results_list[i] if isinstance(results_list[i], dict) else {}-            try:-                score = max(0.0, min(1.0, float(entry.get("score", 0.5))))-            except Exception:-                score = 0.5-            relevant = bool(entry.get("relevant", score >= 0.5))-            out.append(verdict(article["id"], relevant, score, str(entry.get("reason", ""))))+    except Exception as exc:+        return refuse("malformed_json", str(exc))++    # Validate top-level structure+    if not isinstance(result, dict):+        return refuse("unexpected_shape", f"top level is {type(result).__name__}")++    # Extract results list+    results_list = result.get("results", [])+    if not isinstance(results_list, list):+        return refuse("unexpected_shape", "results is not a list")++    # Build set of expected article IDs+    expected_ids = {article["id"] for article in articles}+    +    # Process results and validate associations+    verdicts_by_id: dict[str, dict[str, Any]] = {}+    +    for i, entry in enumerate(results_list):+        if not isinstance(entry, dict):+            return refuse("invalid_type", f"entry {i} is {type(entry).__name__}")+        +        # Get article_id from entry+        article_id = entry.get("article_id")+        if article_id is None:+            return refuse("missing_id", f"entry {i} missing article_id")+        +        # Check for duplicates+        if article_id in verdicts_by_id:+            return refuse("duplicate_id", f"duplicate article_id: {article_id}")+        +        # Check for unknown article_id+        if article_id not in expected_ids:+            return refuse("unknown_id", f"unknown article_id: {article_id}")+        +        # Validate and extract score+        score_raw = entry.get("score")+        if score_raw is None:+            return refuse("invalid_type", f"entry for {article_id} missing score")+        +        score = finite_unit_score(score_raw)+        if score is None:+            return refuse("score_out_of_range", f"entry for {article_id} has invalid score: {score_raw}")+        +        # Extract relevant (default based on score if not provided)+        relevant_raw = entry.get("relevant")+        if relevant_raw is None:+            relevant = score >= 0.5         else:-            out.append(verdict(article["id"], False, 0.0, "scoring incomplete"))+            if not isinstance(relevant_raw, bool):+                return refuse("invalid_type", f"entry for {article_id} has non-boolean relevant")+            relevant = relevant_raw+        +        # Extract reason+        reason = entry.get("reason", "")+        if not isinstance(reason, str):+            reason = str(reason)+        +        verdicts_by_id[article_id] = verdict(article_id, relevant, score, reason)+    +    # Check for missing article IDs+    received_ids = set(verdicts_by_id.keys())+    missing_ids = expected_ids - received_ids+    if missing_ids:+        return refuse("missing_id", f"missing verdicts for articles: {sorted(missing_ids)}")+    +    # Check for count mismatch (shouldn't happen if we got here, but safety check)+    if len(verdicts_by_id) != len(articles):+        return refuse("count_mismatch", f"{len(verdicts_by_id)} results for {len(articles)} articles")+    +    # Build ordered list matching input article order+    out: list[dict[str, Any]] = []+    for article in articles:+        article_id = article["id"]+        out.append(verdicts_by_id[article_id])+         return ok(out) 

Reproduce this run

cd backend
EVAL_OFFLINE=1 venv/bin/python -m lab.orchestrate \
  --candidate agent-k2-generous-out-06 --tag clean
cd ../web && npm run export:lab -- --check

Source under test

backend/lab/contract/candidates/agent_k2_generous_out_06.pysha256 94b1a5983d44ee85… · 5119 bytestranscribed from unknown

04

Timeline

1 attempt
  1. 01succeededsandbox yellow-minimum-badger-Vb6BFb in iad1, exit 0 in 411msvercel-sandbox · started 2026-09-30T04:05:54.778Z · ended 2026-09-30T04:06:00.896Z

Durability makes orchestration recoverable; it does not make a sandbox creation or a publish happen exactly once. An attempt the orchestrator never saw finish is recorded as unknown-outcome rather than assumed to have failed.

05

Cases

60 scored, 4 not applicable
Recorded cases
39real batches, replayed
Fault-injected
21labelled synthetic
Correct
60of the scored cases
Wrong
0see the table

Every scored case was correct. That is what the verdict above is asserting, and nothing more.

06

Unscored

Measured, and deliberately not graded

A criterion decides; a diagnostic reports. Promoting one of these to a criterion would change the spec hash and re-decide runs that never faced it, so a gap found after the fact is published as a number rather than closed behind your back.

0/4

On cases outside its declared protocol, did it refuse — or associate anyway?

It produced no association on any case outside its declared protocol.

07

Provenance

What can and cannot be established
Executed at revision
1c97bc0483f43673fc9f31820d8187d53659a0ff+dirtyrecorded when the harness ran, not re-derived at export
Inputs sha256
080d784000c4dd4ecases, records, candidates and event logs
Evaluator sha256
277ec81521ba14d8
Spec hash
f027762ab4d08b35
Execution mode
offline-replayThe harness reads committed responses from disk and makes no network call. The candidate imports nothing beyond the standard library.
Python
3.13.1
Model calls
10
Spend for this run
$0Grading replayed committed recordings and made no inference call; `replay_spend_usd` is that, and is $0. The investigation that authored the candidate did call a model: 10 call(s) to moonshotai/kimi-k2 through the AI Gateway, 189448 tokens, $0.01784616 as the gateway reported it per call. That is in `provider_reported`. The microVM is metered compute billed at a rate this repository does not record: 1935ms of active CPU across 715ms wall clock.
Recording cost
unknown
Sandbox
yellow-minimum-badger-Vb6BFbpython3.13 in iad1, booted in 304ms
Network policy applied
deny-allread back off the microVM, not the value that was requested

Case suites

backend/lab/cases/observed.json42 cases · sha256 3d7f4b4143d85ab3…backend/evals/.cache/llm via offline replay

backend/lab/cases/synthetic.json22 cases · sha256 21c92da332a9837e…lab/build_synthetic.py — fault injection, ground truth by construction

  • Note. Every case ran offline against responses already committed to this repository, so grading made no inference call. The investigation that wrote the candidate did, and its spend is recorded under usage.backend/lab/runs/agent-k2-generous-out-06-sandbox.trace.json
  • Caution. The case suite is public. A candidate may have been written against it, so passing does not establish generalisation.backend/lab/cases/
The full artifact, as published

Validated against the schema in web/lib/lab/artifact.ts before it was written. Download the JSON.