Daily Lab

Test a scoring parser.

Run a parser for Daily's scoring responses against 64 cases in Vercel Sandbox. Inspect the failure, then edit it.

candidate.py4,140 / 65,536 bytes

"""The count-mismatch guard, transcribed from PR #59 (head 81b2019).

Source: backend/app/services/openai_service.py on
`mmarufov/batch-scoring-alignment`:

if len(results_list) != len(articles):
        logger.warning("... discarding the batch rather than assigning "
                       "verdicts by position")
        continue                              # retry, then report unscored

That commit also stops swallowing the harness's stop signals:

except Exception as exc:
        if type(exc).__name__ in {"CacheMiss", "BudgetExceeded"}:
            raise

Both are reproduced. The guard is a real improvement over positional-v0 and
this experiment is expected to show that — and also to show its ceiling: it
compares *lengths*, so an equal-length reordered response still passes through
and is still associated by position.
"""

from __future__ import annotations

# --- contract prelude -------------------------------------------------------
# Inlined rather than imported. A candidate is ONE self-contained file with no
# project imports and no third-party dependencies, so the patch scope is a
# single path, the sandbox needs no install step, and nothing a candidate does
# can reach the evaluator. The canonical definitions live in
# lab/contract/types.py, and tests/test_contract_prelude.py asserts that every
# copy still agrees with it.

import math as _math

def ok(verdicts):
    return {"ok": True, "verdicts": verdicts}

def refuse(kind, detail=""):
    return {"ok": False, "refusal": {"kind": kind, "detail": str(detail)[:400]}}

def verdict(article_id, relevant, score, reason):
    return {"article_id": article_id, "relevant": bool(relevant),
            "score": float(score), "reason": str(reason)[:500]}

def finite_unit_score(raw):
    if isinstance(raw, bool) or not isinstance(raw, (int, float)):
        return None
    value = float(raw)
    if not _math.isfinite(value) or value < 0.0 or value > 1.0:
        return None
    return value
# --- end prelude ------------------------------------------------------------

import json
from typing import Any

VERSION_ID = "count-guard-v1"
PROTOCOL = "positional-v0"

def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:
    error = response.get("error")
    if error:
        # CacheMiss / BudgetExceeded propagate instead of degrading to zeros.
        if error in {"no_recording", "budget_exceeded"}:
            return refuse("no_recording" if error == "no_recording" else "retries_exhausted", error)
        if error in {"timeout", "cancelled"}:
            return refuse(error, error)
        return refuse("retries_exhausted", str(error))

content = response.get("content")
    if content is None:
        return refuse("retries_exhausted", "no content returned")

try:
        result = json.loads(content)
    except Exception as exc:
        return refuse("malformed_json", str(exc))

if not isinstance(result, dict):
        return refuse("unexpected_shape", f"top level is {type(result).__name__}")

results_list = result.get("results", [])
    if not results_list and "scores" in result:
        results_list = [
            {"relevant": float(s) >= 0.5, "score": float(s), "reason": ""}
            for s in result["scores"]
        ]
    if not isinstance(results_list, list):
        return refuse("unexpected_shape", "results is not a list")

if len(results_list) != len(articles):
        return refuse(
            "count_mismatch",
            f"{len(results_list)} results for {len(articles)} articles",
        )

out: list[dict[str, Any]] = []
    for article, entry in zip(articles, results_list):
        if not isinstance(entry, dict):
            return refuse("invalid_type", f"entry is {type(entry).__name__}")
        try:
            score = max(0.0, min(1.0, float(entry.get("score", 0.5))))
        except Exception:
            return refuse("invalid_type", "score is not numeric")
        relevant = bool(entry.get("relevant", score >= 0.5))
        out.append(verdict(article["id"], relevant, score, str(entry.get("reason", ""))))
    return ok(out)

Python · Tab indents. Press Esc, then Tab, to leave the editor.

Limits: 5 runs/hour per address, 3 at once, and 50 runs or 20 minutes of microVM CPU/day across all visitors. Live runs have a separate inventory.

How a run works

The scope gate checks that only candidate.py is written.
A fresh microVM boots with networking denied and no credentials. It receives the harness, the 64 responses with their answers stripped out, and your file.
The harness calls your parse() on every case and records what it returns.
Four probes check that DNS and HTTPS fail and that neither the evaluator nor a credential is present.
The records are graded outside the microVM by the independent evaluator.

The case suite

Ready to test · 64 cases

42 recorded · 22 fault-injected

Inspect cases

Recorded · 42real batches, replayed

observed-2026-09-02-000: not run yet
observed-2026-09-02-001: not run yet
observed-2026-09-02-002: not run yet
observed-2026-09-02-003: not run yet
observed-2026-09-02-004: not run yet
observed-2026-09-02-005: not run yet
observed-2026-09-02-006: not run yet
observed-2026-09-02-007: not run yet
observed-2026-09-02-008: not run yet
observed-2026-09-02-009: not run yet
observed-2026-09-02-010: not run yet
observed-2026-09-02-011: not run yet
observed-2026-09-02-012: not run yet
observed-2026-09-02-013: not run yet
observed-2026-09-02-014: not run yet
observed-2026-09-02-015: not run yet
observed-2026-09-02-016: not run yet
observed-2026-09-02-017: not run yet
observed-2026-09-02-018: not run yet
observed-2026-09-02-019: not run yet
observed-2026-09-02-020: not run yet
observed-2026-09-02-021: not run yet
observed-2026-09-02-022: not run yet
observed-2026-09-02-023: not run yet
observed-2026-09-02-024: not run yet
observed-2026-09-02-025: not run yet
observed-2026-09-02-026: not run yet
observed-2026-09-02-027: not run yet
observed-2026-09-02-028: not run yet
observed-2026-09-02-029: not run yet
observed-2026-09-02-030: not run yet
observed-2026-09-02-031: not run yet
observed-2026-09-02-032: not run yet
observed-2026-09-02-033: not run yet
observed-2026-09-02-034: not run yet
observed-2026-09-02-035: not run yet
observed-2026-09-02-036: not run yet
observed-2026-09-02-037: not run yet
observed-2026-09-02-038: not run yet
observed-2026-09-02-039: not run yet
observed-2026-09-02-040: not run yet
observed-2026-09-02-041: not run yet

Fault-injected · 22each built to catch one mistake

syn-keyed-in-order: not run yet
syn-keyed-reversed: not run yet
syn-keyed-rotated: not run yet
syn-duplicate-id: not run yet
syn-unknown-id: not run yet
syn-missing-id: not run yet
syn-score-nan: not run yet
syn-score-out-of-range: not run yet
syn-score-string: not run yet
syn-score-null: not run yet
syn-relevant-not-bool: not run yet
syn-malformed-json: not run yet
syn-truncated-finish-reason: not run yet
syn-not-an-object: not run yet
syn-empty-results: not run yet
syn-exec-no-recording: not run yet
syn-exec-timeout: not run yet
syn-exec-cancelled: not run yet
syn-exec-budget: not run yet
syn-positional-short: not run yet
syn-positional-long: not run yet
syn-positional-reordered: not run yet

Recorded examples

Recorded response

Articles sent: 40one batch
Verdicts returned: 254the model kept going
finish_reason: stopnot even truncated
Article ids sent: 0position is the only link

What was sent · first 6 of 40

00Late-stage deals revive in Southeast Asia, but early-stage founders remain under pressuree27
01NASA-linked, MIT-trained founders’ nSWX raises US$2M for AI chip packaginge27
02Cloud Nine: Singapore's newest rooftop bar opens with infrastructure-themed cocktailsStraits Times
03Bringing tech and people together for Singapore’s AI futureBusiness Times SG
04The creator economy is distribution, not marketing. Most Asian businesses are still scaling it like a campaigne27
05Economists raise Singapore’s growth forecast, lower inflation projections in latest MAS surveyBusiness Times SG

What came back · first 6 of 254

00“This article discusses the current state of Southeast Asia's venture capital market, which is highly relevant for a senior backend engineer interested in the startup scene.”
01“The article covers a Malaysian startup focused on AI chip packaging, which intersects with AI engineering and cloud infrastructure interests.”
02“This article is about a rooftop bar opening in Singapore, which does not align with the user's interests.”
03“The article discusses the creator economy, which is not a primary focus for the user.”
04“This article focuses on economic forecasts and inflation, which do not align with the user's technical interests.”
05“This article discusses new capital rules for payment firms in Singapore, directly related to fintech regulation.”

Recorded during an offline replay of the production runner. No article identifier appears anywhere in the request or the response: position is the only thing associating a verdict with an article.

The recorded response · 44,411 bytes, first 600 shown

{
  "results": [
    {
      "relevant": true,
      "score": 0.8,
      "reason": "This article discusses the current state of Southeast Asia's venture capital market, which is highly relevant for a senior backend engineer interested in the startup scene."
    },
    {
      "relevant": true,
      "score": 0.7,
      "reason": "The article covers a Malaysian startup focused on AI chip packaging, which intersects with AI engineering and cloud infrastructure interests."
    },
    {
      "relevant": false,
      "score": 0.0,
      "reason": "This article is about a rooftop bar opening in Sin

Case observed-2026-09-02-040, corpus 2026-09-02, read from backend/evals/.cache/llm.

Grading criteria

Does a candidate parser associate every returned verdict with the article it was actually about, and refuse when that association cannot be recovered?

What it measures

association correctness on cases whose ground truth is known by construction
refusal on responses that are truncated, malformed, or miscounted
refusal on duplicate, unknown and missing article ids
refusal on non-finite and out-of-range scores
termination without crash or hang
refusal of inputs outside the protocol the candidate declares

What it does not measure

relevance quality — no keyed recordings exist, so no protocol-v2 feed has ever been built
latency and cost — reported separately and never traded against correctness
generalisation — the cases are public and a candidate may be written against them

Criteria are versioned and hashed. Both generations retain their verdicts; the second closes a gap found in earlier runs. Acceptance qualifies the parser for human review under those criteria.

Results apply to these public fixtures. Candidates can target the cases. Grading uses recorded or injected responses and makes no model calls; agent-authored runs report candidate-writing calls separately.

Evaluating news quality with an ID-based scoring request needs new model recordings.

Published runs 39 records

Published records: 39Including records with no execution
Sandbox executions: 9Within the published inventory