RECIPE / VERIFIED RETRY

Retry until the tests pass, or stop early.

An agent loop that retries until its tests pass needs a cap, but a fixed cap spends the same budget on a nearly finished task as on one that cannot succeed. After each failed test run, this recipe asks two questions: how likely one more attempt is to pass, and which strategy to use next.

Only a passing test run marks the work as done. Decision can end the loop early by stopping it. It never ends it by declaring success.

The loop

Run an attempt with the current strategy.
Run the tests. If they all pass, stop: the work is done.
If the attempt budget is spent, stop and hand the last failure to a person.
Otherwise ask Decision for a score (1–5, likelihood of passing next time) and a choice (same approach, change approach, or stop).
Stop when the choice is stop, when the expected score is below your threshold, or when the model abstains. Otherwise continue with the chosen strategy.

The request

Send the attempt number, the budget, the latest test output, and a short history of earlier failures. The same error repeating across attempts is the strongest sign that the loop should stop. Remove secrets from test output before sending it.

{
  "model": "decider-4b-v2",
  "state": {
    "attempt": 2,
    "max_attempts": 5,
    "tests": "1 passed, 2 failed",
    "test_output": "FAILED test_retry.py::test_backoff - AssertionError: backoff not applied",
    "previous_failures": ["AssertionError: retry count 0 != 3"]
  },
  "questions": {
    "next_attempt": {
      "type": "score",
      "instructions": "How likely is one more attempt to make all tests pass?",
      "criteria": {
        "1": "Very unlikely: the same failure will repeat",
        "2": "Unlikely: nothing new was learned from the last attempt",
        "3": "Even odds",
        "4": "Likely: the failures are narrowing",
        "5": "Very likely: one small fix remains"
      }
    },
    "strategy": {
      "type": "choice",
      "instructions": "Choose the strategy for the next step.",
      "criteria": {
        "same_approach": "Keep the current approach and fix the reported failure",
        "change_approach": "The current approach is stuck; try a different one",
        "stop": "More attempts cannot fix this; hand it to a person"
      }
    }
  }
}

A complete loop in Python

This uses only the standard library and pytest. run_agent_attempt stands in for your own agent.

import json, os, subprocess, urllib.request, uuid

ENDPOINT = "https://api.verdictdecision.com/v1/decisions"
MAX_ATTEMPTS = 5
QUESTIONS = {
    "next_attempt": {
        "type": "score",
        "instructions": "How likely is one more attempt to make all tests pass?",
        "criteria": {
            "1": "Very unlikely: the same failure will repeat",
            "2": "Unlikely: nothing new was learned from the last attempt",
            "3": "Even odds",
            "4": "Likely: the failures are narrowing",
            "5": "Very likely: one small fix remains",
        },
    },
    "strategy": {
        "type": "choice",
        "instructions": "Choose the strategy for the next step.",
        "criteria": {
            "same_approach": "Keep the current approach and fix the reported failure",
            "change_approach": "The current approach is stuck; try a different one",
            "stop": "More attempts cannot fix this; hand it to a person",
        },
    },
}

def decide(questions, state):
    body = {"model": "decider-4b-v2", "state": state, "questions": questions}
    request = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(body).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["VERDICT_DECISION_KEY"],
            "Content-Type": "application/json",
            "Idempotency-Key": str(uuid.uuid4()),
        },
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)["answers"]

def run_tests():
    proc = subprocess.run(["pytest", "-q", "--tb=line"], capture_output=True, text=True)
    return proc.returncode == 0, proc.stdout[-4000:]

history, strategy = [], "same_approach"
for attempt in range(1, MAX_ATTEMPTS + 1):
    run_agent_attempt(strategy, history)   # your agent; not part of this API
    passed, output = run_tests()
    if passed:
        print(f"Done on attempt {attempt}: all tests pass.")
        break
    if attempt == MAX_ATTEMPTS:
        print("Attempt budget spent. Hand the last failure to a person.")
        break
    state = {
        "attempt": attempt,
        "max_attempts": MAX_ATTEMPTS,
        "test_output": output,
        "previous_failures": history[-3:],
    }
    history.append(output[-800:])
    try:
        answers = decide(QUESTIONS, state)
    except Exception:
        continue                           # Decision unavailable: keep the fixed cap
    score, choice = answers["next_attempt"], answers["strategy"]
    if score["abstained"] or choice["abstained"]:
        print("Decision abstained. Ask a person.")
        break
    if choice["decision"] == "stop" or score["expected_score"] < 2.5:
        print(f"Stopped after attempt {attempt}: expected {score['expected_score']:.1f} of 5.")
        break
    strategy = choice["decision"]

Design notes

Tests are the authority. The loop ends as done only when the tests pass. A high score never replaces a test run.
Decision failures do not stop the loop. If the call fails or times out, the loop falls back to the fixed cap. A cold worker can take longer than your deadline, so keep the loop running while Decision is unavailable.
Tune the threshold on your own runs. The 2.5 cutoff is a starting point. Record each score next to whether the following attempt passed, and adjust.
Use a new Idempotency-Key for each new question. Each attempt sends a different state, so it is a new request. Reuse a key only for a manual retry of the same body.

This recipe keeps a fixed attempt cap and adds the ability to stop early on a task that cannot succeed.

Retry triage recipe · API docs · Playground