StopLoss: Teaching Agents to Predict Failure and Stop Early

On SWE-bench Pro with GPT-5.6 Luna, StopLoss (the Qwen2.5-Coder-7B-Instruct ensemble) captures 50.7% of failing runs at about 0.7 times the full-execution cost per retained solve, while the three GPT-5.6 judges capture at most 29% at a cost per retained solve equal to or above full execution; StopLoss spends $0.15 on monitoring versus $2.03 to $15.36 for the judges. — **StopLoss stops more failing runs than LLM judges and spends far less on monitoring.** SWE-bench Pro tasks executed by GPT-5.6 Luna; StopLoss is the Qwen2.5-Coder-7B-Instruct ensemble. (a) It stops 35 of the 69 failing runs (50.7%), while no judge stops more than 20. (b) Its monitoring costs $0.15 across all 213 runs, versus $2.03–$15.36 for the judges.

StopLoss is a small monitor that learns from finished runs to tell when a coding agent is likely to fail, and stops the run early, at a fraction of what it costs to ask an LLM judge.

$0.00024

estimated cost per check of the primary ensemble, one to two orders of magnitude below every LLM judge we tested

99%

of successful runs kept on SWE-bench Verified with GPT-5.6 Luna, while cost per solve drops 4.5%

−48%

cost per retained solve on SWE-bench Pro with Claude Sonnet 4.6 (Qwen3-8B backbone, keeping 83% of successful runs)

4 / 4

settings where all three backbones cost less per retained solve than every judge

Abstract

Can early stopping pay for itself?

Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5% on Verified while preserving 99% of successful runs, and by up to 31% on the costlier Pro subset, reaching 48% with a Qwen3-8B backbone, and in all four settings its cost per retained solve is below that of every judge policy. Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.

01Method

Learn from outcomes, stop at a calibrated threshold

StopLoss needs only completed runs and their verifier outcomes. It watches a running agent from the outside, reads nothing but the trajectory text, and never changes the agent.

1Label

Outcomes label earlier windows

Every window of a failed run is labelled ABORT. In a successful run, windows at the last two tool steps are DELIVER and earlier ones CONTINUE. A window is the step count plus the three most recent action–observation pairs.

2Train

A small model scores one window

A 7–8B model (Qwen2.5-7B-Instruct, Qwen3-8B or Qwen2.5-Coder-7B-Instruct) is fine-tuned with LoRA. The failure score is the probability of ABORT among the three labels, read from the next-token logits: one forward pass, no generated text.

3Stop

Calibrate once, then stop early

A threshold τ is set on successful runs so that at most 5% of them would be stopped, then frozen. At tool steps 5, 8, 11, …, the run stops as soon as the ensemble's average score reaches τ.

StopLoss workflow. Offline, final outcomes label earlier windows, 7-8B models are trained with LoRA, and a threshold is calibrated on successful runs and frozen. Online, at tool steps 5, 8, 11, ..., each ensemble member reads the step count and the three most recent action-observation pairs in one forward pass; the run stops once the averaged failure score reaches the threshold, otherwise it continues to the next check. — **StopLoss turns final verifier outcomes into early-stopping decisions.** Offline, the outcome of each completed run labels its earlier windows, 7–8B models are adapted with LoRA, and the threshold τ is calibrated on successful runs and frozen. Online, each ensemble member reads the step count and the three most recent action–observation pairs in one forward pass; the run stops once the averaged failure score reaches τ, and the executor itself is unchanged.

02Results

Cheaper per solve than every tested judge, in all four settings

StopLoss is trained on SWE-rebench runs of GPT-family executors and evaluated on SWE-bench Verified and a three-repository SWE-bench Pro subset, with GPT-5.6 Luna and Claude Sonnet 4.6 as executors. Six LLM judges read the same windows, and every method pays for the checks it makes and for the successful runs it stops. Costs are estimated from frozen list prices for API calls and from GPU time for StopLoss.

Change in cost per retained solve

Relative to running every task to completion, monitoring included. Left of zero saves money.

SWE-bench Verified · GPT-5.6 Luna

SWE-bench Verified · Claude Sonnet 4.6

SWE-bench Pro · GPT-5.6 Luna

SWE-bench Pro · Claude Sonnet 4.6

StopLoss bars are seed ensembles of each backbone (5, 4 and 4 seeds). GPT-5.6 judges read GPT-5.6 Luna runs and Claude judges read Claude Sonnet 4.6 runs. Hover or focus a row for its monitoring cost and stopping outcomes.

Full results

Table 1 of the paper. Bold marks the best cost and stopping values within a setting.

	Cost		Prediction	Stopping outcomes
Method	ΔCPS (%) ↓	Monitoring ($) ↓	Tier AUC ↑	Success retained (%) ↑	Failure capture (%) ↑

ΔCPS · change in cost per retained solve versus running every task to completion, monitoring included; negative is a saving.

Monitoring · estimated total cost of the checks a method makes during the stopping runs.

Tier AUC · how well failure scores rank eventual failures above successes at similar spend; 0.5 is chance.

Success retained / failure capture · share of otherwise successful runs kept, and of otherwise failed runs stopped.

Training lifts failure prediction

Tier AUC of each untuned backbone and its trained ensemble (Table 2 of the paper).

SWE-bench Verified · GPT-5.6 Luna

SWE-bench Verified · Claude Sonnet 4.6

SWE-bench Pro · GPT-5.6 Luna

SWE-bench Pro · Claude Sonnet 4.6

0.5 is chance. The step-count baseline scores a window by its tool-step index alone, without reading the trajectory. The right column gives each trained ensemble's gain over its untuned backbone (training plus seed ensembling).

03Findings

What makes early stopping pay off

A signal from the first check

0.642 AUC at tool step 5

The primary Qwen2.5-7B-Instruct ensemble already separates failures from successes at the first check: an ordinary AUC (one observation per run reaching step 5) of 0.642 on Verified/Luna and 0.645 on Pro/Luna, against 0.494 and 0.450 for its untuned base.

Ranking well is not enough

+34.7% for the best ranker

On Verified/Sonnet, Claude Opus 5 has the highest tier AUC in the study (0.747), but its checks cost 44.9% of the execution budget while it avoids only 16.0%, so cost per solve rises by a third.

The break-even rule

g > m + r

Stopping lowers cost per solve only when the share of execution cost it avoids (g) exceeds monitoring cost as a share of full execution cost (m) plus the share of successful runs it stops (r). A small monitor keeps m small.

Thresholds do not always transfer

1.0% vs 27.8% false kills

On Luna runs, the primary StopLoss ensemble has a similar tier AUC on Verified (0.622) and Pro (0.614), yet its frozen threshold stops 1.0% and 27.8% of successful runs. One plausible contributor: successful Pro runs are longer (median 16 vs 13 tool steps), so they face more checks.

04Release

Models, data and code

Three trained monitors, the training windows, and code for trajectory collection, window building, training, scoring, the LLM-judge baselines and the evaluation.

Model · primary StopLoss-Qwen2.5-7B-Instruct LoRA adapter for Qwen2.5-7B-Instruct View on Hugging Face → Model StopLoss-Qwen3-8B LoRA adapter for Qwen3-8B View on Hugging Face → Model StopLoss-Qwen2.5-Coder-7B-Instruct LoRA adapter for Qwen2.5-Coder-7B-Instruct View on Hugging Face → Dataset StopLoss-Training-Data Training windows from SWE-rebench runs and the checkpoint-selection set View on Hugging Face →

Each adapter is seed 0 of its backbone; the main results of the paper use seed ensembles.

Score your own runs

Build windows from finished Harbor runs and score them with a released monitor. A run record is one JSON line with task, bench, worker, reward and the path of the run's trajectory.json.

Training and full reproduction in the README →

# install
git clone https://github.com/TIGER-AI-Lab/StopLoss.git && cd StopLoss
pip install -e ".[model]"

# windows at the checks k = 5, 8, 11, ...
python scripts/build_windows.py --runs my_runs.jsonl \
    --out windows.jsonl --ks checkpoints

# probability of failure for every window
python scripts/score_windows.py --base Qwen/Qwen2.5-7B-Instruct \
    --adapter TIGER-Lab/StopLoss-Qwen2.5-7B-Instruct \
    --windows windows.jsonl --out scores.jsonl

05Citation

BibTeX

If you find StopLoss useful, please cite our paper.

@article{stoploss2026,
  title   = {StopLoss: Teaching Agents to Predict Failure and Stop Early},
  author  = {TBD},
  journal = {arXiv preprint arXiv:2610.25525},
  year    = {2026}
}