We scored 341,054 agent trajectories from 11 public datasets on this hub (SWE-smith, SWE-rebench, Open-SWE-Traces, mini-coder-trajs, agentic-coding-trajectories and others; full list in the repo) for loops, blind retries, oversized tool output, abandoned runs and cost by run length. Every run is scored by the same detectors on the same estimated cost basis, capped at the first 20,000 rows per dataset/config/split.
Loops and blind retries are a weak-model artifact: 14–22% of runs on the 2024 Llama agents, under 1% on Claude 3.7 Sonnet and under 1.3% on Qwen3-Coder-480B.
Oversized tool output tracks the scaffold: Open-SWE-Traces v1.1 through mini-swe-agent, OpenHands and SWE-agent carries a >20k-character observation in 0.1%, 50% and 55% of runs.
The longest fifth of runs by step count takes 40% of estimated spend and resolves at 0.15× to 0.91× the shortest fifth’s rate.
Pipeline, detectors and a sweep command that rebuilds the whole Index from the hub are at GitHub - metermaidai/audit · GitHub ; tables at The Agent Waste Index — edition 1.1 — 341,054 agent runs . If you published one of these datasets and a number looks wrong for your data, open an issue; the method notes carry a corrections list and we would rather add to it than be wrong.
Useful dataset. The longest-fifth result may mix waste with task difficulty: hard runs should be longer and may resolve less often even with efficient behavior. Have you considered matched cohorts by repository and task, plus marginal spend after the last verified state change? That could separate necessary exploration from loops that keep consuming tokens without producing new evidence. I would also love the detectors to emit reproducible spans so scaffold authors can trace each waste label back to exact steps.
The dataset and detectors are extremely useful, especially because most agent papers talk about “success rate” but almost never about waste. Looking at loops, blind retries, oversized tool output and long-run cost is exactly the kind of analysis agent frameworks need.
One thing I’d add from practical experience building agent pipelines: waste is not always the same as failure. Some trajectories are long because the task is genuinely hard, not because the agent is looping. Matching cohorts by repository/task, as you suggested, would help separate necessary exploration from pure waste.
A few ideas that might strengthen the Index:
Matched cohorts by task difficulty
If two runs solve the same repo/task, you can compare marginal spend after the last verified state change. That isolates “exploration that produced new evidence” from “steps that consumed tokens without changing the world state.”
Reproducible spans for each waste label
If detectors emitted exact step spans (e.g., loop boundaries, oversized-output steps, blind retry sequences), scaffold authors could trace the waste back to specific tool calls or prompts. That would make the Index actionable for debugging agent frameworks.
Tool-output normalization
Oversized tool output often comes from scaffolds that return full diffs or entire repo trees. Normalizing tool output before scoring might help distinguish “scaffold design issues” from “agent behavior issues.”
Long-run cost vs. resolution rate
The longest-fifth result is interesting, but as you noted, mixing difficulty with waste can blur the signal. Marginal cost after the last successful state change would give a cleaner measure of “pure waste.”
Overall, this is one of the most useful datasets for agent behavior I’ve seen.
To contribute something practical, here’s a minimal reproducible-span detector you can plug into your pipeline:
def detect_spans(actions, observations, max_repeat=3, max_output=20000):
"""
Minimal reproducible-span detector for agent waste.
Emits exact step ranges for loops and oversized tool outputs.
"""
spans = {
"loops": [],
"oversized_outputs": []
}
# --- Loop detection (reproducible spans) ---
start = 0
for i in range(1, len(actions)):
if actions[i] != actions[i - 1]:
if i - start >= max_repeat:
spans["loops"].append((start, i - 1))
start = i
if len(actions) - start >= max_repeat:
spans["loops"].append((start, len(actions) - 1))
# --- Oversized tool output detection ---
for idx, obs in enumerate(observations):
if isinstance(obs, str) and len(obs) > max_output:
spans["oversized_outputs"].append((idx, idx))
return spans