Tuesday 3 November 2026
Agents finish 95% of single requests but 51% of chains
Five bullets
The one thing
BOBA-Bench tested ten coding agents on 280 chains of 20 consecutive requests. The chains come from 140 real open-source projects. Taken one at a time, agents completed 95% of requests on average. Chained together, they completed 51%.
Worse code is not the cause. Once checked, agent code was nearly as good a base as human code. Agents declare a request done once their own tests pass, even when those tests miss a mistake. The next request then fails about twice as often.
Self-review does not fix it. Asked whether an implementation was correct, agents were right 53% of the time. One external "that's wrong" plus one repair round lifted chain completion to 59%. A repaired mistake stopped hurting the requests after it.
This matters most when you hand an agent a series of tasks in one session. A second benchmark, Foresight-SWE, found agents also miss bugs nobody asked them to look for. The skill that grows is placing checks between requests, not only at the end.
Put a check your agent did not write between chained requests, before the next one starts.
Try this · 15 minutes
See where your agent's "done" breaks across a chain
- Pick three small requests where each one builds on the one before.
- After each request, run a check the agent did not write, such as your own tests.
- If the check fails, tell the agent it is wrong once and allow one repair.
- Note each mistake its own tests missed. Those spots need a standing check.
Stories
- TRYClaude Code hooks can now block an action when they failClaude Code 2.1.317 added onFailure: "block" for Agent SDK hook callbacks , so a failing callback stops the action instead of letting it run. Python SDK users set on_failure="block" on a HookMatcher in v0.2.177, where the default stays "continue". The release also asks before a web or shell call sends your account email address to the network. It matters to anyone whose hooks act as guardrails.
- TRYOne in six matched instruction-file sections gave conflicting guidanceToward an Evidence-Based Template for Agent Instruction Files studied 3,387 instruction files from 1,000 GitHub repositories. Of 386 matched section pairs across formats, 16.1% gave divergent guidance. Also, 40.4% of files had only one recorded change. Check for drift if you keep both an AGENTS.md and a CLAUDE.md.
- TRYInteractive prompts make agents hang or falsely report successRECEIPT found that 7 of 8 harnesses close input when no terminal is attached. With closed input, 11.5% of tested commands returned status 0 without doing the requested work. An unanswered question at an open terminal multiplied token use by 2.9. Check how your harness handles commands that ask before acting.
- TRYllm 0.37 adds a command for OpenAI-compatible endpointsRun llm openai endpoint add to configure extra models. The default model is now gpt-6-luna, half the price of gpt-5.6-luna.
- WATCHOpenAI built Pando, a Git-compatible backend for Codex-scale reposOpenAI says Pando is designed for the scale of development Codex makes possible. The post drew 236 points on Hacker News.
- WATCHAn LLM judge barely noticed when trained models broke their PythonTrained checkpoints produced invalid Python 63% to 67% of the time. The judge's Pass@32 fell only from 96.67% to 93.95%.
- WATCHPoisoning 0.1% of pretraining data planted a backdoor in a CLI agentThe 4B model ran the attacker's command about 92% of the time when triggered. It behaved normally on clean inputs.
- WATCHAmp agents now write editable Artifacts instead of long repliesEach artifact belongs to one thread. When the agent returns, it reads your edits first and works them in.
Safe to ignore
57 sources read. 125 items. 14 shortlisted. 5 bullets.