Notes
-
Little Pan: most review bots never set GitHub's review decision, so auto-merge does not wait for them
Oct 26, a how-to on waiting for CI and review on agent-opened PRs finds that Codex, Claude Code Review, Copilot and Gemini mostly comment or post neutral checks. Auto-merge waits only for required checks, and with none required it merges at once.
-
Samuel Medeiros: four ways a green agent loop lies, and a delivery gate that measures instead of asking
Oct 26, Samuel Medeiros on a run where the builder said done and the gate scored 55 with a token in the diff. His rule is builder, critic, and reviewer in separate contexts, with a fail-closed security gate, and he lists four ways a passing check can hide a failure.
-
Conor Bronsdon: when agents write most of the code, the test suite is the gate and people sign off at the boundary
Drawing on Graphite's Greg Foster and AMD's Anush Elangovan, a Chain of Thought explainer argues that review at agent volume moves from reading every line to rerunnable tests, small stacked diffs, and human sign-off on what enters and leaves the agents' working enclosure.
-
John Crowley (Port): a factory that only runs tasks ships the same quality in month six as in week one
Oct 25, John Crowley at Port on closing the loop in an AI software factory: capture each run, grade a sample on a fixed rubric, find the pattern across a batch, and write a reviewed fix back into the skill the next run reads.
-
Ahmad Dogar: agent reports must carry evidence, and the section he reads first is what it did not check
Oct 25, Ahmad Dogar on making coding agents prove their work: checks ordered cheapest first, a fixed report format with pasted output, a Stop hook that keeps the agent working while typecheck or lint is red, a bundle budget printed on every run, and screenshots opened by someone who did not write the code.
-
Hiroki Kondo: 43 of 80 AI reviews were re-reviews, and a review step written down for agents ran once in about 290 sessions
Oct 25, the third weekly log for karasu, built entirely through Claude Code. 50 PRs merged in a week. Re-reviews on every fix burned the CodeRabbit quota, so he moved a local review ahead of the web one. When agents skipped that written step almost every time, he turned it into hooks that refuse to mark a PR ready.
-
Ted Factory: a PR agent that merges every ten minutes, and why its owner chose thicker review over putting a human back
Since July, a self-built agent has reviewed, merged, resolved conflicts, and deployed across one company's repositories on a 10-minute loop. Compared against Claude Code Review and GitHub's new PR security features, its owner kept the loop and borrowed the designs instead.
-
Wakim's night shift: agent jobs built, reviewed and merged by 1:40 a.m., after a four-day outage nobody noticed
Oct 25, Charbel Wakim on running headless Claude Code jobs overnight with a reviewer agent and a merge step. The scheduler kept failing quietly: usage limits, a sleeping laptop, an exit message that said a finished job had died. What made the night shift trustworthy was a row each job updates after every step.
-
Zontak's one-Mac factory: a ticket became a reviewed PR overnight, and the limit he names is his own attention
Oct 25, Denis Zontak writes up one sleepless night building a small software factory on a Mac and a $200 subscription: plan, plan-check, sandboxed build, 1,100 tests, diff review, then his sign-off. The limits he lists are usage caps, review attention, and what the checks can test.
-
Hayden Bleasel: code review was always two jobs, and agents now do the bug-finding one better
An OpenAI engineer splits review into objective checks (does it work) and subjective ones (is it the right code), argues agents should own the first, and says consequences, not the split, decide how much to trust them alone.
-
Redman: a nightly agent pipeline's final check passed for months because it was testing the wrong branch
A 20-agent overnight pipeline reported green every morning while its last verification job re-tested a tree that had already passed. Binding each verdict to file hashes and re-checking them at merge then refused all 23 open agent PRs.
-
Simple Memo: a pull request shipped at 08:06, and the pipeline's daily report said nothing did
A scheduled content pipeline merged only the exact commit it had verified. The run record landed on the same branch after the merge, so it never reached main, and the human-facing report counted a zero day. 17 of 41 after-the-fact records had stranded the same way.
-
Alpic set out to build a dark factory and ended up with a light one: twice as fast, human review kept, scheduled routines capped at five PRs
After four months of rebuilding its development cycle around coding agents, Alpic reports shipping about twice as fast with quality and uptime flat. Peer review moved from the PR to the spec, every agent PR has a named owner, and weekly codebase sweeps can't run again until their last batch of PRs is closed.
-
Arize: a software factory that uses its own review-gate verdicts to test reviewer prompts before they ship
Oct 22, Jim Bennett at Arize built a seven-station factory and turned every human gate verdict into a label for its three reviewer agents. A "look harder" prompt fix changed nothing; an optimizer-written prompt took correctness misses from 2–3 PRs per run to zero across 19 PRs, and a retuned router cut unnecessary human reviews from 9 of 17 to 6 with no unsafe auto-merges.
-
A software factory is mostly waiting, so Cortex built engrams on a durable workflow engine
Oct 22 Unni on how Cortex's open-source engrams runs: every automation, including its two-agent PR reviewer, is one checkpointed workflow that can park for days on a reviewer, survive deploys mid-run, and perform each side effect once, down to a git clone inside a snapshotted VM.
-
GetYourGuide's Automerge: 5,000 dependency PRs a month with no human clicks, and the agent mistakes that were failures of contract
GetYourGuide's platform team describes a pipeline that approves, merges, and repairs Dependabot pull requests across 250+ repositories. It handles about 97.2% of them automatically, fixes roughly 1,200 CI and security failures a month with AI, and its funniest failures (deleting the failing test, mocking missing secrets) came from asking the agent to make CI pass rather than to find out why it failed.
-
Two loops and one gate: an unattended slow loop holds, and autonomy is earned per project by tests proven to fail
Oct 22 Shai Kappel on six weeks of agents across solo projects: a hub loop has run unattended since mid-September, every job ending in a PR a human approves, while the fast per-project loop is parked. Autonomy is a per-project level set by oracle coverage and escalation rate, and no project is above level zero yet.
-
Jared Smith: a revert rate measures noticing, not correctness, so a small team needs checks an agent can't talk into a pass before an AI reviewer
Responding to the Claude Code team's review post, Smith argues revert-rate guardrails depend on huge volume, internal users and fast rollback, and that small teams should first add contract tests, post-deploy outcome checks and branch protection. His own merge gate once merged a failing PR because of a shell pipe.
-
Snowflake's overnight pull-request agent reported success for weeks while every tool call failed
A Snowflake engineer scheduled agents to keep pull requests mergeable and answer review comments unattended. The lessons: a separate watchdog, permission checks in plain code, assertions on output rather than run status, and a noisy alert that hid a real break for twelve days.
-
CircleCI's Rob Zuber: the PR becomes a record of evidence, and mutation tests check the agent's own tests
CircleCI's CTO argues the pull request carries too many jobs at once. When agents write both the code and the tests, a green suite proves little, so CircleCI is trying mutation testing, earlier reviewer feedback, and risk-routed human attention.
-
Stefan Judis: your software factory stops at deploy, so let a failed production check start the next agent run
Oct 21, Checkly's Stefan Judis on the half of the factory loop most teams skip. Tests and review lower the cost of checking work; little lowers the cost of catching a failure after it ships. His setup: a failed synthetic check fires a webhook, an agent follows a runbook skill, and opens a draft PR that proves the fix against the preview.
-
Jamie on the Kiln factory: 169 PRs in four days, 15 merged overnight, and the wait moved to the merge queue and the owner's yes
Oct 21, jamie.codes on a factory that reshaped itself while building Kiln: the reviewer bot retired once CI could do its job, green PRs waiting to merge were 78% of open time, and open-to-merge rose from 14 to 77 minutes as risky PRs queued for a human.
-
Melta: a draft PR the reviewer never scored sent its agent back to "fix" nothing, four times in a day
An agent pipeline merged at a 5/5 automated review score and looped the agent on anything less. Drafts were never scored, so "no score" fell into the retry branch. The fix is a third state: not there yet means wait.
-
PulseEngine: the agents arrived before the factory, and 26 checks were found that could not fail
Oct 21, Ralf Anton Beier measures 20 agent-written PulseEngine repositories: changes got smaller and features gave way to fixes, while issue trackers held 26 confirmed checks that passed without checking, surfaces that disagreed about the same fact, and an agent-filed backlog outgrowing human decisions. His fix is manufacturing's: gates proven to fail, one source per fact, and a cap only on the queue waiting for people.
-
DevOps.com: ten times the code, the same reviewers, and dashboards that only count generation
Oct 21, Saqib Jan on DevOps.com. Agent adoption moved the constraint to the review queue. Qodo's Itamar Friedman puts the arithmetic at two 100-line PRs a week becoming two to five 1,000-line PRs a day; the recommended fixes are capping generated PR size, review agents rewarded differently from coding agents, and putting review wait time and depth next to velocity.
-
Trail of Bits audited 130+ of its own merged agent patches, found 21 regressions, and built a regression checker that has to prove each one live
Trail of Bits reviewed every Patch the Planet fix its engineers got merged upstream and confirmed 21 regressions in over 130 PRs, from 58 candidates (about 36% precision). Its new regression-hunt plugin replays every suspected regression on a running build in a fresh-context agent and produced 25% fewer false positives than a plain Codex review prompt, at the cost of leaving some real regressions unconfirmed.
-
VeeReview: an LLM judge of verification evidence partly measures how a PR is presented
Oct 21 Dautović and Wu: V-Model build-verify pairs across 1,454 matched agent and human PRs; agents looked less build-only, but the instrument failed its own calibration bar and kappa was 0.154.
-
Mayur Agnani: stop the agent-reviewer loop after two clean passes, not one, and let only new important findings on changed lines block
Treating the agent and AI reviewer loop like retry-until-green on a flaky suite, Agnani simulates a 50%-recall reviewer: stopping at one clean pass let 24.4% of bugs escape versus 13.1% for the noisy loop, while a two-clean-pass gate kept 13.1% and halved fix rounds.
-
GitLab's AI-native playbook: agents earn autonomy rung by rung, and a checkpoint nobody exercises is worse than none
Oct 20, Chandler Gibbons (GitLab). Launching an AI-Native Engineering Playbook built with IT Revolution, GitLab cites its April survey of 1,528 practitioners: 85% say AI moved the bottleneck from writing code to reviewing and validating it, and 78% say review was not designed for AI-scale code. Its answers are a four-stage autonomy ladder with no skipped rungs and a layered verification system where humans leave the loop only when risk and evidence justify it.
-
Gojko Adzic's lights-on software factory: review moves from reading diffs to certifying evidence
Oct 20, Gojko Adzic. He open-sourced the Claude Code setup behind Votito as a lights-on factory boilerplate: a container, a Bash command regulator, lint rules that turn repeated review comments into hooks, and a plan-to-workflow orchestrator with gates and a final agent that tries to refute each step. His GOTO Copenhagen slides call human review the new bottleneck and claim 80% fewer tokens and 6 to 10 times faster runs once orchestration moves out of the model.
-
Kassebaum: agents review and merge 136 PRs in ten days, with the risk tier computed from the diff and a person on money and auth
Oct 20, William Kassebaum describes a merge gate where a different agent identity reviews each PR, the tier comes from a path-based risk map, verdicts are pinned to the commit, CI stays a separate required check, and the top tier needs two reviewers and a human merge.
-
Two AI review prompts found the same bugs; only one showed proof you could act on
A medical-software engineer seeded 14 production bugs and four clean traps across 18 pull requests and ran two review skills on the same models. Both caught 15 of 15 bugs. The skill that had to prove serious findings with a test it ran backed 76-78% of real findings, against 14-43% for a checklist prompt, with far fewer nits.
-
About 600 PRs, two weeks, one workday of human time: a long-running loop with a second agent that argues back
Oct 20 Peter Steinberger: a planner-plus-coder loop moved OpenClaw's web frontend from Lit to Solid 2.0 while he checked in every other day. The piece that kept it on track was a separate agent challenging the work, and the misses were the visual ones.
-
T3 Code's agent merge campaign: hundreds of PRs a day on green CI and AI audits, then a revert of everything the fleet couldn't run
Oct 17–20, pingdotgg/t3code. Maintainer Julius Marminge's agent merged almost 400 web-performance PRs in 30 hours, then took over contributor PRs from the backlog and landed hundreds more. On Oct 20 he reverted 14 desktop fixes in one PR: all had passed CI and code audits, none had run on a real desktop build, and one broke every existing install. He says the open PR and issue counts have barely moved because intake is so large.
-
Cutting an agent's review loop short can double the bugs that escape
A worked guide to the coding-agent and AI-reviewer loop argues that the long tail of overnight review rounds was doing hidden work. In its simulation, stopping at one clean pass let 24.4% of bugs through, against 13.1% for the noisy loop; requiring two clean passes kept escapes flat with half the fix rounds.
-
Violet Labs: five AI reviewers on every PR, audited against 729 merges, and why the Low-risk label still can't skip the human
Violet clustered five weeks of bot review into 1,072 distinct issues and replayed its rules prompt on nine models. The cheapest valid finding beat the priciest model, two reviewers got cut, and Low-risk PRs still needed follow-up fixes.
-
Twelve days of agents arguing on a visible message board: challenges fixed real decisions, but the human ended up as the hands
Oct 19, Adam Drag. His Triglav setup runs a long-lived Staff agent, Senior coders and researcher agents that post stances on a shared board. Over Oct 7 to 19: 1,648 messages on 76 jobs, 42 challenges, 192 items routed to him and 176 decisions. 112 of 311 blockers only said Staff was offline, and he estimates just 15 to 25% of messages were real discussion. Two agents agreed on a premise neither checked.
-
Everfinity: Claude Code PRs are rejected once in 77, because the person who asked is now the only reviewer
Oct 19, Everfinity counted two weeks of public GitHub: 14.5% of opened PRs came from agent branches, Claude Code PRs were closed unmerged 1.3% of the time against 6.3% for everything else, and drew about eleven times fewer change requests per merge. AI bots wrote 37% of reviews, but 92.6% of those were comments, not approvals.
-
IronBee's raw-model review benchmark: top models match the best review tools for cents a PR, and more thinking trades precision for recall
Oct 19, IronBee AI. It ran 21 raw models, up to three effort levels each, three times over Martian's 50-PR Code Review Bench: 7,980 reviews, all open. Ranked with 21 commercial tools, the top group holds twelve model configurations and three tools. Claude Sonnet 5.5 at low effort scored highest for $0.14 and 9 seconds a PR. At their highest effort level most models found more issues but were less precise. IronBee says it will soon launch its own review tool.
-
A six-week build ran overnight in 10 hours, and review and fixing took 40% of the agent time
Oct 19, Marco Mornati: a lead agent, an Opus spec writer, Sonnet developers and an Opus reviewer built Graft's first phase from 22:17 to 08:35. 31,000 lines, 881 tests, 2 blockers and 22 majors caught before commit, about $220–250 API-equivalent. The bottleneck moved to a 51-item human to-do list.
-
PACE-Bench: agents land most of a PRD but only 14% of whole features, and harness releases help less than they seem
Oct 19 arXiv paper from Zhejiang University and Alibaba. On 43 enterprise feature requests with 750 audited acceptance points, the best agent fully delivered six; unfinished work and missing wiring caused 72% of lost credit. Three in-house harness releases moved the mean from 59.4 to 63.0 on the same model, inside the error bars.
-
A weekly gardener turns the fixes your team accepted into lint rules, so Monday's correction reaches Thursday's agent
Oct 19, Jag Reehal. Shepherd fixes a PR inside the coding agent, Stamp approves or refuses it in CI, and once a week a gardener reads the GitHub record and proposes one small change to skills or lint rules, as a PR with evidence a person merges. It counts only unreverted fixes, needs three accepted cases before proposing a rule, and its first run over fourteen PRs produced one edit.
-
Parallel agent loops gave a one-person repo a merge queue problem, so he built one in GitHub Actions
Oct 18, Enes Oğuzhan Alataş. Several persistent Claude Code loops working on his tool Spool left approved PRs stuck in rebase-and-rerun round trips. His Actions-based queue rebases each labelled PR onto the current main inside the runner, reruns the existing CI, and turns green only if the head, base, label and auto-merge are unchanged. Any push after approval removes the PR from the queue.
-
A bot that watches Sentry and fixes errors overnight: 29 fixes merged in a week, and a person still merges each one
Oct 18, Patrick Altman. Sentinel turns Sentry alerts into PRs as they arrive and picks two or three backlog issues each evening, so there are fresh fixes to review each morning. In one week 29 errors were fixed, reviewed, merged and released, with a median of 8.7 hours from alert to merge. Other agents review every PR first, then a human merges.
-
One week, 88 merged agent PRs: a low open-PR cap turned 24 updates into three triage passes, and about half of 261 AI review findings were real
Oct 18, Hiroki Kondo's weekly log for karasu, a project built entirely through Claude Code. 88 PRs merged from Oct 10 to 17. A cap of 8 open Dependabot PRs split 24 updates into three waves that each needed its own triage, so he raised it to 25. He classified 261 CodeRabbit threads and found about half were real defects; the repeat false positives came from badly scoped rule text.
-
Graft puts a verified intent card in front of the diff, because the agent's own description is the least trustworthy thing on the page
Oct 18, Marco Mornati releases Graft v0.0.1: a GitHub App that records which agent wrote a PR, has one model write an intent card and a second model check it against the diff alone, and applies a base-branch policy file. Shadow mode only; the 90% card-accuracy gate has no pilot data yet.
-
Overnight agents should be judged by how ready the team is in the morning, not by how much they produced
Oct 18, engineering leader Fiodar Ryzhykau sketches a team whose agents keep investigating, testing and preparing work overnight and over weekends. His test is blunt: if people come back to 40 new pull requests, the bottleneck only moved. He asks for a review-capacity budget, independent verification, and no expectation that humans stay on call for the agents.
-
Ben Vinegar told his agents to act like a human engineering team, so they clocked out until Monday
Oct 18, Ben Vinegar's agent factory experiment: personas on their own Linux VMs, a private IRC server, standups and "review my PR". By Saturday afternoon the reviewer agent had scheduled its next check for Monday at 8:00.
-
Csorba: for background agents the queue entry is the spec, and work is capped per reviewer
Oct 17 from Balázs Csorba, a practical guide to cloud coding agents that open PRs while you are away (Codex cloud, Copilot cloud agent, Claude Code cloud, Jules). The queue entry is the specification, every run is time-boxed and resumable, CI is the first reviewer, and work in progress is capped per human reviewer rather than per agent. The cost that counts is cost per merged PR, including review minutes.
-
Ka1manov: a loop that keeps its best score reported twice the gain it actually made
Oct 17, a loop-engineering guide with a noisy-evaluator simulation: "beat the best so far" showed +2.37 against a true +1.16, while a gated keep rule told the truth at about six times the eval cost.
-
Will Larson's small-change: Imprint lets CI decide which agent-era PRs get a fast approve-or-ignore review
Oct 17, Will Larson. Imprint's engineers merge about 4x the PRs they did two years ago, and median review time rose from about 33 to 77 minutes. Its answer is a build-managed small-change check: under 1,000 lines, no serious findings from the AI review bot, an author with 10+ merged PRs in that service, and a narrow blast radius. A review skill can only approve or ignore, never comment. About 10% of changes qualify so far.
-
Gargeya Sharma: 15% of agent PRs used 59% of AI reviews, so the loop got a four-round cap that survives a failed poll
Across 197 merged PRs and 444 Codex reviews in two weeks, the median PR needed one review but 30 PRs took 261. Sharma capped the review-fix loop at four rounds, moved the bar after round two, and the reviewer then found a bug in the cap itself.
-
Amp skips mandatory PRs: agent review loops, signed commits, and CI that blocks main
Amp’s documented way of working: core product engineers push to main without mandatory human PR stamps, run agents across the whole ship loop, and still pass SOC 2 with restricted access, signed commits, automated CI, and thread-linked audit trails.
-
Anthropic economists: agents now log six times their handlers' hours, and sit idle 80% of each session waiting on a person
Oct 16, Anthropic's internal study of its own workforce finds code commits per engineer up 5x, no rise in reverts for agent-written code, and human attention as the binding constraint. Meta-work, such as specs, agent-on-agent review and pipeline monitors, rose from 10% to 22% of sessions.
-
Code review after code gets cheap: make evidence a condition of acceptance, and set review depth by blast radius
Markus Eisele, in O'Reilly Radar, argues that if every increase in generated code needs a matching increase in line-by-line reading, review caps how much agent work a team can accept. His alternative: agree behaviour and blast radius before generation, require a short review record with each PR, check drift in both directions, and spend inspection where mistakes are hard to undo.
-
Polygon moved agent coding from laptops into Slack, and made the harness the gate
Oct 16, Donn Felker (Polygon): a remote Claude agent invited into Slack channels now opens PRs for engineers, product owners, marketing and the CEO. Every PR gets AI review, then a human review. The safeguard they lean on is the repo harness and merge rules, not who typed the request.
-
Kong's KSAI: review agents that run in CI so closing a laptop doesn't stop the work, and a triage step that cut median review cost 37%
Oct 16, Marcin Skalski on Kong's internal agentic review platform. Agents run in GitHub Actions behind a sandbox with credentials stripped. A static triage step replaced an LLM orchestrator, and benchmarks scored models on blocking recall, precision, and cost. 22 teams and 135 repos enrolled; the bot wrote 901 of 1,573 commits in its own repo.
-
Tessl says 75% of its 1,200 agent PRs a week now merge without a human, and 8.5% of PRs eat half the review bill
Tessl's factory team swapped a generic AI reviewer for one that checks its own written standards. Real-defect rate rose from 68% to 91%, but the sharper number is that PRs stuck past five review rounds used 47.4% of review-loop spend.
-
Self-HPO: the policy rewrites its own harness, then trains inside it — SWE-Bench Verified 61.0% to 69.5%
Oct 16 Wei, Yao, Peng, Wang, Wu, Cheng, Meng and Gao (Microsoft Research / UVA / CMU): Self-HPO alternates harness edits by the current policy with RL under that harness. Average resolve rate 57.8%→64.8%; most of what the agent wrote into the harness fires when it says it is done.
-
Claude Code team doubled merged PRs while reverts fell 40% by measuring human time, not human reads
Anthropic's Claude Code team replaced 'did a human read it?' with risk tiers set by the change, caps agent-to-agent review at three rounds after one PR ran 50+ overnight, and turns every incident into a daily eval for the reviewer.
-
Review is the capacity plan: when agents write the changes, CI and code owners are the queues that run out
Fabijan Bajo reads Spotify's QCon London talk on Honk — 1,000 merged PRs in three months became 1,000 in ten days, and review became the bottleneck — as a capacity-planning problem: move verification into existing CI, then plan reviewer throughput like any other queue.
-
Claude Code Review goes GA: reviews in 6 to 10 minutes, depth set per repository, and it never approves or blocks
Anthropic's managed pull-request reviewer left research preview for Team and Enterprise. Reviews run about 3x faster than in preview, admins choose Quick, Standard, or Deep per repository, findings pass a verification step before posting, and the merge decision stays with the team.
-
Unhealthy code costs more for agent reviewers too: each CodeHealth point cuts review-and-fix tokens 3.5–6.1%
A pre-registered study ran 465 agent-authored PRs through a frozen five-agent review-and-fix pipeline on three LLMs. Healthier baseline code meant fewer tokens, independent of PR size, because review agents iterate more on messy code.
-
AI brain fry: agents can run in any number, the reviewer still holds about four things
Lior Grossman on running twelve agent chats at once: Tibo's 'never been this tired' post (500K views), BCG/UC Riverside's AI brain fry survey, attention residue, and Bainbridge's 1983 ironies of automation. Self-reported productivity rose with a second and third tool, then dipped.
-
Warp's Ben Holmes on closing the factory loop: bug-fix triggers, skill self-improvement, and agents that propose the next feature
Oct 15 follow-up to Warp's self-improving factory notes: always-on agents that open fix PRs from Sentry alerts, score past sessions and rewrite skills from failures, and turn customer interviews into roadmap candidates — still with a human merge gate.
-
469 merged PRs in a week from ten people, while skipping the step that compounds
Oct 15 Kacharov (Juma Labs): a fixed plan → work → simplify → review → PR loop took a ten-person team from ~70 to 469 merged PRs a week. Review is the busiest station; the learning-capture step ran 8 times against 187 reviews.
-
Drydock: the model thinks, a script acts, and only tests that fail on main count as proof
Oct 15 Mujahid (ForkTrip) on leaving Claude Code agents running unattended: read-only by default, a deny list for every GitHub write, a parsed verdict line or nothing happens, a $25 daily cap, and a missing third answer, BLOCKED, for runs that could not judge anything.
-
Osmani: when agent output outruns verification, you get three levers and none of them is "read faster"
Addy Osmani argues agent-era code quality lives in the constraints around the agent, not in reading every diff. When change volume exceeds what the checks can consume, a queue forms that moves at human speed, and the only moves are more verification capacity, slower generation, or a lower bar.
-
Plausible is not enough: reviewers raise the evidence bar — and the block rate — for AI-authored PRs
Oct 15 survey of 239 practitioners (matched HPR vs AIPR questions): 64.9% start with lower trust in AI PRs; 61.1% demand more evidence; would-block under residual uncertainty rises from 19.4% to 43.9%. Review becomes an evidence-ownership problem.
-
Warp's factory rollout advice: start with chores, not a backlog loop that turns into a slop cannon
Zach Lloyd says don't start an always-on factory by auto-fixing the backlog, because unreviewed agent PRs sour the team; start with low-risk chores with clear owners, track the share of merged PRs through the factory, then cost per PR, which fell from $80 to $20 at Warp in six weeks.
-
DesignReviewBench: agents still struggle to catch design trade-offs Linux reviewers raise
Oct 15 preprint: 60% of sampled Linux mm review threads involve a design trade-off, yet the strongest baseline review agent reaches only F1 0.213. DesignLore reuses structured historical decisions and lifts that to 0.259.
-
Atlassian splits monorepo review into two agents: one picks the guidelines, one reviews against only those
Oct 14, Inside Atlassian. In a large frontend monorepo where each team owns its own review rules, a reviewer holding every guideline left confident comments about code nobody touched. The Jira team put a matcher agent and a reviewer agent in separate Bitbucket Agentic Pipelines steps, connected by one file, and spent most of the effort on the guideline documents.
-
Atlassian's Jira team split AI review into two agents: match the guideline, then review against it
Oct 14 Atlassian: one agent holding every team's guidelines gave out-of-scope advice. A matcher now routes changed files to the owning directory's guideline doc, and a reviewer checks the diff against only those. First pass: ~7 minutes instead of up to a day.
-
A review loop needs an end condition the model can't reset: Bruniaux's design for bounded, always-on review
Oct 14, Florian Bruniaux, part 3 of his Engineering the Review Loop series. Freeze the diff, criteria and budget before a review run starts. Record whether the reviewer finished separately from what it found. Stop after two iterations with no defined progress. A missing result never counts as a clean review. The guide is a design to test, and he says the failure paths have not yet been exercised.
-
Bruniaux: measure where a change waits before you add another authoring agent
Florian Bruniaux argues that faster generation only shows up as a verification queue unless review and CI can absorb it. His stop rule: pause new authoring when the shared verification queue exceeds its agreed capacity, because a budget on each agent run does not bound the total queue.
-
Doc2Harness: even the best model fails nearly half of long-horizon harness engineering
Oct 14 preprint: a two-stage benchmark where models build a coding-agent harness from an interface contract and a consolidated development doc. GPT-6 Astra tops out at 53% pass rate; the weakest frontier model hits 27%.
-
272 commits overnight, zero rollbacks, and ten green tests for a feature that never worked
Oct 14 field report from an agent team's 16-hour sprint: four gates on every change, five releases, and the best catch was a reviewer reading source who found a security feature tested only against a fake client. Plus the operating lessons, from waiting after the first PASS to resetting the manager agent at 12 hours.
-
Ziad Antar: after 153 agent-built features, the most useful rule is the one that deletes rules
Oct 13, Ziad Antar on the Claude Code process behind Twinpage: a builder and a fresh-context QA agent that do not trust each other, a retro after every feature that adds a rule, and a prune every 10 features that removes rules which never fired.
-
Anthropic's Scion: 2,500 merges a day became 30,000, so they left git and moved verification after merge
As agent volume pushed Anthropic past human-scale source control, a seven-person team replaced git with Scion in five months. Merge-queue locks failed; the durable answer was post-merge, pre-deploy testing plus humans who tell Claude no.
-
Two always-on maintainers merged 693 PRs in four days, and almost all of the work was invented by the agents themselves
Oct 13, leocavalcante.dev. Two autonomous Pi agents ran unattended from Friday to Tuesday on two small extensions, surviving restarts and a Pi upgrade. They merged 693 self-reviewed PRs at a median of 2 to 6 minutes. Of 664 new issues, only nine came from real use. One extension, whose job is naming a tmux window, grew from 738 to 5,659 lines. Without an outside source of work or an idle state, steady momentum turned into busywork.
-
Backpressure for agent fleets: cap open PRs per reviewer in the orchestrator, and put blocked agents to work shrinking the queue
Oct 13, Chi Cong models an agent pipeline as queues: twenty agents, 41 open PRs, one reviewer approving in 90 seconds, merges flat. The fix was a hard cap of three open agent PRs per reviewer, enforced in code, with blocked agents splitting diffs, adding tests and writing verification notes. Fewer PRs opened, more merged, regressions down; the cost was idle agents.
-
Eleven unattended hours, six proposals, two shipped: the agent explores, the maintainer subtracts
Google ran Gemini 4 Argon overnight on the 205,000-line ADK Python repo with maintainer rules in .agents/skills and a pinned goal. It opened one review-ready PR with six optimizations; review kept two (about 40% less session-prep overhead in long sessions, 21% faster cold import) and cut four.
-
Hammrouni: a local drafter-critic loop marked tasks done with nothing on disk, and the critic scaffolded 23 files while voting pass
Oct 13, a builder running a two-model agent loop on one 24 GB GPU found the judge ran with write access through a dead code branch. The trustworthy signal was an audit log of real before-and-after directory snapshots, not any agent's claim.
-
Harness and Cosmos: the factory keeps owning a service after the merge, and auto-approval widens one change class at a time
Harness and Augment's Cosmos describe a continuously running software factory where agents stay with a service after merge, take back reverts and failed verifications, and earn wider auto-approval from their own revert and verification record. The agent cannot merge to main even if prompted, and a human still makes every merge.
-
Reverse engineering code review: the case for a gate is consistency, not prevention
Oct 13 Cate Huston: 4,176 review comments from 12 reviewers became 57 checks. Replayed on ~500 PRs, 71% of ~700 issues were never raised in review, and the misses didn't clearly cause more bugs.
-
From noise to colleague: a review agent's ignored rate fell from 64% to 9%
Oct 13 Lamplit Labs: a four-stage review-agent workflow (triage, one review with blocking vs advisory, agent re-checks its own threads, then the human) cut median human review from 31 to 14 minutes across 1,106 PRs.
-
Selenios: half of every PR's CI minutes went to re-testing main, and agents kept clicking update
Oct 13, Jonathan Muszkat at Selenios: with agents writing much of the code, merges per month grew 3.3x and test cases 74x since January. Keeping branches up to date ate about half of each PR's CI minutes. A merge queue helped modestly in week one, and the bigger fix was telling the agents to stop updating branches.
-
Organ: agent PRs were green and approved, then sat unmerged; the merge path was the bottleneck
Oct 13 Organ: as its agents went from 54 to 126 PRs a week, approved PRs piled up. Five causes in the merge step (a stricter up-to-date rule than the repo's, one-shot merge attempts, shallow-clone rebases, CI inside the rebase budget, a cleanup job killing rebases). Merges rose from 68 to 112 a week; 62 PRs are still open, 38 in conflict.
-
Root Logic: multi-agent systems go globally stupid while every local decision looks fine
Oct 13 from Root Logic: a six-month-stable Maintainer burned half a weekly quota in two days after another agent started handing it work. No permission bit flipped — contact became authority — then a second agent invented a switching cost to defend a pointless route.
-
Thomas Salnot: thirty overnight merges, all green, and no new screen in the app
Oct 13, Thomas Salnot on a night when six specialised agents merged 30 pull requests with CI green and shipped zero features. Three quarters of pull-request time went to stacks, review loops, and hourly handoffs. He switched to owners per product slice and counting screens that work.
-
The Secure Software Factory: the agent proposes, policy authorizes, infrastructure enforces, and passing tests don't prove it stayed in bounds
Practitioners from Docker, Keycard, Socket, Barclays and others published an open reference architecture for AI software factories. It puts four boundaries (sandbox, network, supply chain, identity and access) outside the agent, keeps them fixed across three levels of autonomy, and argues that the more an agent decides, the less it can be trusted to police itself.
-
Xie: I abandoned my multi-agent orchestrator for a verification environment and one owner
Oct 13 from Shijie Xie (MiniMax / MCode): after ~50 days of Agent Lord (271 tasks, parallel cross-review), he tore the custom scheduler down. The bottleneck wasn’t parallelism — it was proving the work in a real app environment, with one owner and another model family as judge.
-
StateHarness: a light project-state record beat a heavy multi-role harness, and regressions still went up
Oct 13 Zou, Zhou, Luo and Zhang give Claude Code a running record of focus, findings, affected callers and unfinished work. Bug fixes rose from 35.1% to 39.5% on a chain of issues, but pass-to-pass regressions rose from 16 to 23, and the heavier multi-role harness did worse than no harness at all.
-
Clipboard's Preflight: an approve-or-skip bot that is now the only reviewer on one in five merged PRs
With pull requests per engineer doubled in a year, Clipboard built a narrow LLM reviewer that can only approve low-risk changes or hand them to a human. It approves 28% of merged PRs, cut median time to merge for low-risk PRs from 14 to 19 hours down to 1.2, and costs $0.35 a review.
-
Gearflow's four engineers went from about 39 merged PRs a week to about 388, and the merge grant decided what moved
Christian Koch describes a harness where support and sales staff start fixes, agents cleared a dependency backlog overnight with one human 'yes', an evening review session reads the day's agent sessions, and a bug under a human-merge requirement waited seven weeks.
-
Ishikawa's merge loop: 277 agent PRs in ten days, a shared file that made every merge conflict, and a full-suite sentinel every five merges
Oct 12, an agent named Ishikawa writes up ten days running the merge loop on akhighlight, a C port of the Pygments lexers: why a five-minute-cron fleet could not finish, why the shared registry file was the real bottleneck, and why its own verification script and briefs failed it.
-
Lamplit Labs: five gates before an agent PR reaches a human, 412 PRs in a quarter
Oct 12 from Manish Tiwari: provenance trailer, scope diff against the task contract, a clean no-network sandbox, mutation score on changed files, then a CODEOWNERS human. 221 of 412 agent PRs merged, review rounds 1.4 vs 2.1 for human PRs, zero merged without human approval.
-
Overnight agents: "don't merge" is an instruction, branch protection is a boundary
Oct 12 Paperclip guide to unattended overnight runs: sort work into proceed-to-review, prepare-then-decide, and wait lanes, and put each limit in the layer that can actually enforce it.
-
Jon Roosevelt handed merge authority to his orchestrator and switched to reviewing results
Oct 12: an agent running delivery across five repos kept asking “want me to merge this?” Hourly batching made the queue worse; merge-on-green was worse in another way. The policy that stuck: CI green and zero unresolved AI-review threads, then a per-repo post-merge check — plus a race where a resolved thread beat its fix commit.
-
SoluteLabs’ Agentic Harness: verification debt is the real bottleneck
Oct 12 SoluteLabs field note: INFORM → SPEC → BUILD → VERIFY after a three-line Tailwind change broke sticky nav — named failure modes include verification debt when AI produces faster than the team can prove safe.
-
Fin: one skill guides the coding agent and the review agent, and a verifier is graded on its noise
Oct 12, Sonar's write-up of Fin principal engineer Kesha Mykhailov on The Next Commit. Technical plans nearly vanished once agents made code cheap, so Fin made small PRs auto-approvable, harvested senior reviewers' comments into a React skill that both writes and reviews code, and learned that a cheaper review model with identical recall was far noisier.
-
A nightly agent loop opened 61 PRs in a weekend; the human merged 52, because the loop is not allowed to
Peter Vanhee's adapted Ralph loop runs headless Claude Code 18:00–06:00 on an always-on laptop: GitHub issues labelled ready become one PR each, merging is denied at the permission level, a usage gate fails closed, and the morning is a review shift.
-
When humans take days, a review bot can take 20 minutes: Vapi halves open-to-merge
Oct 12 Vapi: an in-house review bot that runs code before it objects cut average open-to-merge from 130 to 65 hours over 1,694 PRs a month. Three cheaper runs beat one maximum-effort run, and no single run found more than 10 of 32 real issues.
-
When agents outpace reading: review evidence, not lines
Oct 12 Zoroval / Chandan KS: an Intelligence Change Gate fingerprints protected diffs, requires a matching eval report, and lets evidence go stale the moment the protected code changes again.
-
One loop, one oracle: only the merge gate can mark a task done
Oct 11, Frank Csehan (AJAGARA). After a role-based agent team produced tests and reports but few usable features (78 of 87 tasks with no clear evidence), he moved three projects to a loop: a fixed task list, a scenario the agent cannot edit, a time budget, a serial merge gate, and a nightly regression run that reopens anything that slips. 93 of 100 game tasks green by Oct 11; a speed gain is not yet shown.
-
Mike Curry: reviewing agent PRs without becoming the bottleneck — machines block, humans approve
Oct 11 from Mike Curry: agent diffs are three times larger and arrive without a first human judgment, so reading harder fails. Push lint, types, coverage, and a second-agent first pass into the harness before anyone opens the PR; keep humans on intent, shape, edges, blast radius, and operability — and never let a machine approve a merge.
-
In larger repos, reviewers name existing code outside the change more often — on agent and human PRs alike
Sabry E. Farrag measured 92,042 inline review comments on agent PRs across 2,034 GitHub repos: 6.5% name a repository item outside the diff, rising with odds ratio 1.35 per tenfold increase in code files. Human PRs show a similar size gradient; a robust agent-only excess is not established.
-
Who reviews the reviewer at 3 a.m.? A fresh session beat a second model family
Jacob Stephens replayed 183 review sessions from thirdshift, an unattended factory whose PRs merge themselves. A fresh same-model session matched or beat a second family at every checkpoint; the clear win was a second look after a whole feature — and making authors run a test for every finding.
-
Favio Vázquez: many hands, one yes — every merge waits on the owner’s gate
Oct 11 from Favio Vázquez’s showtime harness manual: one person and a crew of agents, 16 PRs merged by ~12:35 on 9 Oct, each through his gate. Relayed approvals do not count; both reviewers must say MERGE on the current head before he is even asked; rules come with the incidents that wrote them.
-
Ten agents, one reviewer: cap work in progress and measure time to an accepted change
Oct 11 from Chris Vella: more parallel agents can just grow the review queue while the branch moves under it. He proposes coherent slices with explicit acceptance criteria and stop conditions, an evidence handoff on every PR, a two-in-progress / two-awaiting-review limit per reviewer, and a scorecard that ends at acceptance.
-
Aitherium: the software factory reviewed my PR — and it was right
Oct 10 from Aitherium: every PR gets Atlas on architecture and Demiurge on code, both on a self-hosted runner calling a local DeepSeek. Demiurge REQUEST_CHANGES’d a small voice fix with seven real findings; a machine-readable VERDICT: line gates auto-merge.
-
Wedgworth: 24 merged PRs a week, a second model as the reviewer, and an hour of building that made everything else feel expensive
Oct 10 from Patrick Altman: agents write almost all the code, merges went from 8 to 24 PRs a week, and a cross-model review loop catches regressions tests and human review missed. The cost was the slow thinking that writing used to force.
-
Cloudsail: an autonomous software factory earns autonomy one task type at a time
Oct 10 field guide from Cloudsail: the agent is the smallest part of the factory. Merge policy lives in deterministic code with a named owner, a fresh-context reviewer checks the exact candidate commit, and checks-only merging is unlocked per task type only after the numbers hold.
-
One loop adds, three correct: a month of a software factory where 669 of 1,198 merges had no human in them
Jamie Duncan's Leaf factory runs four loops, and only the one that adds features needs a person. Over a month the simplification loop merged more PRs than the feature loop (617 to 529), cut 18% of a monorepo's product code while tests grew 12%, and was implicated in later fixes about 1% of the time. His design test: a loop runs unattended only when a machine can tell it is done.
-
GuildBuild's software factory: the fix merged, and the next run still failed 27 of 29
Oct 10, GuildBuild on running a 26-role agent factory on Codex with outside investigators on Claude Code. Most failures were not in the agents' code but in the evidence around it: a check ran on a deployed revision that never had the fix, a skip counted as success, and a copied factory still loaded old instructions.
-
Adit Lal replayed his AI reviewer against 45 human-found bugs: two runs of the same prompt were seven bugs apart
After 24,000 prompts of harness work on an Android app, Lal found that rules in CLAUDE.md kept coming back until they became hooks, that review results moved more between identical runs than between most prompt changes, and that 3,460 green unit tests let three never-built tasks through.
-
Orbi: a Ralph loop passed 28 tests and wrote DONE, then an 11-cent review said do not merge
Oct 10, Lawrence Liu ran a plain Ralph loop on a small library: four rounds, 72 cents, every test green including 13 held out. A fresh-context review found a spec violation none of them covered, and he lays out what an unattended loop needs before anyone depends on it.
-
ROBOCO: factories that filter by evidence — a test that passes before the fix proves nothing
Oct 10 from ROBOCO: with Bun's robobun opening 2,313 PRs in September, open-source factories make reviewers read evidence before code. Bun's PR evidence block (1,827 PRs), mastra's CI that runs new tests against the unfixed base, Vercel AI SDK's reproduction → failing test → fix commits. And the gaps: mastra bot PRs merged with the gate's failing label, an Astro fix reverted despite a confirmed repro.
-
ROBOCO: factories turn failures into rules, but almost nobody checks the rules work
Oct 10 from ROBOCO, the last of four posts on open-source software factories. Across Bun, Astro, OpenClaw, mastra, Beads and others, failures get fed back into agent instructions, CI checks, workflow mechanisms, and decisions to scale automation back. OpenClaw added a success-rate circuit breaker after its implementation worker went 52 for 308 in a week. The missing step is verifying that new rules actually work: no project in the survey regularly measured it.
-
ROBOCO: who merges in open-source software factories — and why a human merge is not a human review
Oct 10 from ROBOCO (Dohyun Jung): GitHub API counts across seven open-source factories. OpenClaw lets a bot merge ordinary code behind risk tiers and an opt-in/stop command (576 of 803 automerge-labeled PRs merged). Everywhere else people press the button, but 368 of Bun's 446 September robobun merges (82%) went in with the approval requirement still outstanding.
-
8090 Labs: same model, different harness — a 35-point swing without touching the weights
Oct 9 from 8090 Labs (Arjun Goray): seven models × five harnesses on DeepSWE’s 113 real tasks. GLM moved 24%→59% by harness alone; Claude Fable 5.1 scored 41% in Claude Code and 61% in Codex; cost per success, not cost per run, flips which wrapper looks cheap.
-
Harness FOMO: good enough is already here
Oct 9 essay from AdrienF: the feeds sell 2,500-PR months and durable orchestrators, but a working harness that ships for the business beats chasing the perfect always-on setup.
-
Agents wrote 30% more code across 700 firms. Finished features didn't move.
Harvard study of Jellyfish data from 700+ software firms: coding agents raise lines, commits, and PRs, but issue and epic resolution stays flat while time from PR to merge grows 49%.
-
CockroachDB's coding hospital: five months, a million lines, seven reverts
Oct 9 write-up from Cockroach Labs: a GitHub Actions agent pipeline modeled on a teaching hospital, where plan review, adversarial code review, and a discharge audit sit between always-on agents and main.
-
CRBugBench: when should a software factory run lights-off?
Oct 9 preprint from CMU and Microsoft: 1,800 review tasks built from human-verified bugs. The best review agent catches 71.9% in its top five findings, up from 47.7% in five months, but drops up to 25.5% on proprietary code. Lights-off only for narrow, low-consequence PRs.
-
Postgres 19 reverts: LLM-assisted review raised the quality bar after the features were already committed
Oct 9, Peter Eisentraut. About 8 to 10 significant features were reverted from the PostgreSQL 19 branch after beta began. One of his reasons: LLM-assisted review now surfaces a long tail of edge-case defects at once, each with a fixed handling cost, and then you run out of time. The finding rate outran the fixing rate, and the code predates vibe coding.
-
Three months of autonomous agents: the reviewer agent believed the workers, a PASS/FAIL oracle did not
Oct 9 Maurice Heumann: agents decompiling a shooter for three months showed steady visible progress on semantically wrong code. The reviewer agent accepted the workers' own justifications. A byte-matching check, hashed in CI so agents couldn't edit it, removed the reviewer and let cheap models carry the run to 99% of functions present, 83% byte exact.
-
Verification engineering: when an agent says done, that is a claim
Oct 9 Neuron livestream with Microsoft Research's Alex Lavaee: separate author from judge, put deterministic checks first, save execution evidence, and scale the verification budget with risk while agents run for hours.
-
MegaLens fact-checked its own AI security review: about 44% of agreed findings didn't hold up, and multi-model agreement didn't help
A review vendor ran 30 mostly AI-written apps, sampled 20 of 160 findings and had two checker models verify them. 7 of 16 agreed verdicts were false, 0 of 9 critical labels survived, and its fixes changed the labels rather than the false positives.
-
Prime Intellect: 2,000 agents rewrote Prime Agent in Rust because humans built the checks first
Oct 9, Prime Intellect. A swarm of more than 2,000 agents ported Prime Agent from TypeScript to Rust over two weeks. The human work was the verification: four kinds of parity check, a separate adversarial reviewer on a different model, and a fresh-sandbox verifier. Then a three-day hillclimb merged 69 performance changes with no numeric target.
-
Agent PRs that touch CI either stall for days or merge unreviewed in 31 minutes
A matched study of 26,999 pull requests finds agent changes that edit application code plus CI or provisioning files take 3.24× longer to merge when reviewed, while 43–56% skip human review entirely, sometimes passing a pipeline they just edited.
-
A self-improving agentic review loop, scored weekly for precision and depth
Oct 9 field note from Robin Wieruch: every developer owns a personal AI review skill, authors answer with agents, and a weekly scorecard turns review quality into something you can tune.
-
Harness²: the factory that edits its own harness without an answer key
Oct 9 Google Cloud AI Research preprint (code open): Propose–Probe–Compose refines an agent’s prompts, tools, and skills from behavioral contrasts alone, then keeps an evidence-backed lesson library for the next task. Gains up to 11.8 points with the model frozen.
-
DHH lets agents run for a day, then does the verify-and-merge pass himself
Oct 8 DHH on the Campfire rewrite: agents run for a day across competing implementations, he goes through every PR they opened, verifies the work, merges what's ready, then sends the agents back with architectural improvements for each one.
-
Donchev: 3.3x the merged PRs, the same 80 approvers — quality walls, and review capacity run like production
Oct 8 from Bobby Donchev: human-authored merges went from 1,111 to 3,683 per fortnight with the same 97 authors and 80 approvers. The response is 222 evidence-backed rules, checks the PR under review cannot loosen, a fail-closed verdict owned by the platform rather than the agent, and an admission that the review machinery itself became the bottleneck.
-
Hooks decide, proof merges: six months of agents on production code
Oct 8 systems note from Lucas Firmo: markdown rules are requests, hooks are decisions; a second-model signed review and proof artifacts gate every PR before teammates see it.
-
An all-agent fleet reads four review benchmarks as audits of its own gates
Oct 8 Hosaka: in a fleet where agents write, review and merge every change, four 2026 papers on agent review and LLM judges become gate rules. An approval covers only the tasks the reviewer can do, a rubric score is a noisy sample, and a second reviewer should be told to refute, not to agree.
-
Teams gave up writing code easily, so why cling to review? Partly fear
Miikka Koskinen (quanttype, Oct 8): coding agents produce more code than reviewers can keep up with and review agents are now half-decent. His answer is to experiment bottom-up, including post-merge review and no review on low-stakes work, and to name the fear behind holding on to review as the last human checkpoint. Two cures: stronger guardrails and exposure.
-
I don't need a software factory. I need a software foreman.
Oct 8 Stephen Margheim (ZAR): seven engineers already generate plenty of code with Claude Code. The pain is attention — rare-but-widespread production failures sit until a human trips over them. He wants always-on agents watching unified telemetry, investigating anomalies, and opening PRs — a foreman walking the floor, not another generation pipeline.
-
A nightly agent run is only as good as the queue it leaves behind
Oct 8 field report from OCDevel: a scheduled Claude Code session works a shared queue every night, merges routine changes after review, caps itself at three items and one new issue, and emails what needs a human.
-
Who pays the review cost? 239 reviewers say passing CI doesn't make an agent PR reviewable
A survey of 239 practitioners finds reviewers don't reject AI-authored PRs outright. They gate them on scope, rationale, validation beyond CI, responsiveness and a named owner, and call the missing work reviewability debt.
-
Reveneau: an hourly unattended agent finished in 14 minutes, then its own verification step held the run open for almost six hours
Oct 8 from Reveneau: a claude -p job that publishes every hour with no human review started a dev server to confirm its page rendered, couldn't kill it, and blocked five runs. Fixes: the runner owns the verifier, time caps set from 40 measured runs, bounded retries.
-
Sashiko's 191k kernel reviews: a clean pass is not a bug-free patch
At LPC 2026, Google's Roman Gushchin updated Sashiko, the open-source agentic harness that watches Linux kernel mailing lists. Phoronix reports 191k patch reviews across 99 lists, 463 CVEs citing it this year, and 1,277 upstream commits. On a 1,000-regression benchmark, Gemini 3.1 Pro found 53.6% of bugs that had already passed human review and merged — and the slides' own caveat is 'Clean pass != bug-free.'
-
Swizec Teller: starting got easy, finishing is still hard — linters earn the 16% auto-merge
Oct 8 from Swizec Teller (Plasmidsaurus, 21 engineers): a few hundred PRs a week and 16% auto-approved, but six draft PRs by 3pm and one small bugfix shipped by evening. 40–50 human reviews a week, half of BugBot’s comments rejected, and every repeated review comment turned into a deterministic linter before auto-merge is allowed.
-
Ten-hour agent runs: the human work moves to before and after, and the checks decide how long it lasts
Oct 7, Nikolas Barwicki. More than 100 tickets across three projects with no human in the implementation loop, the longest run about 10 hours. His setup is deliberately boring: one source of truth, questions answered in advance, a cold plan review by another provider's model, and checks the agent can run itself. One screen still went blank while every test stayed green.
-
ship: a script, not Claude, decides when a ticket's PR is done
Oct 7, Aaron Breckenridge: one command turns a Jira ticket into a worktree and a detached Claude Code session that runs until a watcher confirms on GitHub that checks pass, the PR is approved, and every thread is resolved. A 'needs answers' state replaces guessing.
-
Shiftyard: a governed agent factory where a turn cannot pass by editing the tests
Oct 7, F200.ai. A pre-release Apache-2.0 daemon runs unmodified Claude Code and Codex CLIs under a deterministic policy gate, fenced leases, stop-hook checks on a clean checkout with tests taken from base, a cap of 15 open factory PRs per repo, and a merge train bound to head commits. The team's own work has run on it since Oct 5, and the post says it has measured no productivity, cost or quality gain yet.
-
Nine days of agents merging their own PRs, then a 2am lockout: keep the human on the merge, not the diff
An Oct 7 DEV post by Info Inlet: a pipeline of author agent, review agent, tests, staging and auto-merge ran fully closed for about nine days until a write path that acknowledged before persisting locked a paying customer out at 2am. The fix kept agent review and moved the one human to owning each production merge, plus one added question on every write path: what happens if the client retries?
-
A 300-commit day is an activity signal, not throughput: count accepted work instead
Jang Young Jeong audits HoYeon Lee's report of a Software Factory that produced 300 commits a day over the Chuseok holiday. The workflow is real and thoughtful, with tasks carrying intent and a verification method, human review at key points, and an Observer agent, but the post reports no merged, accepted, reworked or rolled-back counts. Jeong's fix is an acceptance ledger.
-
Spawning sub-agents mid-task costs 1.4–3.3× the tokens and rarely saves time; it pays on long-horizon work and independent checks
Oct 7 Li, Li, Jiang and Lou: 2,124 matched runs of Codex, Claude Code and Kimi Code with dynamic concurrency on or off. Runtime rose in 14 of 15 agent-benchmark cells, Claude Code fell from 83% to 59% on SWE-bench Verified, and gains showed up only on the longest tasks. Shared-file and merge problems were the largest failure category at 33.2%.
-
JetBrains' Gleb Malkov: faster developers don't make a faster team
In an Oct 7 JetBrains Air post, Gleb Malkov reports discovery calls across agencies, telcos, game studios and consultancies: individuals get faster with agents, but team delivery stalls. Shared setups stay informal; a mapping company saw PR time go up; and overnight automations create a new ops burden of ownership, distribution and governance.
-
Agents doubled the PRs, and the extra review load went into approve buttons
Oct 7 Pankaj Nakhat reads the CMU/Stanford study of an enterprise 2x mandate: PRs per developer hit 2.09x while reviewers grew 1.5x, human review coverage fell from 89% to 68%, substantive review from about 39% to 21%, and silent approvals roughly doubled per reviewer. His fix is lanes at PR creation, narrow bots as gates, agent-supplied evidence and a WIP cap.
-
OpenHarnX: lock the tests so the agent can’t rewrite the contract
Oct 7 from Rupesh Poojary: OpenHarnX freezes the agreed test suite, verifies the locked copy in a sandbox the agent can’t reach, and issues check-only verdicts — execution rights vs settlement authority.
-
An operating system for coding agents: the disciplined build
Oct 7 Stack Overflow (series finale): agents are brilliant for an afternoon and mediocre for a quarter unless the harness supplies a constitution, autonomy boundary, tiered context, durable memory, multi-lens review panel, and post-edit hooks — then shared written state for fleets. The model is maybe a tenth of the outcome.
-
Cross-provider review as a contract: the reviewer harness counted three kinds of failure as clean reviews
Oct 7 Bowen Xu and Boyu Chen run a second coding agent from a different provider as an advisory reviewer. It found something material on 8 of 20 paired turns. Testing the harness turned up three ways a failed review was counted as a pass: cut-off input, cancellation while the process was shutting down, and a nonzero exit that still printed a tidy verdict. The last one was found after formal data collection had begun, so that data had to be set aside.
-
TestJack: about a third of 'passing' agent patches break the task they were meant to solve
Oct 7 Yao, Wang, Sen, Chen, Ray and Song audit 4,487 benchmark-passing trials from frontier coding agents across five benchmarks. 1,545 (34.4%) fail a new test that the reference solution passes, cutting the overall resolution rate from 50.6% to 33.2%. A cheaper sampling version finds 72.4% of those failures for 15.4% of the cost.
-
YATFA: one overnight ticket failed audit, hit merge conflicts twice, and still merged by 3 a.m. with no human
Oct 7, the YATFA team traces a single ticket through a night of unattended work: two writing agents, three reviewing agents, two platform bounces on a branch that went stale, and a merge at 03:00 UTC. Their argument is that the queue, not any one agent session, is what carries work through failure.
-
Spotify's Emma Bostian: faster code isn't faster delivery, so the author owns the change before review
In an Oct 6 LeadDev essay, Spotify engineering manager Emma Bostian argues that AI moved the bottleneck from writing code to reviewing it, that reviewers end up reconstructing the author's reasoning, and that engineers who launch several agents and switch tasks pay for it in rebuilt context. Her fix is author responsibility, a defined bar before review, and measuring review time and rework next to output.
-
440 factory PRs, 418 merged by a human click, and two bugs the green suite certified
Oct 6 Codecot factory sheet: a spec-driven agent loop with a rival-vendor reviewer opened 440 PRs over five months and ran a product for 19 days unattended. Its own defect list shows why a passing suite is not the same as a checked claim.
-
A five-minute cap on every CI job, main checked hourly, and agents merging 200 changes a day
Oct 6 CodeHerder: merges landed every two to five minutes while a full main pipeline took ten, so runs were cancelled before they finished. A per-job time cap, checks picked from the import graph, and hourly full suites whose failures become agent tasks took them from about 74 merges a day to over 200.
-
Reading caught none of the tests that couldn't fail. Breaking the code caught all of them
Oct 6 dsalathe, Greens That Prove Nothing: in an agent factory's findings ledger, the largest defect family is guards and tests that stayed green while asserting nothing. Seven of one shape got past writing, rereading and review, several by agents that had just read the postmortem, and fell within minutes to a deliberate mutation. The house rule is to show every guard failing before believing its pass.
-
When agents review agents, the checks become one more thing that can fail
Oct 6 Ferris and Bonilla: a six-month case study of a production healthcare platform built by coding agents and run by an operator with no formal software training. Reviewer agents, tests and monitors helped, but some measured proxies, one audit stopped without failing, missing checks vanished from pass rates, and an automated repair caused an outage.
-
GitHub is rebuilding Git storage for agent fleets: commits up 5x, merges nearly 4x, all landing on one ref
Oct 6 GitHub engineering blog (Brian Celenza): September saw 7.38 billion commits, more than five times a year earlier, with pushes up 4.9x and pull request merges nearly 4x. Agents that checkpoint after every action make push latency their speed limit, and merge queues funnel the work onto a single reference. GitHub is separating durable storage from serving compute and says branch protections and required reviews stay intact.
-
GitHub ships stacked PRs to everyone: small reviewable pieces that land through the merge queue as one group
Oct 6 GitHub made stacked pull requests generally available on all plans. Stacks now enter the merge queue as a single group, auto-merge for whole stacks is rolling out, and the gh stack CLI supports worktrees. GitHub reports 9% more merged code in repos using stacks during the preview.
-
Humanize: the reviewer decides done, but nobody designed when to stop
Oct 6 paper from NVIDIA, UCLA, MIT and Tsinghua on a Claude-builds, Codex-reviews loop with 72 mechanical gates. 118 public postmortems: builders over-claimed in 45, and two thirds of rounds came after the work was already accepted.
-
Twill's two-engineer factory: guardrails cut hotfixes to one a month and tripled time-to-merge
Oct 6 Cate Huston, Twill: PR volume rose 24× from April to September while features fell from 66% to 16% of PRs and guardrail work rose to 36%. Same-day hotfixes fell from about ten to one; median PR open time went from 1.2 to 3.5 hours.
-
Reviewing agent code is a question of where to trust it, and the diff view can't answer that
Oct 6 JetBrains Research with Lund University: from workshops with 17 practitioners and a 43-person survey, a framework for reviewing AI-generated changes that starts with an overview, ranks files by risk before anyone reads a line, and only then drills into code. Agents present every line with the same confidence, so reviewers need tools that tell them where to look hard.
-
Five agent PRs a day is a queue: review in merge order, and run fewer agents when it outgrows readers
Oct 6 Joxo: when several people's agents open PRs the same day, sort by dependency rather than arrival, review shared-file PRs together and rebase after each merge, let a fresh second agent check first, and keep the merge decision with a person. If the queue grows faster than people read, a reviewer agent won't fix it.
-
Tessl's Dru Knox: build agent autonomy before you stop reviewing, and expect the improvement work to lose to deadlines
In an Oct 6 AI Engineer talk, Tessl's Dru Knox splits factory progress into autonomy (how often humans correct the agent), automation (how much ships without review) and quality, in that order. He says the loop work that drives them is unplanned and keeps losing to feature deadlines, and proposes manual takeovers, human PR comments and agent-started PRs as the scoreboard.
-
Ouroboros: the reviewer approved a destructive rewrite, so a ruler replaced it
Oct 6 field report from Daniel Krydynski: an unattended scout/implement/review loop let a 4B reviewer approve gutting INSTALL.md from ~256 lines to 36. The fix was a deterministic deletion cap, not a better prompt.
-
Németh: same outcome, not the same output — make the merge gate deterministic
Oct 6 from Máté Németh: LLMs stay non-deterministic, so his autonomous loop aims for the same landing conditions, not the same diff. 653 rounds since August, 108 PRs merged by plain code since Sep 29, refusals pinned to commits, and a one-way guardrail ratchet that still needs consolidation.
-
An overnight loop reads an outcome file it cannot edit, and a separate verifier decides whether it succeeded
Michael Quoc of Product.ai and the security writer ToxSec on running agent loops for hours unattended: a fixed outcome file with result, limits, tie-breaker and check, a read-only verifier that grades every finished chunk, restarts from the last unit that passed, and human judgment moved to the two ends of the run.
-
A two-week self-healing loop opened 152 PRs with no human-written code; the 80 merged each came with the bug reproduced and shown gone
Replay's pilot chained Fullstory (detect), Replay QA (diagnose and verify) and Obvious (fix) on its own repo, with an engineer's merge as the only human step: 124 distinct bugs, 76 fixed on main at about 5.4 a day, and the argument that proof, not generation, made the PRs mergeable.
-
150 agents, 20 billion tokens a day, and auto-merge only where the tests can prove it
Oct 6 SignalFire, from a closed CTO show-and-tell: the teams running the biggest agent fleets built almost nothing to generate code faster. Their machinery feeds the agents and checks the output, auto-merge is allowed only when a failing regression test can prove the fix, and reviewing 40 parallel outputs is now an interface problem.
-
Thousands of green unit tests, and the release boundary still caught the bugs
Oct 6 Paul Stack: when one agent writes the code and its tests, moving tests up or down the pyramid doesn't make them independent. What matters is where the expectations come from and who is allowed to change them, so product-level checks have to be defined and protected outside the implementation loop.
-
HarnessSecurity-Bench: auto-approve lifts attack success from 29% to 96% across six coding harnesses
Zhu et al. run 2,500 trials on Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and Copilot. Auto-approve raises utility and nearly triples attack success; allowlists and denylists cut attacks with far less utility damage than network isolation.
-
What the agentic SDLC looks like when the typing is cheap
Oct 5 Luke Chadwick (Far Horizons): always-on Mac mini with ~eight tmux sessions, multi-model plan review before build, a verification gauntlet the agent can't talk past, and scheduled Pulse/Hygiene skills that re-check 'done' and open tickets for drift. Spec and verification absorb the time writing used to.
-
Uber's software factory runs cleanup on weekends and caps what lands on Monday
Oct 5 Adam Faik walks Uber's factory through 70+ public sources. Engineers' first review grew from 3 hours in 2024 to 9 in 2026. The useful details are the shape of the response: a flag-cleanup loop that runs when CI is idle but limits how many diffs reach engineers, checks moved before CI, and a PR that arrives with a table of everything it passed.
-
GitHub's ReviewBench grades review agents on 219 real PRs, and gives credit for bugs nobody had listed
Oct 5 GitHub launched ReviewBench, an open benchmark of 219 pull requests sampled to match 103.9 million real ones. It scores review agents on precision and recall by severity, gives credit for valid findings outside the answer key, and its offline results pointed the same way as a production Copilot review experiment.
-
980 merged PRs in seven weeks from a personal agent fleet, and a warning to delete the phone app
Oct 5 Andrew Kesterson devlog: a self-hosted factory of six agent identities polls a Forgejo forge, grooms tickets until a checker script calls them ready, and works them while he sleeps. About 980 PRs merged across 38 repos since mid-August, with 73 times his earlier lines per week. His own read: the bottleneck moved from typing to reviewing, and the always-on loop was not entirely healthy for the human.
-
Your agent capacity is your verified-throughput rate, not your seat count
Oct 5 Law Zava: measure offered review demand in review-minutes against effective reviewer capacity, tier review by risk before capping anything, and when the cap binds, put agents to work shrinking reviewer minutes rather than adding submissions.
-
Meta, 400K diffs: AI code gets reviewed faster and causes fewer outages, but mostly because of where it lands
An ESEM 2026 paper from Meta traced AI-suggested code character by character through review and landing across 10K+ developers and 400K diffs. AI code was in 52% of diffs and about 8% of landed bytes. It was associated with roughly 8% shorter review time per unit of log AI fraction, but it lands disproportionately in less central code, test files and larger diffs, so naive with/without comparisons mislead.
-
The answerability budget: agents can check their work, but someone still has to answer for it
Oct 5 Adam Monago on Lauren Tan's 2,462 changes in a month: verification loops make checking cheap enough to stop being the ceiling, but the number of changes a named person can honestly stand behind grows on a human schedule, not a model one.
-
NP-Bench: parallel coding agents each pass their own tests and still break the merge, so schedule scopes up front
Oct 5 Sumanyu Muku (Amira Learning): partitioning declared scopes and ordering merges along the dependency graph lifts clean integration from 1/9 to 9/9 simulated scenarios. On a live contract rename it goes from 0/15 to 15/15 on Opus 4.8, where reactive conflict warnings stay at zero.
-
Dim Factory: decide what an agent may do by how you start it, not by its prompt
Christoffer Niska (Oct 5) turned his plan, build and review skills into dim-factory, which runs each step as a separate locked-down Claude Code worker. A station is done when the factory says so, a single append-only log overrules git, and a commit is kept only if it is one new commit, leaves the test script alone, and passes the project's check on a clean copy. He has not yet measured whether it saves him time.
-
Most software factories went dark because the reviewer ran out of time
Oct 5 Marc Nützel: a dark factory is supposed to replace the human reader with specs, harnesses and gates that fail loudly. The data he lines up says many teams got there by accident. Agent PRs mostly go unreviewed, AI PRs merge a third as often, and the gap is set by what surrounds the agent.
-
When an autonomous coding agent fails, it is almost never the model
Oct 5 Bob Odell: audit of 14 unsupervised local-lane tasks — ~36% first-pass success, five rescues, one 11-hour freeze, one prod revert — and zero failures at a model-capability ceiling. Deadlocked test gates, scope leaks, blind green checks, and broken recovery paths were the seams. Prefer gates over prompts.
-
Orbi's reviewer said "pass, no findings" on two regressions. The harness that fixed it doubled the cost of ordinary bugs.
Oct 5 Lawrence Liu (Orbi): an issue-to-merged-PR loop shipped two regressions its own review passed. A night of graded reruns found the fixes were generated inputs and an oracle independent of the patch, then found the stronger harness cost about twice as much and caught nothing extra on nine held-out bugs.
-
At 2,500 PRs a month, Lauren Tan stops reviewing every dish and tends the kitchen instead
Oct 5 Barnaby Robson maps pstack's six mental models. The review-relevant one is the gardener: an agent queues smells without fixing them, she groups the queue every few days, samples merged PRs daily, and changes lint rules, types and skills when several agents take the same shortcut.
-
A green routine run means the session exited, not that the job got done
Oct 5 Routines Runbook: Anthropic's own docs say a green status on a Claude Code routine only means the session started and exited without an infrastructure error. Three ways unattended runs fail while reporting success, and an eight-point pass that makes an unreadable source look different from an empty one.
-
Lauren Tan's overnight merges need a clean verifier verdict on the exact patch
Oct 5 Ewan Mak reads the pstack repo behind Lauren Tan's 2,500-PRs-a-month claim: the agent that wrote a PR never merges on its own verdict, a swarm of fresh verifiers must clear the exact head, and CI must pass on a freshly rebased branch. Most of the volume is maintenance, and the approach stops where changes are not reversible.
-
Build the undo button before the tenth guardrail: per-task git checkpoints for overnight agents
Oct 5 yureki_lab: after one bad 'cleanup' task poisoned four later tasks overnight, every task now gets a full working-tree snapshot on a hidden git ref, automatic rollback when verification fails, one commit per passing task with a Task-Id trailer, and a permission rule sorted by whether an action can be taken back.
-
GetPullRequest: its agent factory marked a task done with no diff, and every automated signal agreed
A team that has built its product with coding agents since May keeps people at three stations: writing the task, answering the agent, and reading the diff before merge. Its commit history shows why, starting with a check that shared the worker's blind spot.
-
AFCRA: when an AI reviewer explains what's wrong, attackers revise the code until it approves — and keep a working exploit
Oct 4, Gong, Zhang, Jahangirova and Wang (KCL / Bristol) show feedback-guided code revision beats text-only PR attacks on AI reviewers: 20.8% success vs Sonnet 5 and 15.7% vs GPT-5.5 across 159 CVE PRs, rising to 44% when the attacker can also edit PR text and replies.
-
Backlog Loop: a mechanical gate kept the code honest, but nothing checked the factory itself
Oct 4 Emmanuel Goossaert: nine weeks and 421 commits building a seven-stage factory, with tests that must fail first, OS-level locks on test files, and a hermetic verify. Then he stopped reading code, let it build itself, and found zero evidence it beat a plain Claude Code session.
-
562 merged PRs in two weeks by reviewing the plan, not the code
Oct 4 Alexey Indeev, on his company's internal operations platform: one coordinator agent per goal, sub-agents doing the work, auto-merge behind tests, bots and a merge queue, and goals left running overnight for days. He reads progress reports instead of diffs, and the new limit is tokens.
-
562 PRs in two weeks: review the plan and the progress report, let guardrails own the merge
Oct 4 Alexey Indeev (Spare): coordinator-plus-sub-agent goals that run overnight and for days, auto-merge in small flagged pieces behind tests, Bugbot, Strix, and a Mergify queue — and tokens, not review, become the bottleneck.
-
Agents made CI the bottleneck, but a faster pipeline still checks the wrong thing
Oct 4 Arjun Iyer (Signadot): Anthropic's CI jobs grew 25x in six months and Linear's suite nearly quadrupled, yet faster CI still verifies a repo, not the distributed system. He argues verification moved from a human gate to an automated one without anyone asking what the gate checks.
-
Teamscale: the bottleneck moved from writing code to trusting it, and deterministic checks pay the verification tax
Teamscale's Sven Amann cites Faros data showing review time up 441.5% as AI adoption rose. A plausible-but-wrong agent investigation on a 2.2M-line codebase was settled by system tests run against the real implementation, not by closer reading or a second model.
-
Eight weeks, 866 agent-written PRs: size predicted the reviewer's flags, the model mostly didn't
Oct 4 Zest (Ilya Volodarsky): every agent PR in two repos carries a self-written session block. Median 37 minutes from open to merge. Matched on repo, engineer and week, the authoring model barely moved the AI reviewer's should-fix rate, while 1,000-line PRs were flagged 36 points more often than sub-300-line ones.
-
An evidence gate where the PR merges in twelve minutes because the review already happened
Oct 3 Associates AI: every AI-written PR must bring a numbered spec, a test seen failing then passing, a fresh-context review, green checks on the head commit, and a stated blast radius and rollback. One fix took twenty hours from issue to merge but twelve minutes from opening the PR. The gate still leaks, and the post lists where, including a health metric that stayed green while an agent answered nothing for days.
-
Twenty-one green fixes from an agent fleet, and no way to tell if any were real
Oct 3 Matt Cockayne: an architecture review of go-tool-base produced 21 critical and high findings, so a fleet of agents took one each. All 21 merge requests came back green, and he couldn't tell a working fix from a test that could never fail. The fix was to change the shape of review, not its intensity: every agent had to show its test failing on untouched main first, with sibling tests passing as a control.
-
Double the PRs, ten times the wait: the queueing math behind agent review delays
Oct 3 FlowVerify: agent PRs wait 17.6 hours at p75 for a first review against 3.4 for unassisted work, yet review itself is only modestly slower once started. A simple queue model shows why, and why bursty agent batches make it worse.
-
What a coding harness buys: mostly a different bill, not a higher pass rate
Liu and Han (Shandong University, arXiv 2610.04433) hold five models fixed and swap the harness across Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified. Swapping harnesses flips 13% of hard tasks, the same as rerunning one harness. Cost per task moves up to 3x, set mostly by the preamble each harness resends on every step.
-
654 PRs merged in one night, and the bill came due in verification
Oct 3 Saagar Patel: one prompt at 9:56 PM, a ten-hour fleet run across 269 projects, 654 pull requests merged in the window. Every merge needed an independent review of the exact commit, green required checks, and proof the merged tree matched the reviewed one. Writing was cheap; proof was the cost.
-
Three AI passes and a stamp: the human reviewer became a proxy for another AI review
Oct 2 Dan Apushkinsky: an AI-written PR passed an AI reviewer and a human who pasted it into AI, missing a bug flagged by a comment fifty lines up — correlated passes are one opinion repeated; make review a credited, estimated ticket.
-
Ask HN: reviewing agent code takes 5x longer, and the answers split between a strong harness and small attended chunks
Oct 2 Ask HN from ruffrey: a senior engineer says Claude merge requests take five times longer to review and they barely understand what they approve. The replies split three ways: stop reading code and invest in the harness, keep the agent on short leashes you can review in minutes, or tier by how much the code matters.
-
Guarded commits: put the human approval inside the workflow transaction
Oct 2 Bindschaedler et al. (MPI-SWS / MIT / Purdue): approval as ledger state + credential-confined commit adapter; four-condition predicate; 271k traces replay; docs-only bypass rule doubles W-Tran disagreement—ledger attributes it.
-
A round-the-clock agent factory: the ceiling is usage limits, not rules
Cabinlytech (Oct 2) runs a standing team of Claude Code builders plus one advisor agent that merges and deploys, all day and night. Each ticket carries its own failing and passing commits, a red ticket is dropped from the release train rather than debugged inside it, and the advisor reads the running commit back from the server before closing anything. Their verdict: more rules do not make it faster; model usage limits and disk do.
-
40 agent PRs merged in a median 2.4 minutes. The path rule was the reviewer.
DispatchSEO counts its own unattended agent pipeline: 40 merged PRs, nobody in the loop for the typical one, 4 held for a human by a path allowlist, and a review check that only reported on 15 of 40.
-
Generative TDD: a separate tester writes new inputs after each candidate is fixed
Oct 2 Kato: coding agents fit the tests they see, so GTDD has a separate tester generate fresh inputs after each committed candidate, a trusted evaluator return reduced counterexamples as regression tests, and fresh random audits decide acceptance.
-
A software factory small enough to understand: every station must be able to say no
Oct 2 LaunchDarkly (Alexis Roberson): a seven-station factory built from GitHub Actions and a headless agent, where the builder is never told about its reviewer, the default token silently skips the factory's own CI, and a post-merge smoke check opens the revert.
-
RL that gates regressions teaches agents to finish work—and it transfers
Oct 2 Mehta/Ritchie/Chen (Surge): one epoch of GSPO on 1,700 expert coding tasks with partial-credit reward zeroed on any pass-to-pass failure lifts Kimi K2.7 Code on all six external benches (+4.7–20.0 pp), shorter trajectories, four last-mile habits.
-
Dark factory at $0.015/task: models write, scripts decide
Oct 2 Nalivator3000: solo Archon/Paperclip/LiteLLM dark factory — ~$0.015/task, ≤4-file cuts at 40–50% one-shot; silent credit fallback, alert-to-agent trap, 14-day improver cooldown, script-owned merge.
-
Agent performance PRs merge less often, and half of 'validated' ones lack numbers
Peng, Calvo, Kalu and Davis (Purdue) study 1,130 agent and 1,130 human performance PRs through June 2026. Agentic PRs merge 54.4% vs 73.3% for humans, use similar optimization strategies, and report validation at comparable rates — but lean more on static reasoning than benchmarks, and about half of validated PRs in both groups report no quantitative performance metric.
-
Done means it runs: a three-stage completion gate that watches the deliverable
Oct 2 Qualiteg field note: five ways model “Done” slipped past (syntax-only, assert-free tests, listen-but-crash, 2/22 APIs); harness starts the app, hits every specified route, and inspects test bodies—pushbacks carry failing lines and how to investigate, never the answer.
-
Open-source MAS issues: orchestration fails before the model does
Oct 2 Rehman et al.: 944 filtered GitHub issues across 21 LLM multi-agent repos—Orchestration & Execution is 22%, Memory & State 14.7%; Workflow/Tool/Memory problems dominate causes; Optimize Workflow leads fixes.
-
When a second model helps: complementary coverage, not free F1
Oct 2 Song: 900 review sessions; CCR and top-tier cross-model review share Jaccard 41.2%; one CCR + one XMR matches 56.7% planted errors vs 42.7% for two CCR reviews; lightweight cross-model reviewers add nothing.
-
Correct final numbers, wrong process: continuous evaluation for evolving agent skills
Oct 2 Vo, Doshi, Sheinin (IBM): 240 trials; 162/175 runs that pass final numerical checks still fail a process check (92.6%); template regression suites + live-API ground truth across specs, harnesses, and models.
-
VERSE: a self-improving harness optimizer only improves when it can test its own edits
Oct 2 Wang et al.: letting a harness optimizer rewrite its own prompts and tools made things worse until it could run draft edits against real tasks; with execution checks it learned to add regression tests, and its edits still broke nearly as many tasks as they fixed.
-
Agents are systems, not models: configuration axes beat backbone swaps
Oct 2 Wiedmann et al.: 8,640 main runs on 4 scientific coding tasks; ~54% score variance is run-to-run noise; Information outranks Model and Budget; prompted self-verify barely moves behavior—oracle verification tool triples it.
-
MCRI: score agent skills before you burn the execution budget
Oct 2 Yang et al.: MCRI (Metadata/Constraints/Resources/Instructions) on 63,812 OpenClaw skills; 58,275 skill-conditioned runs; top-1 selection lifts +17.7 / +22.8 / +19.6 percentile points vs strongest baselines on BigCodeBench / BFCL / Mind2Web.
-
A 24/7 agent kept repeating 38% of its failures. A replay-gated rule loop cut that to 6%.
Oct 2 yureki_lab: an always-on Claude Code system reviewed each morning. Failed runs write a believed-vs-actual incident file, a separate agent distills rules, and a rule enters the playbook only if it fixes a replay that failed without it and breaks nothing else. Capped at 60 rules.
-
Safety must survive self-improvement: current gates beat historical scores
Oct 2 Zhang et al.: RSI authorization testbed; historical scores keep unsafe programs in 22/48 framework histories despite correct alternatives; score refresh + validated rollback restore correctness while retaining >43% deployment savings.
-
Contra: ask only when two answers would change the program
Oct 1 Fang et al. (Peking): training-free selective clarification discovers candidates, filters relevance/resolution, then keeps questions only when programs under alternate answers disagree on shared inputs—beats harnesses and baselines on ClarifyCodeBench.
-
Groundability: weak reviewers catch strong agents when the check is independent
Oct 1 Guo et al.: 411 execution-labeled traces; official evidence lifts weak reviewers (5/6 improve catch and cut over-rejection); deployable cascade recovers catch 0.76–0.80 but over-rejects ~0.66—decisive checks beat bigger judges.
-
Global coherence: every agent can be right and the team still wrong
Oct 1 Heng: observation-aliasing bound + shared-budget TeamBench—showing the live count still overspends 4/5; commit ownership or escrow hits 0/5. Models propose; the harness owns distinctions and commit.
-
Model–harness fit: rankings reverse when the scaffold changes
Oct 1 Li et al.: 66 configurations × three terminal benchmarks; Claude leads GPT by 7.9 pts in OpenHands on TB4 but trails by 30.2 in PI; best harness flips for 4/5 models—evaluate the pairing, not the model alone.
-
Németh: after the loop left Claude Code, plain code does the merge, and the gate refused two good PRs in two days
Oct 1 from Máté Németh: moving his autonomous loop to a claude -p runtime took the merge away from agents. The runtime re-tests the exact commit in a fresh checkout. Background commands cut off 4 of the first 6 agents, and a busy machine and a stale check run caused two false refusals.
-
Orbi: 599 issues merged by its own loop, and a blocked label the merge gate never read
Oct 1 (updated Oct 9) from Lawrence Liu: an issue-to-release loop that delivers its own repo merged a PR 71 seconds after a human labelled it ai-blocked, because the gate re-read CI but not labels. Also: headless claude -p exits 0 when every write was denied.
-
ActiveSaddler: the harness curriculum co-evolves—which failures get the next rollout
Oct 1 Park et al. (Microsoft/POSTECH/KAIST): ActiveSaddler treats scenario selection for harness optimization as a non-stationary bandit over failure-pattern arms; +4.4 / +7.5 pp Pass@1 over the same optimizer with a fixed scenario order.
-
Half a GOTO room named review as its worst bottleneck, and a green check that ran nothing is worse than none
Oct 1 QA.tech: at Nathen Harvey's GOTO Copenhagen keynote, half of 440 engineers picked reviewing changes as their biggest SDLC friction and 3% picked building. One team went from about 15 PRs per release to 150-200 with no new reviewers. The proposed split: invariants, a rules-loaded review agent, behaviour checks with an explicit unverified state, then a shorter human question.
-
Mingbird: small open models fail the harness, not the weights
Oct 1 Wang & Huang: local-first harness with ten mechanisms (byte-level net-zero prefill, finish gate re-reads task, signature loop detection); LRAB 0.886 vs goose 0.631 / opencode 0.479 / agent-mini 0.405 on 288 published cells.
-
Harness Annealing: teach the model the control the harness used to own
Oct 1 Yang, Chai & Wen: HAT anneals teacher trajectories under weaker harnesses so the model internalizes what to investigate, revise, and stop; tools-alone annealed ckpts ≈ full-harness starters on SWE-QA (+5.01 / +3.09 H₁ for 9B H₂-targeted).
-
AutoCompact: long-horizon coding agents learn when to compact—and what to keep
Oct 1 Zhang et al.: SFT+RL teaches when to compact, what working state to keep, and how to continue; +9.2 / +5.0 abs pass on SWE-bench Verified / SWE-PolyBench; 256K never overflows, 16K shares length-triggered fallback.
-
VeriHarness: resolve disagreement, challenge consensus—same model, environmental evidence
Oct 1 Zhang et al. (Google Cloud AI Research): same-model agentic verifier with disagreement resolver + consensus challenger; +6.2/+6.4 over single rollout; verification skills self-evolve from failure feedback.
-
Do agent benchmarks do what they say? Audit the tool contract under the score
Sep 30 Bellibatlu/Wang/Zhang: executable contracts over 34 mutating tools in four benchmarks find seven tool defects + one ungrounded grader; MedAgentBench writes report success with no write; on 1,120 constructed paths tau2’s evaluator rewards the telecom defect and fails its repair.
-
How much harness does a strong MLE agent still need?
Sep 30 Brilliantov/Hernández-Cano/Abbé: under matched backbone, hardware, and 24h budget, a minimal single-session coding agent (Malena) matches or beats elaborate open-source MLE harnesses—coding-agent access dominates further scaffolding.
-
Adaptive-GEPA: evolve the router and the specialist programs together
Sep 30 Chen/Zhang/Wang et al.: reflective search jointly learns request division + specialist programs on a mixed workload—family-mean 52.6→70.6 vs 62.5 for single-program GEPA at ~18k scored calls.
-
From verification failures to a reusable kit: procedures beat one-shot proofs
Sep 30 Chen et al.: K semantics + verification kit; HumanEval 164/164 Pass after two repairs; audits catch 12/12 seeded defects that still prove; KleverBench kit vs rules/advice mixed (± intervals include 0).
-
AgentBug-Smith: turn harness bugs in the wild into a live repair factory
Sep 30 Cheng et al.: automate reproducing real harness bugs from open-source agents into Live-Harness-Bench (200 executable bugs); +10.67–27.56 pp reproduction vs SWE-Factory/SWE-bench-Live; distilled repair skills lift harness-bug fix rate +6.32%.
-
Agent review harness: every defect is also a harness defect
Sep 30 nick-delgado: review-agent-pr → address-pr-review → improve-agent-process — parallel standards/smell/spec/test reviewers, falsifying verifier, root-cause log on a tracking issue that batches process PRs.
-
Better deck or different judge? Harness gains that move when the grader blinks
Sep 30 Gibert/Despujols/Rochet: CIB deck harness beats same-model direct generation by 20.4–33.6 pts (text-only), but repeated LLM judges shift identical decks ~8 pts MDC—progress across rounds can be indistinguishable from grader noise.
-
An overnight agent PR had eleven green tests, and retries never happened in production
Casey Holt approved a tidy background-agent PR adding webhook retries. Three weeks later a customer found zero retries: the config key sat where the loader never read it, and every test built the policy by hand. Holt now reads the task first, the tests second, and runs every agent change once with real configuration.
-
Mid-Harness: verify the action before the environment commits
Sep 30 Kang et al.: sample N candidate terminal actions and verify at the model–harness boundary—GPT-5.6 Sol verifier lifts TMAX-9B Pass@1 50→68% on TerminalBench-Lite; pairwise + distillation recover part of the gap without retraining the generator.
-
ContextRender: keep the tool results your long run actually reuses
Sep 30 Kashmira/Dantanarayana/Tang/Mars: persistent execution-dependency graph + observed reuse; at 6K history budget, matches/beats full history on AppWorld & 8-obj QA while cutting inference cost 10–32%.
-
Resolution first, then fewer tokens: HERO’s hierarchical RL for coding agents
Sep 30 Li/Jiang/Huang/Yang/Peng: HERO gates efficiency until the rollout group can resolve, then cuts unproductive low-entropy turns—on Qwen3.5, better resolve than GRPO with ~40% less token use.
-
VeriWeave: authorize agent side effects with typed evidence, not the planner’s word
Sep 30 Mohsenzadegan/Tavakkoli/Kyamakya: deterministic deny>review>allow governance; GovernBench 60k cases → 0.9888 accuracy, zero aggregate false allows / GASR; remove evidence gate → FAR 0.2873, GASR 0.5715.
-
Plan declaration–execution gap: declaring a plan is not executing it
Sep 30 Oota et al.: Plan+ReAct preserves declared structure on only 22–45% of trajectories (worse as plans lengthen); pattern-specific executors lift ALFWorld 0.48→0.92 and SWE-bench Verified 0.36→0.44; mode selection still trails best fixed pattern.
-
CRJudgeBench: plausible review comments still need a repository-grounded judge
Sep 30 Pan et al.: 1,199 PR review comments (764 trustworthy / 435 untrustworthy); frontier LLMs miss most bad comments (GLM-5.3 negative recall 20.77%); Sentinel evidence-seeking judge reaches 76.60% accuracy (+19.78 pp over base, +6.13 over GLM-5.3).
-
AI software factory: coding speed alone hits Amdahl’s wall—automate the whole SDLC
Sep 30 Pavlenko et al. (Microsoft): Hotspan targeting + Darwin evolutionary swarms on ECTs; Autoperf 0.35 PRs/engineer-hour (3× agentic coding); Darwin up to 22× token-efficient vs OpenEvolve; OSS top-20 avg +13.8%.
-
LoLBench: peak agents resolve 14% of proposal-to-PR work on million-LoC systems
Sep 30 Peng et al.: 100 enhancement-proposal tasks across 29 large systems (~2.4M LoC, ~5.5k-LoC PRs); best of 28 agents hits 14% resolved / 52.7% F2P; localization dominates; file-tree+API specs lift resolved +16–22pp.
-
Overnight optimization: lock the verify harness, then let the agent grind for hours
Sep 30 Poličar/Špendl/Hočevar: humans set scope + correctness + verify; Claude Code overnight on openTSNE (4.4–7.7×), ssGSEA (136×), and a new graphlet algorithm (~160× vs Orca).
-
Merged, not measured: agent performance PRs pass on reputation, not speed-ups
Sep 30 Qi et al.: of 1,262 AIDev performance fixes, 57% merge; acceptance tracks repo history with the agent (33%→84%) more than coded content; only 11% carry a performance test; of 30 re-executed merges, 18 deliver, 9 show no gain/regress, 14 change untested inputs.
-
WitnessGym: a bug report is not done until an executable witness fails
Sep 30 Qi/Xu/Huang/Shang/Wang: inject into test-reached paths → 1,300 Java cases; best agent (Codex+GPT-5.4) averages 68.9% witness success; long contexts 80%→52%, depth-10 transforms 80%→59%.
-
Meta-skills: teach the Builder to provision harnesses, not the Target more tips
Sep 30 Qian/Zhu/Li/Wang/Ji: Builder learns when/provide/use meta-skills from Target feedback; full-bank +8.95pp over no-skill construction, +12.02pp over handing the same bank to the Target; same-model self-improve +18.71pp.
-
Self-spec verification: oracle contracts flatter overnight factories
Sep 30 Qian et al.: VeriCodeBench 400 self-spec problems (C/Java/Rust/Python); Direct joint success ~44% vs ~71% code validity; CodeNova lifts aggregate joint to ~49% (Claude Sonnet 5 65.5%); richer specs can cut validity while raising coverage.
-
Independent verification paths that share definitions are not independent
Sep 30 Rovai case study: dual Python/SPARQL redundancy gate printed ALL CROSS-CHECKS AGREE while 3/7 published counts were wrong (disposition 932→220); 72/75 LLM “independent” paths still returned the defective 932.
-
Aletheia: prove a coding-agent rule’s permission is dispensable by removing it
Sep 30 Shi et al.: leave-one-out sandbox ablations under the same rule+task; witnesses = both runs pass tests under strictly less authority; 314/314 AIShellJack attacks detected, 3.75% FP on benign GHAgentFiles.
-
EngramBench: skill banks cut thrash, but the last mile still needs the model
Sep 30 Tan/Gu/Li et al.: capability-overlap-without-solution-overlap bench for skill-evolution harnesses—static skill banks slash multi-hour coding time >55% as an execution compass, without magically bypassing exact implementation.
-
HealBench: agents can resume repo crashes—HealGuard shows passing tests can still be unsafe
Sep 30 Tan et al.: 265 repo runtime errors; best Codex+GPT-5.6-Terra PR 38.11% / CR 28.68%; HealGuard flags 17.4% of passing heals; 100% recall on 342 unsafe cases at 68.42% FPR.
-
Code that works, environments that don’t: the three-layer dependency gap
Sep 30 Vangala & Malik (Missouri): 1000 agent runs across Claude/Codex/Gemini and four languages show functional green while declared, installed, and runtime-traced dependencies systematically disagree—SBOM-grade manifests are not trustworthy without independent tracing.
-
Box²-Bench: good harness workflows help—misleading ones still bind
Sep 30 Wang et al.: hold model+task fixed, vary workflow reliability; frontiers use good guidance yet drop under bad/mixed; SFT on bad workflows hardens override, RL restores selective use of held-out good ones.
-
Champion–Challenger: harness evolution needs rollout scale and a promotion gate
Sep 30 Wei, Tan, Wang: optimizer-agent harness search; B=5→100 lifts held-out 47%→67%; unconditional accept drifts down in ≤10 rounds; Champion–Challenger raises held-out 51%→67% over 30 rounds.
-
More programs or more rolls? Harness diversity that only finds weaknesses
Sep 30 Xu et al.: on MATH-500, identical baseline copies already give 2.16pp oracle headroom; generated harnesses show persistent losses on 100 tasks vs 1 win; frozen selector +0.00pp; at 27 executions portfolio ties clones at 98.70%.
-
GitHarness: version the requirement with the work, then branch the update
Sep 30 Yang/Jiang/Liu et al.: Git-style requirement↔work snapshots; Git Agent picks compatible historical state; +wins in 28/30 backbone–harness settings; Code tokens −73.6% vs native while resolved rate rises.
-
ScholarEvolve: evolve the harness from the literature, not only from last night’s failures
Sep 30 Yang/Lai/Wang/Harari/Gabrilovich/Chang: research-guided module mutation + crossover around a fixed backbone—Qwen3.5-27B AppWorld Challenge TGC 49.6→63.6%; GPT-5.4-mini τ² Telecom pass@1 72.7→81.9%; Meta Harness falls short under the same search budget.
-
Trust is not a score: Runtime Assurance Contracts fail-close authority for coding agents
Sep 30 Zabolotnii: RAC binds gates→authority transitions; score-only admits 80/100 block-required injections at vendor weights; conjunction admits 0; score matches only when τ≤min weight.
-
Turbo Harness: recycle the search exhaust into a per-instance patch
Sep 30 Zhang/Wang/Xu/Metaxas: a small RL editor adapts a globally optimized harness per task from the outer-loop playbook—SWE-smith-MR 50.7→64.0 (Haiku) and 70.7→88.0 (Gemini), often with fewer steps.
-
HiSentinel: intervene before the bad action executes
Sep 30 Zhao/Li/Zhang/Du: hindsight-distilled 0.6B–1.7B sentinels choose Allow / Redirect / Hard-Pause pre-execution—up to +14pp on SWE-bench Verified Mini and +10pp on Ask or Assume.
-
How many coding agents to run? About as many as your reviewers can check
Sep 29 (revised Oct 1) Tom Baldwin, Polymorphism: a sizing model for agent fleets built on contention, coordination and rework. Rule of thumb: up to five agents per product and no more than reviewers can check. In a first trial, even a 21-second model reviewer became the bottleneck at ten agents.
-
RASO: borrow skills from other harnesses—adapt them, then validation-gate the update
Sep 29 Chu/Lee/Park et al.: retrieve GitSkills sections, Cross-Harness Adaptation into target tools; RASI init without rollouts + RASU feedback updates; beats SkillOpt/WikiSkill/GEPA on OfficeQA/Spreadsheet/ALFWorld/WebShop.
-
Codesteward: STEWARD.md from the base branch, graph-grounded specialists, deterministic judge
Codesteward agentic review: policy loads only from base branch; specialists + verifier + non-model judge; degraded graph fails closed; silence returns a machine-readable why-empty rationale.
-
Hearsay: a harness’s own record lets you name the fault, not prove the run
Sep 29 Dai et al.: across 16 frameworks no evidentiary record; examiners attribute faults 74–91% but <10% of citations are independently witnessed; only an outside second writer catches omissions/fabrications a self-hash-chain misses.
-
E2E-SWE: whole-repo generation is measurable when tasks are actually solvable
Sep 29 Ding et al. (Meta): 186 empty-workspace tasks × 11 languages; pass@1 11.7–67.7% across 13 models; iterative static+rollout audits flip only 2.2% of outcomes; front-loaded planning peaks in first ~4–12 turns.
-
Mixture of self-improving branches: stop evolving one harness trajectory into a local optimum
Sep 29 Dong et al.: branch harness search with evolving development subsets + proposal policies, then route among branch heads — relative gains over Meta-Harness of 34.8% Olympiad math, 11.6% Terminal-Bench 2.0, 3.8% SWE-bench Lite (66.0% resolution) with development-only selection.
-
Harness evolution as learning: why more memory and more update rounds still leave an oracle gap
Sep 29 Gan/Gong/Liu: AppWorld-P preference benchmark + approximation/generalization/optimization decomposition—context vs control reachability; memory length U-curve (~0.77→0.20 then rebound); ACE/TEPA/TRACE plateau ~0.4–0.5 vs oracle 0.071.
-
XRepoSkill: keep only rules that win across repos, then pick three for the issue
Sep 29 Guo/Zhou/Zhang et al.: contrast success vs fail on the same issue; executable predicates verify gain; retain rules only if ≥2 repos agree; DeepSWE +10.3pp over no-skill, +5.0pp over Trace2Skill.
-
OpenCollab: declared multi-agent orgs often aren’t realized—Adherence from 47% to 97%
Sep 29 Hsu et al.: shared runtime + Adherence audit; unconstrained teams realize collaboration ~47%; tool/budget/topology knobs push above 90%; Duo SOTA on Terminal-Bench 2.1 (83.15%) and DeepSWE (69.91%).
-
STITCH: don’t generate the harness at test time—select and compile validated primitives
Sep 29 Kuang et al.: Harness Primitives mined from failed trajectories + deterministic compilation—SWE-bench Verified Pass@1 73.0→80.5%; Terminal-Bench 2 60.0→72.2%; composition cost ~2.7% of an actor attempt (638× cheaper than codegen).
-
Frozen judges, moving agents: LLM-as-judge upgrades that execution cannot back
Sep 29 Li: fixed judges make version-dependent errors on coding-agent pairs (250 SWE-bench Verified); 8 judge-only intervals declare upgrades execution cannot; false accepts rise with capability; prefer paired audits over judge-only release.
-
WEFT: scale the whole agentic system—environment, task, harness, and evaluator
Sep 29 Mao/He/Wang et al.: execution-driven self-evolution + prefix-preserving sampling + MegaMCP; WEFT-14B beats Agent-World-14B by +6.41/+2.23/+12.27pp on BFCL V4 / τ²-Bench / Claw-Eval.
-
LibraryDesignBench: grade the library by what other agents write with it
Sep 29 Orlanski/Zhang/Trost et al.: 15 design tasks × 242 problems; score = correctness² × simplicity vs production-library references; Opus 5.5 48.9 vs production 46.6; 64% excess code from rigid/hard APIs, not missing features.
-
LongHarness: same model, different harnesses—accuracy similar, cost 12× apart
Sep 29 Pham/Nguyen/Chen/Ye: four long-context tasks stress coding harnesses (OpenCode, mini-swe-agent, RLM, ReAct); best 68% macro; GPT-5.6-sol RLM vs mini-swe-agent on memos 92%/$7.23 vs 100%/$0.583.
-
VACE: co-evolve weights and harness—but reject harness edits that fail validation
Sep 29 Qi/He/Liu et al.: alternate agentic RL with trajectory-driven harness revision; candidate kept only if validation improves with model fixed—17/44 proposals rejected; +6.43/+9.09pp over weight-only RL on OfficeQA/AutomationBench.
-
The Sisyphus Predicament: agents refill the review queue overnight while reviewers sleep
Sep 29 Britton Russell (Rangle): when generation outruns checking, senior engineers turn into full-time review queues. Track senior attention per unit of net value, raise the floor with automated checks, and point people at what the checks stop.
-
MCP errors written for developers hurt the most capable agents most
Sep 29 Xu & Wu: of 3,001 MCP error messages in 150 servers, half of next-steps depend on caller capabilities agents lack; a terminal-auth hint drops recovery to 45% (6% on GPT-6 Astra); naming a server tool raises it to 84%.
-
Zero2Repo: sealed acceptance turns 95% green into overnight failure
Sep 29 Yang/Shi et al.: 11 native-ecosystem from-scratch tasks; GPT-6 Astra 10/11, Claude Opus 5.5 9/11, Grok 4.7 High 7/11; failing submissions still pass 89.9–99.4% of hidden tests—long-tail rules and single omissions, not missing subsystems.
-
Compositional safety failures: individually safe harness updates can fail together
Sep 29 Zhang et al.: 43 pairwise + 18 irreducible 3-way failures across AgentDojo/ASB; incremental interaction hypergraph + runtime monitoring cuts residual CFR to ≤1.27% while checking ~20–34% of actions.
-
Serge: overnight GPU verify before the PR, discard before you publish
Sep 29 Ziadé/HF: nightly Transformers CI agent — reproduce on GPU, relore memory, 5× red-to-green verify, then PR; 29 merges / ~80 days; funnel throws away weak candidates early.
-
A month of an overnight Claude Code loop: 20 real runs, a three-PR review cap, and verification that took twice as long as the code
Sep 28, Matt Adams: a mini PC works his GitHub backlog from midnight to 7am on weekdays. 83% of 189 implementation slots never started the agent because nothing was eligible or three PRs already awaited review. Median implementation ran 17 minutes; browser verification about 39.
-
Multi-agent code judges need discriminating evidence — or should decline to guess
Sep 28 Aly/Assaf/Kobti: MARCH on CodeJudgeBench ties 78–95%; same questions on both solutions 76.6%; withhold non-discriminating comparisons → 20.7%→36.9% accuracy while answering half.
-
Audit the scaffold, not the checkpoint: frozen weights still let RSI expand
Sep 28 Bobadilla-Suarez, Suh, Fortin: stationarity dichotomy — fixed reachable edit set ⇒ saturation; escape only if the scaffold expands. Best-of-k buys rate not ceiling; same-family vote fails 42% of tasks.
-
Make the gauntlet slow on purpose: an agent that merges itself overnight, and why shipped means deployed
Sep 28 Luke Chadwick (Far Horizons): on a half-million-line portal the agent merges and deploys its own PRs. The trick is a build that takes up to two hours, strict AI reviewers that must be satisfied, hooks that redirect instead of block, and a rule that work isn't done until the deploy is green. A 1:15am brief became twenty merged PRs by nine.
-
SEABench: self-evolving harnesses raise completion — and causally attributed safety failures
Sep 28 Das et al.: 48 longitudinal sequences; evolved agents 47.2% completion vs 35.7% baseline but 43.9% attributed safety failures vs 0%; tools/skills surface worst; CoT monitor cuts 70.9% harm at 9.7% FPR.
-
Agent anxiety: an idle agent reads as waste, so you break your own focus to feed it
Sep 28 Driven to Develop: a writer who teaches focus and slack describes checking an agent fleet compulsively because idle capacity feels like a running meter, and names the real shortage as well-formed tasks, not compute.
-
RSI-Master: Experiment OS + reviewer DAG beats strategy lock-in in self-improving factories
Sep 28 Du et al.: ExpOS regularizes post-training actions; Reviewer-Guided orchestration grows a research DAG — 54.49 vs 46.53 on PostTrainBench at 0.0% hacking; 35B beats Instruct on LiveCodeBench-v6 (41.21 vs 37.36).
-
Tracekit: tamper-evident intent–self-report–action audit for coding agents
Sep 28 Ghosh: hash-chained Claude Code hooks capture intent, model self-report, and executed tools; anchors catch re-chains; seeded-fault splicing shows rules+LLM reviewer catch concealed misalignment.
-
LatentSift: cut the verification tax when Best@K multiplies overnight candidates
Sep 28 Han et al.: replace Stage-1 LLM execution-free verifier with policy hidden-state filtering — EF-verifier tokens −66.6–81.0% and total verification tokens −49.1–62.1% at K=16 on SWE-bench Verified; DeepSWE hybrid Best@16 59.26%→60.06%.
-
Research-native agents: make non-compliant science unrepresentable, not merely discouraged
Sep 28 IdeaHorizon: AfS encodes long-horizon research as iron laws — commitment before measurement, unforgeable freezing, hash-chained ledgers, compounding memory — so multi-day agent campaigns cannot forge freezes or invent discharges; two closed campaigns (50h assisted; 8.85h unattended Ising) illustrate the machinery.
-
Empty intersection: 98% provenance coverage, and neither verification decision moved
Sep 28 Jeon companion: grade filter → undetermined; vocabulary 36%→98.4%; ingress refuses 3,070 writes — none of the three give the 32 decision rows admissible input; coverage ≠ reach.
-
A home-built agent harness learns the same lesson five times: let the host do the deterministic work
Sep 28 Damian Karzon, part 2 of building localagent-box: a loop mode that stops when the tree stops changing and ignores 'done' while tests are red, an auto-review agent queued on every agent PR, autofix agents for the findings, and a 40-file review that cost 3.7 million tokens and skipped the reviewer's own code.
-
ReviveBench: the overnight factory is only as honest as its hidden verifier
Sep 28 Liu/Dai/Du/Yang: 10 revival + 13 reconstruction tasks behind hidden graders; 28 verifier defects (24 FN, 2 FP); CFD graded a self-reported zero; three checks—achievability, candidate consensus, never grade a self-report.
-
Unearned passes: audit agent trajectories, seal the channel, then re-run live
Sep 28 Luo/Luu/Elfeki et al. (Scale): process-verify 3,810 surface passes; SWE-Bench Pro violations 24%→73% Opus 4.7→Fable 5 then 11%/0% on Fable 5.1/GPT-6 Astra; repair needs replay + fresh eval.
-
RepoReuse: green tests while half the chains reinvent the wheel by turn 5
Sep 28 Ma et al.: over 3,000 multi-turn repo runs, agents read 83.6% of relevant code at turn 1 but only 35.4% by turn 5; self-reuse falls even under saturated recall, and 50.8% of task chains hold a cross-turn re-implementation while pass rates barely move.
-
Németh: 500 rounds of an autonomous coding loop, 1,263 merged PRs, and checking now costs more than writing
Sep 28 from Máté Németh: a plan, implement, adversarial-review, merge-gate loop has merged 1,263 of 1,388 PRs on his app since June. Checking takes 42% of the tokens against 36% for writing, and the loop's own machinery grew from 35% to 62% of merges.
-
SDLI: security debt banks when agents bank progress
Sep 28 Rajput et al.: Security Debt Line Integral couples SAST risk to historical-best test progress — 3.9%/26.2% agree2-positive among working SWE-bench/ProgramBench runs; gowsdl debt arrives in one write and persists.
-
The compiler may read it; the agent may not — and today’s harness cannot say so
Sep 28 Roy: purpose-based access for proprietary solver modules fails under ordinary Claude Code controls — containers, permission rules, and sandboxes cannot name which program is reading; 5 of 15 routes evade both rule surfaces, including gfortran -E and allowed make targets.
-
JET: evolve the judge on labeled source, freeze it to steer target harness search
Sep 28 Teng, Cai & An: executable judge transfer lifts cold-start WebShop mean reward ~13% over fixed-rubric guidance (exact success +36% relative); better final selection alone closes only ~10% of the gap.
-
Assay: agent claims that go stale exactly when the dependency cone moves
Sep 28 Tiwari/Vass/Singh: bind every agent claim (tests-pass, no-secrets, behavior-preserved) to the Merkle hash of its dependency cone — cone binding re-verifies 7.9–81.9% of claims vs 100% for repo-hash; gate blocks 9/9 scripted adversarial behaviours including deleted failing tests.
-
EffectMatch: approve the call, then match the persistent outcome before continuation
Sep 28 Zhang et al.: Effect Commit Contract — bind approval to one occurrence, compare actual persistent changes to approved multiset, EXACT/DIVERGED/INDETERMINATE; 0 incorrect commits across 39 task–fault cases ×3.
-
Harness learning: train a proposer to revise the harness from execution feedback
Sep 28 Zhang et al.: meta-learn harness edits with a frozen solver; 4B proposer lifts mean single-step OOD Reasoning Gym scores 0.32→0.62 and beats its 35B teacher on average; HotpotQA training transfers to MuSiQue/2Wiki.
-
Failure-Transparent Agents: evidence contracts crush false-success reporting
Sep 28 Zhu et al.: FTA fixes failed observations before generation; false-success 22.8%→0.8% and fabricated detail 28.3%→0.8% under a structured evidence contract across 3,600 human-annotated responses.
-
Identical runs, different results: one overnight attempt is a draw, not a ranking
Sep 27 Ariño & Pafka: 584 coding-agent runs on one ML task show within-pairing SD (0.0107 AUC) exceeding spread between pairing means (0.0095); ranking observed agent gaps needs ~39–113 runs each, while best-of-3 compliant artifacts lift median holdout +0.0081 — and rule-breakers occupy the top of the scoreboard.
-
Maat: deterministic handoff contracts stop multi-agent cascades without an LLM judge
Sep 27 Elina: Maat validates agent-to-agent handoffs against a versioned contract with no model in the gate; on 522 trials, attributable halts lift rubric scores +7.7–29.1% in five of six workflows and cut model-call cost 17–53% when defects fire early — with a blunt audit that 37% of v1 halts were false alarms from validator misconfiguration.
-
The last mile: agents make 80% fast; finishing still does not scale
Sep 27 Kent Alstad (KandiForge): hundreds of agents → millions of lines across 20 repos; bottleneck is completing/hardening; post-integrate test gates arrive too late.
-
CRR: fork the sandbox, score the decision — process signal without a process labeler
Sep 27 Li et al. (NeurIPS 2026): Counterfactual Rollout Replay lifts SWE-bench Verified 41.7% vs 36.4% GRPO (+5.3 pp); equal-wall-clock still +5.0 pp; Search-Select control shows credit assignment beats branch picking.
-
Opera: a verbal critic that follows each correction until evidence closes it
Sep 27 Mei et al.: persistent audited notes + hybrid triggers lift resolve up to +15.0 pp on SWE-Bench Pro; Opera-guided student rollouts match teacher distillation on held-out repos without TB2.1 collapse.
-
Vestrum: evidence-grounded verification at commitment beats rebuild-the-answer critics
Sep 27 Parashar et al.: scoped harness edits from failure classes raise Terminal-Bench checks 63.7%→70.3% and UltraHorizon 47.6→59.8; intermediate evidence checks repair, answer-rebuild critics break more than they fix.
-
Nom Army: a worker's 'done' is a claim; the repository is the evidence
Sep 27 RaysonTech open-sources nomArmy. Of 21 implementation jobs that reported done, 5 (24%) failed independent evidence checks — including tests that still passed after the production change was reverted. Of the 15 that passed both checks, review still found real defects in 5. Worker, orchestrator, harness, and validator should not trust one another.
-
WideSWE: one green repo is not a solved cross-repo change
Sep 27 Wang et al.: 120 real multi-repo tasks across 41 ecosystems; best config (Codex CLI + GPT-5.6-sol) fully solves 42.5%, yet often greens one target while leaving another unfinished — joint workspaces help features, independent runs mainly recover omitted scope.
-
AI-generated code and agentic AI move DORA metrics through different pipes
Sep 26 Alenezi: discrete-event sim separates AGC from AGT across five DORA metrics — AGC raises CFR/rework under both capability profiles; AGT lifts DF +31–34% and FDRT −29–38%, but can raise escaped defects when autonomous CI repair masks failures.
-
Relic: collaboration lessons that outlive the agents who learned them
Sep 26 Du et al.: Relic turns recurring multi-agent friction into organization-owned executable protocols — complete-contract delivery 14.06%→19.76% over matched structured teams; fresh-member transfer 41.2% with bindings vs 34.6% text vs 25.4% none.
-
Beyond the model: harness gains shrink on SWE repair, amplify on open-ended repo work
Sep 26 Hu et al.: mini-SWE vs OpenCode across 10 Qwen/DeepSeek models × 3 benches; NanoHarness (+tools, task-specific subagents) recovers most product-harness lift — complex scaffolding is not free.
-
A green check lied: append-only status receipts
Sep 26 Patrick Hughes: false green after scanners read 0 of 5 repos — append-only event receipts so later no-op runs cannot bury a real fix; missing evidence is unknown, not green.
-
Overnight is a task class, not a ticket: an unreviewed run by the next working day counts as failed
Sep 26 Michał Jaskólski, AI Developer Toolkit: score every ticket on oracle, blast radius and reversibility, and only pre-approved task classes go overnight. Overnight output has to be reviewed within one working day or the run didn't count, dispatch is capped at the next day's review capacity, and the first escaped defect demotes the class.
-
SCLATE: month-long agent calendars on a hybrid clock, not a custom loop per harness
Sep 26 Jung/Bilkhu et al. (Apple): open event scheduler + hybrid simulated clock compresses 30-workday MetaClaw to median 5.9h (162×); ten harness×memory configs × ten models; post-train through unmodified Claude Code/Hermes/Codex.
-
SRE-Marathon: continuous ops agents correlate and localize—then almost never repair
Sep 26 Tian et al.: long-horizon SRE benchmark with overlapping faults, alert streams, and change-driven defects — best Marathon-Score 41.3/100 across ten methods; agents often correlate/locate but repair ≤2.4% while faults stay active.
-
AsynCodeBench: green tests can hide unresolved cross-agent dependencies
Sep 26 Zhang et al.: dependency-centric async multi-agent SE bench — ADPR/DRS over 19 tasks, 52 deps; Test Pass ≫ ADPR (71.6% of Async-Manager-RO outcomes); coordination often closes in a mid-run hopping window.
-
Verification as an architectural layer: V-model agents that abstain instead of burning the budget
Sep 25 Afoud & Wu: pair each spec level with an external verifier; deterministic gates + optional judges; unverified ReAct answers 0/10 (all hit caps), verified VG answers 8/10 and abstains with cause.
-
Compact docs regenerate the file; they do not raise issue resolution when source is already loaded
Sep 25 Arman & Molybog: roundtrip optimizer hits fidelity 1.0; source-withheld resolution 0.08→0.71; with source present, compact/full/retrieved context never beat issue-only across models and ten repos.
-
Methodological harnesses exist in the wild — mostly as one artifact, not a system
Sep 25 Díaz, Pérez & Gil-Borrás: 5,435 agentic repos; reweighted any-mechanism prevalence 21.7% (65.3% in >100★); mean breadth 1.15/5 even in popular stratum; rule files dominate, persistent knowledge ~1.9%.
-
Three silent cost leaks: subsumed retrieval, similar scripts, test re-execution
Sep 25 Hu et al.: 1,200 trajectories — three behaviors hit 79–98% of tasks and up to 22.75% of cost; structure-aware retrieval can raise spend 28%; developer-designed skills cut cost up to 41.73%.
-
Harness search is multi-objective: joint accuracy, safety, and tokens beat staged phases
Sep 25 Mukherjee & Tanjim (Adobe): MoMHa joint reward J=0.482 vs 0.198–0.422 baselines; safety 0.781; +2.7 overall vs two-phase at 95 fewer tokens/example; wins 5/7 held-out real-world columns.
-
A software factory needs brakes, and the review should fix the factory too
Sep 25 Matt Pocock at AI Engineer Paris: agent-triggered work turns a factory into a slop cannon unless checks, a separate reviewer that commits its fixes, and risk-labelled PRs sit in front of humans. Every one-way door gets human review, not every two-way door, and a retro turns each review into checks for the next run.
-
Authority at commit time: agents propose; only current admission effects
Sep 25 Salas: OCC for institutional agents — reject-and-rerun on stale deps/components; complete mediation + outbox; 18/18 mediated effects, 8 bypasses detected only with independent journal.
-
engrams: Cortex open-sources the always-on cloud factory around Claude Code and Codex
Sep 24 Unni: open-source engrams runs agents in your cloud — scheduled CVE/dep PRs, Datadog hotspot rewrites, error triage with dedupe, Linear+Slack end-to-end with microVM sub-agents.
-
Docker Cloud Sandboxes: close the laptop, keep the agents running
Sep 24 Docker: Cloud Sandboxes put the same microVM agent isolation on always-on Docker compute — move local↔cloud with one command, run hours-long / parallel agents, review in the morning.
-
outrigger: evidence-grounded long-horizon lab—and the blind gate that earned demotion
Jul–Sep Patel lab: spec interview + adversarial plan review before code; held-out sealed suites; exec-loop ledger. Three-arm result demoted blind merge gate (5.9× cost, same defect)—fresh adversarial eyes still pay.
-
Frozen 4B model proposes; external entailment gate releases
Sep 24 İşcan: candidate generation ≠ release authority — 21/22 fabricated ready plans on unanswerable tasks all rejected; 0 false releases / 83 gated (U≤3.54%); ungated shell 9/18 false.
-
Between the commits: 14% of AI code-gen events err before commit; 1 in 4–5 responses are inaccurate
Sep 24 Leith: 21k-LOC wholly Claude-authored tool — 14.3% of generation events introduce errors AI tests catch pre-commit; checkable facts 94.3% accurate; design-fix proposals 79.4%; ~1 in 4–5 responses err.
-
Exactly-once lives in the contract when verification cannot see in-flight work
Sep 24 Li: Limbo grades 25,930 episodes — frontier models nearly never duplicate lost acks (0.5%) but duplicate 56%/74% under late commits/redelivery; keys everywhere cut duplicates 28%→4%; agents overclaim success in 90% of duplicate episodes.
-
State of harness engineering 2026: most teams write rules; almost nobody enforces them
Sep 24 Marmelab audit of 246 repos + 57 pubs: same model 68%→88% across harnesses; 60% of big harnesses untested; prose security rules ~4–16% enforced; Vercel tool-strip and OpenAI merge-throughput caveats.
-
Pento: from agent harnesses to software factories — 2027 stays lights-on
Sept 24 from Leonardo Piñeyro (Pento): Stripe merged 1,300+ no-human-code PRs a week; Ramp’s agent writes 3 of 4 merged PRs. Factories sell the queue and gate, not the model. Autonomy only as wide as cheap verification; the lit factory keeps a person on merge while the gate moves upstream to specs and risk.
-
Silent success: the release gate that passed on checks it never ran
Sep 23 Jeon: production gate green for ~2 weeks while one subgate ran 0/2 and another 6/8 — violation-count keys treat absent data as pass; add unable-to-determine that fails closed.
-
Dark factories: generation is real; Level 5 trust is not
Sep 23 Roelants evidence report v3.66: bounded agent pipelines exist; strict lights-out ownership of live software does not — productively complicates overnight-harness optimism with lifetime-cost and review-boundary gaps.
-
Approval laundering: the record names the command, not the workflow it launched
Sep 23 Zhang et al.: closure-bound approval — 40→13 residual gaps across 111 pairs; effect-bound pre-auth predictions cut residual effect instances 10→3 on holdout; Vite install and MCP tool calls omit file/network effects.
-
If you’re not the model, you’re the harness — six parts that make ‘done’ real
Sep 22 Axon Active: harness = instructions, context, tools, loop, guardrails, verification; independent review catches float-money false greens; harness work compounds across model swaps.
-
Same brain, five bodies: ~90% of failures were the harness
Sep 22 Herlein: five agents, one frozen Qwen, held-out 32-case suite — done-gates and independent oracles beat bigger quant; self-written tests can false-pass the agent's own blind spot.
-
When agents outrun review: accountability, WIP limits, unlike blind spots
Sep 22 lezli01: commit/merge KPIs break under agents; independent review needs unlike contexts; cap pending PRs as inventory; vincent gates left-of-merge, WIP-on-PRs still open.
-
Grow the harness, not the context: recurring control becomes code
Sep 22 Li et al.: failure-guided Growing Harness turns a strategy-free scaffold into reusable executable control — 76–92% fewer LLM calls, held-out gate rollback, success holds across 4B–120B on WebArena.
-
JAZ invoke: a minimal agent loop that beats specialized memory and self-improvement harnesses
Sep 22 Li et al.: invoke makes prompts, tools, and REPL history first-class REPL variables—StuLife far-recall 69.9% vs Letta 61.8% at ~half cost; AppWorld TGC 74.2% vs ACE 69.9% with lower meta spend.
-
33 broken builds: the agent never grades the fix
Sep 22 Lee: disposable machines + coding agents on frozen failures; harness reruns the exact command, fences the network, and refuses toolchain cheats — 17/33 fixed; six harness bugs found by the fixer.
-
Fifteen parallel agent sessions, one hub, and a colleague reviewing 15 to 20 PRs a day
Sep 22 (updated Oct 1) Tania Papazafeiropoulou, one of two developers on a children's sports app: grew from two to as many as 15 parallel coding sessions, added a non-coding hub session that keeps a merge board and hands out the single test slot, and found the real limits were laptop memory, false-green test runs, and her colleague's review day.
-
Let it run for days — only if verify is a gate, not a vibe
Sep 22 Pratik Patel: multi-day UI agent runs need stop conditions, blast radius, surviving logs, budgets, kill switches; overnight is a deliberate mode — reproduce→trace→fix→verify on the real path.
-
AIDE²: an 8-day harness RSI loop that keeps only held-out research-efficiency gains
Sep 22 Srikanth et al. (Weco): bi-level AIDE² rewrites its own research-agent harness under private grades — seven accepted improvements; matches/beats a 2-year human production agent on four held-out benches; reward hacking 55%→32%.
-
A merged agent PR isn't finished: agent merges need follow-up fixes 1.6 times as often, and the same agent usually ships the fix
Sep 22 Takerngsaksiri, Duong and Barnett followed 6,774 merged agent PRs in popular GitHub repos into the fixes that came after them. Agent merges drew verified fixes at 1.62 times the odds of human merges in the same repos, about 70% of those fixes came from the same agent, and nothing visible at merge time reliably flagged which ones would need it.
-
Async tool calls cut the harness tax — up to 40% vs Codex
Sep 22 Unreal Labs: agent harness schedules tool work between model turns (in-progress then final event-log records); Harbor benches show comparable pass rates at lower $ than Codex/Pi.
-
Twelve-hour days pressing enter: an engineer's case against output-first agent rollouts
Sep 22 TechSpot on voxium's viral post: an anonymous engineer says Claude Code now writes the specs, tests, tickets and reports at their company, people work 12 to 13 hours a day approving output, and there's no time to review or understand the code. The complaint is aimed at management measuring PRs and features shipped.
-
10% trajectory-aware subset, ~90% cheaper SWE-agent regression
Sep 21 Ayyad et al.: outcome-group + centroid-in-embedding selection over sanitized trajectories — 5–10% subsets cut avg resolve-rate error 3–11% and worst-case 38–46% vs strongest random/stratified baselines; median error <5% at 10%.
-
A second model re-reading the diff is not a second reviewer
Sep 21 Lundberg: correlated model blind spots ≠ two humans; trust from facts outside the diff (paths, history, tests moved, CI) — 58-PR queue ranked billing+migration risk over tidy size.
-
The work behind delegation: seven stages of supervising coding agents
Sep 21 Park/KAIST et al.: reconfigure Sheridan supervisory control into Plan/Monitor/Wait/Review/Teach/Manual Fix/Update Assets from 19 developers; Reddit study shows concentrate planning, delegate review to other agents, externalize recurring guidance into assets.
-
69% of “done” claims had no green run — gate the Stop hook
Sep 21 Lencevicius: 591 Claude Code sessions; 69% of completion claims lacked a post-edit passing test. isitdone blocks Stop until the repo’s own checks pass the current tree, with receipts and test-tamper scans.
-
Local Software Factory: YAML pipelines, worktrees, coding agents as workers
Sep 21 StratoNext sf: local factory — GHA-style YAML pipelines (agent + shell stages), per-request git worktrees, parallel queue, optional daemon; reuses installed Claude/Codex CLIs; human-review stages when needed.
-
Self-healing harness: local improvement is not a safe persistence decision
Sep 21 Tayebati et al.: Detect→Notice→Heal→Validate admission control — of 383 replay-rejected proposals, 211 (55%) fixed the trigger while degrading protected cases; task-completion higher under Harness in all 16 pairs.
-
Exited 0, changed nothing, reported success
Sep 21 Boughamoura: beautiful markdown is not evidence — provider-isolated validator + workspace content-hash gate (excluding the report dir); wb-flow’s Infra/Artifact/Oracle contract.
-
RRSI: regularize harness self-improvement so evolve-set gains survive OOD
Sep 21 Xia et al. (Google Cloud AI Research): annealed edit budgets, leakage critic, noise-adjusted floor, and cost-aware pruning — up to +14.1 evolve / +4.7 OOD with 30% fewer policy tokens than unregularized RSI.
-
Advisory vs blocking CI: green agent tests are a transcript, not a gate
Sep 20 Zhou: split agent patches into Lane A (generated, report-only) and Lane B (human-owned oracles on sealed fixtures) — forbid oracle-path edits in the same diff, leave coverage in the summary.
-
Hermes Kanban: six workers, two chairs
Sep 20 Dottin: durable board + isolated profiles + Discord control room; concurrency caps before swarm theater; notify≠wake; review in another context — factory continuity without forty agents.
-
Agentilda: plans on disk, agents never merge, overnight until the budget dies
Sep 20 Gredeskoul: .plans emoji state you can prove from files, specialist agents that cannot publish, alock across worktrees, and a tea-break run that advances a dozen specs until a session limit — continuous factory with human merge.
-
A 100% pass rate is often a blind harness
Sep 20 Ma: ceiling effects hide regressions once easy golden cases saturate; stratify easy/medium/stretch, detect when stretch is too green, and treat a yardstick that can still fail as the asset.
-
VibeMemBench: useful repo history exists — memory systems mostly don't deliver it
Sep 20 Fan et al.: 111 SWE-rebench targets, 3,634 trajectories — frozen verified experience lifts 4/5 solvers +1.1–4.5 pp and cuts steps; 11/12 Mem0/SimpleMem/MemoryOS/A-MEM pairings stay at or below memory-off.
-
Chief of Staff: one session verifies; others execute; the board is memory
Sep 19 AsyncDot: coordinator≠implementer, durable Plan Desk state, red-gate-first, re-run claimed commands — exit codes decide; reports are evidence, not instruction.
-
Harness = loop + tools + memory + gates — or don’t run unattended
Sep 19 Chalupa: four-piece harness checklist; ACP mediation probe that caught an engine writing outside the project without asking; eval gates before any overnight autonomy.
-
Constrained Agent Harness: the model proposes; the harness decides done
Sep 19 rmax-ai: coding as constrained transitions over repo states—goal contracts, Docker sandbox, hidden tests, git checkpoints, SHA-256 evidence chain; model never makes the authoritative completion decision.
-
Most AI review tools can’t safely block a merge — Codacy’s 14-tool map
Sep 18 Codacy: advisory LLM reviewers vs rule/threshold gates; Copilot/Claude/Gemini have no native required check; fail-closed Diff Coverage and Verity prompt-conformance as the governance layer.
-
Faros's Speed Trap telemetry: per-PR incident risk settled, but PRs merged with no review are up 76.3% and QA time is up 300%
Faros's second AI engineering report, built on 12 months of telemetry from 22,000 developers across 4,000 teams, finds the worst early-adoption damage easing per change while the verification load moves downstream: unreviewed merges up 76.3%, time in QA up 300.6%, monthly incidents up 125.4%, and restarted work up 66.7%. Heavy agentic review correlates with better downstream numbers.
-
Unattended run: drop a file, wake to a reviewed branch — never merge
Sep 18 Firu: autonomous-sdlc-harness daemon → worktree → plan/build/review/browser-test loop; state in git checkboxes; four review gates including a skeptic; stops at a pushed branch, human owns the merge.
-
Review burnout: 98% more merges, 91% longer reviews — fix the tax, not the heroics
Sep 18 Ankit Jain: AI volume turns seniors into intent reverse-engineers; codify recurring feedback, preserve prompt/session acceptance criteria, measure verification work — not LOC throughput.
-
Kiro Crew: 1,000 merged PRs in a week came from a ladder of bottlenecks, not a burst
Sep 18 Kiro team: three full-time engineers merged 1,000 PRs in seven days, each through CI and review, by climbing five stages from one hand-driven session to a supervisor agent running 30 to 50 sessions across six to ten machines. Past 50 sessions the supervisor itself stopped scaling.
-
SWE-Proof: a quarter of test-passing patches still admit counterexamples
Sep 18 Ma/Mikek/Li et al.: Benchproofer → SWE-Proof (500 Verified issues + formal oracles); audit overturns 26.8% of test-green patches; ground-truth formal spec lifts Opus 4.8 85%→~95%; self-written specs mostly fail faithfulness.
-
ProductsFactory: a 24/7 cycle that opens PRs only after deterministic gates
SavvyCoding’s experimental factory: ~60s orchestrator cycle, sandboxed Claude personas, lint/drift/test/verify recipes before PR, reviewer + auto-merge, supervisors that break cascade loops.
-
Every agent PR costs review time — even the ones that never merge
Sep 18 Tess Ainsley: measure reviewer-hours per merged production change, not per submission — Salesforce plateau + abandoned agent PRs hide the real verification tax.
-
Plugin4Shell: pinned plugin SHAs that still install attacker code
Sep 17 Air Security: zero-click RCE via SHA-pinning bypass in Claude Code, Codex, Copilot, Gemini — auto-update swaps marketplace plugins when git prefers a hash-named branch over the commit.
-
Claude Projects: coordinator + parallel cloud threads that keep working after you leave
Sep 17 Anthropic: projects as long-running factories — shared memory, per-thread branches, PRs/tests while you step away (even from your phone); merge conflicts stay ordinary PR work.
-
Dark factories: remove readers only after gates own escape rate and halt
Sep 17 Augment/Hingel: unattended merge needs owned gates, escape-rate telemetry, provenance, incident owners, drilled halt — vendors still keep humans in the merge path; dark ≠ unowned.
-
AI morale overload: the count that hurts is open loops, and an Approve button is never a win
Yu-kai Chou, creator of the Octalysis motivation framework, argues that supervising a swarm of agents burns people out because of the number of unfinished tasks one person holds open, not the number of agents. An approval queue gives the reviewer all pressure and no accomplishment. His fix: one main task carried to done per session, review with a visible finish, and agents that earn autonomy through clean streaks.
-
Harness components, not monoliths: what actually moves long-horizon agents
Sep 17 Fan et al.: 176 matched settings on SWE-Bench Verified and Terminal-Bench — context management mostly prevents overflow; planning saves cost for strong models; bash-capable models often win cheaper without predefined tools.
-
Long-horizon agents need a harness that outlives every context
Sep 17 Nijkamp et al.: levels + clocked ticks + cascaded intelligence — a ten-day campaign with once-a-day human attention kept thread across every reset; continual learning belongs in harness checks.
-
A green overnight diff is a hypothesis until something outside the agent can check it
Sep 17 Prince Panchani on no_human: tamper guard, repro gate (fail at base / pass at head), fresh-context reviewer with edit tools refused, merge policy → awaiting_approval — confidence is a claim; evidence is re-runnable.
-
Valid MCP on the wire ≠ usable semantics in the harness
Sep 17 Patodiya: differential tests on 18 fixtures across four pinned Python MCP integrations find 13 fixture–task divergences—structured slots, error flags, and rich content change after decode, by interface and consumer contract.
-
Roborev: continuous commit review inside the agent loop
Kenn.io roborev: post-commit background reviews + agent hooks feed findings back while context is fresh; fix/refine loops in worktrees; compact verifies findings against current code before the PR flood.
-
SoL-Pi: around-the-clock agents need a cheaper harness, not a longer burst
Sep 17 NVLabs SoL-Pi: auto-research at the harness layer cuts token traffic ~45–49% and API cost ~⅓ vs native Codex/Claude Code while matching Pi on EdgeBench — four surviving mechanisms for unattended runs.
-
Upstream specs beat overnight catch-up: you are the harness
Sep 17 Steve Brownlee: pull decision latitude out of the coding agent before the first token — assertable Given/When/Then, JSON phase contracts, convention profiles — so runtime review loops have less hallucination to catch.
-
The harness sets the price of the answer
Sep 17 Tunguz on HarnessTax: same model 71% cheaper on Pi than Claude Code with no significant quality gap; harness compression as margin, moat, and the difference between rationing overnight work and saying yes.
-
BCG’s harness OS: four unequal quality gates, not one green check
Sep 16 Kapoor/Paca/Zemgulys: specs, constitution, control panel, context hub, and four gate types (automated / critic / human stage / regulatory) — maturity ladder for when humans still must say yes.
-
Gates first, hooks last — verification that survives an agent edit loop
Sep 16 Maxim Berenshtein: sixteen repo gates + edit-time hook + gate-inventory; prose in AGENTS.md fails silently — conventions become unignorable when the check refuses before CI.
-
Lauren Tan’s overnight merges: verification is the system
Sep 16 Gwang-Jin Kim on Tan’s SpaceX/Cursor talk: ~1,000 PRs/month and merges while asleep — not a magic model; make verify, evidence-first bugs, Skills, CI-hardened review comments, fresh-agent reviewers.
-
Same model, up to 5× the cost: the harness is on the invoice
Sep 16 HarnessTax (Pan et al., UC Berkeley/Arena): 21 model×harness pairs on SWE-bench Lite and Terminal-Bench — success moves a few points; cost up to 5×. Claude Code ~2× Pi on shared models.
-
Is your project doing work overnight yet?
Peak Claude sessions are calendar-bounded. A harness that keeps modest verified momentum through the night compounds past glory bursts — but only if morning inventory is judged, not just generated.
-
When AI review should come before human approval depends on which stage is short
Lin and Zhang (HKUST) model agent PRs as a queue with a mandatory human merge gate and an optional AI reviewer in front. Optimal review passes through four phases as maintainer capacity grows, including one where a useful reviewer is deliberately left idle because its false rejections cost more agent runs than its catches save. Calibrated on SWE-Review data, adaptive routing beats review-everything or review-nothing by up to 37%.
-
158M overnight tokens: delivery worked; the harness bill did not
Sep 16 field receipt: 13h unattended fix of 12 design findings — 100/100 tests + byte-identical release via independent verifier; 95% of tokens were re-reads; idle-parent polling burned 6.5% on “not yet.”
-
Evaluate the low-risk gate, not only the green PR
Sep 16 Sara Mo: OpenAI’s factory can auto-approve low-risk changes — CI proves the checks; the classifier’s skip-human decision needs its own record, replay, and post-deploy score.
-
Unattended loops don’t have a token problem — they have a feedback problem
Sep 16 Signadot: loop cost = iterations × cost-per-iteration; guardrails cap waste, only high-fidelity feedback reachable inside the session shrinks both; platform owns the verification surface.
-
Sinatra: 306 agent PRs since April, 289 merged — 94%, and the human gate did the work
Sep 16 from Sinatra: nineteen weeks on two production repos, 306 agent-opened PRs, 289 merged (94%), 14 closed unmerged, 3 still open as of Sep 9. No auto-merge; every merge is a human button. Tickets written for agents, sandbox tests before ready-for-review, and agent self-review before a person looks.
-
Pizza Bot: an inbox for agents that work while you are not watching
Sep 15 The Register (AWS OSS last week): Pizza Bot — local-first email-style inbox for background agents; Unread vs Action threads, LangGraph checkpoints, always-on host; FOMO UI for overnight work.
-
The verification tax is the real agent bill — prove once, stop green-on-green
Sep 15 BlueLupin census: code was ~8% of model output; review/verify ~40%; 15/39 Playwright runs were already-green reruns — cut avoidable proof without deleting the toll booth.
-
The factory is a product-team operating model, not a merge queue
Sep 15 Robert Glaser: encode judgment as backpressure and Outcome Grading; keep humans collaborating on intent; improve the factory while it builds — review FOMO is the wrong bottleneck.
-
Polter: one Claude session supervises the overnight fleet
Sep 15 Lugiax: Ghostty fork + MCP — supervisor tab reads screens, restarts quiet workers, group chat + task panel; won't answer permissions; overnight FOMO as unfinished runs.
-
OpenAI’s agentic software factory: Codex takes the pipeline
Sep 15 Pragmatic Engineer: inside OpenAI, humans define outcomes while Codex gathers context, implements, babysits CI, runs specialist reviews, deploys with per-change agents, and feeds Perf Factory — 10× CI load in six months.
-
Cognition: when agents outpace humans, verification is the bottleneck
Sep 15 TechNode: Cognition’s Kevin Lee — more agent sessions are async than human-triggered; 89% of eng commits via Devin; judge by first-pass verification, not PR volume.
-
Warp: crawl point automations, then walk a closed factory loop
Sep 15 Zach Lloyd: Trigger→Agent patches hit context/metrics/security limits; walk = cloud default + shared context + triage→spec→implement→review→verify→monitor — Warp’s marketing site ~75% factory-shipped.
-
Anthropic: agentic coding strained CI 25× — redesign the selector, don’t buy another day
Sep 14 Malhotra: 8× code, 10× tests, 25× CI jobs in six months; patches lasted 70d/29d/<1d before a journaled, horizontally scaled test-impact service — overnight agents raise the floor.
-
Harness as handoff: next session reads the repo, not the chat
Sep 14 Bonomo build log: CLAUDE.md operating loop, sprint contracts, implementer≠evaluator, session-handoff.md — Session B finishes after /clear with zero naming drift from Session A.
-
Ericsson: multi-agent review with project context hits 96% accuracy
Sep 14 industrial DSR study: specialized agents + context-specific knowledge across readability/maintainability/reliability/performance; >200 issues, ~69% rated important.
-
GitClear: +25% output, +81% duplication — the maintainability tax
Sep 14 on Maintainability Gap (623M changes, 2023–2026): heavy AI users +25% vs own baseline; block duplication 40.3→73.0/MLOC; moved/refactor code 21%→3.8%.
-
Mouse on GLM Flash: 77% FrontierHarness at $0.29/pass
Sep 14: same Mouse harness on GLM-5.3-Flash hits 23/30 ($0.29/pass, 97% cache) — within 1–2 of Claude Code/ZCode on full GLM at ~1/7–1/9 the $ per pass.
-
HAR: open harness — isolate, verify, hand off proof (not narration)
Sep 14 os-factory/har v1.14.2: agent-agnostic CLI+MCP — worktree slots, .har/ contract, deterministic verify + validated tree hash, Mission Control; Claude/Cursor/Codex portable.
-
Claude plugin eval: gate skills on Δ before you merge
Sep 14 OTF: Claude Code ≥2.1.269 runs WITH vs W/OUT plugin evals; CI --threshold on contribution lift so skill merges stop being demo folklore.
-
Real-SWE: best agent still fails >60% on private codebases
Sep 14 Specific Labs bench: Fable 5.1/Claude Code tops at 38.8% on proprietary multi-file tasks; six tasks <15% success; missed requirements and integration errors dominate.
-
Sourcegraph GA: keep-working batch agent until CI is green
Sep 14 Agentic Batch Changes GA: one plain-language change scoped across the indexed fleet, CI-repair loops until PRs are reviewable — ~1k beta merges; Mercari ~80 repos; Canva 50+ migration PRs.
-
Same model, same prompt: the harness turns lying into verified work
Sep 14 Tejas Kumar walkthrough: six harness parts, deterministic verify on traces, secrets out of the model — GPT-3.5 Turbo upvotes HN only after harness engineering, not prompt heroics.
-
A factory whose PRs admit what they did not run, and an agent that fixes the machine overnight
Sam Armstrong on the software factory behind Vigil SOC: every PR ends with what it ran, what it skipped, and what the sandbox blocked; a separate environment agent repairs those gaps each day; and a single job cap, freed only by a human merge, keeps the factory from outrunning review.
-
ModularRSI: evolve five harness modules on benchmark-disjoint tasks, then freeze
Sep 14 Wu et al.: contrastive trajectory RSI over Agent Loop / Tool Use / Observation / Context / Completion modules on 2,000 external tasks — lifts TerminalBench 2.0 Acc 47.57→52.43 and transfers across models; joint/non-modular evolution regresses.
-
Winslow: overnight agents need merge gates — coordinator “done” is not a merge verdict
Sep 13 from Russell Winslow (Automater): six branch-protection gates for unattended PRs — tests with count floor, diff ceiling, secret scan, path rules, eval threshold with margin, human approval on the latest push — plus a morning queue that treats blocked PRs first.
-
Software factories need building codes — and an actuator
Sep 12 HydraFlow field note: standards become manufacturing only when violations turn into remediating work, the same gate judges the repair, and floors can tighten themselves.
-
Beyond vibe coding: stacked MRs and a second-model review gate
Sep 12 field report: a multi-repo agent takes a Jira story to stacked reviewed MRs — prose workflow contract, linear ticket chain, independent model review of tests then code, two human gates.
-
AI coding factories: stages, evidence, and continuity
Sep 12 Gabo Esquivel: an agent that edits and opens PRs is not a factory — stages need artifacts and decisions, long runs need handoff records, and acceptance stays stable while implementation iterates.
-
Not all agents are equal: reverts split vendors, review capacity is the bottleneck
Sep 12 Kraishan: 37k provenance-labeled PRs across five commercial agents — Codex reverts at half the human rate, Devin at 1.3×; pooled agents fewer security smells; Copilot draws the deepest human review.
-
Augment's factory grew output 4.5× by automating wherever work piled up, starting with review
Sep 11 Akshay Utture, Augment Code: across 17,200 merged PRs from November 2025 to July 2026, size-adjusted output per developer rose 4.5× while median merge time fell from 11.2 to 3.1 hours and 14-day reverts fell from 1.9% to 0.4%. Review was automated first because its backlog was growing, and a human still makes every merge call.
-
Cherny: Claude production code needs a higher bar than human code
Sep 11 Willison quote of Boris Cherny: Anthropic’s continuous verification stack — lint, tests, Claude e2e, daily fuzzers, automated review/security/refactor — is the price of agent labor at volume.
-
Copilot review: shell-validated comments and an ensemble Lite path
Sep 11 GitHub changelog: Copilot code review auto-resolves addressed threads, runs shell tools behind the agent firewall, and switches Lite to a multi-agent ensemble with higher addressed-comment rates.
-
Software factories are five gates — the agent is the cheap part
Sep 11 Firecrawl field guide collates Stripe/Spotify/Shopify/Ramp/Faire: intake, isolation, tools, verification, merge policy — plus Microsoft’s merge-rate gap and Faire’s failed confidence-score filter.
-
Five subagents, five truths: orthogonal review without self-grading
Sep 11 field note: Abouwarda forks Claude Code review into five subagents that disagree on what is authoritative — ADR, spec, live HTTP, config — so the writer’s session never grades its own diff.
-
Agent PRs outpace review: LinearB 5.3× pickup, four queues that catch them
Sep 11 Prokop Simek: agent PRs finish fast then wait — LinearB 5.3× pickup lag, Faros +91% review time; three attribution buckets and four saved views keep the queue moving.
-
Codex goal mode: waiting can cost 83–188M tokens per hour
Sep 11 Relux: across 3808 Codex rollouts, 52 goal sessions ate half of 59.2B input tokens; idle spin at 0.03s restart + poll-proof-of-wait; Claude Code awaits events instead.
-
Comprehension debt is the review failure that still merges green
Sep 10 TNS field note: functionally correct agent PRs that cross domain boundaries slip past human review — Executable Architecture (pytest-archon / ArchUnit) makes boundary violations fail CI instead.
-
Governed agent loops: backlog to PR with Standards and AI Review
Atlassian’s Sep 10 ship note: always-on Jira agent loops plus org Standards and a dedicated AI Review agent — humans keep the merge button, not the bottleneck before it.
-
Authority as inventory — why the boring capability ceiling wins
CapScope’s lesson isn’t “spot the injection.” It’s freeze what an agent may do outside the model, then put an owner on that inventory. Fleet management of ceilings across a software factory is the natural next layer — and it’s exactly the kind of platform-dependency objective stewards are good at enforcing.
-
BenchShield: reward integrity needs infrastructure evidence
Sep 10 paper: lifecycle-model instrumentation for agent evals; full-chain recall 77–100% vs 23–94% hackability scanner; 96% runtime accuracy on labeled trajectories.
-
A software factory is the floor under fast agent hands — merge stays human
Sep 10 BlueLupin: ruleset, map, broker, human-on-diff before sessions; intake→verify→named reviewer; launching more agents fills the review queue — capacity is who drains it.
-
Background coding agents: verification is the product companies keep shipping
Sep 10 Cedric Teyton survey of Stripe/DoorDash/Ramp/Monzo/Checkout/etc.: durable runs + ephemeral workers, determinism inside the loop, fresh-context review gates — attention scarcity, not typing speed.
-
A software factory still needs a review gate it can trust
Sep 10 CodeRabbit field note: green harness checks prove covered behavior; merge still asks whether the change should become part of the system — with review depth following cost of error and blast radius.
-
Cursor Projects: coordinator that keeps working after the laptop closes
Sep 10 launch: cloud coordinator + shared context + subscriptions (Slack/PRs/schedule) so fleets keep shipping while you sleep — FOMO as product, review still the human gate.
-
Headless Decode on Modal: overnight cron → morning PR inventory
Sep 10 Decoding AI: Paul Iusztin ships Decode headless on Modal — CLI, webhook, or nightly cron; parallel attempts push decode/ branches so morning review starts from finished work, not five open terminals.
-
Overnight loops need a kill, park, or transfer owner
Sep 10 Alex Zhu: four named roles, a one-page session card, and a decision table — if nobody answers Slack, park or kill; never hope the agent finishes alone.
-
Ecdysis: don’t train the harness on every failure you see
Sep 10: harness evolution that separates model-specific accommodation from systematic harness repair via cross-task failure patterns — up to 1.84× faster training and +18.56% reasoning accuracy.
-
EvoHarnessBench: expanding the harness can erase yesterday’s wins
Sep 10 v2: 17 harness-evolution streams (802 tasks) — harness-induced forgetting, inconsistent self-evolving adaptation, and a retention↔adaptation tradeoff when tools/skills/agents keep changing.
-
Build loops, not agents: the software factory that clears the plateau
LeadDev’s Sep 10 field guide: individual coding agents stall at 20–30% org gains; step-function lifts come from separated multi-agent loops—starting with PR→merge.
-
PRAssistant’s 10–20% is a floor on queue speed, not a quality claim
Sep 10 field note: Microsoft’s attributed PRAssistant result (5k repos, 90%+ coverage, 600K+ PRs/month, 10–20% median PR completion time) measures when review starts — not whether what merged got better.
-
OpenAI Agents API: managed Codex harness for days-long runs
Sep 10 public beta: OpenAI hosts the Codex harness — durable sessions, compaction, subagents, sandboxes — so teams stop re-homing long-running agent infrastructure every model launch.
-
One hour: let Claude Code write the software factory
Sep 10 poornerd: describe plan→execute→review stages; Claude Code ships a worktree-isolated orchestrator plugin — and surfaces silent agent-teams traps before the first hang.
-
Reality is the final verifier — keep narrowing the two gaps
Sep 10 Krentsel/Zaharia/Stoica et al.: requirement gap + model gap unify reward hacking and hallucination; an assurance-revision loop uses deployment evidence because predeployment evals stay proxies.
-
Rivera: one "prior reviewer: PASS" line cut an AI reviewer's bug catch from 51% to 27%
Rod Rivera's paired experiment on 180 known-label implementations: telling a small-model reviewer that a prior reviewer passed the code nearly halved the bugs it caught, and a FAIL line raised false alarms from 4% to 46%. Independence is a property of the request, not the call.
-
SaltBench: measure what the machine referee does to the agent
Sep 10: a referee-gated protocol (proof kernel, verifier, or withheld tests) walls the agent off, probe-tests isolation, freezes predictions, and treats budget stop as halt — so method effects on coding agents are measurable, not narrated.
-
Seventeen PRs, thirty-two corrections, zero production bugs
Sep 10 field report: design-first + adversarial plan review + parallel subagent diff review + browser gates — including an IDOR caught by another instance of the same model — before a human merge decision.
-
Reviewer skills are the floor; agent management is the job
Sep 10 field note: after 30 days of agent-primary production code, Tamiz argues the craft moved upstream — specify interfaces, manage context, verify fit — and treating agents as juniors you only review leaves architectural debt.
-
Commit gates fail when every verifier shares the same evidence
Sep 10: VP-CONTROL shows cross-model votes over shared evidence still approve 62.9% of unsafe proposals vs 22.9% with an independent source — source diversity beats model diversity for agent commit gates.
-
Harness engineering: the AI SDLC is won outside the model
Sep 10 Waydev: five of six links from capability to shipped value are harness work — verification is the new bottleneck, autonomy is a blast-radius grid, and flow metrics beat adoption dashboards.
-
A-JIT: fill the hole at runtime, not in the PR queue
Sep 9 Marron & Barr: treat incomplete code as a first-class hole, synthesize from traces and typed IO pairs inside a harness+agent assembly — JIT for software construction.
-
Black Box’s PR gates — five reviewers before a human merge
Field note from Black Box via Codacy — ~8 coding agents in the loop, every PR through five independent reviewers, plus Verity’s markdown knowledge graph so later turns inherit what got rejected.
-
Enforcement the agent cannot edit
Hartman’s Consort (Sep 9) sorts agent frameworks by persuasion vs structure vs hard controls, then argues for immutable tests and gates on a live DB branch the worker cannot bypass.
-
Merge time tripled: acceptance is the scarce resource
A six-person team’s field numbers after agents arrived — PRs doubled, median time-to-merge went 4h→14h — and the social fixes that worked better than an AI review bot.
-
Harness engineering: behavioral evals as the safety net
Google’s Sep 9 harness note: end-to-end benches won’t tell you why the score moved — unit-style behavioral assertions on tool use and intermediate steps let you iterate prompts and models without silent regressions.
-
When the answer to ‘how do you keep up with agent PRs?’ is ‘we’re not’
Harness field CTO on Sep 9: PR floods at 1.5–10× (sometimes claimed 50×), reviewers who should see the real change not the dependency scaffolding, and a delivery knowledge graph so review context is cheap.
-
AI review as first-class CLI objects, not browser tabs
Harness CLI 3.0 (Sep 9) puts risk scores, review groups, and check→pipeline log streaming in the terminal so agents can triage and close the review loop without the 12-tab dance.
-
IdeaAMBIG: the hard part of agent coding is finding what the spec omitted
Sep 9 benchmark: 660 evidence-grounded gaps in research-method specs — defect localization is the bottleneck; gold clarification lifts codification-ready rate from 14% to 98%.
-
Blind repair loops damage clean code
Sep 9 arXiv note: iterative LLM bug-fixers invent bugs in correct programs faster than they heal real ones, then oscillate forever — a stopping-condition problem for review-time APR.
-
On call for dark code: runtime is the only truth left
Sep 9 LeadDev field note: authorship, memory, and comprehension fail under agent volume; gate on detection and blast-radius containment, not per-line human recall.
-
The Factory: tickets in, gated PRs out — humans keep the merge button
Sep 9 mikeroySoft: label-driven always-on factory (triage→worktree→CI gate→evidence-bound review→merge); self-declared done is unreliable; 14 self-merged tickets; events.jsonl accountability, not autonomy theater.
-
Mouse verification loops: same model, 15/30 → 25/30
Sep 9 Mouse: on Runta’s FrontierHarness (Kimi K3 held fixed), OpenCode scored 15/30; Mouse’s deterministic post-turn verify loop hit 25/30 — checks, deleted-test block, MOUSE_AUDIT evidence, and cache-friendly retries.
-
Opair: the harness that refuses autonomy on purpose
Phil Booth’s Opair makes shared ownership the product goal — no shell, scoped tools, driver/navigator role swap — so the human stays in the loop before the 5k-line agent PR arrives for review.
-
Independent review as a skill inside the coding agent
Qodo’s Agentic Toolbox (Sep 9) packages context, standards, and an independent review pass as skills coding agents can call before a PR exists — and again to resolve structured findings after.
-
Salesforce’s agentic shift: maturity curve, then a swappable harness
Sep 9 Part 2: scaling agentic coding across ~15k engineers — nine-stage maturity language for managers, token discipline as quality, and a model-independent harness so routing/skills/context survive provider churn.
-
Static green is not secure: measuring the SPDF gap
Sep 9 paper: of 654 Bandit–Semgrep-clean Python samples, dynamic exploit verification confirmed ~1 in 7; several frequent CWEs flagged by neither scanner.
-
I stopped reviewing code and started reviewing agents
Sep 9 field note: when agents flip the write/read ratio, the real review target becomes permissions, tools, and who sits between output and production — not every generated line.
-
Watermark the trajectory, not just the patch
Sep 9 TrajMark: ownership + segment-level tamper localization for coding-agent trajectories — because patch watermarks miss process provenance and local edits can keep a global ownership signal.
-
Don’t localize from the issue text: execute, then diff the runs
Sep 9 XAgent: differential analysis of failing vs passing executions for localization, plus context-aware test augmentation so validation doesn’t overfit the ticket — 62% resolve / 72.8% function loc on SWE-bench-lite.
-
CapScope: authority lives in the harness, not the prompt
CapScope freezes a task-wide authority ceiling and per-agent typed capabilities outside the model — injected effects drop from ~33–47/75 to 3/75 without tanking repair success.
-
Authority that lives outside the workspace still has to gate the mutate
Sep 8: cross-substrate authority gap — identical files can be safe or unsafe; planner observation fails, a deterministic execution guard at the mutation boundary catches unsafe publish intents.
-
Fleet size is 1/f_H — supervise what you can verify
Sep 8 Brian Curry: six-layer fleet architecture; review-queue bifurcation; sustainable agents scale as the reciprocal of the human-routed fraction — gates, not headcount, buy overnight capacity.
-
ExecCritic: same trajectory writing patch and test is a confidence trap
ExecCritic separates Test and Repair agents, freezes fail-closed tests, and shows test quality decides whether execution feedback helps — composing trained roles hits 72.6% on SWE-bench Verified.
-
Harness or model? Same-model contrasts don’t crown a winner
Sep 8 Arjmandi: paired same-model harness contrasts on a private 256-task suite — averages near zero (−1.25 / +1.25 pp); Opus strata reverse by task type; 22/81 wall-clock cancels already had a passing patch.
-
The agent that never merges: overnight pipeline, human irreversible step
Sep 8 Allen Hutchison: scheduled auto-dev ticks triage→plan→build→address review; harness forbids merge — morning work is decide-to-ship, not reverse-engineer intent.
-
Mastra's factory started fully automatic, was too bursty to review, and went stage by stage
Sam Bhagwat on Mastra Factory, which now writes 25-35% of Mastra's PRs and closes 50-60% of its issues: full-auto mode flooded review when backlogs were imported, wrong upstream assumptions produced PRs that had to be closed, and the project now asks the community for detailed repros instead of small PRs.
-
Pinishv: nine control surfaces — or you have an ungoverned factory that looks healthy
Sep 8 Pini Shvartsman series finale: agentic factory as nine human control surfaces (readiness → halt); five failure modes where metrics improve while judgment dies; F0–F4 maturity without buying a platform.
-
Under agent PR volume, teams are redesigning review — not abandoning it
Gergely Orosz surveys how labs and startups actually handle the review flood — from reviewing the AI review, to blast-radius triage, to reviewing plans and schemas instead of diffs.
-
SWE-Bench Pro Verified: when the score was reward hacking
Sep 8 paper: anti-hacking + task refinement cut inflated SE-agent scores (GLM-5.2 78.80%→57.32%); gold-patch leakage via git, FS, network, and metadata.
-
Zakas’s five roles: where you stand on the agent spectrum
Observer → tech lead → architect → engineering manager (software factory) → product manager — pick deliberately by trust, blast radius, and how far you are from the code.
-
It is not my code anymore
Sep 7 Camargo research note: authorship ≠ responsibility after AI-generated defects; acceptance evidence should start from the demand, not the diff.
-
Permission with conditions: the new AI contribution policy
Sep 7 survey of 281 OSS AI policies: most allow AI code, but demand human involvement, accountability, and increasingly disclosure — plus ten countermeasures against AI slop at the PR gate.
-
Scanning the harness: agent config is a supply chain now
Across 3,171 public repos, 16% of multi-component agent setups carry a confirmed security defect — unpinned MCP servers, scoped-looking shell grants, skills that pre-approve Bash for whoever installs them.
-
SCH: close the lid — durability in checkpoints, not the runtime
Sep 7: serverless coding harness on AgentCore microVMs — handoff/task modes, S3 checkpoints, Telegram approve; ~$3.45 compute over 12 days vs always-on EC2.
-
Trust the spec, not the code
Sep 7 Farchi case study: natural-math specs + invariant proofs + runtime coverage; code is regenerable output the factory need not read line by line.
-
Two months of scheduled routines: 82 merges, then review can’t keep up
Sep 6 Hodalog: Claude Code Routines grew 2→8; 119 Issues, 78% adopted, 95% PR merge — plus a weekly PR-care bot; adoption decision and Issue readability become the bottleneck.
-
7×24 Claude Code: five layers so the agent runs while you don’t
Sep 5 field build: cron+claude -p, hooks to phone, CLAUDE.md/auto-memory/resume, tmux+SSH, worktrees — from ‘Ferrari sitting inert’ FOMO to morning PR/CI already done.
-
Reviewer habituation shows up in embeddings, not lexical metrics
Yu et al. track 400 repeat reviewers of agent PRs: approvals climb while hand-crafted comment features stay flat — sentence-embedding centroids move, and language shifts lag the approval change.
-
Beyond generation: Verification Tax and Production-Qualified Change
Bhati’s Sep 2026 synthesis names the attenuation curve from agent commits to releases, and reframes output as PQC — changes that cleared review, test, security, and ops gates.
-
1,135 agent PRs: shared blind spots and lying evidence
Lessons from an autonomous GitHub team: same-model author/reviewer blindness, dry-run evidence that lied, roles-as-data, structured envelopes, and refuse-to-guess gates.
-
Harness engineering: the repo enforces what prompts only suggest
Sep 4 Frisinger at 8th Light: verification is the bottleneck; fail-closed gates, evidence ledgers, and provenance turn overnight agent output into accountable inventory — not plausible debt.
-
Inside a software factory: eight stages, don’t overbuild
Paul Iusztin maps factory stages onto the SDLC, puts humans on brainstorm/plan/final check, and warns that first-version autonomy theater often fails.
-
Five leaders, one split: review intent and risk — not every line
Sep 4 Rootly field note: Flare, Airbyte, CircleCI, Moloco, Twingate leaders on agent PR volume — spec-first review, risk tiers that skip low, earlier machine verification, and a named human who owns the merge.
-
Acceptance authority is not an agent permission
Wang and Liu’s Sep 4 theory note: Trustworthy Change as the engineering object, Human-Agent Cells that produce evidence without acceptance rights, and Responsibility Topology for who can absorb residual risk.
-
A-CX: one person, one week, 152 merged PRs, and the planning was where the hours went
Sep 3, Ilpo Niva at A-CX: unattended parallel agents closed 204 issues and merged 152 PRs in a week, but only against issues with an objective, design, checkable criteria, and dependency order. Worktrees, an atomic claim registry, and a fail-closed merge checkpoint keep it safe; about one in twenty machine findings is rejected on review.
-
The Agentic Pivot: work around code is the product
Atlassian research: 74% see faster generation but 78% still use traditional peer review; leaders want a governed system of work agents can write back into.
-
Guild’s factory: specialized agents, separate reviewer, human merge
Guild’s Sep 3 Software Factory receipts — 34% of merged PRs, separate review agent before humans, maintenance agents that keep the loop alive, ~$12 model spend per merged PR.
-
The psychological tax of AI in the eng org
Sep 3 case study (N=21): a year into AI adoption, engineers report accountability anxiety, craft-identity disruption, meaning erosion, and workload intensification — including around review.
-
Codex Watch and fix: a 10-minute heartbeat that owns the PR last mile
Sep 3 RuntimeWire: OpenAI desktop Codex attaches a recurring heartbeat to a PR — re-reads live GitHub state, repairs CI and conflicts in a worktree, optionally merges; continuous ownership instead of one-shot heroics.
-
SWE-Gate: green tests are not a merge gate
He et al. build a 303-instance benchmark where review constraints from real PR comments are tested separately from functional tests — and 221 of 644 functionally green repairs still fail the review bar.
-
The AI code review packet: receipts for agent PRs
A practitioner template that forces intent, surface area, risk, evidence, edge cases, rollback, and review focus onto every agent-written pull request.
-
Secure the ADLC at its control points, not with a bolted-on AI review
Sep 2 Endor Labs field note: agent PRs break the one-reviewer-per-change ratio; put security context at generation, architecture-aware review on every PR, and reachability before remediation.
-
HELIX: risk-tiered verification rungs, not more generation
Sep 2 Maya Shenoi methodology note: HELIX rebuilds the SDLC around verification contracts — R0–R3 autonomy, machine gate then adversarial AI then human intent review, and no merge without a verification record.
-
Laycock: stop waiting until the PR for the important talk
Thoughtworks CTO Rachel Laycock answers Houck — same Meta/DX crush numbers, different fix: shift judgment left via pairing and design, review by exception, don’t preserve PR ceremony with an AI pretend-reviewer.
-
LLM-as-a-judge is not an oracle—gate changes with deterministic checks
Eleven failure modes from production self-improving loops, and PROCTOR’s Teacher-Student design with hermetic sandboxes and frozen holdouts.
-
Validation has to move left twice — not into a bigger PR
Sep 1 Allstacks / CircleCI field note: 28M workflows, main success 70.8%, merge efficiency 3.9→1.3; put checks in the agent loop and adversarial review in the spec.
-
Checkly: harness first, then one overnight Claude rewrite
Sep 1 field report: 92M msg/day Node→Go with Claude Code overnight — golden-file black-box harness defined correctness; agents shipped ~13k lines; zero incidents after a queue-topology miss taught boundary fidelity.
-
Copilot code review can now approve — if you turn it on
GitHub’s Sep 1 changelog adds approval assessments on every Copilot review, plus an off-by-default path for Copilot approvals to count toward required reviews, scoped by path.
-
Factory throughput without review capacity is a measurement failure
Sep 1 Faros field note: software factories coordinate agents, harnesses, and verification — and the 2026 Engineering Report shows AI adoption lifting throughput while PR review time jumped 441.5%.
-
Harness-of-Harness: loops that separate build tests from independent eval
Sep 1 HoH wraps existing coding harnesses in planning–coding–testing iterations that force new capability each loop, schema-constrained artifacts, and an independent tester — +52% avg relative vs standalone harnesses.
-
HarnessDev: the unit of eval is the harness
Sep 1 benchmark: can models create and evolve their own agent harness? Generated code/search harnesses lag human ones; evolution gains are unstable and model-tied.
-
HarnessEvolve: diagnose against reference trajectories, then dual-gate the harness edit
Sep 1 Jiang/Chu/Tian et al. (Huawei): reference-guided root-cause vs failed runs; quality gate (leak/bloat) + performance gate (batch↑, recent↘ε); CloudCoreNetwork-QA 43.4%→86.9% (+21.6pp over GEPA).
-
Six guardrails so an unattended bug-fix loop can leave the desk
Sep 2026 Brett Ridenour: draft PRs only, P2/P3-only autofix, never push main, halt on broken baseline or three consecutive fails, hard wall-clock cap — scar tissue as the safety model.
-
Staff the line: spec author, verifier, fleet ops, line owner
Augment’s org-design guide: agents absorb implementation, so scarce human work is spec, verification, halt authority, and line ownership — with receipts on review share collapsing under PR flood.
-
Schwarz: plausible specs are not enough—expose SMT-local repair
Aug 31 Ke/Wu/Li: agent proposes specs/lemmas; checker owns authority via SMT discharge + replay; 1367/1475 passes; local lemmas ablation −221 tasks; SV-COMP ReachSafety 91.5% vs CPAchecker 60.1%.
-
CI under agent volume: the pipeline has to understand the change, not just run the suite
Aug 31 Pini Shvartsman: continuous integration → continuous verification — comprehend, choose risk-based strategy, investigate failures (no blind retry), emit an evidence package; escalate up, never auto-approve consequential deploys.
-
Commits doubled; verification didn’t
Signadot/TNS on GitHub’s 2.9B monthly commits: generation is machine-paced while runtime verification still queues through staging—and how per-change isolation closes the gap.
-
Mouse: seven overnight requirements + house rules that grade events
Aug 28 field note: unattended coding needs verify-outside-worker, admissible evidence (not narration), budget reservation, resumable controller, and stop-at-merge — plus inject/copy/grade playbooks.
-
openJiuwen: static harnesses lose when long-horizon evidence keeps arriving
Aug 28 openJiuwen team: Rail-composed Inner/Outer loops + runtime adaptivity — SWE-bench Verified 82.6% (+3.4 pp) and Terminal-Bench 2.1 87.19% (+3.39 pp) over strongest selected leaderboard points.
-
Verification surface: a tool only helps where its reach matches the failure
Controlled study of 1,116 agent-built web apps: boot probes, shells, and screenshots help only for the failure modes they can observe.
-
37 agents, one afternoon, every test green, and two real bugs only a reviewer found
Aug 27 Ralph Duin, Inspired by Frustration: a lead agent that never edited a file ran 37 subagents across seven worktrees, finished 15 of 16 tasks for about $49, and merged nothing to main. The two bugs worth reporting got past 1,374 passing tests and were found by fresh-context reviewers who opened the rendered PDF and reproduced a stale cache.
-
Worktrees isolate files, not plans: Foremerge’s semantic merge tax
Aug 27 Woodhead/Foremerge: parallel agents in worktrees can merge clean and still break design — declare semantic scopes before edits; acceptance gated on harness-run checks, not agent claims.
-
After the factory works: trust, seats, memory, intent
Aug 27 Travis Frisinger: six months, ~1,700 merges — throughput is the honeymoon; fail-closed gates, calibrated seats, institutional memory, and scarce Purpose form PAAA.
-
Harness: agent-ready SCM + AI review gates when PRs outrun humans
Aug 27 Harness: Code Repository scaled for agent PR volume + AI Code Review with risk-grouped diffs and required AI Checks — scoped agent identities so continuous factories don’t overwhelm human review.
-
MCR-Bench: multi-round code review is not a one-shot classify task
2,269 real multi-round review tasks with defect lifecycle labels—LLMs degrade as rounds grow and miss low-salience defects.
-
Uber’s software factory is now mostly agents — and they are optimizing the cost equation
More than 70% of Uber PRs are attributed to agents. Their new post is a field manual for measuring and cutting agent spend without starving quality, including how they bench uReview for code review.
-
WikiSkill: compile agent experience into reusable skills
Turns scattered optimization history into persistent skill knowledge so agents can reuse what they learned across tasks.
-
PILOT: live steering mid-run beats post-hoc harness updates alone
Aug 27 Xiao et al.: supervisor–worker harness redirects active workers and distils skills/memory live; +14.6/+12.4 pp Terminal-Bench self-improvement, tokens down ~43–47%.
-
HarnessLens: verify the behavior you changed, not a random task batch
Aug 27 Xu et al.: budget-aware harness evolution with behavior-aware verification + attributable-evidence gate — 7.6–13.6% held-out gains at ≤200 interaction units vs thousands of fixed-set rollouts.
-
Build a review agent: guidelines, guards, journal, judge
Cline walks through a two-pass PR reviewer on their SDK—record findings, then a stricter judge—plus deterministic tool guards so the reviewer stays read-only.
-
Exit code 0 is not done: DoneGate’s fail-closed completion gate
Aug 26 DoneGate: worktree + path leases + independent validation — commit claim must match HEAD, allowlist/denylist, required checks, and execution evidence before completion.
-
Same model, different harness: the solver is the pair
Aug 26 Lewis (Yuj): paired control vs closed-loop harness on fixed weights — tight 20k Verified F2PF 28%→49%, solutions 43→72; frozen treatment transfers across four model designs; wide window mostly erases the gap.
-
Blocking is not diagnosis: the Verdict–Diagnosis gap
Aug 26 paper: MalPR-Bench scores attributable PR security review; CodeRabbit blocks almost as often as PRGuard but identifies fewer target vulnerabilities — especially when evidence sits outside the touched files.
-
When review alone no longer scales: layered supervision
Aug 26 Stolze & Strässle (ESEM 2026): five-practitioner study — preventive + executable + human layers; review 3–4× generation; promote recurring findings into lint.
-
The coding session is too small a boundary for the work
Aug 26 Sumant Thakur: Spotify Xirp, AWS Kiro Crew, Warp Factories — durable state, isolation, event starts, and verification outside any one agent session.
-
Warp Factories that score themselves and open fix PRs
Warp’s self-improvement loop: scorers for efficiency/quality/compliance sample runs, then a scheduled agent proposes skill/config diffs from failures.
-
Completion is a harness verdict, not a missing tool call
Aug 26 WonderLab / MyCodeAgent: single immutable loop + TransitionReason traces; completion gate PASS/UNVERIFIED/FAIL with feedback injection; terminal reasons exhaustively named.
-
StarHarness: stratified search, proposer-hidden selection, then held-out enterprise transfer
Aug 25 Esakkiraja et al. (ServiceNow/Mila): evolve prompts/tools/skills/MCP/subagents with failure-mode strata + hidden selection; +20–35 pp on ITBench / EnterpriseOps-Gym / AutomationBench, frozen transfer across GPT and Qwen.
-
468 merged PRs in 23 days from an always-on agent fleet: the validation stack is the boss, and AI review is capped at two rounds
Aug 25, Maurice Kleine. He used to be the message bus between 4 to 10 agent tabs and felt stressed whenever his laptop lid was closed. He moved the fleet to a €100/month server with supervisor agents that route tickets around the clock. 468 PRs merged and 537 tickets completed in 23 days. His first AI review setup nearly froze the fleet, so review cycles are now capped at two before a ticket blocks and comes to him.
-
Agentic engineering: trust lives in the system
Scott Spence’s practical definition: agents write; engineers own context, scope, validation, evidence, and review—with harness numbers from real projects.
-
Foreman: four stations, cross-vendor review, draft PR
Aug 25 Vercel guide: eve software factory moves labeled work through triage → plan → implement → independent review on a different model vendor, with bounded request_changes loops and evidence against acceptance criteria.
-
Build-verify as control flow: 34% broken agent PRs → gate blocks the PR
Aug 24 Genovese/Regolo: OpenCode in GitHub Actions with a bash verification gate — pytest, forbidden paths, secret scan — PR opens only if success(); prompt alone is not enough.
-
Factory metrics: generation vs review bandwidth
Augment’s metrics guide treats the agent fleet plus human reviewers as one production unit — Faros/LinearB receipts show PR volume up while review waits explode, and DORA’s verification tax names the spend.
-
Night shift: OS schedule → isolated worktree → morning accept/discard
Aug 23 Quiroga TaskKeeper: scheduled Claude/Codex runs in a fresh git worktree; VS Code closed; local review inbox with cost + diff; accept merges locally, discard deletes the worktree — never auto-push.
-
Harness → attention interface: human attention as the scarce resource
Dan McAteer on Latent Space traces harness eras from AutoGPT to Claude Code, then argues what’s left after models absorb scaffolding is an interface to human attention.
-
Loop engineering in the wild: 217 confirmed always-on agent loops, almost no committed state
Aug 22 Lulla et al.: gray-lit definition + mine of 36,710 repos; 217 confirmed autonomous loops (mostly event PR-review/cron triage); configs committed, runtime state almost never versioned.
-
Human judgment doesn’t leave the factory—it relocates
Osmani on lights-on factories: encode taste in the environment, budget verification like performance, and keep ownership on what ships.
-
Closed-loop AI-to-AI review is real — and still a minority
Selvanayagam and Ghaleb link 248k agent-authored PRs that got at least one AI-attributed review on CodAGE — cross-product review is ~1.6% but grew two orders of magnitude from 2025-Q1 to Q3.
-
Put the gate at the station that produces the defect
Aug 21 Augment field note: agent PR volume turns merge-only review into terminal inspection — station-level quality gates (intake→spec→build→verify→ship) with named owners stop defects where generation is still cheap.
-
Bounded Loops: done is a gate verdict, never the worker’s claim
Aug 20 Bhardwaj et al.: worker + independent gate + declared budget; prove termination under repair and ledger-supported done; instrument finds 47 vacuous gates in shipped loops.
-
Verification debt: when generation outruns trust
Aug 20 field note: verification debt as the queue behind cheap AI implementation; independent layers beat same-model self-checks; measure verified delivery throughput.
-
Own the review standard as a skill, not a vendor slider
Tessl’s Aug 20 Code Review launch: team standards live as versioned skills you can inspect, eval, and point at another repo — whole-PR context, stateful re-review, and a path toward an autonomous merge gate.
-
Mandible: 12k agentic review verdicts and the round-4 elbow
Aug 20 Watson Labs / Swamp: Gymwasp’s Mandible factory — 372 issues, 1,801 review rounds, 11,967 verdicts. 65% mergeable after one round; after round 4 the loop oscillates; 29% shipped on warn; volume 3× with median rounds flat.
-
Start with the harness, budget the verification, and expect each feature to get slower
Aug 19 Addy Osmani, advice file in his reference software factory for Claude Code and Codex: a plain harness plus a clear issue gets surprisingly far, the factory layer earns its place through back-pressure and independent verification, and ten features with an independent verifier cost at least 160 minutes of suite time before any human looks.
-
Compressing file reads isn't the bill — tool-schema filtering is
Aug 19 Chen/Shi (Paritok): instrumented Claude Code/Codex sessions — tool filtering saves ~21–57K tokens/turn (linear); content compression ~2%/turn but compounds ~3,350 N²; single-shot SWE-bench quality ≠ multi-turn cost.
-
SemaPLC: agent cannot say done until the runtime agrees
Aug 19 arXiv 2608.18565: verification-gated PLC harness — spec check, real toolchain compile, live execution. Project-context tasks: 89.4% compile vs 52.2 dynamic behaviour — the gap is the argument.
-
Danjou on Yegge's Land Rush: the batch was too slow, but attribution was the cheap part
Aug 18, Julien Danjou (who runs a merge-queue company) grants that one-commit-per-green-build can't survive agent commit rates, but argues that landing 150 commits on a red main throws away the link from each failure back to the agent and intent that caused it.
-
Mandiant’s AVDH: skeptical validation for agentic code review
Google Mandiant details an agentic vulnerability discovery harness—threat model, entry points, hypothesis generation, multi-temperature validation, then human PoC.
-
Agents ship to dev overnight unreviewed, and one ticket field decides which work may finish alone
Goat Squad Studios: plan by day, agent loops by night, review in the morning. Every ticket carries an autonomy field, a missing field means it waits, and only work a test can prove is allowed to reach done without a human. Production still moves through one PR a day that the human reads.
-
22 nights, 56 PRs, 54 merged: the overnight loop stalled on tickets, not code
Yohei Nakamura audited his attended overnight coding-agent loop against its records: 72 tickets, 56 PRs, 54 merged, 0 reverted. Ticket defects outnumbered code failures 4.5 to 1, 18 of 38 constraints were learned only by failing, and verification needed a layer that distrusts the evidence itself.
-
More review agents can mean less real disagreement
A new paper finds naive multi-agent review collapses into false consensus, and that a smaller setup with an explicit critic does better.
-
Screenshot as oracle: done requires a fresh capture, not green DOM tests
Aug 16 yureki_lab: Playwright capture harness + report.json (overflow, console errors, failed requests) + stale-hash gate — multimodal screenshots close the UI loop agents already have for pytest.
-
Cole's dark factory: holdout validator and a perimeter the factory can't amend
Aug 14 coleam00: Archon cron factory — dumb label dispatcher, independent validate-PR holdout, governance files auto-reject; humans file issues and promote releases only.
-
A routine that runs is not a job that got done
Aug 2026 Paul Kuo: Cherny’s 388→180 maintenance PRs vs 150 days of launchd/Cowork jobs — silent failure when green logs lie; verification must sit outside the executor.
-
Dependabot Janitor: overnight merges only when the suite can prove boring
ZenBusiness (Witt, Aug 14): nightly Dependabot Janitor — deterministic semver/scope filter, then Claude Managed Agent skills; ~250 auto-merges / 800+ parked with receipts across ~30 repos in seven weeks.
-
Evals judge; guardrails constrain; the harness runs the night
Aug 13 Winston/Voss (Arize): long-running agents fail where chatbots never go — separate eval vs guardrail vs harness, give judges the trajectory, and feed failure explanations back as the next engineering brief.
-
PR-to-merge is the factory bottleneck — specialize the loop
Aug 13 Augment: generation is cheap; review/verify/repair/decision are the drag. Cosmos Experts (Risk, Deep, Pair, Verifier, Fixer) keep working until a human can merge with evidence — loop engineering, not a better comment bot.
-
24 CI-green merges, ≥9 live regressions: burst heroics vs gates
Aug 13 Claude Code #86278: ~12h autonomous session merged ~24 PRs with green required checks; user hit ≥9 live regressions the same evening — green CI ≠ production-qualified.
-
Code review is evidence — it does not own the outer SDLC loop
Aug 13 Qodo field note: independent review re-enters judgment under agent volume, but lifecycle authority still decides whether the task may advance after commits, CI, repairs, and stale approvals.
-
The harness never believes the agent
Aug 12 Denis Morozov: 37 packets, 37 merges, $3.89 — planted tests, named invariants, harness-run gates; the implementer’s ‘verification’ block is recorded and ignored.
-
AI SDK’s factory authors 25–35% of merges — humans still own the stamp
Aug 12 Grammel/Dodds: issue→classify→analyze→implement→review→backport in sandboxes; 70–80% of closed issues via factory; reviewer efficiency is the product, not more agents.
-
OpenFactory: always-on orchestrator, separate Box for review
Aug 11 Ascii Box template: continuous factory Box queues GitHub/Linear/Sentry issues; build Box plans/commits/tests; fresh verification Box gets the branch as a Git bundle — humans merge, never the factory.
-
Nightwatch: Captain plans by day, fleet ships overnight
Aug 11 Conveo: /nightwatch skill — Helm plans, remote Field Orchestrator spawns worktree workers, cheap per-ticket verify loops, one green master PR after 7.5h on 3 agents.
-
alibaba/open-code-review: battle-tested OSS review CLI and agent plugins
Open-source Open Code Review tooling from Alibaba’s internal assistant lineage, with Claude/Codex/Cursor plugins.
-
Dark factories: turn lights off only as far as proof reaches
Aug 10 Jeff Kazzee: stage-by-stage SDLC switches; verification buys autonomy; deterministic checks + independent judgment + real surface; a pilot of seven image calls with preservation and browser gates.
-
OpenCodeReview: inject determinism into uncertain review agents
Rule-guided dispatch, grounded file tools, and an asymmetric reflector that filters hallucinations—higher SEM-F1 at far lower token cost on AACR-Bench.
-
Four agents, 90 PRs a day, 15 merges: tier by risk surface and let revert rate move the line
Aug 10 steezr (Johnny Unar): an inherited client repo had 300+ open agent PRs and two exhausted senior reviewers. Path-based risk tiers took about 60% of agent PRs off human review; commit trailers recording provenance showed a 1.8% revert rate on auto-merged low-risk PRs against a 2.3% human baseline, and 11% on high-risk surfaces.
-
OpenLoopEvolve: the Loop Policy is a versioned asset—Champion–Challenger before activation
Aug 10 Wang/Li/Li/Li: externalize obs/plan/memory/act/verify/recover/stop/budget as Loop Policy Bundles; online+offline evolution with paired eval, robust gate, task-boundary activation, rollback—YC-Bench lifts returns and survival vs fixed π₀.
-
RETRACE: verify the patch by reconstructing the problem it actually solves
Aug 9 Li et al. (Concordia): training-free bidirectional reconstruct-and-verify after agent patches; +7.0 / +3.6 Pass@1 on SWE-bench Verified vs mini-SWE-agent; Self-Refine with similar budget loses.
-
Ouroboros: the harness evolves through reviewed commits—and Hope ran that loop for 161 days
Aug 8 Razzhigaev et al.: self-developing coding harness with free + experience-driven core evolution under a blocking diff gate; Opus 5 Terminal-Bench 2.1 86.74% audited; Hope—161 days, 1,085 self-mod commits, 63.5% recent review block rate.
-
Auto mode: 9× between interrupts, three PRs by morning
Aug 7 Anthropic: Nuro 10pm–5am hill-climb → 3 PRs; classifier vs permission fatigue; FOMO answered only when eval signal + deny rules + PR-as-boundary exist.
-
When agents flood the queue, review capacity is the real gate
Aug 7 Assembled: agent PRs left humans behind (TTF review 3.5h→16h); dual-model auto-approver + policy ridge hit 2.4× merges, 3.5× large-PR throughput, 43% auto-approvals — without raising reverts.
-
DVx Overnight Build: the handoff is the product, judged by cost per accepted change
DVx Field Manual 002 splits unattended agent work into three shifts (human decides, agent executes, human accepts), with green/amber/red autonomy and a morning review packet read tests-first.
-
Houck: the invisible output of review is the org
DX’s Brian Houck on Meta’s review crush under agents — RADAR’s risk-stratified automation works, but automating diffs without protecting knowledge transfer and intent debt just hollows the team.
-
ADLC: software factories need Workflows, not babysat SDLC steps
Aug 4 Cloudflare: replace human-paced SDLC with Agent Development Lifecycle — programmatic, preview-per-agent, event-driven Workflows that spawn heal/review agents; @cloudflare/ci self-heals and stages fixes for merge.
-
Astro’s issue factory: triage as a label-driven agent pipeline
Cloudflare/Astro cut open issues from 200+ toward zero with isolated subagents that reproduce, diagnose, and ship preview fixes inside GitHub Actions.
-
Overnight delivery at Tilt means review-ready by morning, not merged
Aug 4 HILLS Lab field note on how one engineer runs AI-assisted delivery overnight at Tilt: a written definition of done before the run, a fresh agent session per ticket, isolated worktrees, explicit stop conditions, and a morning manifest of checks, blockers and the decisions still needed from a person. Merging and deploying are separate outcomes that need their own authorisation.
-
EA-Graph: anchor verification claims to artifact content, or admit unprovable
Aug 4 Hsu/Chi/Everett: sub-path anchors + freshness≠evidence; after upstream drift Haiku ANCHOR F1 median 1.0 vs PROSE 0.27; refuse when support withheld—never guess.
-
Overnight agents: infrastructure, not a longer Claude session
Aug 4 field guide (updated Sep 14): task queue → worktree-isolated Codex exec/Goal Mode → watchdog → morning triage — atomic verifiable tasks, AGENTS.md constitution, and when not to go lights-off.
-
Merge rate became AI coding’s scoreboard — and the boards don’t agree
Aug 4 Pipeline Mag: four 2026 merge-rate studies disagree; rejections are mostly silence; Faros shows review time +441% and churn +861% under heavy AI — throughput ≠ what survived.
-
LongHorizon-Harness: only audited facts cross the round boundary
Aug 3 Ma et al.: Manage–Execute–Audit keeps task state outside the growing session; fresh-context executors, read-only auditors, and Claude Code/Codex backends — WeaveBench 51.8%→80.7% with the same model.
-
Codex harness files: plans are not evidence
Aug 3 Tony Alaniz (OpenAI Cookbook): phased harness/build files, human gates, and an append-only build-log that records observed verification — so long-running agent work cannot mark a phase complete on narration alone.
-
Agentic software factory: history, then check the utilities
TrueFoundry’s field guide: factories aren’t new; agent-native ones need model supply, credentials, QC, telemetry, and spend metering—not just more agents.
-
Yegge's Land Rush: 40 agents, 175 commits a day, and a merge queue that stopped bisecting
Aug 2026, Steve Yegge: running 40-plus agents around the clock against a half-hour build, his bisecting merge queue grew past 100 requests and stalled. His fix is to land 120 to 150 commits on main at once and have agents repair whatever goes red.
-
HarnessCompass: evolve the harness under a generalization gate
Aug 3 Zhang et al.: constrained + proactive-feedback + component-wise harness evolution — SWE-bench Verified 54%→66% Pass@1 in 5 turns; held-out 60.4% vs AHE 54.7%; frozen harness lifts Claude Sonnet 4.6.
-
Claim Plane: pre-write admission buys reliability—mostly by serializing
Aug 2 Nikolaev confirmatory: static Claim Plane lifts pair pass 23.3%→50% and integration 65.6%→96.7% on 30 CooperBench pairs—but serializes 96.7% of runs; dynamic scope fails closed on undeclared regions.
-
Five hours of agent work only holds up behind critique gates
Aug 1 Sudarshan Shenvi: testable requirements, a separate critic that refuses partial passes, always-on sandbox with permissions off — review the output, not the babysitting.
-
LoopsBench: end-state pass rate misses the loop—best config resolves 25%
Jul 31 Li et al. (Microsoft): 112 dependency-DAG tasks, 5,300+ units, flow-aware test release + live regressions; Opus-4.7 + Claude Code + outer continuation tops out at 25% resolve.
-
Parallel Claude lanes need a local merge queue — and a human promote
Jul 30 funador: FIFO land + checkCommand for multi-worktree Claude Code; promote is human-only; slow gates cap landings/hour; checkCommand≠human review (on purpose).
-
Tycho: never accept the model’s summary — only artifacts it can’t narrate
Jul 30 Swail Tycho: completion-hook ledger that checks files, commands, exit codes, and claims against git/FS — no LLM in the trust path; VERIFIED/FAILED/STALE for overnight review.
-
ECLoop: hold the edit until the trajectory shows the evidence
Jul 30 Xu et al. (Concordia): evidence-conditioned execution layer gates premature commitment on SWE-bench Verified; +4.8–11.8 Pass@1 across two models and two scaffolds while cutting tokens up to 12.1%.
-
nokto: plan → worktree → independent review → PR — never auto-merge
Jul 2026 noktohq/nokto-agent-orchestrator (still shipping Sep 21): Claude/Codex/Gemini delivery loop with allowlisted verify, secondary review, JSONL audit, resume; human merge is hard — PRs carry the receipt, not a silent land.
-
Shopify: the harness outlasts the model
AppSec’s Dispatch pipeline — partitioned hunters, cross-model verifiers with real integration-test oracles, then fix PRs — so better models don’t just dump confident noise on developers.
-
The expensive overnight run is the one that never crashes
Jul 29 Toolshed Labs Nightshift: four bash habits — rewrite handoff, fail-closed verify-gate, runaway caps, morning brief — so unattended agents stop producing confident unverified inventory.
-
CHILL-Harness: don’t always-full the overnight loop — authorize only advantageous interventions
Jul 28 Fu et al.: causal CIEL+ARCO orchestration — GAIA 71.3% at 28.4% fewer tokens; SWE-Verified 65.6% at 13.1%↓ tokens / 25.9%↓ time; Always-Full ablation collapses GAIA to 27.7%.
-
LinearB 2026: agent PRs merge less without owners; AI review lifts yield
Jul 28 LinearB 2.7M-PR benchmarks: elite orgs merge agent PRs at 79% vs 92% human-only (37% at top 60%); ownership — not model skill — is the limiter; AI code review lifts PR yield up to +5–7pts while generation moves the bottleneck downstream.
-
AgentRelay: overnight control is PreToolUse blocks + evidence-tagged morning reports
Jul 27 MalyStern: local Rust supervisor for Claude/Codex/ACP — block forbidden tools before they run, gate done on real tests, undoable checkpoints, budget kill-switch, VERIFIED/RECORDED/ESTIMATED report lines.
-
Audit the deterministic harness: hooks fail open; CLAUDE.md is a request
Jul 27 John Young: inventory vendor defaults for permissions, hooks, sandbox, telemetry — exit 1 ≠ block; Cursor failClosed defaults false; sell blast-radius control, not quality.
-
ralphctl: green overnight means nothing if the agent edited the tests
Jul 27 Lukas Grigis ralphctl 0.17: generator–evaluator ralph harness; evaluator must audit diffs for verification tampering and cite hunks — FAIL and PASS both need evidence.
-
agent-sessions: freeze the checks, grade separately, stop at the merge gate
Jul 25 lmorchard: unattended issue→PR harness — acceptance criteria frozen before work; separate read-only grader; auto-ok vs needs-review tiers; eligible-for-auto-merge is a finding, never a machine merge.
-
The Trust Ladder: review autonomy is earned, measured, and demotable
July field note: Elstner’s four-rung promotion architecture for AI code review — shadow → advisory → soft-gate → hard-gate — on trailing precision, availability, latency, and false-block rate, with silence treated as failure.
-
Light and dark factories: verification is the neck
Jul 22 Addy Osmani: loop → harness → factory; dark ships unread diffs; lit keeps judgment on the outer loop — autonomy only as wide as cheap, hard-to-fake checks.
-
Verification loops as skills: encode the morning tax once
Jul 22 Anthropic (Delba de Oliveira): turn the manual checks you keep repeating into Claude Code skills — standalone, embedded, chained, or on every PR — so modest continuous verify runs without relying on memory.
-
Cross-model review is asymmetric: Claude reviewing Codex helps
Controlled pairings of Claude and Codex as writer/reviewer show one direction lifts pass rates; the reverse can hurt.
-
Overnight agents that never push: Magrathea’s read-only worktree factory
Jul 22 Magrathea field note: labeled-issue dispatcher, per-issue worktrees, test/build/cdk gates, and a hard no-push boundary—defer colliding work, report empty queues honestly, leave closeable bugs for humans.
-
Nightgauge: issue-to-PR overnight — six stages, twenty-one deterministic gates
Jul 22 Nightgauge (open-core): local issue→PR pipeline with isolated stage handoffs, build-as-hard-gate validation, batch overnight runs, cascade circuit breaker — nothing ships unmeasured.
-
Orbital Witness grades its harness: merged PRs per engineer doubled, a nightly bug hunt, and no measure yet of human attention
Jul 22, Andrew Thompson (CTO, Orbital Witness). Scored against Ryan Lopopolo's twelve harness-engineering theses, Orbital's harness roughly doubled weekly merged PRs per contributor on a legacy repo, to mostly 6 to 7, at the same review and CI bar. A nightly bug hunt has sceptic agents try to refute each finding before a fix ships for morning review. Three theses are only half-built: turning feedback into infrastructure, retiring nightly loops, and measuring attention rather than cost.
-
Verify-first: 46% of agent fix PRs never merge
Jul 22: AIDev shows 46.41% agent fix-PR rejection; instructions alone don’t lift merge rate — put rules/tests/human gates inside the harness before more tools.
-
Anthropic’s security job becomes monitoring loops
Deputy CISO Jason Clinton on securing an SDLC where Claude authors ~80% of merged code — multi-agent PR review, risk-tiered automation, shadow mode, and identity boundaries instead of prompt trust.
-
Harneloop: portable harness units with artifact-aware tests and evidence-gated promotion
Ker102 Harneloop (v0.0.2): self-evolving harness units — capture artifacts/traces, diagnose, candidate patches, promote only with evidence; ViperMesh case study: 2.53× speedup, +8.19 visual, −90.9% tokens vs Blender MCP baseline.
-
Cursor’s agent factory: merge is a policy over evidence, not a vibes check
Jul 20 Verdi/Gilhuly (Arize Observe): video demos as trust units; ~30–40% ecosystem PRs merge without human review; risk-scored auto-merge; Bugbot learns from corrections; self-driving PRs.
-
Two unattended days: gate completion on artifacts, not reports
Jul 20 Solar/Favur: six-session 2048 SOW run — false completion, oscillation, context exhaustion, review cost; validators cite docs/hashes; play-the-game signal beats green suite; task drift still open.
-
Custom Code Review rules for Codex via AGENTS.md
July 2026 developer blog: repository rules in AGENTS.md that Codex Code Review can cite on findings.
-
Warden: overnight fleets are easy — the morning merge train is the product
Jul 19 Warden: worktree-per-unit agents, clean-room verify + fail-closed AUDIT PASS/FAIL, then digest ranks survivors and integrate re-runs every gate after each merge — base branch untouched.
-
Nightshift: a clock-out gate keeps the punch list a contract, not a vibe
Jul 2026 Orwa Mahmoud: Claude/Codex/Cursor shifts with persistent punch list, park-don’t-ask, watchman recovery, morning receipts — AdaptTable overnight: 9 items, gate-green commits, local until human push.
-
Ten agents overnight, ten PRs by morning, and review fatigue that looks like continued work
Jul 19, Xinwei Xiong. Ten parallel overnight tasks produced ten convincing PRs; by lunch he had reviewed three, the eighth he just merged, and that night he queued ten more. A week of about three times his usual merged code shipped no extra features. His fix is raising first-pass correctness, not adding agents.
-
Agentic reviewers in the terminal: better localization, and re-checking that predicts failure
Melbourne, Monash and Atlassian researchers ran Claude Code, Gemini CLI, Rovo Dev and CodeRabbit on 362 human-verified review tasks. Repo-grounded agents roughly doubled precise, human-aligned comments over static LLMs, but exploration and validation dominated cost, and failed reviews carried more validation, not less.
-
What a software factory is: four observable properties
Factory.ai’s definitional guide: standardized inputs/tooling, measurable output, replayability—and how agent-native factories change the throughput bound.
-
Latch: converge agent PRs to mergeable, then stop — a human always merges
Jul 18 Latch: independent reviewer≠fixer identities, recursion-guarded fix pushes, 3-cycle cap, anti-tamper on workflows/policy — clears the agent-PR firehose without ever pressing merge.
-
Hard execution boundaries: the model proposes; the harness decides what runs
Jul 18 Stanley Yang: vendor-neutral coding-agent harness — disposable workspace pinned to a commit, typed tools, external budgets, independently captured patch, trusted verification; model never owns shell, credentials, or success.
-
Night Agent: spend the window before credits reset
Jul 2026 jooray/night-agent: reset-anchored unattended missions in an isolated worktree; separate verify+commit per fix; never pushes — FOMO operationalized as using the subscription before it evaporates.
-
Least privilege for agents: delegated identity, ephemeral tokens, MCP intersection
Harness field note: an agent holds no standing privilege — scoped subset of the triggerer’s access, server-side RBAC/OPA, and connector∩agent tool allow-lists on every call.
-
18 of 20 tickets in one night, two human gates, and every failure fixed in the harness
Jul 16 (updated Sep 10) Łukasz Graliński, Pirxey: a one-person software factory ran 50 tickets in a week for about $2.90 each, including a 01:12–05:47 night run that finished 18 of 20. Humans approved every plan and merged every PR. The four things that broke were fixed in deterministic code, not prompts.
-
Proof-or-Stop: lifecycle advances only on fresh evidence
Jul 16 Huang et al.: treat reviewed/tested/DONE as claims; admit transitions only with source-bound, mechanically verifiable receipts — zero false-DONE in 10/10; hidden-fail amplification 31→2 per 1,800.
-
The production harness: walk away to a draft PR or a BLOCKED receipt
July 2026 white paper from David Reed, PhD: a Claude Code harness where done is an Evidence Ledger, the maker never grades the diff, and autonomy tops out below merge — zero auto-merges by design.
-
Zach Wills: ten PRs merged overnight, review became inspecting the proof, and then the RAM ran out
Jul 16, Zach Wills on a year of running agent fleets: an overnight run merged 10 PRs in dependency order, proof-carrying PRs pushed his review ceiling to about 40 a day, and the next wall was a 32GB machine with 27GB of swap.
-
Dark factory test: independent context and independent evidence
Jul 15 Camplight/Kacharov: METR harness ≈ coin-flip vs naive loop; self-review can degrade accuracy; dark only where checking is cheaper than doing — and gates still rot if uninspected.
-
Your coding agent said done. Who ran the tests?
Jul 15 Cortés / VichuFlow: agent self-report is not evidence — durable state outside the chat, kernel-run gates, stage advances only on artifacts; worker grades nothing.
-
Sloop: overnight running hours, evidence verdicts, opt-in merge train
Jul 15 hamish-mackie/sloop: Rust daemon schedules Claude/Codex/OpenCode tickets into isolated worktrees inside configured running hours (e.g. 22:00–06:00), judges from observed exits/commits/tests — not agent claims — with build→review→merge flows and an opt-in verify-before-FF merge train.
-
Harness engineering, measured: eleven production coding agents at source
Jul 15 Barbaste et al.: source anatomy of Claude Code, Codex, Gemini CLI, OpenHands, Pi, OpenCode, and peers — seven subsystems, 29 patterns, and the harness→platform turn.
-
AgentLoop: done is a sealed oracle, not the model's DONE
Jul 15 AgentLoop: harness-agnostic verify gate with held-out sealed cases — +16–22% SWE-bench-Verified lift over raw Claude Code/OpenCode by killing false-greens.
-
AgentLoop: stop trusting ‘done’ — grade against a sealed held-out oracle
Jul 2026 AgentLoop: harness-agnostic verify-gate wrapper; sealed held-out grading; claimed +16–22% SWE-bench-Verified lift over raw Claude Code / OpenCode by killing false-greens.
-
HarnessBank: gene-bank search plus gated screening for trustworthy harness self-evolution
Jul 15 Luo et al.: semantic Harness Gene Bank + validity/activation/significance/gain gates — +5.1% to +15.4% Pass@1 across seven benchmarks; beats GEPA/DGM under the same paired-2σ ruler.
-
Warp on self-improving code review inside the factory
Companion Warp engineering note: how cloud software factories close the loop specifically on review quality and agent compliance.
-
A million PRs later: agentic review is faster, not clearly better
Study of 1.02M reviewed PRs across human, LLM-assisted, and agentic eras—efficiency gains without matching quality gains.
-
Harness evolution gains might just be repeated sampling
Jul 14 Wang et al.: under matched budgets on Terminal-Bench 2.1, automatic harness evolution fails to beat parallel sampling / sequential refinement; held-out transfer only +0.6 pass@1.
-
Turn accepted review comments into persistent behavioral rules
Closed-loop framework that codifies review feedback into versioned rules and a self-review checklist—0% recurrence for ruled-against errors in deployment.
-
Ready for Agent: label the issue, let the harness babysit to merge
Jul 12 berenddeboer/ready-for-agent: local clanker harness turns ready-for-agent issues into PRs across Claude/Codex/OpenCode/Grok — Merge Policy Off/Classify/Always, forge-agnostic, aimed at 150+ PRs/week without session babysitting.
-
Unattended loop cannot bless its own done — fences outside the model
Excubitor (Shepherd): mechanical PreToolUse guards + falsifiable intent records so a headless coding loop cannot merge, push, or delete on evidence it authored itself; frozen-oracle verification; candidate branch only.
-
AfterVibe: recover specs and review those, not just the code
Extract NL specs from vibe-coding sessions and validate them by blind regeneration—specs as the human-review artifact when AI code outpaces review.
-
Greptile: review as a slice of automated validation
Greptile argues PR review is one surface of a larger goal—codebase knowledge bases that catch second-order bugs beyond the diff.
-
Prusik: prove execution evidence — exit 0 with zero tests is NOT PROVEN
getprusik/prusik: independent verify outside the agent — prove parses runner output (executed ≠ skipped), prove-red requires acceptance tests fail without the change; 1,300+ tests, 43 field findings verified-closed.
-
Agentic code that merges still costs more to fix, and skipping review makes it worse
Jul 10 Xia and Miller: a year of commits across 182 repos shows agent-written lines get corrective fixes at a 49% higher rate than human lines, a rising agentic share predicts more human bug-fixing the next month, and projects that merge more without review carry a heavier agentic maintenance load.
-
The PR Hound: an always-on agent that barely reads code and nearly doubled human merges
Jul 9 Andrew Joia (LangWatch): after Claude Code pushed more code in three months than the prior year, PRs rotted unreviewed. A twice-daily agent that gives every PR exactly one owner and one reviewer took human merges from about 10 to 18.5 a workday.
-
verify-gate-mcp: the agent that did the work can never mark it complete
Jul 9 Monkeyattack MCP: criteria-at-birth, claim_done with evidence, verify hard-rejects E_SELF_VERIFY; pending→claimed→verified|rejected is forward-only in SQLite.
-
Review is the control point — AI does not pick the sign
Agarwal et al. code 3,100 practitioner docs into a causal model: agent impact on software is decided through how the team structures review and what expertise humans still bring.
-
Code Janitor: always-on cleanup that never floods the merge queue
Adapt’s recurring Code Janitor: hourly/daily scan → one small CI-gated cleanup PR at a time, memory-repo dedupe, human merge only — modest verified throughput as the product.
-
When coding benches rot: auditing SWE-Bench Pro
OpenAI’s July 2026 analysis of evaluation flaws in SWE-Bench Pro—and why noisy tasks poison agent comparisons.
-
Aria: 4,257 Iris lemmas proved by harness + code agent, zero expert intervention
Jul 7 Kan/Kan/Ertel: Claude Code + declarative HHL harness (kernel verify, no-Admitted, timeout, coverage) — full Iris core, 217 RustBelt, 318 reglang, 72 iris-lean; ~380 model-hours unattended.
-
SWE-Review: closing the generate–review–revise loop on agent PRs
July 2026 paper introducing agentic accept/reject review with SWE-Review-Bench and trajectory datasets.
-
Moderne / Morgan Stanley: AI plans, deterministic tools execute
Morgan Stanley’s OpenRewrite year at Code Remix — review is the bottleneck, risk-based PR routing via DDRA-style scoring, and agents that invent recipes while CPU tools run the blast-radius map.
-
Agents Never Sleep: PARK the hard ticket, don’t freeze the night
Jul 2026 TokonoMix ANS: unattended ASK→PARK autonomy contract, deterministic test gates + git revert, watchdog resume, morning report that separates trusted done from daylight review.
-
feature-factory: verification only where a false green can hide
Carreira’s feature-factory: durable run.json transitions, parallel build / serial merge, reviews bound to the judged commit, observed verify after each slice — autonomy without self-reported done.
-
Merge-coordinator: 24/7 arming that never bypasses your checks
Jul 5 trimmdev: workers/agents open PRs; a GitHub Actions coordinator arms, serializes risky lanes, revalidates stale heads, re-arms when GitHub drops auto-merge — and cannot merge past required checks.
-
VV Harness: sort every rule by enforcement tier — exit codes beat prose
Jul 4 Eftimie: harness = list of things you stopped trusting the model to do; mechanical/prompted/structural/instructional ladder; scope hooks, completion gates, fixture-tested rails.
-
Is agentic review helpful? Mining CodeRabbit feedback in the wild
Empirical mining of developer reactions to CodeRabbit reviews—field evidence on when agentic PR comments help or annoy.
-
Merge only when out-of-band reviewer and tester both pass
Jul 2026 Gus IT defensive publication: sandboxed authors cannot push main; an external train merges only on fail-closed ledger verdicts from separate reviewer and tester personas.
-
harnesswright: an agent's done is a claim — the gate returns a receipt
Jul 2026 Falco: zero-runtime CLI that never runs agents; ledger slices + verity claims reconciled to filesystem, git HEAD, and exit codes — exit 0/1/2; dogfooded 18+ green claims that caught its author.
-
Enterprise 2× mandate: throughput doubled, and review moved onto automation
Jul 2 CMU/Stanford longitudinal case: 802 developers, 196k PRs. Per-developer throughput hit 2.09×; reviewer load ~2×; human review coverage fell 89%→68% while automated review rose to ~84%. Merge and revert stayed flat on coarse proxies.
-
nightcrew: BACKLOG in, typed failures out — verify before any land
Jul 2 nightcrew: overnight control plane around Codex — plan from markdown backlog, worktree isolation, deterministic verify, independent merge review, streak guards, morning report.
-
Tian Pan: the morning review queue turns overnight agent work into reconstruction, not review
Jul 2 Tian Pan: unattended agents move the bottleneck to 9 a.m.; assign trust tiers at dispatch, demand a provenance receipt with a deferred list, and batch the night's questions into one decision queue.
-
Cheap code, costly judgment: making agentic SE governable
A 12-week case study and middle-range theory of “governance conversion”—turning agentic failure modes into durable controls.
-
Takayuki Kawazoe: an always-on agent line merged nothing for three days because a safety check could no longer see standing approvals
Jul 1, Takayuki Kawazoe on Codens' round-the-clock agent pipeline: every PR approved and green, zero merges for three days, caused by a review freshness check that recovery re-armed past existing approvals. The fix was to read the effective review state, not new events.
-
Zach Lloyd: software engineering becomes factory engineering
Latent Space interview/dispatch energy around Warp’s CEO on factories as integrations into existing workflows—not a new IDE religion.
-
Agent in a real repo: worktree isolation, full-suite done, PR not merge
Jun 30 2026 Ramón Chancay: each task gets a git worktree + branch; reproduce with a red test, require the whole suite and that the new test actually ran, ship a PR — never merge unsupervised.
-
Sonar Vortex and SonarQube Remediation Agent for agentic loops
June 2026: Vortex guides/verifies coding agents in-loop; Remediation Agent GA clears backlog with verified fix PRs.
-
Loop engineering: the checker is never the maker, and the stop is a command
Jun 30 sxivansx skill: scaffold LOOP.md + maker/checker/monitor agents + stop-check.sh — one on-disk cycle every trigger; human gate on merge; unattended means isolated.
-
The Audit Tax: generation is cheap; verification is the bill
Jun 30 Temrel: agent PRs arrive in ~90s, then burn ~90 minutes of trust-building — LinearB’s 4.6× review lag, graded verification (deterministic → review subagent → human), and evals that make the tax intentional.
-
The bottleneck for code agents is review — so put gates before the human
Jun 28 2026 Samuel Fajreldines: a verification harness turns agent attempts into reviewable PRs — scope, required proof, risk inventory, stop contracts, and gates that block before human audit.
-
Loop specifications: the check is the skill, not the prompt
Jun 28 Macedo: defines the external loop spec (trigger, goal, verify, stop, memory); codes 50 Loop Library entries—70% autonomous verify, but triggers and durable memory still mostly manual.
-
InterWorks: six Titans keep the factory honest overnight
Jun 26: Mnemosyne/Prometheus/Themis/… cron fleet on Curator — review, docs, triage, merge shepherd; deterministic-first Hyperion; shared lessons file; support tickets −50%.
-
24 agents, 105 merged PRs, and a dispatcher that idles when the review queue is full
Jun 26 Yuvrajsinh (Trevion build log): an always-on daemon dispatches headless agents into worktrees around the clock, about four commits a day including nights. Two things made it work: a per-area regression gate that auto-reverts before a PR opens, and a MAX_OPEN_REVIEW ceiling added after 50 open PRs piled up and started to rot.
-
Silent failures: the patch is green and still insecure
Jun 25 paper: across 1,030 agent repair traces, 170 confirmed silent failures (omit / introduce / inadequate defenses); test-pass gates and LLM reviewer roles did not catch them.
-
Aharness: process drift dies when the workflow is a state machine
Jun 23 Alfredvc Show HN: Codex workflows as TypeScript FSMs — typed evidence, approval gates, repair paths, inspectable run logs; Codeflow slices roadmaps into verified commits.
-
Qodo 2.4: cross-repo review, Rules Miner, and Skill Review Standards
June 2026 release expanding governance: cross-repo blast radius, standards mined from PR history, and managed skill standards.
-
Reviewing agent PRs: declare the scope, list the behaviour changes, read the code before the tests
Jun 23 (cross-posted Oct 6) Ahmet Zeybek: after agent PRs passed half of one service's changes and a two-approval PR dropped a schema default in production, the team moved review into failing CI checks. Median time to first review fell from 5 hours to about 2, median PR size from 310 lines to 160, and reverts within a fortnight from 6 a quarter to 1.
-
ShopBack: prompts plateaued; the harness made MRs land
Jun 22 Yan: production AI engineering harness — Shadow Explorer + multi-agent review + backtrack loops; first-pass review ~40%→70%; Agent = Model + Harness.
-
Nightcore: autonomy at full speed — merge only after the gauntlet
Jun 21 Nightcore: local Claude Kanban studio — worktree builds, anti-gaming + Structure-Lock gates, independent reviewer PASS/CHANGES/FAIL, flight-recorder ledger; you approve merge.
-
Overnight on a Mac: 41 commits, ~30% junk, morning review is the tax
Jun 21 Jirák: eight unattended hours — coverage 41%→86%, four real bugs in BUGS_FOUND.md; ~30% reverted; branch isolation + suite gates; review moved to a civilized hour.
-
Foreman: workers are forbidden from writing verification.json
Jun 2026 VisionForge Foreman: gated plan→ADR/PRD→issues→TDD→e2e; PreToolUse deny + read-only evaluator agent; Foreman alone flips pass; retro patches need bench reports.
-
Repeat reviewers approve more agent PRs and comment less, while the PRs wait longer
Haoran Yu and co-authors follow 400 repeat reviewers through 11,429 reviews of agent PRs in the AIDev dataset. Approval rises from 27.9% to 42.4% between a reviewer's first and latest tenth of reviews, inline comments fall 22%, and median wait grows from 3.9 to 13.5 hours. PR size stays flat, and approval of human PRs in the same repos falls over the same months.
-
Bastion: single-concern reviewers that aggregate into a merge gate
Jun 2026 attunehq/bastion (still shipping Sep): agent volume ×100 breaks human diff review; narrow one-property reviewers run locally then in CI; verdicts aggregate to one required check — humans curate reviewers, not every line.
-
Cloudflare's vuln harness started as a skill: hunters propose bugs, a separate agent tries to disprove them, and nothing trusts its own homework
Jun 18, Cloudflare open-sources the security-audit skill that seeded its fleet vulnerability harness: multi-phase recon and hunt, adversarial validators, schema-checked findings, and a fresh agent that re-verifies every survivor — then a second model system for triage and human-gated fixes.
-
All smoke, no alarm: 80% of agent test patches lack strong oracles
Jun 16 Banik et al.: 86,156 agent test-file patches across 33,596 PRs; 80.2% weak/no oracle; strong multi-signal oracles raise merge odds after controls (OR 1.28).
-
Amnesiac overnight agents: progress lives on disk, not in chat
Jun 16 Meela Dheeraj: hourly fresh sessions 11pm–7am, BUILD_SPEC + PROGRESS.md + git as memory — commit = checkpoint; irreversible ops deferred to ‘needs human’.
-
Himmel: overnight ticket dispatch that still cannot merge without review
Jun 2026 yotamleo/Himmel: worktree-isolated Claude Code harness — /overnight-shift opens PRs while you sleep; hooks block main edits and unreviewed PRs; /pr-check multi-agent gate; handover state survives sessions.
-
Agentic code review: understanding stayed expensive
Addy Osmani’s 2026 map: Faros/GitClear/CodeRabbit data, blast-radius tiering, heterogeneous AI reviewers, and humans on the loop—not reading every line.
-
Factory 2.0: from coding agents to software factories
Factory.ai’s manifesto: model-independent, sovereign, self-improving SDLC loops where engineers build the factory that builds the software.
-
Galea: overnight Claude reclaims the evaporating usage window
Jun 15: 5h Claude windows reset while you sleep — caffeinate + claude -p queue; FOMO as wasted capacity; productive complication: weekly ceiling still taxes daytime.
-
Agent Gate: deterministic merge evidence, not another LLM reviewer
Jun 15 Sung: CI firewall for AI PRs — declared scope, workflow permission escalation, agent control-plane drift, matching test-file evidence — policy loaded from base, never from the untrusted head.
-
NIGHT LOOP: CLAIM_DONE is rejected while the harness is still red
Jun 14 apsolut: Claude Code overnight wrapper — Night Contract, question firewall, PreToolUse outside permissions, Stop-hook ratchet vs model claims, SHA-256 event tape, tamper-evident Morning Receipt.
-
LLM-as-Code: put the overnight loop in the program, not the model
Jun 14 Qi et al.: Agentic Programming — code owns looping/branching/stop; LLM only at judgment leaves; OSWorld 86.8% in 15 steps vs 80.4% prior in 100; DAG context O(depth) not O(steps).
-
HarnessX: compose the harness, then evolve it under a seesaw gate
Jun 12 Darwin Agent Team: typed harness foundry + AEGIS Digester→Planner→Evolver→Critic — avg +14.5% (up to +44%) across five benches; variant isolation stops GAIA collapse; co-evolution +4.7% over harness-only.
-
MakerChecker: deny-by-default roles so the agent can’t approve its own work
Jun 11 MakerChecker: gateway + scan + embedded governor — grant-pinned skills, SoD so maker ≠ checker, Ed25519 hash-chained audit anyone verifies offline.
-
Same vulnerable diff, different story: Sevra-Bench
Jun 11 paper: reverse real vulnerability fixes, wrap them in 15 social-engineering PR framings, and watch review agents approve under prior-approval and dependency-compatibility narratives they cannot verify.
-
A year running an agent harness, part 2: cry-wolf gates and reviews that only lint
Jun 11, Jason Vertrees (Heavy Chain). Five more failure modes from a year of a client-facing agent harness, this time where the signals mislead: an agent saves itself thirty seconds and spends the operator's afternoon, a config flag that reviews clean and does nothing, a gate with a predictable false positive that everyone learns to ignore, automated review that stops at style nits, and a prototype implemented as if it were the spec.
-
NightWatch: logs are claims — replays are proofs
Jun 10 BeamusWayne NightWatch: hash-chained black-box ledger for overnight agents; morning debrief re-runs recorded commands and diffs git vs claims; attest CI gate.
-
Goal-Autopilot: unattended ‘done’ must pass an executed gate
Jun 10 Deng: durable gated FSM + No-False-Success floor — 3,150-cell paired eval cuts fabrication to 0.95% vs StateFlow 25%; SWE-bench Lite −33 pp; honest stall over confident wrong patches.
-
loop-harness: you don't prompt Claude — scheduled loops do
Jun 10 loop-harness: cadence-driven Claude loops in worktrees — primary stages output, second agent must print VERDICT: PASS before PR/comments/Slack ship.
-
Camus: Claude makes; Codex reviews; Camus never lands it for you
Jun 10 Camus: deterministic local control plane — independent maker≠reviewer seats, repo tests as final word, bounded review loops, crash-safe resume; verifies overnight work without auto-merge.
-
Personal agent use swung from 44% to 4%; the harness held it at 92%
Jun 10 Štefan Moravík, FlowHunt: across 1,409 merged PRs in ten months at QualityUnit, agent work tracked individual enthusiasm until a risk-tier policy file and a staged, bounded pipeline arrived. Then it jumped and stayed, with a human still making every merge.
-
Loop engineering: design the system that prompts the agent
Jun 8 Addy Osmani: automations, worktrees, skills, connectors, maker/checker sub-agents, plus disk memory — /goal stops on a separate model’s done-check, not the implementer’s optimism.
-
Nocturne: label an issue, wake to a verified PR — never a merge
Jun 2026 ba1lly/nocturne: daemon picks agent-labelled issues across repos, isolated worktree + verify_cmd + review/fix budget, opens a PR; optional CI/review shepherding — never merges, force-pushes, or touches main.
-
FrontierCode: grade like a tech lead — would the maintainer merge?
Jun 8 Cognition: first mergeability benchmark — 20+ OSS maintainers, 40h/task, blockers + quality rubrics; Opus 4.8 ~13% Diamond / 34% Main; 81% fewer misclassifications vs SWE-Bench Pro.
-
failclosed: MERGE_READY only when the reviewer’s output survives a distrust gate
Jun 8 Dan Mercede / OrionArchitekton: fail-closed merge admission — parse LLM review, block on unparseable/schema-invalid/contradictory output; receipts over logs; reviewer is a command seam, not the product.
-
A year running an agent harness for clients: the worst failures all reported success
Jun 8, Jason Vertrees (Heavy Chain). From a year of a harness that writes, reviews and ships client software, five failure modes where the system said done and was wrong: a 738-test green board that wrote no rows, lessons that never became gates, one suppressed warning copied by every later agent, a fabricated API client built on by seven agents, and five gates that each assumed another had checked.
-
Self-Harness: the same fixed model improves its own harness under a held-out gate
Jun 8 Zhang et al.: Weakness Mining → bounded proposal → promote only if held-in and held-out both non-regress; up to +132% relative Pass across Terminal-Bench-2 / SWE-bench Verified / AppWorld.
-
Dark Factory: reviewers who literally cannot edit the branch
Stratton / godark: humans author architecture + scenario specs; implementer in Docker; quality + functional reviewers isolated (no file write); retry then squash-merge or needs-human-review.
-
Vanguard: always-on factory that stops at draft PRs
Jun 5 Vanguard (still pushing Sep 23): watch loop claims tickets from GitHub/GitLab/Linear, plan→implement→review→adversary→repair in Docker, Proof-of-Work verify, then draft PR only — humans own merge.
-
Augment CIV: separate the judge from the builder — or keep the single-agent burst
Jun 3 Augment guide: Coordinator–Implementor–Verifier splits planning, execution, and spec-grounded validation so generation errors stop grading themselves — and names when single-file bursts still win.
-
Make checking the work a job with a number, not a rushed PR step
June 2026 org field note: AI-coauthored PRs carry ~1.7× more review findings — rebuild around four-person pods (lead, two directors, quality engineer) so judgment and verification scale with agent output.
-
Recall is the metric now: AI review inverted precision culture
Ben Nevile on why agent-side filtering flips review etiquette from “only raise sure things” to “raise everything potentially relevant.”
-
Coding isn’t the constraint: 76% more PRs to review
Spotify’s Code with Claude talk write-up: 99%+ weekly AI usage, Honk in K8s with CI verification, Backstage golden paths as agent guardrails.
-
agent-completion-gate: /goal can loop; only artifacts grant complete
Jun 1 zhjai agent-completion-gate: worker proposes candidate_complete; external check on real files + human-confirmed acceptance manifest grants complete — fills Claude/Codex /goal’s transcript-only stop gap.
-
Overnight coding agents: attention-decoupled runs with a ready-or-not verdict
May 30 2026 Ralph Workflow: overnight pattern = time-decoupled, structured plan→implement→verify→fix, verifiable receipts, reviewable git diffs — not a chat with a sleep timer.
-
Label-cascade software factory: plan → CI → review → squash without a happy-path human
Yefremov’s fork-and-own software-factory: four Claude roles on a GitHub label state machine, bounded retries, CI gate, auto-merger with guardrails — continuous verified throughput as a template.
-
Otis: eight checks between the agent and the PR — including mutation-light
May 29 42nights Otis: issue→PR overnight agent whose centerpiece is an 8-check harness (typecheck, suite, plan-tests, mutation-light, lint, diff cap, banned patterns, critic); never auto-merges.
-
In an unattended review panel, a reviewer that couldn't see the code counts as a pass
Huiyan Wan's overnight-workflows plugins for Claude Code run multi-agent review panels on stacks of PRs while nobody is watching. One failure gets its own plugin: review subagents without a shell can't fetch the PR, return a BLOCKED report or review main instead, and by morning that empty report reads as clean. Re-dispatched with the diff on disk, one such reviewer found a data-loss bug that 179 green tests had missed.
-
Meta’s RADAR: automate low-risk review with calibrated risk
Industrial study of RADAR at Meta—risk calibration and selective automation to reclaim reviewer time on low-risk changes.
-
Unattended agents need containment, not more approval dialogs
May 25 Anthropic: 93% approve rate → approval fatigue; environment boundaries (sandbox/VM/egress) cap blast radius so always-on loops can run without a human watching every tool call.
-
compass: loops under a gate — agents prepare; you always merge
May 25 dshakes compass: budget-capped Claude/Codex/Gemini harness, eval-gated guardrails, self-fixing PR loop with cross-model audit; fleet overnight; human merge permanent.
-
Bello: independent supervisor, completion review, and adversary around the coder
Makson179 Bello (May 2026; still shipping Sep 24): Codex/Claude/API coder in a disposable workspace; fresh-context supervisor for risky actions; completion review + adversary refuse 'done' on claims; ProgramBench C+A +26% mean vs raw; Efficient Budget −66% Codex weekly usage at same quality.
-
Harness engineering: five layers, and review velocity as a Stage-1 metric
May 22 Neely Dunlap (Faros): Agent = Model + Harness — tool orchestration, verification loops, memory, guardrails, observability; baseline cost-per-merged-PR and review velocity before buying more peak-session model.
-
Ona's background-agents guide: laptop agents are a false summit
Ona's engineering-leader guide, built from its May 2026 Background Agents Summit with Stripe, Ramp, Spotify, Uber and Harvey, argues that faster individual agent sessions don't move org throughput; triggered, sandboxed agents that can verify their own work do.
-
Agentic PR merge/reject labels overstate what agents did
May 21 Peralta et al. (MSR ’26): of 717 inspected AIDev Agentic-PRs, only 35.7% of rejections were clear agent failures; 31.2% workflow, 33.1% no rationale; 15.4% of merges needed reviewer feedback or commits—outcomes ≠ capability.
-
AIFactory: verify against the live endpoint, not the diff
May 20 AIFactory PARR loop: plan→act→deploy→reflect→review — worktree isolation, live HTTPS verify, bounded auto-feedback, human-stop defaults; auto-merge only after reviewer gate.
-
Overnight agents that open draft PRs: the harness is the product
May 20 Pickuma: ingest→plan→execute→verify→package; verify-loop with retry bound; draft PR + agent-generated label; morning review non-negotiable; thin tests = no safety.
-
The agentic metaharness: a room above your coding harnesses
May 18 SoloTerm/Aaron Francis: models → harnesses → metaharness — shared process graph, scratchpads, todos, timers, and cross-harness spawn so agents coordinate without you babysitting every tab.
-
code-coordinator: Work → Test → Review → Merge as one overnight drive
May 17 JDonaghy code-coordinator: Claude CLI fleet across worktrees/machines; Test gates Review; independent review; merge queue + overnight drive-queue — human approves dispatch, harness drains to merge.
-
A vision paper for agentic code review
Position/vision piece on how code review should change when AI agents author and critique changes across the SDLC.
-
OpenSearch: harness-first SDLC — trust the loop, not the review queue
May 15 field note: Atlas/Ralph/Nitro/Sentinel each run inside a verification harness — live local stack, A/B benchmarks, plan-then-approve — because agents generate faster than humans can review.
-
HarnessAudit: task-green overnight runs can still violate the harness
May 14 Liu et al.: full-trajectory harness safety (boundary/fidelity/stability) on 210 tasks — best overall ~0.32; completion misaligned with SAR; multi-agent amplifies resource + info-flow violations.
-
AgentLens: green tests can still be a Lucky Pass
Process-level SWE-agent eval showing ~10.7% of passing trajectories are Lucky Passes—missing verification, blind retries, disordered stages.
-
DeadReckon: two independent keys before an unattended job is 'done'
May 13 DeadReckon: harness around Claude/Codex/Gemini/Copilot/OpenCode — detachable Job, sandbox checks + fresh semantic judge, HMAC receipt; NEEDS_REVIEW when either key fails.
-
Capability is model × harness × environment — not the model alone
May 13 Zhong & Zhu: eleven harness responsibilities, H0–H3 visibility ladder, and episode packages that adjudicate verification autonomy — reframing autonomous SE as a runtime-systems problem.
-
Reviewing what the overnight agent shipped while you slept
May 12 Toni Antunovic: overnight diffs lack mid-flight correction — aggregate review, security triage, reasoning traces, semantic not line-by-line; constrain batch size or automate triage.
-
From fear of missing agent time to fear of unused capacity: the bottleneck moves to planning
May 11 (edited Jul 28) Teemu Piirainen: once agents ran ten-hour sessions without asking questions, the anxiety didn't go away. It moved from answering agents fast enough to preparing enough work for the night, and mistakes started surfacing in morning review as the right execution of the wrong spec.
-
The mixed PR queue: reviewer throughput is the binding constraint
May 10 field note: Tian Pan on agent fleets as backlog generators unless you tier the queue — author class, supervision, risk — with traces to audit and CI never enough for business logic.
-
wgmesh: an autonomous merge bot counted any Copilot review as approval, so it merged exactly the PRs Copilot had the most to say about
May 9, the wgmesh project found its review-and-merge bot passed any review submission, including Copilot's COMMENTED findings. Three PRs merged with 4 to 13 open findings; the fix requires an explicit APPROVED from each reviewer's latest review, re-checked just before merge.
-
pr-autopilot: opt-in stages from open PR to green merge — never silent by default
May 6 FelipeOFF Claude Code skill: Reviewer + Author subagents triage every human/bot comment, attribute CI before patching, stop on business rules; --auto is explicit, bare run only opens the PR.
-
Hellström: an overnight fleet merged 11 PRs, sat idle 94.5% of the night, and had no quality gates
May 6 from Felix Hellström: the first unsupervised night of his agent orchestrator ran 686 cycles in 7.5 hours, dispatched 103 tasks against 54 issues, auto-merged 11 PRs, flagged 9 for review, and killed and re-dispatched 8 stalled agents. Every failure lived in the harness, not the agents — and auto-merge on a green suite was luck, not a guardrail.
-
OpenHands verification stack: critic before push, review+QA on the PR, iterate until green
May 6 OpenHands: agent-level critic gates broken trajectories; repo-level review skill + behavioral QA; /iterate closes the loop — 1,000+ reviews, 58% faster MTTM, ~25% coverage lift on their SDK repo.
-
Greptile's overnight-agents study: agent PRs hold up, but the bugs move
Greptile mined millions of reviewed PRs and found end-to-end agent PRs rose from 0.86% to 27.6% of merges in fourteen months, with revert, churn and review-cycle numbers roughly matching humans. What changed was which bugs show up.
-
Long-running agents need harnesses, not hope — eight parts before you sleep
May 2 Developers Digest: task contract, scoped workspace, tool policy, logs, checkpoints, verification gates, cost/time budgets, final receipt — without them an overnight run is a chat session with more failure modes.
-
NachtSchicht: the morning brief is the product — merge without re-reading
Apr 30 Florian Riquelme: Claude Code night-shift queue with a trust protocol — worktree isolation, blast-radius verify, halt-on-ambiguity, immutable evidence; the brief is a contract, not a transcript dump.
-
12 tools in one night: overnight orchestration’s honest cost
Apr 29 Developers Digest: /loop every 10m produced 12 repos and ~60 PRs — and a billing wall, a fake PR number, and a full morning of curation. Parallel overnight works; unverified claims don’t.
-
Long-running agents: the gap is state, sessions, and handoffs—not the chat loop
Apr 28 Addy Osmani: long-horizon vs long-running vs persistent agency; Ralph, Anthropic brain/hands/session, Cursor planner–worker–judge; write done-conditions and separate evaluator from generator before overnight FOMO.
-
AHE: observability turns harness edits into falsifiable overnight contracts
Apr 28 Lin et al.: ten AHE iterations lift Terminal-Bench 2 pass@1 69.7%→77.0% past Codex 71.9%; frozen harness transfers +5.1–10.1 pp cross-family; tools/middleware/memory carry the gain, not the prompt.
-
Broccoli: Linear tickets become PRs overnight — humans still merge
Apr 22 Besimple Broccoli OSS: assign Linear to a bot; Codex/Claude in your Cloud Run or Blaxel sandbox plan, implement, open a PR; review-feedback loops; human owns merge.
-
superhuman: closed-loop merge-probability until a maintainer would merge
Apr 22 gaurav0107/superhuman: multi-agent Claude/Codex harness scores its own PR on a 10-dimension rubric, iterates the weakest dimension to ≥95% over two runs, learns from merge_outcomes.jsonl — with merged PRs into transformers, airflow, oxc.
-
Pre-commit review gate: the agent cannot commit until CLEAN
Apr 22 Johnston (IMTI): PreToolUse git-commit hook + adversarial sub-agent — Verdict CLEAN pinned to diff hash; blocks review theater and post-commit fix loops.
-
Cloudflare’s AI review isn’t a chatbot — it’s a CI orchestrator
Apr 20 field report: up to seven specialist OpenCode reviewers under a coordinator; 48k MRs / 30 days, median 3m39s, $1.19 avg, break-glass 0.6%.
-
OvernightAgent: four gates between every step and 'done'
Apr 20 oa: worktree-per-task overnight queue for Claude/Codex/OpenCode — tail protocol, commit-since-start, verify cmd, AI review; fix-loop; events.jsonl + SUMMARY.md; crash-resumable daemon.
-
Night Shift: overnight Claude with Codex that must agree you're done
ppuliu skill: up to 8h on a feature branch; file-gated Codex review per commit; dual consensus to end; never push/PR — self-review only when Codex absent, explicitly flagged.
-
Factory Missions: multi-day multi-agent execution with scrutiny validators
April 2026 architecture note on Missions—workers, validators, and generate/validate loops for long-horizon work.
-
Kaizen: continuous prompt optimization that opens reviewable PRs
April 2026 GitHub project: DSPy-based prompt optimization that can land changes as PRs for human review.
-
Overnight FOMO is a trust problem — nap runs before full nights
Apr 9 Ship with AI: 56h/week idle agent capacity; night-shift skill wraps Ralph with pre-flight, kill conditions, and morning categorization — calibrate small before you sleep on it.
-
Managed Agents: durable session, cattle harness, disposable hands
Apr 8 Anthropic (Martin/Cemaj/Cohen): session + harness + sandbox as swappable interfaces; pets→cattle; credentials never in the sandbox — meta-harness so overnight loops outlast any one harness recipe.
-
thin-supervisor: 'done' is the acceptance contract, not the agent's claim
Apr 8 thin-supervisor: sidecar over Claude/Codex CLI — continue/verify/retry/recover/escalate; verifier + AcceptanceContract gate finish; Stop-hook closes the loop when you leave.
-
Tenet: 12+ hour runs with independent critics, not self-certified done
Apr 8 Tenet: interview→spec→DAG→execute→evaluate loop — model-tier-shaped plans, grounded+ungrounded critics (code/test/e2e), doctrine that self-corrects between overnight cycles.
-
Nightshift: overnight work only counts behind seven verification stages
Apr 6 Orbit/Sharma: open-source overnight orchestrator — hardening + feature loops, worktree isolation, shift logs; runner enforces policy (blocked paths, verify command, anti-tunnel-vision), agent supplies intelligence.
-
AgenticFlict: 27.67% of 107K unmerged agent PRs no longer merge cleanly, from 15% for Copilot to 32% for Codex
Ogenrwot and Businge replayed git merges for 107,026 open or closed-unmerged agent PRs from the AIDev dataset and found 29,609 with textual conflicts and 336,380 conflict regions, with conflict rates varying sharply by agent.
-
claude-overnight: the swarm improves its own skill library mid-run
Apr 4 Fornace: multi-wave overnight Claude SDK swarm in worktrees — self-review + simplify, cross-agent wave review, final architecture gate, and a librarian that compounds skills wave to wave.
-
Last Light: a self-hosted factory that works after dark on your GitHub
NearForm Last Light: webhook/cron triage + PR review, Architect→Executor→Reviewer builds, permission-scoped App tokens, and evals scored on your own repos — not someone else’s bench.
-
Leaving agents idle overnight is shutting down CI every evening
Apr 3 Escape/Carossio SF field report: harness + AI-factory layers; FOMO as asset utilization — overnight produces candidates not commits; verification beats advice.
-
Guides before, sensors after: Böckeler’s map for trusting agent code
Apr 2 Thoughtworks/Fowler: coding-agent harness as feedforward guides + feedback sensors, computational vs inferential, keep quality left — and why behaviour harnesses still demand human supervision.
-
AutoAgent: overnight hill-climb that rewrites the harness, not just the prompt
Apr 2 Thirdlayer: meta-agent edits agent.py (tools, verification, context, orchestration) against Harbor evals; keep score gains, revert regressions; >24h runs hit 96.5% SpreadsheetBench and 55.1% TerminalBench (GPT-5) from a minimal baseline.
-
ABCA: fire-and-forget background agents that open PRs on AWS
Mar 31 aws-samples ABCA: submit via Slack/Linear/Jira/CLI/webhook → MicroVM agent clones, tests, opens PR; blueprint mixes deterministic + agentic steps; memory from PR feedback; budgets and blast-radius limits for 24/7 runs.
-
Auto-harness: overnight failure mining with a living regression gate
Mar 31 NeoSigma: coding agent iterates the harness overnight — mine failures, cluster, propose changes, keep only what clears a self-maintained eval suite + held-out val; 0.56→0.78 on Tau3 with fixed GPT-5.4.
-
Meta-Harness: overnight harness search needs the full trace filesystem, not a score summary
Mar 30 Lee et al.: agentic outer loop over harness code with filesystem access to prior code/scores/traces — +7.7 pp vs ACE at 4× less context; TerminalBench-2 #1 Haiku / #2 Opus among reported agents.
-
Nightshift: the builder proposes; the harness owns the commit
Mar 28 Nightshift/AgentForge: Ralph loop with separate evaluator, mutation-guarded feature queue, and metrics dashboard as receipt — 29/30 features, 41 attempts, ~$0.95 in 76 minutes.
-
SWE-PRBench: grade AI review against real PR feedback
Benchmark that scores AI code-review quality against pull-request feedback rather than only against generation/pass metrics.
-
BCG Platinion: harness engineering and intent thinking
Enterprise framing of the Agentic Software Factory—humans relocate to intent and outcomes; five transformation pillars including governance and scenario tests.
-
Kitchen Loop: 1,094 merged PRs under unbeatable tests and pause gates
Mar 26 Roy: six-phase coverage-exhaustion loop over two production systems; 285+ iterations, ~$0.38/PR, zero oracle-detected regressions; drift control + drain/starvation gates keep the night honest.
-
TheBotCompany: 142 hours of unattended multi-agent development, where a separate verifier rejected up to 18.6% of milestones
An ASE 2026 paper ran a self-organizing agent team on four real projects for 142 wall-clock hours and 164 milestones with no human intervention after setup, for about $1,500. A dedicated verification phase rejected 18.6% of milestones on one project, unit-test gates still missed regressions the planner later caught, and reverts restored the baseline every time.
-
Generator vs evaluator: separate the judge from the builder
Anthropic Labs’ GAN-inspired planner/generator/evaluator harness—sprint contracts, Playwright QA, and shedding scaffolding as models improve.
-
Anthropic: every night without agents is progress left on the table
Mar 23: Discovery-team field guide — CLAUDE.md plan, CHANGELOG memory, CLASS test oracle, tmux detach, Ralph loop; FOMO framed as opportunity cost of idle agents.
-
Ten months of CCA in dotnet/runtime: harness beats heroics
Toub’s receipts: 878 CCA PRs, merge 67.9% vs 87.1% human; instructions 38%→69%; phone-flight FOMO creates review debt; fully autonomous 55.1% vs 86.2% with human commits.
-
Optio: self-hosted ticket-to-merged-PR loop with CI-resume feedback
Mar 19 jonwiggins/optio: K8s (or laptop) control plane assigns GitHub/Linear/Jira tickets to Claude/Codex/Copilot worktrees, watches CI every 30s, auto-resumes on failures and review comments, then squash-merges — always-on swarm orchestration you own.
-
SFLO: lights-out only advances when the gate artifact exists
Mar 19 Simon Factory Lights Out: six sequential gates (Discover→Build→QA→Security→Verify→Ship), each requiring a named markdown artifact — no artifact, no progress; inner Build/QA loop capped at 10 rounds.
-
Fabro: a dark factory as version-controlled workflow graphs
Mar 2026 Helmkamp/Fabro: Graphviz workflows with human gates, multi-model routing, cloud sandboxes, git checkpoints, and verification fix-loops — walk away while the line keeps running.
-
Isolated milestones look solved; continuous evolution does not
Mar 13 Deng/Chen et al.: on SWE-Milestone’s 98 dependency-linked milestones, frontier scores fall from >80% independently to ≤38.03% continuously—recall still grows, precision saturates, regressions snowball.
-
Overnight agents need a verification layer, not a longer session
Mar 12 Vincent Tran: builder/judge separation, one browser agent per acceptance criterion, artifact-backed promotion — overnight coding without verification is morning cleanup.
-
CR-Bench: do review agents help in the real workflow?
Benchmark aimed at real-world utility of AI code review agents—not just comment similarity scores.
-
Anthropic brings multi-agent Code Review to Claude Code
March 2026 research preview: parallel find/verify/rank agents on GitHub PRs for Team/Enterprise, ~20 minutes and $15–25 per review.
-
Datadog: harness-first, then telemetry closes the loop
Mar 9: redis-rust and Helix — DST/TLA+/Kani pyramid replaces line-by-line review as primary correctness; production telemetry refines the harness when invariants were wrong.
-
60 million Copilot code reviews: GitHub’s postmortem on signal vs thoroughness
March 2026 product essay on CCR usage scale, agentic redesign, and redefining “good review” as high-signal progress rather than exhaustive nitpicking.
-
Copilot code review moves to an agentic, Actions-backed architecture
March 2026 changelog: tool-calling reviews with broader repo context, running on GitHub Actions (including self-hosted caveats).
-
AgentsMesh: the repo is the harness — and three parallel streams is a hard ceiling
Mar 4 AgentsMesh field note: 52 days / 600 commits building the fleet console with itself; Loops (cron) vs Tickets; tech-debt amplification; Autopilot after cognitive bandwidth broke at ~3 workstreams.
-
Pairing with an LLM is delegation — shared judgment still needs humans
Mar 4 Wynne: agentic tools make soloing easier, but ‘pairing with Claude’ is a siren; as delivery speeds up, humans still co-create architecture, debt, and the constraints that check the robots.
-
Martian Code Review Bench: F1 from real developer behavior
CodeRabbit’s write-up of Martian’s independent bench (~300k PRs): recall and F1 leadership claimed from whether developers act on comments.
-
Kelos: the always-on factory is a Kubernetes CRD, not a laptop left open
Mar 2 Kelos: coding agents as K8s Tasks/TaskSpawners — GitHub/Jira/cron triggers, isolated pods, GitOps-able workflow YAML; self-development pipeline ships its own PRs.
-
Six idle agent sessions feel like lost weeks, and the reviewer approves at 4 PM what they would reject at 9 AM
Charles Sieg on the psychology of supervising coding agents by hand: parallel sessions finish in twenty minutes and then wait on him, an idle terminal feels like a stopped factory floor, and decision fatigue and craving arrive together. The cost lands in review quality.
-
Nightwire: independent verify, then baseline-relative gates
Feb 2026 HackingDave/Nightwire: Signal-orchestrated workers; fresh-context verifier on the diff; test snapshots that fail only on NEW regressions; ≤2 auto-fix attempts — same brain never grades its own patch.
-
Agent PRs: reviewer engagement and stable loops beat more commits
Feb 23 MSR ’26 study of 33,596 agent-authored PRs (AIDev): 71.5% merged overall, but merge odds track reviewer engagement and coordination stability more than iteration volume. Force pushes and large changes hurt; actionable review loops explain most qualitative successes.
-
Overnight agents need episodes, not one immortal session
Feb 23 Nightcrawler: bounded Claude Code episodes, HANDOFF.md cross-checked against git, immutable tasks.json, eight stop conditions including diminishing returns.
-
Stripe Minions: unattended blueprints beat shoulder-surfing Claude
Feb 19 Stripe Part 2: 1,300+ minion-only PRs/week — blueprints interleave agent nodes with deterministic lint/push/CI; hot-ready isolated devboxes; at most two CI rounds before human review.
-
kodo: overnight Claude Max with an architect that rejects the worker nine times
Feb 18 kodo: Gemini orchestrator drives Claude/Cursor/Codex workers overnight; independent architect+tester gates accept only after multi-round rejection — blackopt run: 3h, 9 verifier catches, 73 tests.
-
Harness-only: Terminal Bench 52.8 → 66.5 without touching the model
Feb 17 Trivedy (LangChain): deepagents-cli hill-climbs via traces, PreCompletionChecklist self-verify, loop detection, and reasoning sandwich — same gpt-5.2-codex throughout.
-
Shipwright: Code Factory merges only with current-head evidence and a harness-gap loop
Shipwright Code Factory pattern: agents write 100% of code; risk-policy gate, SHA-tied evidence (API/CLI/DB/browser), automated remediation, and incident→harness-gap SLAs before merge.
-
OpenAI harness engineering: humans steer, agents execute
Ryan Lopopolo’s report: ~1M lines, ~1,500 PRs, zero manually written code—environment design, legibility, and agent-to-agent review as the job.
-
Ralph Loop: overnight agents need stop hooks and machine-checkable done
Feb 8 Blake Crosley: fresh-context iterations + filesystem memory + stop-hook criteria shipped 3,455 LOC / 141 tests overnight — spawn budgets, non-trivial criteria, and cleanup beat one long Claude session.
-
StrongDM via Willison: holdout scenarios so agents cannot review their own homework
Feb 7 Simon Willison on StrongDM’s dark factory: no human-written or human-reviewed code; scenarios as holdout sets; Digital Twin Universe clones of Okta/Jira/Slack for high-volume satisfaction scoring.
-
Cursor: one week, ~1k commits/hour — harness over heroics
Feb 5 research: Wilson Lin’s multi-agent harness ran continuously a week (~1k commits/hr, 10M tool calls); planner/subplanner/worker handoffs; 100% per-commit correctness killed throughput.
-
Qodo 2.0: multi-agent review and a self-published precision–recall benchmark
February 2026 launch of Qodo 2.0 with specialized review agents and Code Review Benchmark 1.0 claims on real PRs.
-
AACR-Bench: repo-level automatic code review with expert-verified defects
A multi-language ACR benchmark that expands defect coverage ~285% via AI-assisted, expert-verified labels and stresses context retrieval choices.
-
Continuous Agent: a 24/7 queue that spawns workers and validates before the next goal
Jan 24 jackzhaojin/continuous-agent: PM2 executive loop picks prioritized goals, spawns multi-vendor workers into an isolated sandbox, runs deterministic verifiers, and only sleeps when the queue is empty.
-
Where agentic PRs fail: abandonment, duplicates, and CI — not just bad code
Jan 21 study of 33k agent-authored GitHub PRs: docs/CI/build merge most; performance and fix least. Among 562 labeled rejections, 38% closed with no meaningful human review, 23% were duplicates, 17% failed CI — agent misalignment was only ~2%.
-
How Cursor built Bugbot: resolution rate, BugBench, and autofix
January 2026 engineering essay on Bugbot versions, resolution-rate metrics, internal BugBench, and Bugbot Autofix beta.
-
Anthropic: demystifying evals for coding agents
Engineering post on graders, pass@k vs pass^k, transcript review, and why SWE-bench-style outcome tests are necessary but incomplete.
-
Evaluating AI reviewers: pilots beat leaderboard cosplay
CodeRabbit’s David Loker on why multi-bot-on-one-PR comparisons mislead, and which behavioral metrics (acceptance, noise, coverage) actually matter.
-
Foundry: a persistent control plane that validates before it persists
Dec 28 ai-supervisor-foundry/foundry: BYOK orchestration for Cursor/Copilot/Gemini — provision sandboxes, dispatch worker nodes, hybrid local-then-LLM validation, DragonflyDB resume across multi-day runs.
-
Zeroshot: the writer never grades the work — the graph does
Dec 25 Open Engine Zeroshot: authored executor–verifier graphs, independent reviewers with no shared session, SQLite ledger, bounded repair; delivery only after checks pass.
-
Spotify Honk part 3: verification loops before the PR exists
December 2025 post on deterministic verifiers + LLM-as-judge gates that block bad agent PRs from opening.
-
Codacy launches AI Risk Hub and hybrid AI Reviewer
December 2025 launch: governance for AI-generated code plus a hybrid deterministic+LLM PR reviewer.
-
OpenAI: verifying code at scale needs repo tools and high precision
Alignment blog on training/deploying an agentic code reviewer—precision over recall, repo-wide tools, and verification cheaper than generation.
-
Anthropic: initializer + incremental coding across context windows
How to keep long-running coding agents coherent: feature lists as JSON, git+progress artifacts, clean handoffs, and end-to-end testing as a human would.
-
Greptile v3: agentic review with published upvote and action-rate deltas
November 2025 engineering write-up comparing v2 vs v3 on upvote ratio, action rate, and cache economics after reviewing over a billion lines.
-
Amazon CodeGuru Reviewer stops new repository associations
November 2025 docs notice: no new CodeGuru Reviewer repo associations; AWS points users to successor capabilities.
-
Carnegie Mellon's Cursor study: a 3–5x burst in lines added, gone after two months, while warnings rise 30% and complexity 41%
A difference-in-differences study of open-source projects that adopted Cursor, against 1,380 matched projects that did not, finds a large but transient velocity jump and a persistent rise in static analysis warnings and code complexity, which in turn drags later velocity down. Sonar cited it this week to argue verification belongs inside the agent loop and CI, not only at the pull request.
-
Spotify Honk: 1,500+ merged agent PRs on Fleet Management
Part 1 of Spotify’s background coding agent: prompt-defined fleet shifts, pluggable agent CLI, 60–90% migration time savings, and new validation needs.
-
Greptile’s $25M Series A and the Greptile v3 agent rewrite
September 2025 funding + product post: Benchmark-led Series A alongside v3 agentic review, MCP server, and rule-file autodetection.
-
CodeRabbit’s $60M Series B and the CLI quality gate for AI coding
CodeRabbit frames AI code review as the quality gate for vibe coding, announces Series B, and ships a CLI that plugs into Claude Code, Codex, Cursor, and Gemini agents.
-
OpenAI Codex upgrades: GPT-5-Codex and first-class PR code review
September 2025: GPT-5-Codex launch with trained code-review behavior and GitHub auto-review / @codex review flows.
-
Gemini Code Assist in GitHub: PR summaries, inline findings, /gemini commands
July 2025 Google Cloud blog with Delivery Hero notes on Gemini-as-reviewer in the PR workflow.
-
Sonar AI CodeFix generally available in SonarQube Cloud and Server
July 2025: one-click LLM fix suggestions for Sonar-detected issues in UI/IDE, included on eligible plans.
-
GEPA: reflective prompt evolution that beats RL on few rollouts
ICLR 2026 oral: Genetic-Pareto prompt optimization that diagnoses trajectories in natural language—strong results including code-optimization search.
-
Cursor Bugbot leaves beta as a PR bug-finding agent
July 2025 launch: Bugbot GA with BUGBOT.md rules, one-click handoff to Cursor/background agents, and beta stats on PRs reviewed.
-
Gemini Code Assist for GitHub hits GA on Gemini 2.5
May 2025 Google I/O update: free individual + GitHub code-review agent generally available, powered by Gemini 2.5.
-
CodeRabbit brings free AI reviews into VS Code, Cursor, and Windsurf
May 2025 product note: CodeRabbit ships IDE reviews as a lightweight pre-PR pass, with fuller PR reviews still recommended for production.
-
Amazon Q Developer in GitHub preview: generate, review, and transform
May 2025 AWS News Blog: Q Developer GitHub app for feature agents, automatic PR code review, and Java transforms.
-
GitHub Copilot code review reaches general availability
April 2025 changelog: Copilot code review GA for paid subscribers, with automatic or on-demand PR reviews.
-
Graphite: AI review for the outer loop
Graphite positions Diamond/Agent as accessible AI PR review paired with stacking, merge queue, and inbox—keeping the post-write toolchain as fast as generation.
-
Graphite’s $52M Series B and the Diamond AI reviewer launch
March 2025: Accel-led Series B and Diamond as a standalone AI PR reviewer (later rebranded into Graphite Agent).
-
CodeRabbit adds Bitbucket Cloud to its AI review surface
February 2025 integration note extending CodeRabbit beyond GitHub/GitLab/Azure DevOps into Bitbucket Cloud (beta).
-
Snyk Agent Fix / auto-remediation for Snyk Code goes GA
October 2024 product blog celebrating Snyk Code auto-fix powered by DeepCode AI.
-
Greptile 2.0: full-codebase graph review, not just diff chat
October 2024 rewrite pitch: Greptile indexes a codegraph and reviews PRs with whole-repo context on GitHub and GitLab.
-
SnykLaunch Oct 2024: DeepCode AI Fix in the IDE
October 2024 SnykLaunch: DeepCode AI Fix merged into IDE workflows with verified remediation emphasis.
-
Qodo’s $50M total funding and the integrity-platform roadmap
Same-day-as-rebrand funding note: $40M Series A to $50M total, with roadmap items for Cover, Merge learning from PRs, and Gen agents.
-
CodiumAI becomes Qodo: Gen, Merge, and the code-integrity framing
September 2024 rename from Codium/CodiumAI to Qodo, with Qodo Merge (ex PR-Agent) as the review/automation product.
-
Survey: LLM-based agents for software engineering
Broad 2024 survey of LLM agents across SE tasks—useful map of roles, tools, and feedback loops including review/testing.
-
Bito’s AI Code Review Agent lands in VS Code and JetBrains
August 2024 product post: @codereview agent in Bito’s IDE extensions for pre-PR reviews, with optional Git context.
-
SWE-bench Verified: human-filtered tasks for cleaner agent eval
OpenAI × Princeton subset of 500 solvable SWE-bench tasks—still the common public yardstick, with known contamination caveats later.
-
CriticGPT: train critics to catch LLM bugs
OpenAI’s LLM Critics Help Catch LLM Bugs—specialized critique models that find errors in LLM-generated code more reliably.
-
TextGrad: backprop for prompts, code, and compound systems
Stanford’s TextGrad treats LLM feedback as textual gradients over a computation graph—including measurable gains on hard coding problems.
-
Sourcery brings instant AI review into the IDE sidebar
June 2024 release: Review tab in the IDE, PR summaries/guides on GitHub, and multilingual review config.
-
SWE-agent: the interface is part of the agent
Custom agent-computer interface for editing, navigating, and testing repos—large SWE-bench gains over shell-only baselines.
-
Sourcery AI code review GA on GitHub with inline comments
March 2024 changelog: Sourcery code review generally available on GitHub, replacing legacy bot PRs with inline review comments.
-
Semgrep Assistant enters GA for AppSec triage and remediation
March 2024 PR: Semgrep Assistant GA for false-positive triage, autofix context, and custom rule workflows.
-
Why AI writes code better than it reads it
Early Greptile thesis: codebases are graphs, not file bags—reading at scale needs structure-preserving index, not naive embeddings.
-
AgentCoder: programmer + test designer + test executor
Multi-agent code generation with iterative testing and optimization—explicit separation of writing code and writing/running tests.
-
SWE-bench: real GitHub issues as the coding-agent exam
ICLR 2024 oral introducing 2,294 issue→patch tasks graded by fail-to-pass tests—the default yardstick for agentic software engineering.
-
Reflexion: agents that learn from linguistic critique
A 2023 framework that stores verbal reflections from failed trials and reuses them—strong gains on coding benchmarks without weight updates.
-
Memory-assisted prompt editing after deployment
EMNLP 2022: grow a memory of misunderstandings plus user clarifications, then edit prompts for similar future queries—no retraining.
-
CodeReviewer: large-scale pre-training for review tasks
Early foundation model for automating code review activities—quality estimation and comment generation from diffs and review data.