ROBOCO: factories turn failures into rules, but almost nobody checks the rules work
Oct 10 from ROBOCO, the last of four posts on open-source software factories. Across Bun, Astro, OpenClaw, mastra, Beads and others, failures get fed back into agent instructions, CI checks, workflow mechanisms, and decisions to scale automation back. OpenClaw added a success-rate circuit breaker after its implementation worker went 52 for 308 in a week. The missing step is verifying that new rules actually work: no project in the survey regularly measured it.
Source: Software Factory Best Practices from Open-Source Projects (4/4): Turning Failures into Rules
Dohyun Jung closes ROBOCO's four-part survey of public software factories with the step that is supposed to make them compound: turning failures into rules. He splits the loop into four parts (detect the failure, diagnose the cause, codify a rule or mechanism, verify the effect) and finds the first three across several repositories. The fourth he could barely confirm from public records at all. Figures were pulled from the GitHub API on October 10.
Where the failures go
The projects feed failures back into four places: agent instructions and skills, CI checks, mechanisms in servers or workflows, and decisions to stop or scale back automation. Bun mined about 2,500 merged PRs where review feedback led to fix commits and kept only the reasons that blocked merging, as a REVIEW.md the agents read. Astro rewrote its factory skill after bot PRs kept skipping changesets, diagnosing wording that let the model rationalize skipping them. OpenClaw found its AGENTS.md was truncated at 20,000 characters when injected, so the agent had been seeing fragments of its core rules; the team cut it to 18,900 and moved detail into skills.
Bun also blocks the same failure twice: an instruction against verbose comments, plus a CI check (comment-cop) that flags new multi-line comments in agent PRs. Jung's recommendation is to fix instructions and CI together.
Knowing when to stop
The most useful idea for anyone running agents around the clock is that scaling automation back is part of the loop, not an admission of defeat. OpenClaw's implementation worker succeeded 52 times and failed 256 times in a single week; with the success rate at 16.9% and failed runs consuming 123 hours, it added a circuit breaker that halts the lane when the success rate drops. Mastra switched its factory to manual mode when PRs arrived faster than it could review them. See also mastra's own account of going manual.
Where the loop breaks
Rules get written, but nobody checks they are followed. Astro built 24 evaluation cases for 8 skills and did not connect them to CI, citing cost and nondeterminism; the changeset and lockfile failures never became evaluation cases. A reverted Astro fix was not remembered, and the factory produced the same fix again when a new issue came in. Beads' rule that schema changes get human review is not enforced by branch rules. And Bun's CLAUDE.md says branches start with claude/ while robobun's actually start with robobun/.
Agents do edit their own instructions (roboclaw-bot files AGENTS.md PRs at OpenClaw; robobun wrote 4 of 50 commits to Bun's CLAUDE.md), but mostly through ordinary PRs that people approve. Jung's last best practice is his own proposal, since he found nobody doing it: add evaluation cases with every new rule and run them regularly.
For steady, always-on agent work, this is the gap that matters. A factory that only accumulates rules gets more constrained, not more reliable. Earlier in the series: who merges and filtering by evidence. Compare Jag Reehal's weekly gardener, which keeps only fixes that went unreverted.