The two-minute version
Everything below expands on this page. If you read nothing else, read this.
Ship a high-quality product at a speed no human team could match, while humans stay firmly in control of what gets built and what reaches production. Speed and quality are not in tension if, and only if, verification is automated.
Trust in AI-written code cannot come from humans reading it. There is too much of it, and reading is the slowest, least reliable check available. Trust comes from the system around the agent: rules that steer it, sensors that catch it, loops that force it to fix its own failures, and measurements that show whether any of this works. If an agent cannot verify its own work, nothing else matters.
Six stages. Assisted (autocomplete), Delegated (agent writes, human reads every line), Harnessed (rules steer, checks catch, loops fix), Verified autonomy (agent proves behaviour in the real app), Managed agent teams (agents on a queue around the clock), Self-improving system (the system captures its own lessons as structure). Most production teams in 2026 sit at Stage 1 or early Stage 2.
The most common failure is not a lack of tooling. It is a team that has the paperwork of a rigorous process and the outcomes of a loose one: rules files nobody's agent reads, hooks that never fire, "review required" that is a checkbox nobody ticked, browser testing deferred in every feature and never done. Every stage transition in this handbook has a test for whether a control actually binds.
Take the self-assessment in section 04. Go to your stage's section. Do the exit checklist in order. Do not skip ahead: a Stage 4 agent team built on a Stage 1 verification layer produces slop at industrial scale. Every checklist item names the evidence that proves it is done.
A team of agents working around the clock on a human-prioritised queue, opening pull requests that arrive with their own evidence, gated by machines, merged by humans who read evidence and metrics instead of diffs, with the system capturing its own lessons and proving its own rule changes. Humans own direction, taste, irreversible actions, and exceptions.
The three moves that matter most, whatever stage you are at
- Make the merge gate bind. Branch protection on the integration branch with required status checks and no admin bypass. This single setting converts every downstream repair into an in-session fix. It is the highest-leverage change on this page and it costs zero engineering.
- Give the agent a check it can run. One command that runs the same lint, type, test, build, and security checks as CI. The difference between a session you watch and a session you walk away from is whether the agent can find out for itself that it is wrong.
- When something goes wrong, change the system. A hook, a lint rule, a CI gate, a type, a regression test. A sentence in a rules file is the weakest possible fix and should be your last resort, not your first.
Why this, why now
The opportunity is real and the failure mode is real
The evidence from teams who have gone furthest is consistent. OpenAI's Codex team reported shipping an internal product over five months with roughly 1,500 merged pull requests and about a million lines of code, none of it hand-written, in about a tenth of the estimated time. Lauren Tan, whose twenty-plus agents handle bug fixes, testing and merging around the clock, says she "almost never looks at code anymore" because architectural constraints, CI, and verification flows took over the checks she used to do line by line. Simon Willison, one of the most careful sceptics in this space, wrote in May 2026 that for well-tested, well-specified changes he no longer reviews every line either. He treats the agent like another engineering team at a large company: a semi-black box that produces reliable output because the contracts around it are tight.
The same evidence says what happens when you skip the system-building. DORA's 2025 research found that AI raises delivery throughput but also raises delivery instability, and that about 30 percent of developers report little or no trust in AI-generated code. Their summary: AI does not fix a team, it amplifies what is already there. Mitchell Hashimoto warned that entire companies are in "AI psychosis," unable to talk rationally about what agents can and cannot do.
Put those together and the conclusion is uncomfortable but simple. Agents multiply whatever your process already is. A team with loose gates and agents gets loose outcomes faster. A team with binding gates and agents gets high-quality outcomes faster. The agents are not the variable. The gates are.
The thesis: trust is manufactured by verification, not by reading
Every serious practitioner converges on the same claim, in their own words.
- Lauren Tan (pstack): verification comes first because everything else depends on it. Verify against the real artifact, never a proxy: run the feature, read the actual value, inspect the diff. "It compiles" does not count.
- Mitchell Hashimoto: every time the agent makes a mistake, change the system so that mistake structurally cannot recur. Not the prompt. The rules, tools, constraints, and feedback loops.
- Birgitta Böckeler (martinfowler.com): Agent = Model + Harness. The harness has two halves: guides that constrain the agent before it acts (rules files, plans, tool definitions) and sensors that check it after it acts (linters, tests, hooks, evals).
- Anthropic's Claude Code guidance: give the agent a check it can run. It is the difference between a session you watch and one you walk away from.
- OpenAI: "Humans steer. Agents execute." The faster an agent moves without constraints, the worse the architectural drift, so encode the project's principles into the repository itself.
- 1Password: the verifier is the bottleneck, and no system should grade its own homework.
The unit of trust is a check that runs, not a person who read. A human reading a diff is the fallback for things you have not yet learned to check mechanically, and this whole ladder is, in large part, a plan to shrink that set.
What "human control" means under this thesis
Control does not mean reading everything. It means humans hold, and never delegate, five things.
| Humans own | Agents own |
|---|---|
| Direction: what to build, for whom, why | Turning a plan into working, verified code |
| Settled decisions: trade-offs once examined and pinned in the spec | Building to those pins, re-opening one only on evidence of a defect |
| Irreversible actions: merge to the shared branch, production deploy, schema changes, deleting data or branches, anything customer-facing | Everything reversible, including committing green work and advancing the plan |
| The bar: which checks exist, what "green" means, what evidence "done" requires | Meeting the bar and proving it with pasted output |
| Exceptions: loops that stop, findings that survive review, metrics that move the wrong way | Escalating cleanly with the evidence attached |
The operating principle has a name in pstack: never-block-on-the-human. Make reasonable decisions, proceed with reversible work, and let the human course-correct after the fact; reserve confirmation for irreversible actions. As autonomy rises, what humans look at shifts from code to evidence to metrics and exceptions. That shift is the ladder.
Operating principles
These are the beliefs that decide arguments. When two options look equally good, pick the one these favour. They apply at every stage.
Verification before everything.
No feature is done until the agent has driven it through the real app and attached evidence. A green build is a proxy. A screenshot of the real flow, a passing test that would have failed before, a grader's PASS line: those are evidence.
Encode lessons in structure, not prose.
When something goes wrong, the fix is a hook, a lint rule, a CI gate, a type that makes the wrong state unrepresentable, or a regression test. Text in a rules file is the weakest form of a lesson. Mechanism hierarchy: unrepresentable state › lint or banned API › canonical helper › runtime check › documentation.
Writer and judge are never the same agent.
The builder never grades its own work. Reviewers get the artifact and the contract, never the builder's reasoning, because independence dies the moment you tell a reviewer what you expect it to find. A test-writing agent should write tests from the spec without reading the implementation for the same reason.
A control is done when it fires, not when it merges.
A review workflow that is ticked as done when the YAML lands, but never executes a turn, is worse than no workflow: it produces a permanently red or permanently green check everyone learns to ignore. Every gate, hook, or routine is verified by a run that proves it fired, and its failure mode must be loud, never
continue-on-error.Measure behaviour, not documents.
Re-reading a rule tells you it exists. Running a fresh agent against a canned task and grading the output mechanically tells you the rule binds. Rule and skill changes should cite an eval run. Trend the score; a single run is weather, not climate.
Fewer, sharper things.
Teams that add a skill or rule for every problem end up with dozens of skills, dozens of rules, and tens of kilobytes of context loaded before the agent reads a single project file. Adding is the failure mode, not the fix. Every proposed addition must name what it replaces or why nothing existing covers it.
Go deep first, then go parallel.
Speed comes from parallelism, and parallelism is only safe when each unit is small, verifiable, and independently checked. Competing attempts plus a separate judge beat accepting the first answer for anything architectural.
Honesty in the paper trail is a hard rule.
A plan may say Complete only when every task and acceptance criterion is checked with named evidence. Deferrals are recorded, never silent. A pull request that was not browser-verified says
UNVERIFIEDand that is a merge blocker. Checklists that lie are the root of the worst incidents: a plan marked fully complete while the code does not compile.
The ladder at a glance
Six stages. In the staircase at the top of this page, the height of each step is the share of the team's attention that moves from reading code to reading evidence.
| Stage | Who writes | Who verifies | What humans read | Attention per PR |
|---|---|---|---|---|
| 0 Assisted | Human, with autocomplete and chat | Human | Everything | 100% |
| 1 Delegated | Agent writes, human reviews every line | Human, plus whatever CI exists | Every diff | High |
| 2 Harnessed | Agent, inside guides; sensors catch violations; loops fix to green | Machine for mechanics; human for behaviour | Diffs for risky areas, evidence for the rest | Medium |
| 3 Verified autonomy | Agent writes and proves behaviour in the real app | Machine end to end, including the browser or the real client | Evidence bundle, review findings, exceptions | Low. Merge is a human act based on evidence |
| 4 Managed teams | Multiple agents on a queue, 24/7, scheduled routines | Same, plus routines watching production and the backlog | Metrics, exceptions, the queue | Manager: priorities and escalations |
| 5 Self-improving | Same | Same, plus the system proves its own rule changes with evals | Trend lines and the rare novel failure | Product owner and editor of principles |
How to read the stage sections
Each stage below has the same five parts, so you can compare across stages.
- What it looks like. The shape of the work, in plain language, with an example.
- A day in the life. A realistic timeline for one developer at that stage.
- Signs you are here. Two columns: what is true, and what typically breaks.
- Exit checklist. The exact things to do to reach the next stage, in order, each with the evidence that closes it. These are deliberately few. If you cannot produce the evidence, the item is not done.
- Do not skip. The mistake teams make when they try to jump this stage.
Timing guidance is given per stage. Confidence in those estimates is low to moderate: the engineering is well understood, but the constraint is almost always human attention on admin and infrastructure steps, and that varies enormously between teams.
Assisted
Developers use AI as a faster keyboard and a smarter search engine. Every line that lands was typed or accepted by a person who read it.
What it looks like
Autocomplete in the editor. A chat window open on the side for "how do I do X in this library" and "explain this stack trace." Occasionally a developer pastes a function in and asks for a refactor, then pastes the result back and adjusts it by hand. The AI never touches the repository directly. There is no rules file for the AI, or there is one that a single enthusiast wrote and nobody else knows about.
Example. A developer is adding a date-range filter to a reports page. They type the component skeleton, accept a few autocomplete suggestions for the state hooks, ask chat how to format a date in the project's locale, and hand-write the tests. Total AI involvement: maybe 20 percent of the keystrokes, zero percent of the decisions.
- 09:00Stand-up. Pick a ticket. Read the ticket, open the relevant files, start typing.
- 09:30Autocomplete fills in boilerplate. Ask chat about an unfamiliar API. Copy the snippet, adjust it.
- 11:30Hit a confusing error. Paste it into chat. Try the suggested fix. It works, or it does not and you search the web.
- 14:00Write tests by hand. Run them locally. Open a pull request.
- 15:00Review a colleague's PR line by line. Leave twelve comments. Half are style.
- 17:00Your PR gets two rounds of review comments. Ship tomorrow, maybe.
Signs you are here
- Humans write every commit. AI suggestions are accepted keystroke by keystroke.
- There is no shared rules file for the AI, or it is not maintained.
- Code review is line-by-line human reading, and it is the main quality gate.
- Throughput is roughly what it was before AI, maybe 10 to 30 percent better.
- Nothing dramatic. This stage is safe and slow.
- Inconsistency: each developer uses AI differently, so conventions drift by person.
- Quiet copy-paste of chat output that nobody tested, especially in error handling.
- Rising frustration: the team can see the potential and is not getting it.
Exit checklist · 0 → 1
- 1
Pick one agentic tool and give the whole team access.
A terminal or IDE agent that can read the repository, edit files, and run commands (for example Claude Code, Cursor, or Codex). One tool first, not three. The point is a shared vocabulary.
Evidence: every developer has run at least one agent session that edited a file and ran the tests.
- 2
Write the first rules file.
One file at the repository root (
AGENTS.mdor your tool's equivalent) with no more than a page of content: how to install and run, how to run the tests, the five conventions that matter most, and what the agent must never do (push, force-push, edit secrets, delete data). Every tool you use should read this same file or import it.Evidence: the file exists, is under a page, and a fresh agent session follows the test command in it without being told.
- 3
Make the test suite runnable in one command.
If a new engineer cannot run all tests with one command in under ten minutes, an agent cannot either. Fix that first.
Evidence: one documented command runs lint, types, and tests locally, and it passes on a clean checkout.
- 4
Agree the review rule for agent-written code.
At Stage 1 the rule is simple: a human reads every line, same as any other PR, and the PR says it was agent-written. Write this down.
Evidence: the rule is in the contributing guide; the first three agent PRs carry the label.
- 5
Run one real ticket end to end through the agent.
Small, well-specified, low-risk. Let the agent write the code and the tests. Review it as you would any PR. Note where it went wrong.
Evidence: one merged PR where the agent wrote more than half the diff; a short written note of what it got wrong.
Do not skip this stage by letting agents loose on the repository with no rules file and no one-command test suite. Everything the agent writes will follow its training defaults instead of your conventions, and nobody will be able to tell you whether it works.
Delegated
The agent does the typing. Humans do the thinking, the reviewing, and increasingly the waiting. Speed is now capped by how fast people can read.
What it looks like
Developers describe a task and the agent implements it: reads the codebase, edits files, runs tests, iterates. The developer watches, corrects, and reviews. Pull requests get bigger and arrive faster. Code review becomes the bottleneck, and reviewers start skimming because they cannot keep up. The rules file grows as people add "never do X again" lines after each incident. CI exists but may not be required for merge.
Example. The same date-range filter. The developer writes a three-paragraph prompt: what the filter does, where it goes, that it should use the project's existing date utilities and follow the existing filter components. The agent finds those components, builds the feature, writes tests, and runs them. Forty minutes of agent time. The developer then spends thirty minutes reading a 400-line diff, finds that the agent duplicated a date-formatting helper that already existed, asks for a fix, re-reads. Total elapsed time is similar to Stage 0 but the developer's time has shifted almost entirely into reading.
- 09:00Pick a ticket. Write a prompt. Start the agent. Watch it work for the first five minutes to make sure it understood.
- 09:30Start a second ticket in another window while the first runs. Feel productive.
- 10:30Agent one is "done." Read a 600-line diff. Find two things it invented and one convention it broke. Paste feedback. Re-run.
- 12:00Agent two has been waiting on a question for an hour. Answer it.
- 14:00Open two PRs. Both are large. Your reviewer sighs.
- 15:00Review three colleagues' agent PRs. Skim the second half of each. Approve. Hope.
- 16:30Something merged this morning broke the integration branch. Someone opens a fix PR. It is also agent-written.
Signs you are here
- Most of the diff in most PRs is agent-written. Humans still author the prompt and review the result.
- Throughput of code is up sharply. Throughput of merged, working code is up less.
- The rules file exists and is growing, mostly by appending prohibitions after incidents.
- Review is the bottleneck and reviewers know they are skimming.
- The paper process. "PRs go through CI before merge" is written down, but PRs merge in seconds and CI goes green half an hour later. This is the single most common Stage 1 pathology and it is invisible until someone measures it.
- Duplicated helpers, drifting architecture, and inconsistent patterns, because the agent never hears about conventions that live only in people's heads.
- Tests that assert whatever the code does rather than what it should do, because the agent wrote both.
- Fix-PRs. A meaningful share of the branches exist only to repair the shared branch.
- Plans and checklists that say "complete" over work that does not compile.
Exit checklist · 1 → 2
- 1
Turn on branch protection with required checks and no admin bypass.
On the integration branch (
main,develop, whatever you merge to). Required: the lint, type, test, and build checks. At least one reviewer for now. Nobody, including admins, can merge red. This is the first item because every other control depends on it.Evidence: a screenshot of the protection settings, and the first PR that was blocked by a red check. Merge-before-green rate is 0 for four consecutive weeks.
- 2
Build one canonical validation command, shared by the agent and CI.
A single script (call it
validate) that runs formatting, strict lint, type check, build, tests, and a secrets scan. Your CI runs exactly this script. Your rules file tells the agent to run it before declaring anything done.Evidence: CI calls the script by name; an agent session, unprompted, runs it and fixes what it reports.
- 3
Split the rules file into a short core and topic rules loaded by path.
The core (under two pages) carries the critical conventions and the never-do list. Topic rules (database, UI, testing, error handling) load only when the agent touches matching paths, so context stays small. Delete any rule that has no example of a violation it prevented.
Evidence: the core is under two pages; each topic rule names the paths it applies to; total rule count went down, not up.
- 4
Add write-time guards for the things that must never happen.
Pre-edit hooks that block writes to secrets files and generated files, and a shell guard that blocks destructive git commands (force-push, hard reset, branch deletion). These are hard blocks, not warnings.
Evidence: each guard has been triggered deliberately in a test session and the block was observed. A hook that has never fired is not a control.
- 5
Add a "tests ship with features" gate.
A CI check that fails any PR touching application logic with no change to a test file. Crude, but it ends the "I'll add tests later" pattern overnight.
Evidence: the check has failed at least one real PR; share of commits touching a test file trends up.
- 6
Introduce the builder-checker loop.
Two separate agent roles: a builder that writes, and a checker that runs the validation command and reports failures verbatim. The builder never judges its own work. The loop stops on: all green, five cycles, the same failure twice in a row, or a fix that breaks a previously passing check. Write these stop rules into the rules file.
Evidence: the stop rules are written down; a session log shows a loop stopping on "same failure twice" and escalating instead of guessing.
- 7
Adopt a spec-then-build workflow for anything larger than a bug fix.
Before the agent writes code it writes (or is given) a short plan with acceptance criteria a human could verify, and the human approves it. Settled trade-offs are pinned in the plan so downstream agents build to them.
Evidence: the last five feature PRs each link to an approved plan with checked acceptance criteria.
- 8
Add a plan-honesty lint.
A CI check that fails a PR whose plan says "Complete" while any task or acceptance criterion is unchecked.
Evidence: the check exists and has fired once.
Do not skip this stage by installing a dozen specialist reviewer agents on top of a process where merge does not require green CI. You will get twelve agents producing findings that land in the PR thread three minutes after the human already merged. Fix the merge gate first. It is one setting.
Harnessed
The agent works inside a system. Mechanical correctness is the machine's job. Humans still verify behaviour, but they read less code and more output.
What it looks like
A short rules file steers the agent before it acts. Hooks block dangerous writes as they happen. One validation command is the gate, locally and in CI. A builder-checker loop runs to green without a human relaying errors. Branch protection means nothing merges red. A handful of read-only reviewer agents (security, architecture, performance, error handling) fan out on a branch before merge and return findings in a shared format. The human reads risky diffs and the reviewer findings, and trusts the machine for the rest.
Example. The date-range filter again. The developer starts the build command with a link to the approved plan. The builder reads the plan and the topic rules for UI components, finds the existing date helpers because the rules file tells it where they live, builds the feature and the tests. The checker runs the validation command, reports a lint violation and a failing test verbatim, the builder fixes both, the checker returns all green. Reviewer agents flag that the new server function does not validate its input. The builder fixes that. The developer reads the reviewer findings, spot-reads the server function, opens the PR. CI runs the same validation command and passes. A colleague approves in a few minutes because the PR body says what was checked. Total human reading time: ten minutes instead of sixty.
Guides
- Rules file · short core
- Topic rules, loaded by path
- Workflow commands
- Plan with acceptance criteria
Loop
- Builder writes
- Checker verifies
- Failures fed back verbatim
- All green → commit, next task
Sensors
- Write-time hooks
- One validation command
- CI required for merge
- Reviewer fan-out
Human gates
- Plan approval
- Push · merge · deploy
- Schema changes · escalations
- 09:00Review the plan an agent drafted overnight for ticket A. Pin two trade-offs. Approve.
- 09:15Start the build loop on ticket A. Start a fix session on bug B in a second worktree. Walk away.
- 10:30Ticket A loop reports ALL GREEN with the checker output pasted. Reviewer fan-out surfaces one real finding and two low-confidence ones. Ask the builder to fix the real one.
- 11:00Bug B loop stopped: the same test failed twice. Read the failure. The agent misunderstood the spec. Clarify in two sentences. Restart.
- 13:30Open PR for A. The body lists what was checked and what was not. Spot-read the one risky file. Manually click through the feature in a local build because there is no automated browser check yet.
- 15:00Approve two colleagues' PRs. Read the checker output and reviewer findings, spot-read one file each. Twenty minutes total.
- 16:00A hook blocked an agent from editing a generated file. Good. It found the right command to regenerate it instead.
Signs you are here
- Nothing merges red. Branch protection is on and has blocked real PRs.
- One validation command is the gate locally and in CI.
- Hooks fire. You have seen them block something.
- The builder-checker loop runs to green without a human relaying errors.
- Most PRs come from agent branches, and the fix-PR rate is falling.
- Browser and end-to-end verification is deferred in every feature and never happens. The agent says "manual QA recommended" and the human clicks through, or does not.
- Rules that are written but do not bind: a fresh agent with no coaching still skips the failing-test-first step, still forgets the page wrapper, still misses the access check. The same eval criterion fails run after run.
- Scheduled automation that never executes. A review-on-every-PR workflow dies on turn one for weeks on a missing API key while its red check is ignored.
- Context bloat. Rules and skills multiply until sessions cannot navigate them.
- Lint rules for real conventions kept at "warn" because the backlog is too big to make them errors.
Exit checklist · 2 → 3
- 1
Make end-to-end tests run in CI without a human, against a real seeded environment.
Whatever your client is (browser, mobile, CLI, API consumer), a suite that drives it against a disposable database with realistic seed data. Gate PRs touching the highest-risk flows (auth, billing, onboarding) on it.
Evidence: e2e green on a PR touching auth; a deliberately introduced regression is caught by CI, not by a person.
- 2
Build a feature map and a project-local verification skill.
A list of the ten to twenty most-used user flows, how to reach each, and the observable end state that proves it works. The verification skill drives each flow and produces a screenshot, console log, and PASS/FAIL. This turns QA from open-ended exploration into a checklist an agent completes and a human audits.
Evidence: the map covers the ten most-used flows; the verification skill produces an evidence bundle for each.
- 3
Make browser verification the default, not the deferral.
The build workflow at full depth spawns the browser checker. A PR that was not verified this way must carry an explicit
UNVERIFIEDmarker, and reviewers treat it as a merge blocker.Evidence: every merged agent PR in the last four weeks carries either verification evidence or the marker; the marker share is trending toward zero.
- 4
Start golden-task evals.
Three to five canned tasks (add a page, fix a bug with a failing test first, add a database change) run by a fresh agent with no coaching, graded mechanically by greps and existing linters. Run weekly. Any rules or skills change must cite a run.
Evidence: a results file with per-criterion PASS/FAIL for at least four runs; a CI check that fails agent-system PRs without an eval citation.
- 5
Mechanise every standing eval failure.
If the same criterion fails in three consecutive runs, the text is not working. Replace it with a mechanism: a lint for the missing wrapper, a check for the unregistered route, a fix workflow that refuses to proceed until a failing-test commit exists.
Evidence: the next eval run shows the criterion PASS without coaching; standing FAILs are zero.
- 6
Introduce the lint ratchet.
For every convention rule with a backlog too large to make an error today: commit the current per-rule violation count as a baseline, and fail any PR that exceeds it. Never update the baseline to absorb an increase.
Evidence: the baseline file is committed; the ratchet has failed one real PR; at least one rule has been promoted from warn to error.
- 7
Prove every scheduled automation fired before you rely on it.
For any scheduled agent workflow (review on every PR, nightly triage): run it once by hand, read the transcript, record the cost. Remove
continue-on-error. If it cannot be made to fail loudly, delete it rather than leave a permanently red check.Evidence: a transcript of the first real run and its cost, or the workflow is deleted.
- 8
Close the lessons file.
Every "lesson learned" entry has a regression test that would catch its recurrence. A lesson without a test is a sentence, not a control.
Evidence: a repo-health script reports lessons-with-tests at 100 percent.
Do not skip this stage by declaring "verified" because CI is green. Green CI proves the mechanics. It does not prove the feature works for a user. Until an agent can drive the real product and attach evidence, a human still has to, and if nobody does, you are shipping unverified behaviour at Stage 2 speed.
Verified autonomy
A pull request from an agent arrives with everything a human needs to merge it without reading the diff, and for routine work humans actually do that.
What it looks like
The agent does not just build the feature. It drives it through the real application, records what it saw, runs the reviewer fan-out, and assembles all of that into a structured PR body: checker output, verification screenshots, review verdict, residual findings, and each acceptance criterion ticked with a pointer to its evidence. The human reads that page, not the code. Merge takes seconds for routine work. Code gets read when the evidence is thin, a reviewer flagged something, or the change touches a place where the team has decided humans always look (auth, payments, data migrations).
Example. The date-range filter PR body now reads: "Acceptance criteria: 1. Filter defaults to last 30 days: PASS, screenshot 1. 2. Custom range persists in URL: PASS, screenshot 2 and URL shown. 3. Invalid range shows inline error: PASS, screenshot 3. Checker: all green, output attached. Reviewers: security PASS, performance one low-confidence note about an unbounded query, architecture PASS. Residual: none. Not verified: dark-mode rendering, noted UNVERIFIED." The reviewer reads that in ninety seconds, glances at the unbounded-query note, sees it is a paginated endpoint, and merges.
- 09:00Four agent PRs are waiting, each with an evidence bundle. Merge three in ten minutes. The fourth has a red reviewer finding; send it back with one sentence.
- 09:30Write the plan for a bigger feature. Two agents produce competing architecture proposals in parallel; a third agent judges them against the spec. Read the judge's comparison, pick one, pin the decision.
- 10:30Start the build loop on it. Start a fix on a production error. Both will verify themselves in the browser.
- 12:00The fix loop stopped: it cannot reproduce the error locally. Read the escalation. Give it the production log line it was missing.
- 14:00Look at the weekly scorecard. Fix-PR rate is down. QA-waived count is up this week. Ask why in the retro notes.
- 15:00Spend an hour actually reading code: the one PR this week that touches the permissions model. This is the reading that matters.
- 16:00The feature loop is green with evidence. Merge. Deploy. Run the production sanity check. Go home.
Signs you are here
- Routine agent PRs are merged on evidence without reading the diff, and nobody feels nervous about it.
- End-to-end tests run on every relevant PR.
UNVERIFIEDis rare and treated as a blocker. - Golden evals run on a cadence with zero standing failures.
- Human minutes per merged PR is measured and falling.
- The team has a written list of the places where humans always read code.
- The bottleneck moves rather than disappears: everything waits on one person's merge, or one admin's infrastructure change.
- Over-trust creeps in: a green evidence bundle for a change the verification map did not cover gets merged as if it did.
- Agents sit idle overnight and at weekends because every session still starts with a human.
- The backlog is prioritised in a tool the agents cannot see.
Exit checklist · 3 → 4
- 1
Make the evidence bundle the PR body, generated by the ship workflow.
No human fills anything in. The ship step assembles checker output, verification artefacts, review verdict, residual findings, and checked acceptance criteria.
Evidence: every merged agent PR in four weeks scores full marks on "acceptance criteria with evidence" without a human editing the body.
- 2
Put agent economics on a weekly scorecard.
Cycle time from plan approval to merge, fix-PR-within-7-days rate, reviewer-finding acceptance rate, human minutes per merged PR, agent spend per merged PR. Review it in a weekly retro.
Evidence: three trend lines visible for four consecutive weeks; the scorecard average is at or above your target for eight weeks.
- 3
Turn on the first scheduled routine: review on every PR.
It is the most valuable routine and the cheapest to make safe, because it only comments. Policy: routines may open PRs on their own branch and comment; they never merge or push to a shared branch. Cost one run before scheduling.
Evidence: the routine has posted a machine-parseable verdict on fifty PRs; cost per run is recorded.
- 4
Turn on the second routine: production error triage.
Nightly, it reads new production errors, reproduces each with a failing test, and opens a fix PR with the failing test committed first.
Evidence: ten triage PRs opened; at least half merged without changes.
- 5
Adopt competing attempts with a separate judge for architectural work.
Anything that crosses a module boundary gets two or three independent proposals and a judge agent that compares them against the spec. Record the rationale as a settled decision.
Evidence: used on every plan tagged architectural in the last month; rationale recorded.
- 6
Promote your convention lints from warn to error as the ratchet burns down.
Schedule a burn-down routine if the backlog is large.
Evidence: the most important convention rule is at error in the strict lint config.
- 7
Spread the merge authority.
More than one code owner per area. Batch approval for evidence-complete routine PRs; a red or contested PR stops the batch.
Evidence: no single person merged more than half of last month's PRs; median time-to-merge for evidence-complete PRs is under an hour.
Do not skip this stage by letting a routine merge its own work. Autonomous merge is a Stage 5 privilege earned by evidence over months, and even then only under strict conditions (see Stage 5). At Stage 3, routines open PRs and comment. Humans merge. Some practitioners, Lauren Tan among them, already let agents merge routine work. This guide is deliberately more conservative: autonomy is earned with evidence first.
Managed agent teams
The team of agents works while you sleep. The human's day is spent on direction, exceptions, and taste.
What it looks like
Issues tagged ready-for-agent in the tracker are picked up by a routine that writes a plan if none exists, waits for a human to approve it, then builds, verifies, and ships a PR with its evidence bundle. Several agents work in parallel, one owner per PR, small verifiable units, stacked where dependent. Other routines watch the system rather than the code: nightly error triage, a weekly retro that opens a PR against a rule when a metric moves the wrong way, a scheduled drift scan that files refactoring PRs. Humans approve plans, merge PRs, and handle escalations. A human looks at code when a metric or an escalation says to.
Example. Monday morning. Over the weekend, the queue routine picked up six tickets tagged ready. Four had approved plans and are now PRs with evidence bundles. Two are draft plans waiting for approval. Error triage opened two fix PRs for a null-pointer path that appeared in production Friday night; both have a failing test committed first and both verified in the browser. The drift scan opened one PR consolidating three near-duplicate date helpers into one. The retro routine flagged that QA-waived PRs rose from 2 to 7 last week and drafted a PR tightening the ship workflow to refuse a waiver without a named reason. The engineering lead spends the first ninety minutes of the week reading evidence and approving plans, and the rest on the one architectural question no agent should decide.
- 09:00Read the overnight queue report: PRs opened, PRs blocked, escalations. Merge six evidence-complete PRs in a batch. One is contested; it stops the batch and gets a comment.
- 09:45Approve two draft plans. Reject one: the ticket was underspecified. Rewrite the ticket instead of the plan.
- 10:30Triage the escalations. One agent could not decide between two data-model shapes. That is a human decision. Make it, pin it, restart the loop.
- 11:30Product conversation with a customer. Turn what you learn into three tickets with acceptance criteria. Tag two as ready-for-agent.
- 14:00Read the retro routine's proposed rule change. It cites the metric and an eval run. Merge it.
- 15:00Spend two hours on taste: the onboarding flow does not feel right. Write a design brief. Nobody, human or agent, could have told you this from a metric.
- 17:00Prioritise the queue for tonight. Log off. The agents do not.
Signs you are here
- PRs are opened while nobody is working. Humans start the day by reading a queue report, not by starting an agent.
- At least three routines run on a schedule under a written policy, with cost per run known.
- The queue is prioritised by humans in the tracker, and agents pick from it.
- Human minutes per merged PR is a tracked metric and trending down.
- Humans read code by exception, and can say exactly which exceptions.
- Cost surprises: a routine on the most expensive model with fifty turns per PR.
- Merge conflicts and duplicated work when parallel agents touch the same area without an owner.
- Measuring proxies: the scorecard looks great and users are unhappy, because nobody put production escapes and customer signals beside the process metrics.
- Lessons still captured by hand. The system does not yet learn from its own corrections.
Exit checklist · 4 → 5
- 1
Add the reflect step to every completed task.
Every build, fix, or triage run ends with a capture: did anything require a human correction, and what structure (hook, lint, test, type, grader criterion) would prevent it next time? The capture is a draft PR, not a note.
Evidence: the last twenty completed tasks each have a reflect entry; at least a quarter produced a mechanism PR.
- 2
Schedule the eval run and wire its failures to tasks.
A monthly (or weekly) golden-eval run whose FAILs automatically open mechanisation tasks in the queue.
Evidence: one eval FAIL traced end to end: run → task → mechanism PR → next run PASS.
- 3
Put outcome metrics beside process metrics.
Production escapes per release (new error groups within 72 hours of deploy), customer-reported defects, human minutes per merged PR, and agent spend per merged PR sit on the same scorecard as the process numbers.
Evidence: all four are on the weekly scorecard with four weeks of trend.
- 4
Write the autonomous-merge policy, narrowly.
Define the one class of PR a routine may merge on its own (for example: a small corrective PR created in the same run, targeting the integration branch, with a regression test that failed before and passes after, full checks green, an independent reviewer agent returning Go with no findings, no human review comments, and no changes to dependencies, migrations, CI, infrastructure, auth, permissions, or secrets). Everything else stays human-merged.
Evidence: the policy is in version control; the first autonomous merge under it is audited by a human after the fact and passes.
- 5
Make every routine's prompt, schedule, and write scope version-controlled.
No private copies of governing instructions in an account-level scheduler. Adding or changing a routine is a reviewed change with an eval citation.
Evidence: every live routine points at a file in the repository; a diff of that file is the only way to change its behaviour.
Do not skip this stage by giving routines merge rights because the queue is moving fast. The Stage 5 autonomous-merge policy exists precisely because it is narrow. Widening it under throughput pressure is how a team ends up with an agent merging a permissions change at 3 a.m.
Self-improving system
The system edits itself, but every edit is evidence-gated and human-merged. Humans are product owners and editors of principles.
What it looks like
When an agent is corrected, the system proposes the structural change that would have prevented the correction, with an eval run attached showing the new rule binds. When a metric moves the wrong way, a routine proposes the rule or workflow change and the eval that proves it. When a golden eval fails, a task is opened and a mechanism PR follows. Humans read trend lines and the rare novel failure. They still own direction, taste, the never-do list, and every irreversible action. The never-do list has not shrunk since Stage 1.
Example. A human corrects an agent for putting a business rule in a UI component instead of the domain layer. The reflect step captures it. Overnight, a routine drafts a lint rule that flags domain logic in the UI folder, runs the golden evals with and without the rule, shows that the "add a page" task now PASSES the layering criterion where it used to be coached, and opens a PR with the rule, the eval diff, and a one-paragraph rationale. The human reads the rationale and the eval diff in two minutes and merges. Nobody will make that mistake again, and nobody wrote a sentence in a rules file.
- 09:00Read the weekly trend lines. Escapes per release flat at near zero. Human minutes per PR down again. Agent spend per PR up slightly; open a question in the retro.
- 09:30Merge three system-improvement PRs from the reflect routine. Each has an eval diff. Reject one: it proposes a rule that would encode a preference, not a defect.
- 10:30The novel failure of the week: an agent shipped a correct feature that users found confusing. No check could have caught it. Write a design principle. This is the work.
- 13:00Customer calls. Product strategy. The queue for next month takes shape.
- 16:00Audit last week's one autonomous merge. It met every condition. Note that in the log.
Signs you are here
- Most rule and workflow changes are proposed by the system with an eval attached, and merged by a human.
- Every lesson has a covering test, and new lessons arrive with theirs.
- Outcome metrics (escapes, customer defects) are stable or improving while throughput rises.
- The never-do list is unchanged since Stage 1.
- Watch for eval drift: graders that pass because criteria stopped firing. Self-test graders before every run, compare per criterion, record denominators.
- Watch for the system optimising the scorecard instead of the product. Outcome metrics are the antidote.
- Watch for the human becoming a rubber stamp. Audit autonomous merges after the fact. Reject some proposals. If you have not rejected one in a month, you are not reading them.
- Re-run this whole ladder's self-assessment quarterly. Stages can regress when the team, the tools, or the product change.
There is no Stage 6. Two guardrails never move, whatever stage you reach. Irreversible actions pause for a human. A check is never weakened to reach green. A check that is genuinely wrong is a human decision made in daylight, not a loop edit.
Which stage are you at?
Answer honestly. Your stage is the highest row where every statement is true. Most teams overestimate by one stage because they count things that are written down but do not bind.
Find out which stage you’re really on.
Each check asks whether something works today, not whether it’s planned or half-built. The engineering edition has 24 evidence checks for CTOs and engineering leads. The leadership edition has 15 questions a CEO can answer without reading code. Take both and see where leadership and engineering disagree.
If you failed a row on exactly one item, you are "late" in the stage below and that item is your next move. If you failed on several, do them in the order the exit checklist gives, because the order is chosen so that each item makes the next one safe.
How human control evolves
Made explicit so nobody mistakes "reading less code" for "losing control."
| Decision | Stage 1 to 2 | Stage 3 | Stage 4 to 5 |
|---|---|---|---|
| What gets built | Human writes or approves the spec | Same | Same; the queue is human-prioritised |
| How it is built | Agent, within guides; human reads diffs, then spot-reads risky ones | Agent; human reads the evidence bundle | Agent team; human reads exceptions |
| Is it correct | Human, then machine for mechanics and human for behaviour | Machine end to end, including the real client | Same, plus routines watching production |
| Does it merge | Human, after green CI | Human, based on evidence, in seconds | Human, batch-approving evidence-complete PRs; a narrow class auto-merges under a written policy |
| Does it deploy | Human, with a post-deploy sanity check | Same | Human-approved release train with automated sanity and a rollback pointer |
| Do the rules change | Human | Human, with an eval citation | Routines propose; humans merge with the eval attached |
| What the agent may never do | Push to a shared branch, merge outside the written policy, force-push, delete data or branches, edit generated files by hand, write secrets, weaken a check. This list does not shrink. |
The scoreboard
Ten numbers, reviewed weekly. Start collecting the first five at Stage 2; the rest come online as the stages that need them arrive.
| Metric | From stage | Target | Why it matters |
|---|---|---|---|
| Merge-before-green rate | 2 | 0 | The gate binding |
| Fix-PR-within-7-days rate | 2 | < 10% | Escapes past the harness |
| Share of PRs with tests changed | 2 | → 100% of logic PRs | Tests ship with features |
| Lint-ratchet backlog per rule | 2 | → 0 | Structural debt burning down |
| Session scorecard average | 2 | ≥ 6 of 7 sustained | Session quality trend (see appendix A for the seven criteria) |
| Golden-eval pass rate and standing FAILs | 3 | ↑ · FAILs → 0 | Do the rules bind without coaching |
| Verified vs UNVERIFIED PR ratio | 3 | UNVERIFIED → 0 | Verification is real |
| Lessons with covering tests | 3 | 100% | Lessons are structure |
| Human minutes per merged PR | 4 | ↓ | The actual autonomy gain |
| Production escapes per release · agent spend per merged PR | 4 | ↓ · known, then ↓ | Quality as users see it; economics of the routine tier |
Risks and hedges
| Risk | What it looks like | Hedge |
|---|---|---|
| Instability rises with throughput (DORA) | More red integration branch, more hotfixes | Branch protection first. Nothing else starts until the merge gate binds |
| Over-trust, "AI psychosis" | Merging on vibes; treating a green check as proof of behaviour | Evidence bundle, UNVERIFIED as a merge blocker, the browser tier actually running |
| Slop volume | Large PRs, duplicated helpers, drifting architecture | Small verifiable units; an architecture reviewer agent; scheduled drift scans |
| Silent control failure | A gate red so often it is ignored, or green because it never ran | Principle 4. Loud failures only. Every control proven by a run that fires |
| Context bloat | Rules and skills grow until sessions cannot navigate them | Principle 6. Evals gate deletions and additions alike; count rules and make the count go down |
| Measuring proxies | Optimising the scorecard instead of the product | Escapes per release and human minutes per PR sit beside the process metrics |
| Eval drift | Graders that pass because criteria stopped firing | Per-criterion comparison, grader self-test before every run, denominators recorded |
| Cost surprise | A routine on the most expensive model with fifty turns on every PR | Cost one run before scheduling; spend per PR on the scoreboard |
| Bottleneck moves rather than disappears | Everything waits on one person's merge or one admin setting | Batch approval for evidence-complete PRs; more than one code owner; admin steps done in the first week of each stage |
| Paper process | The handbook says one thing and the repository does another | Every exit-checklist item names its evidence. No evidence, not done. Re-run the self-assessment quarterly |
Day-to-day habits at every stage
- Start every session with a workflow, not a blank prompt. A build command, a fix command, a plan command. The workflow carries the loop, the stop rules, and the checkers. Freeform sessions are how gates get skipped.
- Use a prompt template. One-sentence goal with the ticket ID, the plan path, acceptance criteria you could verify yourself, an explicit evidence demand ("paste the checker output"), and "do not merge."
- Never tell an agent to skip a gate silently. If a skip is needed: "skip X and record the skip and the reason in the plan changelog."
- Merge on evidence, after green. If the PR body says
UNVERIFIED, treat it as a blocker, not a note. - Feed failures back verbatim. Paste the error. Paraphrased symptoms send the builder guessing.
- Ask for the generalisation. After a bug fix: search for the class of bug repo-wide, add a lessons entry with a covering test.
- One session, one surface, one branch. Check what is already in flight before spawning another agent on the same area.
- When something goes wrong, change the system. A hook, a lint rule, a test, a grader criterion. If you only wrote a sentence in a rules file, you are not done.
Appendices
Templates and reference, collapsed. Open what you need.
A · Starter templates
# <Project> — Agent rules Stack: <language, framework, database, hosting>. Package manager: <one>. ## Run - Install: `<cmd>` Dev: `<cmd>` Validate (lint + types + build + tests + secrets): `<cmd>` - Run validate before declaring anything done. Paste its output. ## Critical rules (violations break the codebase) 1. <error-handling convention> 2. <import / module-boundary convention> 3. <data-access convention, e.g. tenant filter on every query> 4. <validation at boundaries> 5. <search before creating: components, helpers> ## Never Push to a shared branch · merge · force-push · delete data or branches · edit generated files by hand · write secrets · weaken, skip or delete a check. ## Loop stop rules Stop on: ALL GREEN · 5 cycles · same failure twice in a row · a fix breaks a previously passing check. Escalate with the evidence attached. ## Topic rules Loaded by path from `rules/`: database, ui, testing, error-handling.
## What <one paragraph, links to plan and ticket> ## Acceptance criteria - [x] AC1 <text> — evidence: <test name / screenshot #> - [x] AC2 <text> — evidence: ... - [ ] AC3 <text> — UNVERIFIED: <reason> ## Checker output <pasted validate output, ALL GREEN> ## Verification <flow → observable end state → screenshot> ×N ## Review fan-out security: PASS · architecture: PASS · performance: 1 low-confidence note (below) ## Residual findings <none, or listed with confidence>
- CI green before merge
- Tests changed alongside the feature
- Plan status honest (no Complete over unchecked items)
- No new convention-lint violations (ratchet held)
- Verification run, or explicitly waived with a reason
- No fix-PR within 7 days
- Acceptance criteria checked with evidence
- A routine may create and push only its own
auto/<routine>-<slug>branch and open a PR targeting the integration branch. - It never pushes to a shared branch or another author's branch.
- Its prompt, schedule, model, and write scope live in version control. Changing them is a reviewed change with an eval citation.
- Before it is scheduled, it is run once by hand, its transcript read, and its cost recorded. No
continue-on-error. - Autonomous merge is prohibited unless a narrow written policy (Stage 5) explicitly permits the specific class of PR, and every condition is met.
A · The harness in five layers, and which stage adds each
| Layer | What it contains | Added at |
|---|---|---|
| Guides (before acting) | Short core rules file · topic rules loaded by path · verb-named workflow commands · plans with acceptance criteria and settled decisions | 1 → 2 |
| Loop | Builder ⇄ checker with stop rules · code-blind tester at spec-first depth · browser checker at full depth | 1 → 2, browser at 2 → 3 |
| Sensors (after acting) | Write-time hooks · one validation command · CI required for merge · tests-with-features · lint ratchet · read-only reviewer fan-out · e2e in CI | 1 → 2, ratchet and e2e at 2 → 3 |
| Measurement | Weekly scorecard · golden-task evals · repo-health metrics · residual-findings ledger · agent economics · outcome metrics | 2 → 3, economics at 3 → 4, outcomes at 4 → 5 |
| Human gates | Plan approval · push, merge, deploy · schema changes · settled decisions · escalations · rule changes with eval | All stages; the list never shrinks |
B · Research sources and what to take from each
| Source | What to take |
|---|---|
| Dan Shapiro, The Five Levels: from Spicy Autocomplete to the Dark Factory, January 2026 | The driving-automation framing this ladder builds on. Where the Five Levels describe where a team is, this ladder adds the evidence-gated steps for moving up. |
| Lauren Tan (@poteto), pstack · github.com/cursor/plugins/tree/main/pstack | Verification first; prove-it-works; never-block-on-the-human; encode-lessons-in-structure; sequence-verifiable-units; verification skill + feature map; competing architectures with a judge; "go deep first"; 20+ agents 24/7 escalating only for human decisions |
| Birgitta Böckeler, Harness engineering for coding agent users, 2026 | Agent = Model + Harness; guides vs sensors |
| Mitchell Hashimoto, My AI Adoption Journey, 2026 | Six-step adoption; change the system so a mistake structurally cannot recur |
| Simon Willison, Vibe coding and agentic engineering are getting closer than I'd like, 2026 | Tests as the differentiator; trust in agent output is earned, not assumed |
| OpenAI, Harness engineering: leveraging Codex in an agent-first world, 2026 | ~1,500 PRs, ~1M LOC, zero hand-written; "Humans steer. Agents execute."; scheduled drift-cleanup agents |
| DORA, State of AI-assisted Software Development | Throughput up and instability up; ~30% low trust; AI amplifies what is there |
| Anthropic, Claude Code best practices; How Anthropic teams use Claude Code | Give the agent a check it can run; autonomous loops driven by tests |
| Berkeley RDI, Autonomous software development; Hicron, The autonomy ladder | L1–L5 framing; most production teams at L2–L3 in 2026 |
| 1Password, Verified loops | The verifier is the bottleneck; no system grades its own homework |
Attribution confidence: high for the pstack repository text and the linked articles; moderate for interview quotes reported second-hand. The "paper process" failure examples in the stage sections are drawn from a real team's internal audits, anonymised.
C · Glossary
- Harness: everything around the model that shapes what it does: rules, workflows, hooks, tools, CI, evals.
- Guide / sensor: a control that acts before the agent moves (rule, workflow, plan) versus after (hook, lint, test, reviewer).
- Loop: builder writes, checker verifies, failures relayed verbatim, until green or a stop rule.
- Ratchet: a committed ceiling on a violation count that may only go down.
- Golden task: a canned task run by a fresh agent and graded mechanically, to test whether rules bind.
- Evidence bundle: the checker output, verification artefacts, review verdict, and checked acceptance criteria that accompany a PR.
- Routine: a scheduled agent run that may open a PR and comment, and nothing more unless a written policy says otherwise.
- Settled decision: a trade-off the owner examined and pinned; re-opened only on evidence of a defect.
- Paper process: a control that is written down but does not bind. The most common way a team overestimates its stage.
- UNVERIFIED: the marker on a PR whose behaviour was not driven through the real product. A merge blocker, not a note.