What it takes to accept an agent-written change

We noticed fairly early that generating code changes wasn’t the difficult part of building an internal software factory. Deciding whether a particular change was safe to accept was harder.

Harder still was showing why.

During the early build phase, coding agents received work without an agreed plan, an acceptance gate or an independent review. The implementing agent also reported what it had done. That let us explore agent-led delivery, but acceptance rested on the agent’s account rather than evidence tied to the resulting software.

That isn’t enough for dependable engineering.

An agent’s account of its own work is not evidence that the resulting software meets the requirement.

We’ve since separated planning, product clarification, implementation, commit-specific continuous integration evidence, independent review and human judgement. Some of those controls are reported as operating. Others are still design requirements.

The distinction matters. Saying that a control exists, saying that it is used and showing it acting on a real candidate are three different things.

Right now, we don’t have an auditable record of that complete path operating on one candidate commit.

The acceptance path we’re aiming for

The proposed workflow starts with an approved interpretation of the requirement. It ends with a person deciding whether to accept the reviewed candidate.

APPROVE PLAN AND ACCEPTANCE CONDITIONS
    |
    v
IMPLEMENT CANDIDATE
    |
    v
RUN RELEVANT CONTINUOUS INTEGRATION CHECKS
    |
    v
VERIFY EVIDENCE AGAINST REVIEWED COMMIT
    |
    |-- evidence absent, malformed or mismatched --> REJECT
    |
    v
INDEPENDENT REVIEW
    |
    |-- accepted --> HUMAN ACCEPTANCE DECISION
    |
    `-- rejected --> REPAIR
                        |
                        |-- budget remains --> IMPLEMENT CANDIDATE
                        |
                        `-- budget exhausted
                                |
                                v
                    HUMAN JUDGEMENT | REPLAN | CHANGE SCOPE

Automated review doesn’t make the final acceptance decision. An accepted review passes the candidate to a person.

The controls currently sit at different stages of implementation, though. The available records don’t show them working together on one change, and they don’t provide enough implementation evidence for an end-to-end audit.

Even the boundary of an assessment remains undefined. We don’t yet have:

• the assessment period;
• the codebase estate;
• eligibility rules;
• the complete record population;
• representative control invocation records;
• a complete anonymised delivery trace.

So we need to be precise about control maturity.

A related view describes controls as intended, partial, reported operational or auditable.

A statement that a control is used doesn’t meet the threshold for deployment. Based on the available material, nothing can yet be classified as deployed, operating effectively or auditable. The source records, workflow artefacts and repository policies needed for those classifications haven’t been supplied.

There’s also no concrete commit-binding protocol or anonymised control trace. We can’t reconstruct either from partial statements without inventing the missing events.

That would put us straight back where we started: trusting an account rather than inspecting evidence.

A plan sets the target; commit-bound evidence shows what happened

Implementation starts from an agreed plan and specification. That plan is the shared reference for implementation and review, covering:

• intended behaviour;
• affected boundaries;
• the validation approach;
• acceptance conditions.

But a technically precise plan can still hide an unresolved product decision.

If ambiguity reaches implementation, an agent has two choices: stop or infer intent. The second option can produce coherent code that solves the wrong problem. Product clarification therefore has to happen before implementation. Business rules, exceptional cases and acceptable trade-offs need to be recorded in the specification, with a person deciding the intended behaviour before code is written.

The aim is to implement an approved interpretation of the requirement against acceptance conditions defined in advance, rather than produce something that merely looks plausible.

The records say implementation and review use an agreed plan, but approved plans aren’t yet universal. Product decision records are also too limited for us to measure coverage. Both agreed plans and pre-implementation product clarification therefore remain partial.

And a plan is still only the target. It isn’t proof that a candidate reached it.

That proof is intended to come from the relevant continuous integration checks. Evidence counts only when it has been independently checked and matched to the exact commit under review.

A test summary copied into an agent response won’t do. It might:

• refer to another revision;
• omit a failed check;
• describe an environment that can’t be reproduced;
• provide no evidence that every required check ran.

The commit-bound interface needs four defined elements.

The intended gate fails closed. If evidence is absent, malformed, incomplete, stale, mismatched or otherwise non-compliant, the candidate is rejected.

That’s the intended behaviour. The records don’t yet show how it happens.

Several design details are still missing:

• how the gate obtains the reviewed commit identity;
• how evidence carries the tested commit identity;
• how those identities are compared;
• how the evidence producer is authenticated;
• which systems may produce or store trusted evidence;
• which attestations are required;
• how attestations and provenance are checked;
• how evidence reuse across revisions is prevented;
• how checking failures are recorded;
• how the workflow blocks acceptance after a failure.

Until those interfaces, checks and trust boundaries are documented, commit-bound evidence and fail-closed enforcement remain specified or intended controls rather than auditable ones.

An assessable record needs enough information to connect the requirement, candidate, checks, review and final disposition.

Those fields describe the minimum record. They don’t mean that the schema has already been implemented.

The corresponding state transitions are:

PLAN_APPROVED
    -> CANDIDATE_CREATED
    -> CHECKS_COMPLETED
    -> EVIDENCE_VERIFIED
    -> REVIEW_COMPLETED
    -> HUMAN_DECISION

EVIDENCE_REJECTED
    -> REPAIR_STARTED
    -> CANDIDATE_CREATED

REVIEW_REJECTED
    -> REPAIR_STARTED
    -> CANDIDATE_CREATED

REJECTED_WITH_BUDGET_EXHAUSTED
    -> HUMAN_JUDGEMENT
    -> REPLANNED | SCOPE_CHANGED

Each transition needs a recorded source state, destination state, candidate identity, reason and time. Those transition records aren’t present in the available material.

Independent review still needs a rejection policy

The reviewing agent is reported to use a different model family from the implementing agent. Implementation and review are separate, and findings are reported as being classified by severity and category. A rejection must give an engineering reason rather than a general expression of confidence.

Those are useful separations. But classifying a finding and deciding whether it blocks acceptance are different controls.

The current records don’t define:

• the severity taxonomy;
• rejection thresholds;
• findings that require escalation;
• findings that permit acceptance;
• how multiple findings are combined.

Separate automated review and findings classification are reported as operational or in use. Supporting workflow records, review artefacts and finding counts aren’t available, however. The assessment period and codebase scope are also undefined.

All recorded review findings in the current phase came from automated reviewers rather than people. But the records neither define that phase nor provide the underlying counts. We don’t have counts by severity, category, reviewer type, period or scope.

People still have specific responsibilities. They define intent, approve plans, resolve ambiguity and decide whether a reviewed candidate is acceptable.

Using a different model family may reduce repeated assumptions, shared blind spots or shared interpretations of the specification. We don’t have comparative evidence that it finds faults a same-family review would miss. For now, it remains an untested design hypothesis rather than a mandatory acceptance control.

Here’s the current position.

A rejected candidate can return to implementation, but it can’t circulate forever. The proposed repair budget sets that boundary.

REJECTED
    |
    |-- budget remains --> IMPLEMENTATION
    |                         |
    |                         v
    |                    EVIDENCE CHECK
    |                         |
    |                         v
    |                       REVIEW
    |
    `-- budget exhausted --> HUMAN JUDGEMENT
                                  |
                                  |-- REPLAN
                                  `-- CHANGE SCOPE

The budget is a safety control, not a measure of engineering effort. To assess it, we need:

• the configured repair limit;
• how that limit was selected;
• how it is enforced;
• a record of every repair attempt;
• the budget state at every rejection;
• the workflow state entered after exhaustion;
• the terminal outcome.

Those definitions and records aren’t available. Bounded repair remains a design principle or specified control until the workflow records its configuration, transitions and exit states.

The repository can’t sit outside the control model

A controlled workflow can’t compensate for an environment that agents can’t reproduce.

Codebase readiness includes:

• written engineering conventions;
• a pinned runtime;
• dependency locking;
• an architecture map;
• testing guidance;
• common failure modes;
• consistently invocable checks.

Agent tooling and default-branch protection are reported as the least consistently satisfied areas. The records identify both as weaknesses, but the supporting repository policies and configurations aren’t available.

We also don’t know the assessed estate, scoring method, dates or underlying counts. Nor do the records say which readiness checks are automated, which are reviewed manually and which block acceptance.

That leaves codebase readiness in a partial state. It can’t yet be assessed reproducibly.

Where readiness differs between codebases, the same workflow may provide different levels of assurance. Every undocumented convention or environment-specific step pushes another decision back into inference, making acceptance harder to audit.

Measurement runs into the same issue. Previously reported percentages and the delivery-activity comparison have been omitted because they can’t be reproduced from the available records.

First-pass acceptance means acceptance through the configured evidence and review path. It doesn’t mean code perfection.

The records don’t support published figures for readiness, approved-plan coverage, first-pass acceptance, verified-evidence coverage or delivery activity. Denominators, dates, reporting windows, work-item counts and eligibility rules are missing.

They don’t show a time saving, productivity multiplier or increase in individual working speed either. And we have no comparative data for cross-family and same-family review.

The next useful artefact is one complete trace

The work now has to proceed in dependency order.

1. Define the assessment scope

   • Set the assessment period.
   • Define the codebase estate and eligible work.
   • Identify the complete record population.

2. Standardise plans and product decisions

   • Make approved plans and acceptance conditions universal.
   • Record product clarifications as durable decisions.

3. Specify commit-bound evidence

   • Define the evidence fields and immutable commit identity.
   • Define required attestations and provenance checks.
   • Document producer authentication.
   • Document identity matching, storage and trust boundaries.
   • Define rejection for absent, malformed, incomplete, stale, mismatched or non-compliant evidence.

4. Complete the review policy

   • Define the severity taxonomy.
   • Set rejection thresholds and escalation rules.
   • Define which findings permit acceptance.
   • Define how multiple findings are combined.
   • Set a comparison method for cross-family and same-family review.

5. Record control execution

   • Store evidence-checking results and failures.
   • Store review decisions and classified findings.
   • Retain the candidate identity at every stage.
   • Provide workflow records and repository policies for controls reported as operating.

6. Enforce bounded repair

   • Record how the repair limit is selected and configured.
   • Record each repair transition and budget state.
   • Ensure budget exhaustion produces a recorded terminal outcome.

7. Produce an end-to-end trace

   • Use an existing delivery record.
   • Don’t reconstruct missing events.
   • Include plan approval, product clarification, commit identity, continuous integration provenance, attestations, evidence checks, review findings, repair attempts and final disposition.

8. Set up reporting

   • Define numerators, denominators, periods, eligibility rules and exclusions.
   • Publish the readiness checklist, enforcement method and scoring method.
   • Report verified-evidence coverage and review findings only when the required records exist.

9. Address codebase readiness

   • Strengthen agent tooling.
   • Strengthen default-branch protection.
   • Record the assessment method and codebase coverage.

Of these, the missing delivery trace matters most.

It needs to show one candidate moving from an approved specification and product clarification through implementation, commit identification, commit-bound checks, attestations, independent review, any repair or escalation, and the final human decision.

Every stage has to link to its artefact: the approved plan, product decision record, commit identifier, continuous integration evidence, attestations, review findings, repair history and acceptance decision.

Could we create an illustrative trace instead? We shouldn’t. Not where the records don’t exist.

That would repeat the original mistake, replacing evidence with an account of what supposedly happened.