Testing whether cross-family code review finds more defects

We noticed a gap between what the review process was designed to do and what the evidence could actually show.

Implementation and code review were deliberately assigned to different model families. That gave us operational separation, which was the point. But it didn’t show that cross-family review found more defects than review by the same family.

The reasoning made sense. A model reviewing its own output may carry assumptions from implementation into review. If it misunderstood a business rule, chose an unsuitable abstraction or missed an edge case, that mistake could survive a second pass.

A different model family may challenge those assumptions.

But a different name doesn’t prove an independent view. Disagreement doesn’t prove either reviewer is right, either.

Cross-family review creates useful disagreement, but its advantage over same-family review remains unproven.

The routing we reviewed

The documented workflow assigned implementation and code review to different model families:

The other responsibilities were split as follows:

• Model C challenged plans for LOW, STANDARD and HIGH-risk work.
• Model A challenged plans for CRITICAL_FINANCIAL work.
• Model E provided visual review for UI work.
• Model D performed verification at every risk level and handled escalation where required.

Plan challenge, implementation, code review and verification remained separate controls. In each case, the reviewer examined an artefact it hadn’t produced.

There was a catch. The routing table had no effective date or model-version context. We could treat it as a snapshot of an operating process, but not as a lasting routing specification.

A different reviewer still needs the right evidence

Changing the model family only helps when the reviewer has enough evidence to tell a real defect from an intentional constraint.

The repository readiness checks described in the material included:

• pinned runtimes;
• lockfiles;
• architecture maps;
• testing guidance;
• common failures;
• one-command validation.

Those checks give reviewers observable conditions against which to assess a change. A failing test, an inconsistent business rule or a missing validation path supports a finding more clearly than a preference about naming, formatting or architecture.

What wasn’t supplied mattered just as much.

The material didn’t include reviewer prompts, context packages, tool permissions or deterministic validation commands. It also left out the risk taxonomy, classification procedure, fallback behaviour, escalation criteria and severity calibration.

That makes comparison difficult. Were two reviewers working under equivalent conditions? Could we reproduce their findings? Could we explain why their results differed?

From the available material, we couldn’t answer those questions.

Where the evidence ran out

The internal measures covered first-pass performance, readiness checks, approved-plan coverage, agent-tool support, default-branch protection and finding severity.

We’ve omitted their reported rates because the material didn’t provide the underlying counts, denominators, definitions or collection dates.

Those measures describe parts of the current process. They don’t answer the comparative question: does cross-family review perform better than same-family review?

The material also lacked:

• a defined population of repositories and changes;
• inclusion and exclusion criteria;
• finding-level audit records;
• false-positive, disagreement and override rates;
• escaped-defect rates;
• a controlled comparison between same-family and cross-family review;
• an end-to-end case linking a change, finding, correction and verification result.

Reviewer agreement wouldn’t settle it. Agreement isn’t the same as correctness, and model families may share training patterns. Both model names and consensus need to be tested against adjudicated outcomes.

That was the part we had to be careful about. The process created separation, but the evidence didn’t yet tell us what that separation was worth.

Comparing the two approaches

We’d use one eligible code change as the experimental unit. It could be reviewed under both defined conditions or assigned to one condition through a recorded allocation procedure.

Repeated observations need careful treatment. If the same repository or model contributes several changes or review runs, those results aren’t independent.

Both conditions should use the same:

• code change, or matched change criteria;
• business requirements;
• repository state and context;
• reviewer instructions;
• tool permissions;
• deterministic checks;
• time and execution limits;
• finding format;
• adjudication criteria.

Where practical, model identity should be removed from finding records before adjudication. Reviewers also shouldn’t see findings from the other condition before submitting their own.

The records would need enough evidence to audit each outcome:

Severity levels and consequential-defect criteria should be defined before collection begins. Style preferences and other non-blocking observations should remain separate from correctness defects.

How large should the sample be? That depends on the comparison being tested, the outcome rate of interest and the required precision. The source material didn’t contain enough information to calculate it.

The analysis should also account for clustering where repositories, changes or models contribute repeated results. Comparative rates should include confidence intervals. If multiple model runs are used, the protocol should say how run-to-run variation is recorded and whether the results are combined or reported separately.

A primary comparison should be agreed in advance. False positives, escaped defects, disagreements and overrides can then stay as separate measures rather than being folded into one success rate.

For now, we can say that cross-family review is an operational control. It separates implementation from review and may expose different assumptions. Whether that difference leads to better defect detection is still open, and only a controlled comparison with reproducible conditions and auditable outcomes will answer it.