Why First-Pass Rate Does Not Measure End-to-End Delivery

Gate success and cohort delivery measure different populations. They answer different questions. A change can pass its first measured gate without reaching delivery.

Code volume appears near the start of delivery. It does not show whether a change survives review, passes later checks, meets its business intent or reaches a usable state.

A high first-pass rate measures gate performance, not end-to-end delivery.

The denominators differ

The supplied measurement draft reports:

• First-pass rate: [reported]
• Delivered share of started work: [reported]

These measures use different denominators:

First-pass rate
= changes passing their first recorded gate attempt
  / changes with a recorded first gate attempt

Delivered share
= changes reaching delivery
  / changes in the defined started cohort

Started work that has not reached the gate is excluded from the first calculation but included in the second. Work can also pass the gate and then remain pending, enter repair, be superseded or be abandoned.

The gate boundary matters. Moving it from implementation to review, or from review to production readiness, changes the meaning of the result. A reported first-pass rate applies only to its defined gate. It is not a proxy for delivery.

Measurement boundaries must remain explicit:

• Use gate attempts to measure first-pass performance.
• Use a defined started cohort to measure delivery.
• Separate pending work from terminal non-delivery.
• Record transitions between the two boundaries.

The reported results are not yet auditable

The source measurement draft and raw counts are unavailable. No citation or link has been supplied. The metric definitions, cohort rules, observation window, collection methods and source systems are also missing.

The figures therefore remain reported observations. They cannot yet support comparisons across teams, periods or work classes.

Other reported measures have the same limitation:

Publication of the source should include a raw-count appendix covering:

• Numerators and denominators for every percentage and zero value
• The exact gate definition
• Definitions of started work, gate attempts, repair cycles and delivery acceptance
• Cohort inclusion rules and dates
• The observation window
• The age distribution of unfinished work
• Finding-level dispositions and linked revisions
• Collection methods and source systems

Until these materials are available, the results should not be treated as a substantive case.

Delivery needs a validated lifecycle model

End-to-end measurement requires consistent states and transition events. The following table is a proposed baseline, not a confirmed representation of the measured workflow.

In this baseline, delivered, superseded and abandoned are terminal states. Pending is not a separate terminal state. It describes work with no later transition at the observation cut-off.

The model must be checked against the actual workflow. Validation must determine how it represents:

• Rollback after delivery
• Rejection after approval
• Partial deployment
• Reopened work
• Work that leaves the tracked workflow

If these events occur, the model needs corresponding transitions. Otherwise, it will hide outcomes or assign them to the wrong state.

Each transition should record a persistent change identifier, timestamp, responsible stage and applicable work or risk class. Each review finding should link to its disposition and any revision that addresses it.

Pending work changes the result

A recent cohort may contain substantial unfinished work. Counting every unfinished change as non-delivery reduces the delivered share without showing whether those changes will later reach delivery.

Reporting should distinguish between:

• Delivered work
• Superseded work
• Abandoned work
• Work still pending at the cut-off

The following example is hypothetical. It does not represent the reported data.

A cohort contains 100 started changes. Its status changes between two observation dates:

At the earlier cut-off:

Delivered share = 40 / 100 = 40%
Pending share   = 45 / 100 = 45%

At the later cut-off:

Delivered share = 58 / 100 = 58%
Pending share   = 17 / 100 = 17%

The cohort size does not change, but its observed outcome does. Some pending work reaches delivery. Other work becomes abandoned or superseded. A delivered-share figure therefore needs both a cohort definition and an observation date.

The method must also define cohort maturity. It may use a documented maturity threshold or a documented time-to-delivery treatment. The available material provides neither, so no method can yet be selected or assessed.

Blocking findings expose a recording gap

The supplied draft reports that automated reviewers raised all recorded findings and that every recorded finding was categorised as a correctness issue. It also reports that a blocking share was reported, while the second-pass and repair-cycle shares were also reported.

These observations do not prove that repair occurred. They raise a narrower question: what happened after each blocking finding?

A finding record should show whether the affected change:

• Was corrected before the measured pass
• Continued without a recorded repair cycle
• Remained pending
• Was superseded or abandoned
• Left the tracked workflow
• Reached delivery

Without these links, the data cannot show whether findings create repair work or how much engineering capacity that work consumes.

Keep the measures separate

Complete transition records support several distinct measures:

These measures describe gate quality, correction activity, workflow progress and final outcomes. Combining them produces an ambiguous result.

Measurement priorities

The immediate task is to make the existing claims auditable:

1. Publish or cite the source measurement draft.
2. Add a raw-count appendix for every reported result.
3. Define the measured gate and each lifecycle state.
4. Publish the cohort rules, dates and observation window.
5. Define the treatment of immature and unfinished work.
6. Validate the proposed state model against the actual workflow.
7. Record transitions against persistent change identifiers.
8. Link findings to dispositions, revisions and final states.
9. Compare teams or periods only after definitions and collection methods remain stable.

What this means

First-pass rate describes performance at one gate. End-to-end delivery requires a defined cohort, explicit outcomes and transition records from the start of work to the observation cut-off.