What a first-pass rate leaves out

We noticed that the reported first-pass rate and delivered share seemed to describe the same flow of work.

They don’t.

A change can pass its first recorded gate and still wait for later checks, enter repair, be superseded, be abandoned or fail to meet the definition of delivered. Code volume has the same problem. It appears near the start of delivery, but it doesn’t tell us whether a change survives review, meets its business intent or reaches a usable state.

A high first-pass rate measures gate performance, not end-to-end delivery.

The supplied material also reports zero second passes, zero repair cycles and a reported share of started work reaching delivery. All three figures may be correct within their own boundaries. But without the underlying records, we can’t reconcile them or treat them as substantive evidence.

The denominators tell different stories

The distinction starts with the populations being measured.

First-pass rate
= changes passing their first recorded gate attempt
  / changes with a recorded first gate attempt

Delivered share
= changes reaching delivery
  / changes in the defined started cohort

Started work that hasn’t reached the gate is excluded from the first calculation but included in the second. Work can also pass the gate and then remain pending, enter repair, be superseded or be abandoned.

A small synthetic example shows what happens. These records are illustrative; they don’t come from the reported data.

That gives us four different measures:

first-pass rate = 3 ÷ 4 = 75%
delivered share = 2 ÷ 4 = 50%
recorded second-attempt share = 1 ÷ 4 = 25%
repair cycles per started change = 1 ÷ 4 = 0.25

The first-pass result is valid for the specified gate. It tells us nothing about whether changes B and D were delivered.

Where the gate sits matters too. Move it from implementation to review, or from review to production readiness, and the result takes on a different meaning even if the engineering work hasn’t changed. So a useful gate definition has to say:

• which artefact is assessed;
• what starts an attempt;
• which checks apply;
• what records a pass or failure;
• how corrections made before the result are treated;
• how reopened or resubmitted work is treated.

Delivery needs the same precision. We’d want an observable event, rather than a state reconstructed later from an incomplete workflow.

That left us with four boundaries to keep explicit:

• Use gate attempts to measure first-pass performance.
• Use a defined started cohort to measure delivery.
• Separate pending work from terminal non-delivery.
• Record transitions between the two boundaries.

We can’t audit the claims yet

The original report, measurement draft, dashboard, dataset and raw counts haven’t been supplied. There’s no citation or link either, so the figures can’t be checked independently.

What’s missing is substantial:

• Raw numerators and denominators, including those behind zero values
• Metric definitions
• Cohort dates, inclusion rules and observation windows
• Exclusion and ageing rules
• Collection methods and source systems
• Definitions of started work, attempt, pass, repair cycle and delivered
• The position of review findings relative to the measured gate
• Breakdowns by risk class, repository, change type or work age
• Repository counts, readiness results and aggregation rules
• A policy linking risk classes to planning, review and approval controls
• A recorded end-to-end change trace
• Cost-attribution rules and comparable cost periods

These are the reported claims under examination:

For now, they remain reported observations rather than verified results. We can’t use them to compare teams, periods or work classes.

The material also lists model-routing assignments. Those describe configuration, not measured performance. They provide no evidence about assurance outcomes.

A source publication would need a raw-count appendix containing:

• Numerators and denominators for every percentage and zero value
• The exact gate definition
• Definitions of started work, gate attempts, repair cycles and delivery acceptance
• Cohort inclusion rules and dates
• The observation window
• The age distribution of unfinished work
• Finding-level dispositions and linked revisions
• Collection methods and source systems

Unfinished work is where this gets particularly easy to misread. A change that’s still open when the observation window closes isn’t necessarily a failed delivery. We need an explicit ageing rule before making that judgement.

Consider another hypothetical cohort. Again, this isn’t the reported data.

At the earlier cut-off:

Delivered share = 40 / 100 = 40%
Pending share   = 45 / 100 = 45%

At the later cut-off:

Delivered share = 58 / 100 = 58%
Pending share   = 17 / 100 = 17%

The cohort hasn’t changed size, but its observed outcome has. Some pending work reached delivery. Other work became abandoned or superseded.

So which delivered-share figure is the right one?

Neither stands alone. Each needs a cohort definition and an observation date. The method also has to define cohort maturity, either through a documented maturity threshold or a documented time-to-delivery treatment. The supplied material gives us neither, so we can’t yet select or assess a method.

Following the work through repair

Once we separated gate performance from delivery, the next need was consistent lifecycle states and transition events.

The model below is a proposed baseline. It isn’t a confirmed description of the measured workflow.

Delivered, superseded and abandoned are terminal states in this baseline. Pending isn’t another terminal state. It simply describes work with no later transition at the observation cut-off.

We’d still need to check the model against the actual workflow, including how it represents:

• Rollback after delivery
• Rejection after approval
• Partial deployment
• Reopened work
• Work that leaves the tracked workflow

If these events occur, the model needs matching transitions. Otherwise, outcomes will be hidden or assigned to the wrong state.

A common event record would make attempts, repairs and delivery states queryable:

A minimum event set could include:

• change_started
• gate_attempted
• gate_passed
• gate_failed
• repair_started
• repair_completed
• change_superseded
• change_abandoned
• change_delivered

Each transition should carry a persistent change identifier, a timestamp, the responsible stage and the applicable work or risk class. Each review finding should link to its disposition and any revision that addresses it.

Corrections made before gate_attempted wouldn’t count as post-attempt repair. Corrections after gate_failed would appear as repair events followed by a later attempt number.

That detail matters. The material reports that automated reviewers raised all recorded findings, that every recorded finding was categorised as a correctness issue, and that a reported share was blocking. It doesn’t show when those findings occurred or what happened next.

For every blocking finding, we’d need to know whether the affected change:

• was corrected before the measured pass;
• continued without a recorded repair cycle;
• remained pending;
• was superseded or abandoned;
• left the tracked workflow;
• reached delivery.

This is why zero recorded repair cycles doesn’t show that no correction occurred. Corrections may have happened before the recorded gate, appeared as continuations rather than repair cycles, or moved outside the tracked workflow.

Without links between findings, revisions and outcomes, the data can’t tell us whether findings created repair work or how much engineering capacity that work consumed.

Two changes can reach the same recorded outcome while consuming very different amounts of engineering work. That difference affects capacity and forecasting.

With complete event records, we can keep the measures separate:

In formula form:

first_pass_rate =
  changes passing the specified gate on attempt 1
  ÷ changes with recorded activity at that gate

delivered_share =
  started changes with a delivered event
  ÷ all started changes in the cohort

recorded_second_attempt_share =
  changes with attempt_number = 2 at the specified gate
  ÷ changes with recorded activity at that gate

repair_cycles_per_started_change =
  completed repair cycles
  ÷ all started changes in the cohort

A recorded pass is evidence that the change reached the gate. We don’t need a separate gate_attempted event when gate_passed already exists.

A full reference query belongs in an implementation appendix or companion repository. The core gate logic is:

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS reached_gate,

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type = 'gate_passed'
     AND attempt_number = 1
    THEN 1 ELSE 0
END) AS passed_first_attempt,

MAX(CASE
    WHEN gate_name = :gate_name
     AND attempt_number = 2
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS recorded_second_attempt

This detects a recorded second attempt. It doesn’t tell us that the attempt was required unless the query also identifies a preceding failure.

Risk, readiness and cost don’t share a denominator either

The material reports that high-risk work makes up [reported] of the mix and critical financial work makes up [reported]. We can’t verify either figure without the source data.

Higher-risk work warrants stronger planning, challenge and review than low-risk implementation. If we combine risk bands in a single throughput comparison, we may end up favouring easier work over the safe delivery of valuable work.

Approved plans reportedly cover [reported] of changes. Is that good coverage? We can’t tell without a policy stating which risk classes require approval. The useful measure is coverage against the applicable policy, not coverage on its own.

Repository readiness has a similar aggregation problem. The material reports an overall pass rate of [reported], along with complete pass rates for:

• Runtime pinning
• Lockfiles
• Observed pipeline checks
• One-command checks
• Testing guidance
• Absence of committed secrets

Agent tools and default-branch protection each reportedly pass at [reported].

We can’t interpret the aggregate without repository counts, repository-level results and the weighting method. Reporting should show every control by repository, then state whether the total gives equal weight to repositories, checks or observations.

Cost needs to stay tied to an accepted outcome as well. Attributable cost should include:

• Agent use
• Human planning and review
• Delivery-pipeline activity
• Verification
• Repair effort

cost per delivered change =
  attributable cohort cost
  ÷ delivered changes in the cohort

Started work that doesn’t reach delivery should remain visible unless an explicit attribution rule excludes it.

The supplied material can’t support a directional cost claim. A comparison would need consistent cost attribution, delivery definitions, observation windows and risk bands.

Before a later report presents first-pass performance as delivery efficiency, it should:

1. Publish or cite the underlying source.
2. Add raw numerators and denominators for every reported result.
3. Define the measured gate, delivery event and lifecycle states.
4. Publish cohort rules, dates, exclusions and the observation window.
5. Define ageing rules and the treatment of immature work.
6. Check the proposed state model against the actual workflow.
7. Record lifecycle events against persistent change identifiers.
8. Link findings to dispositions, revisions and final states.
9. Publish the risk policy for required planning, review and approval.
10. Show repository-level readiness results and aggregation rules.
11. Use comparable cost periods and attribution rules.
12. Include at least one anonymised change with a complete recorded trace.
13. Compare teams or periods only after definitions and collection methods remain stable.

No owners or delivery dates were supplied, so we can’t assign any.

First-pass rate still has a useful job: it tells us how work performed at a defined gate. The work left open is connecting that gate to the rest of the lifecycle, without losing pending work, correction effort or terminal outcomes somewhere between the two.