A coding assistant can make the first draft arrive faster and still make the finished change more expensive. The missing work appears in review, correction, defect repair and the time needed to understand unfamiliar code. If a team measures only typing time, suggestions accepted or lines changed, that cost stays invisible.

This issue asks a practical question: how can an engineering team determine whether an AI coding assistant saves net time and money on its own work?

The reader outcome is a small, reproducible audit. It follows comparable tasks from start through a seven-day follow-up, rejects incomplete records, and reports total labor minutes and cost per accepted task. It does not produce a universal score for a product. It helps a team decide whether to expand use, restrict it to particular task families or pause a pilot until the evidence improves.

This issue was researched, written, technically checked and edited by separate AI roles at the Publish Haven Editorial Desk. Owner Primus Vekuh authorized the publishing workflow but is not represented as having performed the technical review. There is no sponsor or affiliate placement in this issue.

Why reported productivity gains conflict

Use is widespread enough to make the measurement problem consequential. The 2025 Stack Overflow Developer Survey received more than 49,000 responses overall. On the AI-use item, 84 percent of 33,662 respondents said they use or plan to use AI tools. On the accuracy item, 46 percent of 33,244 respondents distrusted AI output and 33 percent trusted it. Sixty-six percent named solutions that are almost right as their biggest frustration. This is a large self-selected survey about reported use and attitudes, not a causal estimate of productivity.

A May 11, 2026 METR survey of 349 technical workers found median self-reported value gains between 1.4 and 2 times, while the median self-reported speed gain was 3 times. METR explains why speed can exceed value: cheaper tasks can become attractive without becoming equally important. The sample was recruited through sources including GitHub, academic directories, METR networks and X, and response rates for emailed participants were about 2 percent. The result records participant beliefs under that sampling process. It does not establish the gain a different team will receive.

Controlled studies also answer different questions. A study published online in Management Science on February 27, 2026 combined three randomized company experiments with 4,867 developers at Microsoft, Accenture and an anonymous Fortune 100 company. The authors report a 26.08 percent increase in completed tasks, with a standard error of 10.3 percent, when developers had access to an AI coding tool. Each company experiment was noisy, and task completion is not the same outcome as downstream defects, review burden or retained skill. Several authors and data contributors were associated with Microsoft, GitHub or participating companies, which is relevant when interpreting the evidence.

METR's early-2025 randomized study reached a different result for experienced open-source developers working in repositories they knew: tasks took 19 percent longer with AI. METR now labels that finding out of date. Its February 24, 2026 follow-up found raw estimates that pointed toward speedups, but confidence intervals included no effect. METR says recruitment, lower compensation, withheld tasks and measuring parallel agent work made the new estimates hard to interpret. The inspected early-study repository was pinned at commit 7ff2d767; the follow-up repository was inspected at commit 55f00721. We inspected their documentation and reported results. We did not rerun those third-party research environments.

Skill formation is another boundary. Anthropic's January 29, 2026 research release reports a randomized trial that assigned 52 Python developers who had never used the Trio library to learn with or without an assistant. The AI group finished about two minutes faster, a difference that was not statistically significant, but scored 50 percent on an immediate mastery quiz compared with 67 percent for the control group. The source reports Cohen's d of 0.738 and p=0.01. The paper's current v2 was revised February 1. The small sample, one unfamiliar library and immediate quiz do not establish a long-term skill effect for all developers. Anthropic produced the study, so it is an originating lab result rather than independent reproduction.

The evidence is not a contest in which one percentage must win. The studies vary in task, tool, experience, repository familiarity, outcome and observation window. Their conflict is the reason to measure the decision locally.

What this audit counts

The unit is a predefined engineering task, not a prompt, completion or commit. Before work begins, record its task family, acceptance gate and follow-up window. A task is accepted only when the same normal checks used for production work pass. These may include tests, review, security checks and product acceptance.

For each task, count four kinds of labor:

  1. Active work: time spent authoring or directing the assistant.
  2. Review: human time spent understanding, testing and reviewing the result.
  3. Seven-day rework: correction after the first proposed completion.
  4. Defect remediation: time spent on an escaped defect attributable under the team's normal incident process.

Add assistant fees and infrastructure expense that can be assigned to the task. Convert labor to cost with a loaded hourly rate that includes the employer's chosen labor boundary. Report that boundary instead of pretending one rate is universal.

Also record a short comprehension check when the task involves learning or unfamiliar code. It should ask the developer to explain the changed behavior, identify a planted fault or predict a boundary case without the assistant. This is not a validated psychological instrument. It is a local warning signal that review capacity may be eroding.

We do not use lines of code, commits or suggestion-acceptance rate as the primary outcome. Those measures describe activity. The ICSE 2026 paper “Beyond the Commit”, based on 2,989 survey responses and 11 interviews at BNY Mellon, argues for a broader view that includes self-sufficiency, cognitive load, task completion, peer review, expertise and ownership. This is an industry case study, not an independent experiment across companies.

A bounded search on October 16 found current guides from Vercel, OpenNash and DevStats. They already advise teams to consider acceptance, review, rework or delivery quality. This issue adds a complete offline audit that refuses to score unfinished follow-up and exposes the difference between faster active work and cheaper accepted work. That is a practical addition, not a claim that the topic was previously uncovered.

Prerequisites and cost boundary

Use Python 3.10 or later, a text editor and a fresh directory. The verified run used Python 3.12.14 on Linux. The program uses only the standard library and synthetic data. It needs no repository, model, network connection, account or credential.

The sample has four tasks in one family and one experience band. It is intentionally too small for a purchasing decision. Dollar values are invented. The allocation alternates between control and AI rather than using random assignment, so the program labels its result descriptive only. Do not place confidential prompts, source code or personnel assessments in the worksheet.

Save this complete listing as value_audit.py:

"""Synthetic AI coding-assistant value audit. No repository or service access."""

import csv
import io
import math
import statistics


SAMPLE = """task_id,task_family,experience,mode,active_min,review_min,rework_7d_min,defect_min,accepted,tool_cost_usd,loaded_rate_usd,comprehension_pct,followup_complete
C1,api-change,senior,CONTROL,90,20,10,0,yes,0,60,90,yes
C2,api-change,senior,CONTROL,100,20,0,0,yes,0,60,85,yes
A1,api-change,senior,AI,55,35,30,20,yes,3,60,70,yes
A2,api-change,senior,AI,50,40,45,30,no,3,60,60,yes
"""


NUMBERS = (
    "active_min", "review_min", "rework_7d_min", "defect_min",
    "tool_cost_usd", "loaded_rate_usd", "comprehension_pct",
)


def load_rows(text):
    rows = list(csv.DictReader(io.StringIO(text)))
    required = {
        "task_id", "task_family", "experience", "mode", *NUMBERS,
        "accepted", "followup_complete",
    }
    if not rows or set(rows[0]) != required:
        raise ValueError("columns do not match the audit schema")
    for row in rows:
        for field in ("task_id", "task_family", "experience"):
            row[field] = row[field].strip()
            if not row[field]:
                raise ValueError(f"{field} cannot be blank")
    if len({row["task_id"] for row in rows}) != len(rows):
        raise ValueError("task_id values must be unique")
    strata = {(row["task_family"], row["experience"]) for row in rows}
    if len(strata) != 1:
        raise ValueError("compare one task_family and experience stratum per run")
    for row in rows:
        if row["mode"] not in {"AI", "CONTROL"}:
            raise ValueError("mode must be AI or CONTROL")
        if row["accepted"] not in {"yes", "no"}:
            raise ValueError("accepted must be yes or no")
        if row["followup_complete"] != "yes":
            raise ValueError("every task needs a complete 7-day follow-up")
        for field in NUMBERS:
            row[field] = float(row[field])
            if not math.isfinite(row[field]):
                raise ValueError(f"{field} must be finite")
            if row[field] < 0:
                raise ValueError(f"{field} cannot be negative")
        if row["loaded_rate_usd"] == 0:
            raise ValueError("loaded_rate_usd must be positive")
        if row["comprehension_pct"] > 100:
            raise ValueError("comprehension_pct cannot exceed 100")
    if {row["mode"] for row in rows} != {"AI", "CONTROL"}:
        raise ValueError("both AI and CONTROL tasks are required")
    return rows


def summarize(rows, mode):
    group = [row for row in rows if row["mode"] == mode]
    accepted = sum(row["accepted"] == "yes" for row in group)
    minutes = sum(
        row["active_min"] + row["review_min"] + row["rework_7d_min"]
        + row["defect_min"] for row in group
    )
    cost = sum(
        (row["active_min"] + row["review_min"] + row["rework_7d_min"]
         + row["defect_min"]) * row["loaded_rate_usd"] / 60
        + row["tool_cost_usd"] for row in group
    )
    return {
        "tasks": len(group),
        "accepted": accepted,
        "acceptance": 100 * accepted / len(group),
        "median_active": statistics.median(row["active_min"] for row in group),
        "followup": sum(row["rework_7d_min"] + row["defect_min"] for row in group),
        "minutes_per_accepted": minutes / accepted if accepted else None,
        "cost_per_accepted": cost / accepted if accepted else None,
        "comprehension": statistics.median(row["comprehension_pct"] for row in group),
        "minutes": minutes,
    }


def compare_costs(control, ai):
    if not control["accepted"] or not ai["accepted"]:
        return "NOT_COMPARABLE", "NOT_COMPARABLE_NO_ACCEPTED_OUTCOMES"
    delta = ai["cost_per_accepted"] - control["cost_per_accepted"]
    observation = (
        "AI_HIGHER" if delta > 0
        else "AI_LOWER" if delta < 0
        else "EQUAL"
    )
    return f"${delta:+.2f}", observation


def main():
    rows = load_rows(SAMPLE)
    control = summarize(rows, "CONTROL")
    ai = summarize(rows, "AI")

    print("design=alternating allocation; causal_status=DESCRIPTIVE_ONLY")
    print(f"tasks={len(rows)} complete_7d={sum(r['followup_complete'] == 'yes' for r in rows)}")
    for label, result in (("CONTROL", control), ("AI", ai)):
        minutes_display = (
            "NOT_AVAILABLE" if result["minutes_per_accepted"] is None
            else f"{result['minutes_per_accepted']:.1f}min"
        )
        cost_display = (
            "NOT_AVAILABLE" if result["cost_per_accepted"] is None
            else f"${result['cost_per_accepted']:.2f}"
        )
        print(
            f"{label} tasks={result['tasks']} accepted={result['accepted']} "
            f"acceptance={result['acceptance']:.1f}% "
            f"median_active={result['median_active']:.1f}min "
            f"followup={result['followup']:.1f}min "
            f"minutes_per_accepted={minutes_display} "
            f"cost_per_accepted={cost_display} "
            f"median_comprehension={result['comprehension']:.1f}%"
        )
    print(f"raw_active_delta={ai['median_active'] - control['median_active']:+.1f}min")
    delta, comparison = compare_costs(control, ai)
    print(f"net_cost_per_accepted_delta={delta}")
    print(f"cost_per_accepted_observation={comparison}")

    broken = SAMPLE.replace("A2,api-change,senior,AI,50,40,45,30,no,3,60,60,yes",
                            "A2,api-change,senior,AI,50,40,45,30,no,3,60,60,no")
    try:
        load_rows(broken)
    except ValueError as error:
        print(f"negative_case={error}")

    nonfinite = SAMPLE.replace("A2,api-change,senior,AI,50,40,45,30,no,3,60,60,yes",
                               "A2,api-change,senior,AI,nan,40,45,30,no,3,60,60,yes")
    try:
        load_rows(nonfinite)
    except ValueError as error:
        print(f"nonfinite_case={error}")


if __name__ == "__main__":
    main()

Run python3 -I value_audit.py. The -I flag isolates Python from user site packages and Python environment variables. It is not an operating-system sandbox.

Expected complete stdout is:

design=alternating allocation; causal_status=DESCRIPTIVE_ONLY
tasks=4 complete_7d=4
CONTROL tasks=2 accepted=2 acceptance=100.0% median_active=95.0min followup=10.0min minutes_per_accepted=120.0min cost_per_accepted=$120.00 median_comprehension=87.5%
AI tasks=2 accepted=1 acceptance=50.0% median_active=52.5min followup=125.0min minutes_per_accepted=305.0min cost_per_accepted=$311.00 median_comprehension=65.0%
raw_active_delta=-42.5min
net_cost_per_accepted_delta=$+191.00
cost_per_accepted_observation=AI_HIGHER
negative_case=every task needs a complete 7-day follow-up
nonfinite_case=active_min must be finite

Read the result correctly

In the synthetic sample, median active time falls by 42.5 minutes with AI. That is a real calculation over the supplied rows. It does not survive as a net benefit. One of two AI tasks fails its acceptance gate, and AI tasks accumulate 125 minutes of rework and defect remediation. Because rejected-task labor remains in the numerator, total labor becomes 305 minutes per accepted AI task, compared with 120 minutes for control. Labor plus tool cost per accepted AI task becomes $311, compared with $120 for the control tasks.

The program then changes one follow-up marker from yes to no. Validation rejects the dataset. Without this rule, a recent AI task could appear cheap merely because its repair window has not finished.

The conclusion is deliberately narrow: for these four invented records, expand is not supported. The data do not show that every assistant is unproductive. They also do not show causation because assignment was not randomized and the sample is tiny.

A numbered method for a real pilot

  1. Choose one task family. Start with comparable work such as small API changes, test creation or routine migrations. Do not mix unfamiliar architecture work with repetitive edits and call the average meaningful.
  2. Write acceptance before allocation. Freeze the tests, review requirements and follow-up window. If the gate changes after seeing results, record the change and restart the comparison.
  3. Choose an allocation rule. Random assignment can support a causal estimate when it is ethical and operationally feasible. Alternation or matched pairs are easier but remain vulnerable to order, learning and selection effects. Never randomize high-risk emergency work merely to improve the study.
  4. Record the whole path. Capture active work, review, rework, defects, tool cost and acceptance. Preserve medians and distributions when the sample grows. Do not hide failed tasks from the denominator.
  5. Protect people and code. Use task identifiers instead of personnel rankings. Follow repository, client and data rules. A measurement program is not permission to send restricted code to an external model.
  6. Segment before deciding. Compare task family, repository familiarity and experience band. Expand only where an acceptance floor, cost ceiling and quality boundary are met.
  7. Recheck later. Tools and user skill change. A result from one product version, policy and team should not become a permanent universal rule.

Evidence ledger and limits

Item What was established What remains unverified
Stack Overflow 2025 survey Widespread reported use plus accuracy concerns in a large self-selected sample Causal productivity and this publication's reader demand
Three company experiments Reported increase in completed tasks under randomized access Universal effect, downstream defects, review and learning
METR studies Slowdown in an early setting; later estimates weakened by selection and measurement problems Current effect for a typical private team
Anthropic skill trial Lower immediate quiz scores in one small randomized learning task Long-term skill retention and effects across tools or seniority
This issue's audit Exact calculations and validation behavior for four synthetic rows Product ROI, team causality and production outcomes

No study reviewed here independently reproduces a universal productivity effect. No customer outcome, revenue claim or product endorsement is made. The local audit can reduce ambiguity, but expert experimental design may still be needed for a consequential purchasing or workforce decision.

Failure cases and troubleshooting

If the script reports a schema error, compare the CSV header exactly. Spreadsheet exports sometimes rename columns or insert an index column. If it reports incomplete follow-up, wait for the defined window rather than changing no to yes. If minutes or cost per accepted task is NOT_AVAILABLE, that group had no accepted outcomes. Report that fact and investigate; do not replace it with zero.

Duplicate task IDs can count one job twice. Negative minutes or costs indicate an input or export error. A comprehension score over 100 is invalid. The script also requires a positive loaded labor rate because zero would remove human effort from cost.

If two groups use different task families, repositories or experience bands, do not interpret their delta as a tool effect. Either collect matched work or report the segments separately. If reviewers know which tool produced a change, their behavior may differ. Record that limitation instead of calling the study blinded.

Completion check and independent exercise

You have completed the lab when your stdout matches byte for byte, including the final newline, and you can explain why the negative active-time delta coexists with a positive cost delta. You should also be able to identify the one validation rule that prevents an unfinished repair window from looking favorable.

For an independent exercise, add four test-creation tasks with different loaded rates. Write the acceptance gate before choosing the numbers. Then compare that family separately from api-change. Introduce one duplicate ID and confirm that validation stops before a result is printed. Explain whether your allocation supports a causal claim.

The next investigation should run a preregistered shadow pilot with enough comparable tasks to show distributions rather than four-row averages. Retain tool and model versions, assignment rule, task exclusions and every rejected outcome. Test whether comprehension and review burden change after four weeks. The decision is not “Does AI code fast?” It is “Under which tasks and controls does this team deliver accepted, durable work at lower total cost?”

Sources and competing coverage were checked October 16, 2026. The AI reviewers' agreement is not independent human peer review, financial advice or a professional software audit.