CodeTrain / Rework report

The free rework report

The rework report is what codetrain metrics prints for one git repository. It shows a team how much of its code is rewritten soon after it lands, how long pull requests wait for a real review, how often checks fail and how often a deployment is a fix. The figures are per repository: never one blended number, nothing that grades anyone.

Run it in one command

One command, no login, no account, and nothing uploaded by default. It reads the local git history. When gh is installed and signed in, it also reads pull requests, checks and deployments from GitHub through your own gh login.

$ codetrain metrics

The five figures and their limits

The command prints four figures. The fifth, rework per $1,000 of AI spend, shows on a team's delivery card instead. Each figure comes with its limits, spelled out below, so you know where it stops being true. A figure that could not be measured shows as not measured, never as zero.

Short-horizon rework

Short-horizon rework is the share of the lines a team wrote in a week that are gone 14 to 21 days later. Each week of commits is checked 14 days after the week ends, so every line is 14 to 21 days old when it is checked. The window ends 14 days before the day the command runs. A line moved to another place is not rework, and a whitespace-only edit keeps its lines. Generated, vendored and documentation files are left out, and a team can add its own exclusions in .codetrain/metrics.json.

This figure counts lines that are gone, not why they went, so a planned rewrite reads the same as a fix. Moved lines are credited to both the old and the new place, so a move can hide a little real rework, and a block moved in a later commit can read as rework. Work on a branch that has not merged yet is not counted. A shallow clone is not measured.

Review wait

Review wait is the time from a pull request being ready, opened or last marked ready for review, to its first substantive review. A substantive review is by a person who is not the author, never a bot: a request for changes, or a review or comment with an inline comment or at least 20 characters of real text. It prints the median and the 90th percentile in hours. It counts pull requests merged with an approval that said nothing, and with no review at all, so an empty approval cannot make the wait look better.

A median needs at least 5 reviewed pull requests; below that only the counts show. A pull request with more than 15 reviews or comments where no substantive one was found among those read is counted as unknown. At most the 200 most recent pull requests in the window are read. With no gh, or with --no-gh, review wait is not measured. On a team's delivery card it shows as a range of hours, because medians cannot be added across uploads.

Validation failures

Validation failures is the share of pull requests with a check run in which a check failed on any of their last 20 commits, so a failure fixed by the next push still counts. When GitHub says which checks are required, only required checks count; otherwise every check counts and the report says "all checks". Failed means failed, errored or timed out, and a cancelled or skipped check is not a failure. Pull requests opened by a bot are left out.

This figure shows that a check failed, not why. A flaky test reads the same as a real fault.

Post-deployment rework

Post-deployment rework is the share of deployments that were a fix: a deployment that shipped a revert or a hotfix, judged by its commit subject and patterns a team can change, or the merge of a pull request labelled bug, hotfix or incident. Deployments are read from the first of these that has any: GitHub Deployments to production, GitHub Releases, the repository's version tags, then a local deploy history file.

A fix that is not named a revert or hotfix and carries none of those labels is not counted. A team that records its deployments in none of those four places is not measured. When the commits of a deployment cannot be read, as in a shallow clone, the share is unknown and the counts stay.

Rework per $1,000 of AI spend

Rework per $1,000 of AI spend is reworked lines set against the monthly AI coding tool spend that a team owner or admin types in, scaled to the upload's own period. It shows on the team's delivery card, not in the command's output. Every member sees the ratio; only the owner and admins see the amount.

CodeTrain cannot see a team's AI tool bills, so the figure is only as good as the number typed in.

Bringing the numbers into CodeTrain

codetrain metrics --upload sends one repository's numbers to a team. It needs a login, and the person must be a team owner or admin. It runs once a day per repository; a second upload the same day is refused. It cannot be combined with --me.

$ codetrain metrics --upload

An upload sends counts only: lines introduced and reworked, review waits as a histogram, pull request counts, failed check counts, deployment counts, distinct authors, the period, an optional label the lead types, and a key made from the team and the repository's first commit. No code, file names, commit messages, names or email addresses. A repository with fewer than 3 distinct authors is held back from the team's card. The team's delivery card shows the figures per repository from these counts, with no team total and no ordering by result.