Twelve Months Into Our AI Shift: How Our Team's Productivity Metrics Actually Moved
A year after bringing AI into our daily workflow, the question we asked ourselves was not whether AI helped. The harder question was this: how do you know it is real, and not just a feeling? This is an account of how one software development team actually measured change across twelve months, which metrics we trusted, and which ones we had to abandon. It is not a performance report dressed up with shiny figures. We deliberately avoid stating specific internal numbers as absolute fact, because most of them depend heavily on the circumstances of each project. Instead, we are honest about what we measured, how we measured it, and how far the direction of movement can be trusted.
What metrics we actually tracked after twelve months
After a year, we focused on four groups of metrics: the time from request to delivery, the rate of rework after delivery, the share of work that passed internal review on the first submission, and how consistent quality was across team members. These reflect real value rather than surface-level activity.
The first thing worth saying is that how we chose metrics changed even more than the numbers themselves. In the early months, we measured what was easy to count. But the longer we measured, the clearer it became that something easy to count is not necessarily something worth trusting. Any team can generate enormous activity while producing little value, and AI makes this more dangerous, because it makes producing large volume almost effortless.
So we returned to a simple principle: measure only what customers and the team actually feel. Customers feel waiting time. They feel it when something is delivered but then needs fixing over and over. They feel consistency, meaning whether this delivery holds the same quality as the last one. All four of our metric groups revolve around those felt experiences, not around how busy the team looked.
We also drew a clear line between leading and lagging indicators. A lagging indicator tells you where a project ended up, but it arrives too late to act on. A leading indicator is an early signal, such as how many issues were caught during internal review before anything reached the customer. What the past year taught us is that investing in leading indicators is far more worthwhile, because it gives you the chance to fix things while there is still time.
Why we dropped simple metrics like lines of code and hours logged
We dropped metrics like lines of code, commit counts, and total hours worked because they measure effort spent, not value created. In the age of AI, one person can produce thousands of lines in minutes, so these numbers no longer speak to capability or outcome, and they can even push you toward the wrong decisions.
For a long time, in many places, hours logged was nearly synonymous with contribution. A developer who sat long and typed a lot was seen as hardworking. AI broke that link. Once a tool can write the first draft of a feature in minutes, a person's value moves away from typing and toward defining the right requirements, judging quality correctly, and deciding what to keep.
If we had kept rewarding lines of code, we would have encouraged exactly the wrong behavior. Someone could let AI generate a flood of code, push the number up, and still produce something bloated, hard to maintain, and full of hidden defects. Beautiful number, bad result. This is the trap many teams fall into during the first year of transition.
Instead, we shifted to measuring outcomes. Does a feature work the way the customer expected? Did it need fixing after delivery? Did it break something elsewhere in the system? These questions are harder to measure and take more effort to record, but they are far more honest.
Another lesson was the seduction of numbers that are easy to present. Lines of code slide neatly into a clean report. Rework rate is harder to talk about, because it admits that something was not done right the first time. But that very discomfort is a sign of a valuable metric. A metric that exists only to be shown off improves nothing.
Lead time, rework rate, first-pass review rate: which way they moved
In the direction we observed, lead time for producing a first draft fell noticeably, rework rate gradually declined as our review process tightened, and the first-pass review rate improved slowly but steadily. We describe direction rather than publishing absolute figures, because the magnitude depends heavily on the type of project.
Let us be specific about each metric. Lead time, the span from receiving a request to delivering, is where AI's effect is most visible. The first-draft stage, which used to take days, became considerably shorter. What is interesting, though, is that total lead time did not shrink at the same rate. Faster code generation pushed more pressure onto the review and refinement stages. In other words, AI moved the bottleneck rather than removing it. If you look only at writing speed, it is easy to draw the wrong conclusion about the real improvement.
Rework rate is the metric we care about most, because it reveals true quality. In the early period, when the team trusted AI output too readily, the rework rate actually rose at times. This is an uncomfortable but important truth. AI writing fast does not mean writing correctly, and a draft that looks complete can hide subtle flaws. Only after we built the habit of reviewing more carefully did the rework rate begin to fall in a sustainable direction.
First-pass review rate is our favorite leading indicator. It answers the question: does a piece of work meet the standard on its first submission, or does it need several rounds of revision? This metric improves much more slowly than lead time, because it reflects genuine capability rather than tool speed. It depends on whether a team member truly understands the standard, checks their own work before submitting, and knows when to trust AI and when not to. The slow but steady improvement in this metric is, to us, the most reliable sign that the team is genuinely maturing rather than simply moving faster because of a tool.
When metrics lie: the traps of measurement and the misconceptions we fell into
Metrics lie when we measure the wrong thing, observe over too short a window, or let the number define success instead of reflecting it. Our biggest trap was believing short-term improvements in speed while overlooking the hidden quality costs that surfaced only months later.
The first lesson about this trap is the problem of the observation window. In the first few weeks, every speed metric looked wonderful. Had we stopped and declared victory then, we would have told a false story. Only months later, when rushed work began demanding maintenance and bug fixes, did the real cost appear. A metric becomes trustworthy only when it is observed long enough to contain both the later consequences and the immediate benefits.
The second lesson concerns Goodhart's law, the idea that when a measure becomes a target, it stops being a good measure. Once we began tracking a metric openly, the team's behavior shifted to make that metric look good, sometimes in unhealthy ways. This is why we deliberately avoid wiring these metrics directly and mechanically into individual evaluation. Metrics are for understanding the system, not for ranking people in a simplistic way.
The third lesson, and perhaps the most important, is the confusion between activity and progress. AI creates a very strong sense of busyness. The screen is always full of new code, everything is constantly moving. But motion is not forward progress. There were stretches when the team was extremely busy while the value delivered to customers did not rise to match. Only by clearly separating doing a lot from finishing the right thing could we see the true picture.
Finally, we learned to be honest with ourselves about what cannot be measured. Some of the most important values, such as clarity of thinking or the ability to ask the right question, are nearly impossible to reduce to a number. The mistake is pretending everything is measurable. Measurement is a powerful tool, but it has to travel alongside humility about its own limits.
The operational decisions we made from twelve months of data, and what we will measure next
From a year of data, we made three decisions: invest more in the review stage than in code generation, train people before scaling tools, and measure by project rather than by individual. What we plan to measure next is long-term quality after delivery, not just speed at the moment of handoff.
The first decision came directly from watching the bottleneck move. Once code-generation speed was no longer the problem, investing more in writing faster yielded almost nothing. The value lives in the review stage, where a person decides what is good enough to ship. So we moved resources, attention, and training toward it. This is a large shift in thinking: in a team that uses AI well, the person who evaluates matters as much as the person who creates.
The second decision reinforced something we had always believed: tools do not create productivity, people do. The data showed that the biggest gap in results was not between those who used the tools and those who did not, but between those trained to use them well and those who used them carelessly. So we put training ahead of scaling. Scaling a process that is not yet understood only multiplies the mess.
The third decision is to measure by project rather than by individual wherever possible. A project is a whole outcome the customer receives, and it is harder to game than individual metrics. This approach also encourages collaboration instead of unhealthy internal competition.
Looking ahead, our measurement plan tilts firmly toward the long term. We want to track how a deliverable lives six months or a year after handoff: how much maintenance it needs, whether it is easy to extend, whether it is quietly accumulating technical debt. These are the hardest metrics to capture and the slowest to arrive, but they are the most honest about real quality. What the past year taught us is that the right question is not how fast AI made us, but whether we are building something more durable and more trustworthy. That is the metric we will keep chasing.
