2,161 commits later, here is what I measure
Between the repository endpoints covering February 15 through November 2, the All Is Vanity history contains 2,161 commits. Comparing those endpoints shows changes across 4,283 files. The period after July 15 accounts for 711 of those commits.
That is a substantial amount of activity around a project I work on with AI agents. It includes product code, tools, tests, documentation, experiments, and generated material. It also includes merges and multiple author identities. I am describing the history of the repository I maintain, not claiming that I personally typed every change.
The numbers are useful context. They are a poor definition of success.
The repository can answer some questions
Git gives me a record of changes and decisions. I can see the tooling expand, trace an architectural choice, or identify when a new validation path appeared. I can compare a fixed pair of endpoints and reproduce the result.
It cannot tell me how many pull requests were reviewed without a separate source for PR data. A directory full of tests does not tell me how many passed. A large line count does not distinguish a valuable system from generated output or expensive rework.
Even the commit count depends on the query. Traversing every local branch can include work outside the history of the endpoint I intend to describe. A credible number needs a scope as well as a value.
I have included the snapshot and counting method so the terms are explicit. There is no defensible conversion from these counts to a claim that the whole project became ten times more productive.
The meaningful change is what I can inspect
The more interesting history is the development of the feedback loop.
Early work added a browser editor toolchain and automated runtime routes. Subsequent work added metrics, visual captures, shared schemas, and targeted checks. In September, I tightened the operating rules around focused smoke validation and reusing passing evidence while the relevant inputs stayed unchanged.
Those changes gave me more ways to examine the product and more precise tasks to give agents. They support iteration. They do not automatically prove that every resulting feature is good.
The next question is how reliably this system turns a selected problem into an accepted result. That is where I would put the measurement effort for an engineering organization.
A scorecard I would use
I would start with a small set of measures tied to delivery:
| Measure | Working definition | What it helps reveal |
|---|---|---|
| Time to accepted result | Task selection to a result that meets agreed acceptance criteria | Whether the complete workflow is getting faster |
| Review waiting time | Time a change waits for an accountable reviewer | Whether generation is outrunning review |
| Rework | Changes reopened or substantially redone after acceptance | Whether apparent speed creates downstream work |
| Escaped defects | Relevant defects found after release, by severity | Whether faster delivery preserves reliability |
| Verification effort | Human and machine effort needed to establish acceptance | Whether tooling makes confidence cheaper |
| Customer or product outcome | The behavior or business result the work was intended to improve | Whether the organization chose useful work |
These are proposed measures, not results from an instrumented study of my project or Samsara. Their definitions would need to be consistent before comparing periods or teams.
I would compare similar types of work and retain the cost of review, testing, and rework. Timing only the implementation step can make a workflow look faster while pushing its cost onto someone else.
Feeling faster is not enough
METR’s early-2025 randomized study is a useful challenge to intuition. In that study, 16 experienced open-source developers handled 246 tasks in repositories they knew. Tasks with AI access took 19% longer on average, even though participants believed the tools had helped them work faster.
That finding concerns a specific set of developers, tasks, and early-2025 tools. It is not a verdict on my workflow or every later agent system. It is a reason to measure the outcome rather than rely on the feeling of acceleration.
The 2025 DORA research approaches a different scale. Its survey-based findings describe AI as amplifying the organization around it, with increased delivery throughput alongside continuing instability. That is an association from organizational research, not a controlled estimate of what one tool will do for one team.
Both perspectives make me more interested in the surrounding system: ownership, platforms, feedback, and the ability to absorb change.
What I want an AI investment to buy
My ambition is to increase the useful capacity of the teams and systems I lead. At Samsara, that means thinking across devices, firmware, testing, automation, data, and backend services. In my independent work, it means learning quickly enough to make a better product with the time I have.
Those settings have different constraints. A successful game-development experiment does not establish a fleet-wide productivity result. It can still produce a practice worth testing in another setting.
The question I would put in front of an executive team is specific: can we deliver a chosen class of valuable changes sooner, with acceptable cost and reliability, and sustain the improvement?
Commits help me investigate how the work happened. The answer to that question tells me whether the investment worked.
See the complete AI engineering series for the practices behind these numbers.