What an AI agent has to prove before I call it done
A clean compile answers one question. It does not tell me whether the right screen opened, the interaction worked, or the browser ran the new build.
That distinction matters when agents can produce implementation and a convincing explanation in the same session. I need a way to evaluate the result independently of the explanation.
For my game, I have built that around targeted runtime checks. The project has harnesses for gameplay, UI, abilities, procedural generation, effects, and other systems. Those tools are part of the development interface: an agent can use them to observe behavior and return evidence I can inspect.
Here is how I expect that loop to work.
Start with the behavior that would prove the fix
For a bug, the preferred sequence is to reproduce it in a test, observe the test fail, change the implementation, and observe it pass. The failing run matters. Without it, I may have a test that never exercised the problem.
Suppose a confirmation button looks enabled but does nothing. A source-text check that finds the button label is not useful evidence. Neither is a screenshot showing that the button exists. The check needs to reach the relevant screen, trigger the control, and observe the expected state change.
That example sounds obvious. It is also the kind of distinction an agent can miss when the task is framed as “add a test” instead of “demonstrate this behavior.”
I remain responsible for the scenario and the assertion. A test generated from the same misunderstanding as the implementation can agree with it perfectly.
Check the build that is actually running
The development loop uses a browser target for quick feedback, while Windows is the release target. For compiled game changes, I use the full build pipeline and verify that the local server is serving the intended output.
This catches an easy source of false confidence: testing an older bundle. It also prevents a shortcut compile from bypassing framework setup that the runtime expects.
Once the build identity is established, the harness must reach the intended state. A process that exits without errors after opening the wrong scene has not tested the change. The route, relevant configuration, readiness signal, and resulting action all belong in the evidence.
Ask for an evidence packet
A useful completion report is short and specific:
1
2
3
4
5
6
7
Change: What behavior changed?
Build: Which output did the runtime load?
Scenario: Which route and deterministic inputs were used?
Check: What action ran, and what observable result passed?
Errors: Were there new console or network failures?
Limits: What was not exercised?
Cleanup: Were test-owned pages and processes closed?
I do not need a wall of logs pasted into every conversation. I need the conclusion and a path to the supporting output when I want to investigate.
This also improves handoffs. Another session can continue from a known state instead of repeating setup or accepting “tested” as an unexplained fact.
Match the check to the change
More tests are useful when they add information. Repeating a passing check without changing its inputs may only add time.
A tool-only change gets the affected tool interaction. A game change gets a build and a relevant runtime smoke. A shared contract or progression change may need wider coverage because its failure can travel. Changes involving native behavior need exercise on the actual target; browser success cannot establish Windows parity.
Visual and gameplay judgment remain separate. A harness can tell me that an effect triggered, stayed within a limit, and cleaned itself up. I still need to decide whether it communicates the right thing. An automated player finishing a sequence does not establish that a human will enjoy it.
This is why I resist both extremes: accepting a compile as proof of everything, and treating an exhaustive sweep as the default price of every edit.
Make the checks sustainable
Browsers, local servers, traces, and test processes consume real resources. If a harness leaves them behind, repeated testing can make the workstation slower and the evidence less reliable. Cleanup is part of the test contract.
So is choosing a cheap route back to the behavior. If checking a small change requires manually navigating five screens every time, that is a tooling problem worth considering. The easier it is to inspect the result, the more often I can exercise judgment while the change is still small.
Simon Willison’s practical guide to coding with LLMs makes testing a responsibility of the person using the tool. My version of that principle is to make the proof part of the assignment.
When an agent returns, I want to know what changed in the running system. That gives me something more useful than confidence: a result I can check.