No change without a way to prove it safe.
This is the hardest capability in the product and the one that makes every other capability defensible. It is built first, not last.
- Precondition
- Code cannot be modified until tests pin what it currently does
- Coverage
- Measured against live paths. Coverage of dead code is worthless
- Merge rule
- An unexplained difference blocks. This is not advisory
Legacy systems have no tests. So the tests are captured, not written.
Before any code is modified, the platform records real inputs and outputs at service boundaries in production, passes them through redaction, and generates tests that pin the current behaviour. Including the rare branches that runtime evidence proves are still live.
This is deliberately shippable on its own. Generating characterization tests for untested legacy code changes nothing about how the system behaves, which makes it the easiest thing in this product to get approved, and the prerequisite for everything that follows.
Grouped by cause, so a human can actually rule on it.
Captured production transactions run through both the old and the new implementation, and results are compared. Two hundred and one failing cases is noise. One rounding rule is a decision somebody can make.
| Cause | Count | Example | Ruling |
|---|---|---|---|
| Rounding on half-cent | 201 | 1284.125 to .12 against .13 | Accepted New behaviour matches the filed rate manual |
| Credit ordering | 21 | Credit applied before the surcharge rather than after | Accepted Signed off against RATE-0042 |
| New York pre-2014 rate | 2 | 0.0725 against 0.0680 | Merge blocked Unexplained, and it stays blocked |
Every difference must be explicitly ruled on by a named person. Accepting one records who accepted it and against which rule. Two unexplained cases out of two hundred and twenty-four is exactly the shape of the defect that reaches production eighteen months later.
Running it is its own problem.
Executed on platform infrastructure and parallelised, so suite length stops being the reason people skip it.
Supported through the same pre-built image catalogue the cloud workspace uses.
Regression checked against the budget declared in the task envelope, not against a general threshold.
The new implementation runs against live traffic without serving it, as a pre-cutover mode.
Unexplained differences permitted to reach a merge. Target and gate, not an aspiration.
Assumptions outstanding at cutover. Every override has to be closed or accepted by name.
Production defects that trace to a wrong or missing assertion. The metric that actually matters.
Verification gates everything, so it gets built first.
The tempting order is to build the agent, then the board, then verification once there is something to verify. That order produces a tool that writes confident changes with no way to tell whether they are correct, which is the third failure mode with better marketing. Test generation ships as a standalone product before any agent writes a line of code.