Once AI raises research output by an order of magnitude, the new problem is trust: how do these conclusions earn confidence? Line-by-line human checking doesn't scale, and 'just trust the AI' is worse.
Make verification a structure, not an attitude: important conclusions get rebuilt from zero by a second Agent that didn't touch the development, working from documents alone; live numbers get computed by two Agents on different data channels and cross-checked; and a scheduled backstop scan watches specifically for things that should have been produced and weren't.
Never reads the original code; rewrites everything from the reproducible spec alone — any mismatched number is a finding.
Maintains the production engine, fixes what reproduction uncovers, and regression-verifies the fixes.
When the two sides disagree, rules which is wrong and which document governs; conclusions go into the ledger.
Checks on schedule that the day's required artifacts exist and timestamps are fresh; missing items alert and convert to tasks.
Rules for verification:
@replication rebuilds the full backtest with zero context and catches a price-adjustment definition bug in the first round.
After the fix, three independent implementations align daily NAV to five decimal places before the definition freezes for production.
On the first live day, two Agents compute NAV independently — one from settlement files, one from position detail — reconciling item by item to the cent.
The post-close backstop checks required artifacts automatically; it once caught a hidden gap where the job ran but the NAV never rolled forward — fixed the same day.
Incident post-mortems go into the self-check list and alert rules the same day; from then on, machines watch for that class of problem.
Scheduled checks on the day's artifact completeness and freshness, alerting on gaps.
A third-party Agent periodically reproduces a core result with zero context.
Every definition ruling's basis and conclusion, archived and traceable.
Extend reproduction from key milestones to a spot-check institution: one core output randomly recomputed each quarter.
Build a provenance chain for external numbers: each one labeled with its frozen definition and production date.
Tier the alerts: missing artifacts are a red line, definition drift a yellow line, with handling windows spelled out.