# OMS benchmark: results and method Updated 21 September 2026. Compliance scores in this report use the model judge. They measure following a specific rule on the first answer, not overall task quality. ## Headline result Agents using OMS guidance followed the current rule in **272 of 320 tasks (85%)**. AI-updated instruction files scored **120 of 319 (37.6%)**. ## Exact first-answer counts | How corrections were handled | Familiar wording | Reworded tasks | Both combined | |---|---:|---:|---:| | OMS | 138 / 160 | 134 / 160 | 272 / 320 (85.0%) | | AI-updated instruction files | 63 / 159 | 57 / 160 | 120 / 319 (37.6%) | | Every correction kept as written | 98 / 160 | 100 / 160 | 198 / 320 (61.9%) | | OMS with existing-rule lookup disabled | 30 / 160 | 38 / 160 | 68 / 320 (21.3%) | | Original files, never updated | 13 / 160 | 9 / 160 | 22 / 320 (6.9%) | Combined percentages use the sum of passing answers divided by the sum of measured answers, not an average of rounded percentages. One familiar-wording answer from the AI-updated-file approach was lost to timeouts, leaving 319 answers for that approach. OMS had all 320. ## What was tested The test started with 40 existing skill documents and 160 synthetic rules. Each approach received the same stream of 640 corrections (seed 11), including restatements and reversals. In 71 cases, a withdrawn instruction was repeated after its reversal. This tests whether guidance keeps the rule that applies now. - **OMS:** finds related rules, reconciles changes, revises the document and lists current decisions at the top. Repeats of withdrawn rules are held for review. - **AI-updated files:** a model rewrites a document for each correction without a memory of earlier corrections. - **Every correction kept:** each correction is appended as it arrives. - **OMS without lookup:** retrieval is disabled, so corrections become new rules without corroboration, replacement or review holds. - **Original files:** the starting documents receive no corrections. Each rule had two tasks: one using familiar wording and one describing a situation where the rule applies. Each task started a fresh Claude Sonnet session with the approach's document. The documents were already built: this measures agents using published guidance, not a live OMS lookup. ## Scoring An AI judge checked whether each first answer followed the rule in force. It produced three votes per answer, with the majority deciding. UNCLEAR answers remain in the denominator but do not count as passes. Four OMS answers were unclear. Both the reading model and judge were labelled Sonnet; the experiment is not evidence across multiple agent platforms or models. This report uses only model-judge compliance scores. Statistical significance figures from the source report's other scoring method are not applied to these scores. The judge can be wrong, and rule compliance is not a measure of the whole answer's quality. ## Run dates and review handling OMS was built and replayed on 21 September. The four alternatives were built and answered on 19 September; their saved answers were reused on the same tasks. This is not a fresh simultaneous run of all five approaches. OMS was built in eight parallel groups of five skills. Checks were scoped to each skill. The distiller's view of existing team rules covered five skills rather than forty. A two-skill serial/parallel check produced the same documents; a full serial comparison was not run. The benchmark supplied OMS with the correct skill hint. Routing is not measured here, and these results do not establish a routing-free maintenance advantage. Review handling was scripted: reject repeats of withdrawn rules, reinstate held reversals and settle conflicts for the newer rule. A real reviewer may make different decisions. OMS retired the replaced rule at compile time in 143 of 160 reversals and held 61 of 71 repeats of withdrawn rules for review. These are processing counts, not extra task scores. ## Automated reminders: a separate measurement The replay harness sent 93 reminders over 320 OMS tasks and 231 over 319 AI-updated-file tasks: 0.291 versus 0.724 reminders per task, rounded. That is about **60% fewer reminders per task**, calculated as `1 - (93 / 320) / (231 / 319)`. These reminders were triggered by the replay's automated rule checks, not by the model judge used for the first-answer scores above. The harness restated the rule perfectly and immediately, allowing up to two retries. No person sent the reminders and no staff time was measured. After retries, 28 OMS tasks remained unresolved by the replay's checks. Fewer reminders does not mean every task was completed. The website's benefit calculator is an illustration using the visitor's assumptions, not measured savings or a verified return on investment. ## Limits and separate experiments This is one controlled synthetic corpus, not a customer trial or independent validation. Some corpus rules overlap; some questions do not force the intended value. Model answers and model judgements vary. The 19 and 21 September run dates add possible day-to-day model variation. The earlier handoff experiment reported on 10 September remains separate. Its 76% rule-following result and 99/99 marker checks have not been rerun or combined with these dense-skill scores. Earlier smaller tests likewise do not describe this version's performance. ## Provenance Summary supplied in `2026-09-21-dense-results.md`. Exact model-judge counts were aggregated from `judge-replay-dense-seed11-improved-r5.json` for OMS and `judge-replay-dense-seed11-improved-r3.json` for the four alternatives. Only YES judgements count as passes. Familiar/reworded groups correspond to the source's near/far tiers. Retry counts were checked against `replay-dense-seed11-improved-r5-merged.json`. This download contains public aggregate results and method, not raw agent answers, private logs or a replacement set of test inputs.