OMS
- 1Find related rules
- 2Update the guidance
- 3Publish instructions
Finds related rules, checks what needs to change, then publishes updated instructions.
Measurement
Lessons that stick.
A correction you give an AI agent should help the next person’s agent too. Making that happen takes more than an instruction file. Teams need a way to submit corrections, record them, update the guidance and make it available to agents on other machines. OMS provides those mechanics, and also uses AI to turn corrections into useful shared guidance.
In our controlled test, we compared OMS with an AI that rewrites instruction files, using 640 corrections across 160 rules.
Agents using OMS guidance followed the rule on the first try in 85% of tasks, compared with 38% for AI-updated files, as scored by an AI judge.
01 / Following the rule
Same tasks. Different guidance.
640 corrections across 160 rules, including changed advice and repeats of old rules. Then the same tasks, scored by an AI judge.
View results and comparisons
Both task groups combined · AI-judged · Higher is better
Less correcting in the test
60%
fewer reminders
About 29 reminders per 100 tasks with OMS, down from 72 with AI-updated files.
Sent by the test’s automated rule checks, separately from the AI judge. Staff time was not measured.
Our controlled test: 160 rules, 640 corrections, 40 instruction files. 21 September 2026.
These scores measure following the tested rule, not the whole task.
Both approaches use AI.
The difference is how they update the instructions.
AI-edited files
- 1Receive a correction
- 2Read the whole file
- 3Return the edited file
Reads and rewrites a file for each correction, without a memory of earlier corrections. It is told to remove old advice, merge repeats and keep unrelated guidance.
02 / Sharing the lesson
Help the next agent learn it too.
In the earlier handoff test, reported on 10 September, we corrected one agent. OMS updated the guidance, and the test gave it to another agent in a fresh session.
View the handoff results
Teach one agent
Submit a correctionOMS updates the guidance
Process and publishAnother agent reads it
Start a fresh session
76%
Next agent followed the rule
The lesson helped in a new session. Receiving an instruction does not mean an agent will always follow it.
99/99
Checks found the rule’s marker
A marker is a word or phrase we looked for. In 6 checks, it was already there. Finding it does not prove the whole rule arrived correctly.
How did shared files compare?
Shared files worked well when kept up to date. OMS handled the tested handoff automatically. We tried 11 recording rates for files. These were test settings, not measured habits of teams.
Separate handoff test. Selected rates; rounded figures.
OMS
- Marker found
- 100%
- Rule followed
- 76%
Files: every correction recorded
- Marker found
- 98%
- Rule followed
- 81%
Files: 90% recording rate
- Marker found
- 88%
- Rule followed
- 70%
Files: 50% recording rate
- Marker found
- 48%
- Rule followed
- 41%
Files: none recorded
- Marker found
- 0%
- Rule followed
- 1%
03 / The evidence
Look inside the tests
In the larger test, agents using OMS guidance followed more rules on the first try and needed fewer reminders. Explore the comparisons, method and limits below.
View the evidence and method
Did the agent follow the rule on the first try?
After 640 corrections across 160 rules, agents used the finished instructions to answer the same tasks. An AI judge checked whether each first answer followed the current rule.
“Familiar wording” uses the rule’s original terms. “Reworded tasks” asks agents to apply the rule in a differently worded task or situation.
| How corrections were handled | Familiar wording | Reworded tasks | Both combined |
|---|---|---|---|
| OMSFinds related rules, reconciles changes and publishes updated instructions. | 86% | 84% | 85% |
| AI-updated instruction filesAn AI reads and rewrites a file for each correction, without a memory of earlier corrections. | 40% | 36% | 38% |
Model-judged first answers · Higher is better · Rounded percentages
OMS
Finds related rules, reconciles changes and publishes updated instructions.
- Familiar wording
- 86%
- Reworded tasks
- 84%
- Both combined
- 85%
AI-updated instruction files
An AI reads and rewrites a file for each correction, without a memory of earlier corrections.
- Familiar wording
- 40%
- Reworded tasks
- 36%
- Both combined
- 38%
“Both combined” adds the results from both groups. These are the 85% and 38% scores in the chart above.
Keeping every correction as written was the strongest alternative: 62% (198 of 320). OMS led all five approaches. These scores measure the tested rule, not the quality of the whole answer.
Additional test comparisons
These checks help explain the result. They are not three equivalent products a team would buy.
| How corrections were handled | Familiar wording | Reworded tasks | Both combined |
|---|---|---|---|
| Unedited correction historyKeeps every correction as written, including conflicting instructions.Tests whether keeping everything is enough. | 61.3%98 of 160 | 62.5%100 of 160 | 61.9%198 of 320 |
| OMS with existing-rule lookup disabledProcesses corrections without finding related guidance already stored in OMS.Tests the contribution of finding earlier rules. This is a test setting, not a separate product. | 18.8%30 of 160 | 23.8%38 of 160 | 21.3%68 of 320 |
| Original files, never updatedUses the starting skill documents without adding any corrections.Checks what happens when the guidance is left unchanged. | 8.1%13 of 160 | 5.6%9 of 160 | 6.9%22 of 320 |
Supporting comparisons · Model-judged first answers · Counts shown below percentages
Unedited correction history
Keeps every correction as written, including conflicting instructions.
Tests whether keeping everything is enough.
- Familiar wording
- 61.3%98 of 160
- Reworded tasks
- 62.5%100 of 160
- Both combined
- 61.9%198 of 320
OMS with existing-rule lookup disabled
Processes corrections without finding related guidance already stored in OMS.
Tests the contribution of finding earlier rules. This is a test setting, not a separate product.
- Familiar wording
- 18.8%30 of 160
- Reworded tasks
- 23.8%38 of 160
- Both combined
- 21.3%68 of 320
Original files, never updated
Uses the starting skill documents without adding any corrections.
Checks what happens when the guidance is left unchanged.
- Familiar wording
- 8.1%13 of 160
- Reworded tasks
- 5.6%9 of 160
- Both combined
- 6.9%22 of 320
How the answers were judged
Scored by an AI judge: three votes per answer, majority decision. Unclear answers are not counted as passes.
The judge checked the rule in force, including any change to earlier advice. Four OMS answers were unclear; they remain in the total of 320. The judge is also a model, so its decisions can be wrong.
Exact counts and test dates
| How corrections were handled | Familiar wording | Reworded tasks | Both combined |
|---|---|---|---|
| OMSFinds related rules, reconciles changes and publishes updated instructions. | 86.3%138 of 160 | 83.8%134 of 160 | 85.0%272 of 320 |
| AI-updated instruction filesAn AI reads and rewrites a file for each correction, without a memory of earlier corrections. | 39.6%63 of 159 | 35.6%57 of 160 | 37.6%120 of 319 |
Main task comparison · Passing first answers out of measured first answers
OMS
Finds related rules, reconciles changes and publishes updated instructions.
- Familiar wording
- 86.3%138 of 160
- Reworded tasks
- 83.8%134 of 160
- Both combined
- 85.0%272 of 320
AI-updated instruction files
An AI reads and rewrites a file for each correction, without a memory of earlier corrections.
- Familiar wording
- 39.6%63 of 159
- Reworded tasks
- 35.6%57 of 160
- Both combined
- 37.6%120 of 319
OMS was run on 21 September 2026. The four alternatives were run on 19 September on the same tasks. Only OMS was rebuilt and rerun; the alternatives’ saved answers were reused.
Each group had 160 planned tasks per approach. One AI-updated-file first answer was lost to timeouts, leaving 159 familiar-wording answers and 319 overall. OMS had all 320.
How we ran and scored the tests
1Teach the rules
Start with 40 instruction files. Add 640 corrections about 160 rules, including reversals and repeats of withdrawn advice.
2Give each approach the same tasks
Some tasks look like the original correction. Others put the rule in a new situation. Each task starts in a fresh session.
3Judge the first answer
Ask an AI judge whether the answer follows the current rule. Use three votes and take the majority. Unclear answers do not count as passes.
Five approaches, 320 planned task sessions each. The chart combines familiar wording and reworded tasks. The separate automated replay sent 0.291 reminders per measured task for OMS and 0.724 for AI-updated files (rounded), with up to two retries. Those reminders were triggered by automated rule checks, not the model judge.
Read the test methodWhat these results do and do not show
- What made the difference?
OMS keeps track of which rules still apply. Agents using its guidance followed the rule first time in 85% of tasks, versus 21% with existing-rule lookup switched off. OMS also held 61 of 71 repeats of withdrawn rules for review.
The test supplied OMS with the correct skill hint. It does not measure routing or isolate a routing-free maintenance advantage. Review decisions were scripted: reject withdrawn repeats, reinstate held reversals and prefer the newer rule in conflicts.
- First tries and retries differ
The automated replay sent about 60% fewer reminders per task with OMS than with AI-updated files: 29 versus 72 per 100 tasks.
Retries were triggered by the replay’s automated rule checks, separately from the model judge used for the first-try scores. The test restated rules perfectly and immediately; no staff time was measured. Even after retries, 28 OMS tasks remained unresolved.
- Ahead of all four alternatives
OMS scored 85%. The strongest alternative, keeping every correction as written, scored 62%. Rewriting files with AI scored 38%.
This is one synthetic 160-rule test, not a customer trial. OMS ran on 21 September; the alternatives ran on 19 September on the same tasks. Model behaviour can vary between runs.
- Our tests, not a customer trial
A model judge checked whether the first answer followed the current rule. It gave three votes per answer; the majority decided. Four OMS answers were unclear and counted as non-passes.
The reading model was Claude Sonnet. Model judges can make mistakes. OMS documents were built in eight parallel groups; only a two-skill check, not the full test, was also built serially. These results are not independent validation or a promise for every setup.
Read the results and method.
Explore the test questions, corrections, scoring criteria and prompt templates, with a guide to the testing methodology.
Download the test materials (ZIP)ZIP · 111 KB · 7 CSV files, methodology guide and checksums
Download the results and methodMarkdown · Model-judge results, test method and limits
Make your own benefit estimate
Separate from the test results: choose how many repeat corrections you think your team could avoid. This is an estimate of staff time, not measured savings, net return or an OMS price.
Estimate the value for your teamIllustrative, not measured
250
2
5 min
GBP 450
You are estimating how many repeat corrections OMS would avoid. It is your assumption, not a measured OMS result. Set it to zero to see a no-benefit case.
Potential hours freed per year
2,000
Value of that staff time
GBP 120,000
Assumptions
- 48 working weeks a year and a 7.5-hour working day.
- Staff time is valued at the loaded day rate you set above.
- Software, implementation and operating costs are excluded.
- This estimates the value of time released. It is not OMS pricing, a cash saving or a net return.
Discuss the results against your own guidance
See what OMS would manage for your team and agree how to evaluate its value.
Book an OMS demo