# OMS benchmark test materials

This pack accompanies the OMS benchmark results reported on 10 September 2026. It contains the questions, corrections, scoring criteria and test procedure so readers can examine what was tested.

The tests ask whether agents retain and follow a team's supplied guidance as it changes, and whether a correction becomes available to the next agent. They measure compliance with specific instructions, rather than the general quality of an agent's work.

## Start here

Open **[01-test-cases.csv](01-test-cases.csv)**. Each row contains one rule, its corrections, two questions and a link to the original source. It includes every case in the two test sets, not a selection of favourable examples.

For a concrete example, find `r-competitive-dimensions`. The starting guidance calls for a positioning map with two dimensions. The first correction changes this to three. Two other messages restate that instruction, and a later change calls for four. The questions ask how to draw the map, once using the original terminology and once describing the situation to a board. The final scoring check looks for four dimensions or axes.

This example illustrates the test design, not a recommendation to use four-dimensional positioning maps. Some changes intentionally depart from usual practice. Otherwise an agent could answer from its existing knowledge without retaining the correction.

## Files in this pack

| File | Rows | Contents |
|---|---:|---|
| [01-test-cases.csv](01-test-cases.csv) | 193 | All 160 larger-test rules and 33 smaller-test rules. Original source excerpt, four correction messages, both questions and source link for each rule. |
| [02-correction-sequence.csv](02-correction-sequence.csv) | 772 | The order of the 640 larger-test corrections and 132 smaller-test corrections. Includes the rule considered current after each message. |
| [03-question-scoring.csv](03-question-scoring.csv) | 386 | Exact text-matching criteria for each question, the intended final rule, and the corrective message used if a replay attempt failed. |
| [04-agent-handoff.csv](04-agent-handoff.csv) | 33 | Agent A's task, the correction submitted, agent B's different task and the marker used to check arrival. |
| [05-starting-documents.csv](05-starting-documents.csv) | 40 | Public sources, pinned versions, recorded licences and content checksums for the larger test's starting skill documents. |
| [06-prompt-templates.csv](06-prompt-templates.csv) | 5 | The question and retry wrappers, plus the file-selection and file-editing instructions given to the comparison model. |
| [07-handoff-recording-schedule.csv](07-handoff-recording-schedule.csv) | 99 | For each of three seeds and 33 cases, whether the text-file alternative was scheduled to record the correction at each recording probability. These are input decisions, not measured arrivals. |

The CSVs use UTF-8 with a byte-order mark for spreadsheet compatibility. Import them as comma-separated text, enable text wrapping and filter by `corpus` or `rule_id`. Embedded commas, quotes and line breaks are quoted as part of a cell. No workbook software or code is needed to read the materials.

For safe spreadsheet opening, 45 cells whose text starts with `=`, `+`, `-` or `@` have one protective apostrophe added. Remove that first apostrophe to recover the original text when inspecting a pattern outside a spreadsheet. There are no other wording changes to the exported questions, corrections or pattern specifications.

## What was compared

Both OMS and the main file-editing alternative use AI. The difference is how they manage the guidance.

| Approach | Procedure and reason for comparison |
|---|---|
| OMS | Processes corrections as structured rules, finds related guidance, reconciles changes and publishes instructions. |
| AI editing instruction files | A practical alternative using a model to maintain ordinary documents. In the larger test, the model selects one file from a list of names and descriptions, reads the whole selected file and returns an updated version. It is explicitly told to preserve unrelated guidance, merge duplicates and remove superseded advice. In the smaller test it maintains one shared file. |
| Every correction kept verbatim | Preserves the complete correction history without editing. This tests whether keeping everything is sufficient, even when instructions conflict. |
| OMS with retrieval disabled | Disables finding existing guidance during processing, to examine the contribution of that part of OMS. This changes the processing path; it is not a separate commercial product. |
| No supplied guidance | Establishes what the agent can do without being taught the rules. |

Ordinary files need surrounding mechanics to collect corrections and make guidance accessible to other agents. The document tests supply the input stream and hand the resulting guidance to the answering model. They do not measure the work of building that distribution system. The handoff experiment separately tests ingestion, processing, publication and use in a fresh session.

## How the tests ran

### 1. Maintaining guidance as instructions change

The larger test started with 40 existing skill documents and covered 160 rules. A skill document is a set of instructions for a subject or task. Each rule had four messages: an initial correction, two restatements and a later change, for 640 messages in total. Corrections were processed sequentially. The smaller test used 33 rules and 132 messages, building guidance from those messages rather than starting with the 40 documents.

The correction order was randomised using seed 11 for the larger test and seed 7 for the smaller test. A seed fixes the input ordering. It does not make the model's responses deterministic. The later change always follows its initial correction, but an old restatement can occur after the change. The benchmark treats that restatement as stale and keeps the later change in force. Such rows are explicitly marked in the sequence CSV. This is an assumption of the test, not a claim that every ambiguous real conversation should be interpreted this way.

After processing the stream, each approach was tested with two questions per rule: 320 questions in the larger set and 66 in the smaller set. One uses familiar wording; the other puts the instruction in a differently worded task or situation. Some questions ask explicitly for the quantity being tested. The split is not proof of broad reasoning ability or transfer to any task.

In the larger test, the answering model received the approach's version of the relevant skill document, not all 40 files. The smaller test supplied the approach's complete shared guidance. Question wording and grading criteria were the same across approaches.

There was a deliberate difference in filing information: OMS and verbatim storage received the correct skill-file hint, while the file-editing model selected its own file. This means the larger comparison includes both filing and maintenance quality. A separate analysis excluding filing errors had only 64 matched cases and was inconclusive. The headline result does not isolate reconciliation or graph structure as the sole cause of improvement.

### 2. Following the finished guidance, with retries

The replay experiments used the same questions and the finished guidance from the corresponding larger or smaller test. OMS did not run during replay. Each task started fresh, with one approach's guidance and the question.

The first answer was scored. If it failed the rule check, an automated message stated the final rule and asked for the work again. At most two retries were allowed, making three attempts in total. Each retry included the guidance, original task, previous answer and corrective message. The loop stopped after a passing answer or the attempt limit.

The larger replay scheduled 320 tasks for each of five approaches, or 1,600 task sessions. Two first attempts were lost. The smaller replay scheduled 66 tasks per approach, or 330 task sessions. Retries are additional attempts within a task, not additional unique questions.

The primary measure was following the tested rule on the first attempt. Corrective-message counts came from this automated loop. No person was timed, and the messages assume the failure is detected and the correct rule can be stated accurately.

### 3. Passing a correction to the next agent

The handoff test used the 33 cases in `04-agent-handoff.csv`. Agent A attempted the familiar question without guidance. The test then submitted the listed correction. Agent B, in a fresh session, received the resulting guidance and attempted the different question.

Each case teaches one instruction once. It does not replay the four-message correction history. The taught instruction is the final version because that is what the shared question graders check. The initial, unaided agent A answers were cached and reused across conditions. Agent B's answer was the measured behavioural outcome.

Guidance accumulated between cases within each run. Three seeds, 11, 23 and 41, were used. The text-file alternative was tested at recording probabilities from 0% to 100% in steps of 10%. A random decision determined whether each correction was submitted to its file editor. A 50% probability does not guarantee exactly half the 33 corrections will be recorded. The schedule CSV exposes those decisions. OMS submitted every correction and had no manual-recording probability.

The test used OMS's ingestion, compiler and publication services, with scripted handling of review decisions. Agent B was then supplied the published guidance by the test system. It was not a field trial of installation and synchronisation across employees' laptops or different agent products.

Arrival was checked by looking for a rule-specific marker word or phrase in agent B's guidance, ignoring letter case. The marker was present in all 99 OMS observations. In six observations it was already present before the current correction, so those arrivals cannot be attributed unambiguously to that correction. The separate behaviour test measured whether agent B followed the rule and averaged about 76%. Marker presence does not by itself establish that the correct instruction arrived intact.

## How answers were scored

These 386 questions use deterministic text-pattern checks, not an AI judge. Each row in `03-question-scoring.csv` gives the exact regular expression and whether it must be present or absent. Some rows also require a second pattern as evidence that the answer addresses the task. All patterns use Python's `MULTILINE` flag; additional flags such as case-insensitivity apply only where included in the pattern itself.

To pass, an answer must satisfy both the main test and any additional required pattern. For example, the positioning-map test looks for “4” or “four” near a word such as “dimensions” or “axes”. It does not judge whether the entire map is commercially useful.

The `intended_rule` column explains the desired behaviour. The pattern columns specify what the automatic grader actually accepted. These can differ in scope: mentioning the right value can pass a narrow check without demonstrating that the whole answer is correct. This pack lets readers inspect that limitation directly. The larger replay did not measure answer structure or overall task correctness.

Timeouts and errors were treated as missing evidence. A missing first attempt is excluded from first-attempt scoring. Eventual-compliance scoring excludes tasks with missing attempts because their full outcome is unknown. The correction-stream test also had processing losses, including 20 corrections in the larger OMS run. Sequence rows show the planned input, not confirmation that every approach processed every message successfully.

Paired comparisons in the reported results used McNemar's exact test, separately for the two question types. The handoff recording probabilities form a sweep of experimental conditions, not estimates of how frequently real teams write corrections down. The smaller experiments found no statistically clear advantage for OMS. The larger replay's first-attempt advantage does not imply a win after retries on every measure.

## Provenance and scope

The original guidance comes from the public documents linked in the CSVs. The benchmark authors wrote the changes and questions. Source excerpts establish where the original conventions came from; they do not imply that the original authors endorse the synthetic changes or the OMS benchmark. Some legacy question identifiers still name an earlier convention. Use the final rule and scoring columns to interpret them.

The reports identify the model as `sonnet`. This is a provider alias, not an immutable model-version identifier. The smaller test records provider-default temperature, and the handoff report notes that the CLI exposes no temperature control. The larger guidance test and smaller guidance test report Neo4j as their store. These materials do not claim that all provider behaviour can be reproduced exactly from the recorded settings.

This export was prepared from the test definitions associated with the September results. Handoff source-file hashes match the hashes recorded in its report. All larger and smaller replay question identifiers match the exported question set, and the exported scoring criteria reproduce all 3,375 available, non-error replay verdicts when applied privately to the saved answers. The sequence and recording schedule were reconstructed from the recorded seeds and the test's deterministic ordering rules. They are not execution logs.

The pack contains no benchmark implementation, JSON reports, model responses or per-approach result logs. It is an input-and-method disclosure: readers can examine the questions, corrections, assumptions and acceptance criteria. Recalculating the published success rates or independently auditing every scoring decision would also require the withheld responses and run records. Full reruns would require the implementation, generated guidance and execution environment.

Checksums in `CHECKSUMS.sha256` identify the files in this release so a reader can check that their copy has not changed. They establish file identity, not independent validation of the reported results.