Release note · August 3, 2026
A benchmark for reviewing Word changes, not just comparing text
A Word document can retain ordinary text while its stored review surface changes. A direct hyperlink can point somewhere new. A HYPERLINK or INCLUDETEXT field can retain its displayed result while its private instruction changes. Hidden text, revision markup, editing restrictions, content-control mappings, custom XML, and opaque macro or OLE payloads sit outside a normal prose diff.
Document Change Assurance Benchmark (DCAB) makes a narrow, useful slice of that problem reproducible. Version 0.1.0 supplies twelve deterministic baseline/candidate WordprocessingML pairs, public target-free truth manifests, a structural validator, a portable observation protocol, and a scorer. It evaluates static document-review claims—not Word rendering, macro behavior, field evaluation, or client security outcomes.
One stored boundary at a time
Each DCAB case preserves its package-member set and the same sequence of stored Word text nodes. The pair changes only declared ZIP members. The initial corpus includes direct Word hyperlink target changes and markup addition; HYPERLINK and INCLUDETEXT field-source retargeting; direct w:vanish and w:ins markup; Track Changes and read-only document-protection settings; content-control binding XPath and custom-XML payload changes; plus inert VBA-project and embedded-OLE payload boundaries.
These are explicit stored forms, not invented metadata. Microsoft’s Open XML reference documents relationship-backed w:hyperlink; its Word documentation describes content controls bound to custom XML; and the API references expose w:vanish, w:ins, and w:documentProtection.
A public oracle that does not disclose what it is protecting
Real document-review output should not need to reveal a target URL, field instruction, XPath, relationship ID, payload digest, or document text just to show that a category changed. DCAB therefore keeps the raw synthetic values inside the generated packages and publishes only a bounded fact such as field_target_changed or hidden_text_run_added. Targets use the reserved example.invalid domain. VBA and OLE package members are inert marker bytes, not runnable samples.
Its validator checks deterministic source bytes, ZIP integrity, the OPC root relationship, content types, local relationship shape, compact WordprocessingML evidence, unchanged stored text, target-free truth, and exact changed-member boundaries. It deliberately never opens Word, resolves or fetches a target, updates a field, parses an opaque payload, activates OLE, executes code, or states a rendering/runtime result.
A score that keeps scope honest
DCAB does not impose one private scanner schema. A tool adapter emits an observation document naming the cases it analyzed, exact declared facts it observed, and an optional allow, review, or block reference disposition. Unsupported and errored cases remain explicit. The scorer reports fact recall, analysis coverage, reference-policy agreement, and complete cases. An observation outside the intentionally partial contract stays visible as unrecognized evidence; DCAB does not call it a false positive or manufacture a precision figure.
The first optional consumer is DocFence 0.27.0. Its local adapter maps public aggregate change categories to DCAB’s facts and reaches a strict 12/12 score. That is a reproducible integration check, not a requirement to use DocFence and not a claim that its normal policy is universal.
Compatibility evidence, carefully bounded
The test suite opens every ordinary .docx fixture through the independent python-docx reader, then opens every .docx and .docm package through its lower-level OPC reader. Hosted CI repeats the package, source-distribution, and wheel checks on Python 3.11, 3.12, and 3.13, with a separate released-DocFence adapter job.
That is useful independent-reader evidence. It is not a claim that Word, LibreOffice, an OLE server, a macro host, a field-update engine, or every renderer will behave a particular way. DCAB records static package state only.
Use the release
python -m pip install \
https://github.com/SybilGambleyyu/document-change-benchmark/releases/download/v0.1.0/document_change_benchmark-0.1.0-py3-none-any.whl
dcab validate
dcab observation-template --output observations.json
dcab score --observations observations.json --output score.jsonThe source generator, validator, research notes, release artifacts, and reference adapter are open at github.com/SybilGambleyyu/document-change-benchmark. The corpus is also mirrored as a public Hugging Face dataset.