Release note · August 2, 2026
A benchmark for reviewing spreadsheet changes, not just editing them
Spreadsheet tools are increasingly good at writing a formula or producing a finished workbook. The harder operational question comes afterward: a workbook changed—what changed, which other formulas can it reach, should it block a review, and what could the tool not determine? Workbook Change Assurance Benchmark (WCAB) is a small, open way to make those claims testable.
Version 0.4.0 contains 18 deterministic scenarios: 17 baseline/candidate workbook pairs and one two-workbook portfolio. Each supplies machine-readable truth about an observable change, a benchmark review disposition, and—where appropriate—a static dependency-impact lower bound. The files are generated from source in the repository, not copied from a financial model, email archive, or other sensitive corpus.
The gap is about change assurance
There are already useful spreadsheet datasets. Modified EUSES injects formula faults and supplies test decisions. VEnron recovered version information from business spreadsheets. SpreadsheetBench tests spreadsheet manipulation. And recent formula-repair research measures whether a tool can fix runtime formula problems.
Those are complementary questions. I could not find an open benchmark whose primary contract is the review boundary between two workbook versions: explicit change facts, a stated accept/review/block convention, static impact that must not be missed, and a way to say “unsupported” rather than silently pass. That is the narrower problem WCAB addresses.
What the first cases cover
The suite includes formula-to-value replacement, wrong-period reference drift, material input propagation, external formula references, defined-name redirection, copied-formula interruption, mismatched SUMIFS ranges, data validation and conditional-formatting removal, hidden-sheet visibility, direct formula-cell unlocking, incomplete manual calculation, direct static cycles, a 3-D formula whose tab span changes without its text changing, an Excel Table whose scope changes while its structured-reference summary text stays unchanged, a direct reference that becomes INDIRECT, a documented structural rewrite, and a local cross-workbook dependency.
That mix matters. A row or column insertion can rewrite many formulas while retaining the declared logical inputs. Conversely, inserting a tab inside SUM(Jan:Mar!B5) can change the formula’s scope while its text does not change at all. A benchmark that treats every text difference as a regression—or every unchanged formula string as safe—cannot measure the review problem honestly.
Truth that can be inspected
Every case carries a truth.json document. Schema version 3 adds a machine-matchable coverage_expectations field alongside facts such as formula_to_value, sheet_visibility_changed, and three_d_scope_changed. Its must_reach section gives formula locations a conservative local A1 dependency walk must reach. These are deliberately lower bounds; WCAB is not a formula evaluator and does not claim complete Excel dependency semantics, dynamic-reference resolution, or numerical correctness.
The bundled validator reads the generated files and verifies those contracts. It does not trust a fixture merely because the manifest says it is right. The schema documents each fact kind and boundary.
Reproducibility is part of the artifact
Openpyxl can build the workbooks, but ZIP timestamps would otherwise change on every run. WCAB fixes workbook metadata and canonicalizes package member order and timestamps, then tests byte-for-byte regeneration. Version 0.4.0 validated all 18 cases and 23 tests locally under Python 3.12 and 3.13; hosted CI passed under Python 3.10 and 3.13, and a fresh Python 3.12 wheel installation reproduced the catalogue byte-for-byte. The validation record contains the exact commands and results.
A distribution that tools can verify
Version 0.4.0 retains a deterministic, one-row-per-case manifest.jsonl catalogue. It retains each case’s facts, review convention, static-impact lower bounds, coverage text, and scoreable coverage expectations, then adds the exact relative path, byte count, and SHA-256 digest for every baseline and candidate workbook. A runner can now record precisely which fixtures it consumed instead of inferring identity from a directory name.
The same release is available as a Hugging Face dataset. Its downloadable catalogue matches the GitHub release byte-for-byte, while the source repository remains the place to regenerate and validate every file.
A score that makes coverage gaps visible
A benchmark needs more than files and truth documents: different review tools emit different report formats. WCAB 0.4.0 upgrades its small normalized observation protocol and scorer. An adapter declares each case as analyzed, unsupported, or errored; then the scorer reports expected-fact recall, coverage-disclosure recall, analyzed coverage, and agreement with WCAB’s reference review convention.
That last boundary matters. WCAB’s truth is intentionally targeted, not a claim that it lists every possible workbook difference. A tool may report another real observation that the case does not name. The scorer keeps such evidence explicit as unrecognized instead of calling it a false positive. And an unsupported case remains visible in the result—it cannot become an unearned pass.
When a reference becomes text, a graph needs a warning
Microsoft documents that INDIRECT returns the reference named by text. The new fixture changes a direct =Revenue!$C$8 output into =INDIRECT(Inputs!$E$12), while an editable cell supplies the address. A tool can still report the formula edit; that alone does not establish that its static dependency graph covers every target the workbook can select.
Schema v3 therefore names a dynamic_reference_static_coverage expectation at that formula location. Protocol v2 lets an adapter attach native evidence to the exact expectation, and the scorer measures it separately. This avoids rewarding a tool for silently treating an indirection boundary as a complete impact graph, without forcing every tool to make the same claim about formula evaluation.
When a Table grows, formula text is not enough
Excel structured references use a Table and column name instead of a fixed cell range. Microsoft documents that those references adjust when data is added to or removed from the Table. The schema-v2 case therefore holds =SUM(SalesLedger[Amount]) constant while the stored Table range changes from A1:D4 to A1:D5. The benchmark asks a reviewer to surface that stored-scope change; it does not attempt to calculate the total or claim a numerical result.
A baseline adapter, without making it the benchmark
WCAB has an optional local adapter for FormulaFence. It invokes a locally installed executable, maps its JSON evidence into WCAB facts, and leaves unmapped cases visible. FormulaFence 0.219.0 observed all 19 mapped facts, including the portfolio impact, the Table-definition change, and the new FF012 dynamic-reference warning; it also produced five targeted lint rules. Its normalized-observation export leaves review decisions unset rather than inventing a policy, so its score is 19 of 20 declared facts, one of one coverage disclosures, full analyzed coverage, and zero policy agreement. The structural rewrite remains intentionally unmapped: a small fixture can document its before/after intent, but it cannot establish a generic proof of Excel semantic equivalence.
That distinction is the point. A tool should be rewarded for finding what it can prove and for exposing what it cannot—not for converting a coverage gap into an unearned pass.
Use or extend it
git clone https://github.com/SybilGambleyyu/workbook-change-benchmark.git
cd workbook-change-benchmark
python -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
wcab validate --fixtures fixtures
wcab manifest --fixtures fixtures
wcab observation-template --fixtures fixtures --output observations.json
wcab score --fixtures fixtures --observations observations.json
pytestThe benchmark is MIT-licensed. Start with the source, the v0.4.0 release, or the dataset mirror. The useful next additions are cases and adapters that retain the same standard: deterministic generation, inspectable evidence, explicit limits, and a validation test.