A pre-registered run on 58 held-out workbooks and four models. The protocol, code and data were frozen and their digest posted before the first call.
32 pre-registered comparisons: 12 wins, 0 losses, the rest no difference. Against agents that open the .xlsx with a shell, cost is mostly no difference. Every number has its limits below.
Accuracy against cost per call for the four models, from the same run. Cellprose in the prompt sits higher than CSV and MarkItDown for most models, and for some it costs less.
Real-world workbooks have formulas, and CSV falls behind. Cellprose in the prompt scores 93.8–99.3% against CSV's 66.7–84.3%, for 0.70–1.38 times CSV's cost.
How to read it: each chart is one model. Up means more answers right. Left means cheaper per call; the dashed line is CSV's cost, and 2x means twice as much. The best place is the top left.
Cellprose 96.5% against CSV 66.7%, at 1.38x CSV's cost; the agent scores 96.9%.
Cellprose 99.3% against CSV 82.7%, at 0.70x CSV's cost; the agent scores 98.7%.
Cellprose 93.8% against CSV 73.3%, at 0.72x CSV's cost; the agent scores 97.5%.
Cellprose 99.3% against CSV 84.3%, at 1.15x CSV's cost; the agent scores 99.2%.
Real-world set, 30 workbooks, 3 repeats, run of 2026-11-11, list price, every token uncached. Four of the six arms are drawn; the table under "Every arm, mean per call" below lists all of them. Each model ran at one reasoning setting, so there is no effort curve. Hover a point for its numbers.
Ratio of Cellprose's cost per correct answer to the baseline's, with its 95% paired-bootstrap interval over workbooks. A win needs the whole interval below 1 and no more than one point of accuracy lost; a loss needs it wholly above 1.
Real-world: 30 workbooks, 163 questions, 15 Damodaran valuation models and 15 Enron trading workbooks; five question types. 3 repeats.
| Against | Accuracy (Cellprose / baseline) | Cost ratio | 95% interval | Verdict |
|---|---|---|---|---|
| MarkItDown | 99.2% / 81.1% | 0.44 | 0.34 to 0.55 | win |
| Agent + shell | 99.2% / 98.8% | 0.68 | 0.47 to 0.92 | win |
| Agent + viewer | 99.2% / 97.4% | 0.72 | 0.52 to 0.95 | win |
| CSV | 99.2% / 82.6% | 0.58 | 0.50 to 0.67 | win |
| Arm | Accuracy | Tokens per call | Cost per call | Cost per correct | Turns |
|---|---|---|---|---|---|
| Cellprose in the prompt | 99.3% | 24,388 | $0.0563 | $0.0567 | 1.0 |
| MarkItDown | 81.4% | 36,155 | $0.1047 | $0.1286 | 1.0 |
| CSV | 82.7% | 22,396 | $0.0804 | $0.0972 | 1.0 |
| Agent + shell on .xlsx | 98.7% | 35,175 | $0.0830 | $0.0841 | 4.6 |
| Agent + formula viewer | 97.3% | 33,170 | $0.0771 | $0.0792 | 4.7 |
| Agent + shell on Cellprose file | 98.3% | 37,671 | $0.0865 | $0.0880 | 4.7 |
| No workbook (guess) | 23.2% | the control: what a model gets right with no workbook at all | |||
Question-averaged accuracy here; the verdicts above average by workbook, so the two can differ by a tenth of a point. Prices are list prices with every token uncached.
CSV and MarkItDown carry values only. The gap opens on the questions that need a formula: what a cell is computed from, what a change would move, and which numbers were typed in where a formula belongs.
| Claude Sonnet 5.5, Real-world | Lookup | Cell address | Formula trace | Impact | Typed-in vs formula |
|---|---|---|---|---|---|
| Cellprose in the prompt | 100.0% | 100.0% | 100.0% | 95.8% | 100.0% |
| MarkItDown | 100.0% | 95.2% | 83.9% | 71.4% | 62.0% |
| CSV | 98.6% | 96.8% | 82.1% | 74.0% | 70.4% |
| Agent + shell on .xlsx | 100.0% | 100.0% | 100.0% | 93.0% | 100.0% |
| Agent + formula viewer | 100.0% | 96.8% | 99.9% | 87.8% | 99.6% |
| Agent + shell on Cellprose file | 98.6% | 100.0% | 97.9% | 95.8% | 100.0% |
An agent given the workbook's index and Cellprose's read, find, trace and what-if tools, with no shell, against the same two shell agents. Frozen and posted separately; 2 repeats. The decision to run it was made after the main results were seen, so read it as a second result, not part of the first.
| Against | Accuracy (own tools / baseline) | Cost ratio | 95% interval | Verdict |
|---|---|---|---|---|
| Agent + shell | 99.5% / 99.8% | 0.49 | 0.35 to 0.67 | win |
| Agent + viewer | 99.5% / 99.9% | 0.50 | 0.35 to 0.71 | win |
Across three overlapping views (the real-world comparisons appear in two of them, so these are not 24 independent tests): 14 wins of 24, no losses, cost ratios 0.29–0.79. The remaining "no difference" verdicts come from accuracy, mostly address questions on Haiku 5.5 and GPT-6 Luna. Impact questions are left out of the primary view because the what-if tool shares its method with the ground truth for them.
benchmarks/site_run.py, and a test checks it against RESULTS.md.