Skip to content
Benchmarks · run of 2026-11-11

More right answers, in one call.

A pre-registered run on 58 held-out workbooks and four models. The protocol, code and data were frozen and their digest posted before the first call.

93.8–99.3%accuracy with Cellprose in one call on the 30 real-world workbooks, against 65.9–84.1% for CSV and 70.0–85.7% for MarkItDown
3.5–4.6turns an agent with a shell needs on the same workbooks (97.0–99.2% accuracy). Cellprose in the prompt answers in one call, within about 0.5 to 3.8 points of it per model (GPT-6 Luna 93.8% against 97.6%)
0.32–0.60×MarkItDown's cost per correct answer; a win in 7 of 8 model-and-corpus pairs

32 pre-registered comparisons: 12 wins, 0 losses, the rest no difference. Against agents that open the .xlsx with a shell, cost is mostly no difference. Every number has its limits below.

Accuracy and cost

Every model, every way of giving it the workbook.

Accuracy against cost per call for the four models, from the same run. Cellprose in the prompt sits higher than CSV and MarkItDown for most models, and for some it costs less.

Real-world workbooks have formulas, and CSV falls behind. Cellprose in the prompt scores 93.8–99.3% against CSV's 66.7–84.3%, for 0.70–1.38 times CSV's cost.

How to read it: each chart is one model. Up means more answers right. Left means cheaper per call; the dashed line is CSV's cost, and 2x means twice as much. The best place is the top left.

Claude Haiku 5.5 ↖ up and left is better
60%70%80%90%100%0.5xCSV2x3xcost per call, relative to CSV (log scale)Claude Haiku 5.5, CSV: 66.7% correct, 1.00x CSV's cost per call ($0.0037)CSVClaude Haiku 5.5, MarkItDown: 70.6% correct, 3.16x CSV's cost per call ($0.0117)MarkItDownClaude Haiku 5.5, Agent + shell on .xlsx: 96.9% correct, 1.16x CSV's cost per call ($0.0043)Agent + shell on .xlsxClaude Haiku 5.5, Cellprose in the prompt: 96.5% correct, 1.38x CSV's cost per call ($0.0051)Cellprose in the prompt

Cellprose 96.5% against CSV 66.7%, at 1.38x CSV's cost; the agent scores 96.9%.

Claude Sonnet 5.5 ↖ up and left is better
60%70%80%90%100%0.5xCSV2x3xcost per call, relative to CSV (log scale)Claude Sonnet 5.5, CSV: 82.7% correct, 1.00x CSV's cost per call ($0.0804)CSVClaude Sonnet 5.5, MarkItDown: 81.4% correct, 1.30x CSV's cost per call ($0.1047)MarkItDownClaude Sonnet 5.5, Agent + shell on .xlsx: 98.7% correct, 1.03x CSV's cost per call ($0.0830)Agent + shell on .xlsxClaude Sonnet 5.5, Cellprose in the prompt: 99.3% correct, 0.70x CSV's cost per call ($0.0563)Cellprose in the prompt

Cellprose 99.3% against CSV 82.7%, at 0.70x CSV's cost; the agent scores 98.7%.

GPT-6 Luna ↖ up and left is better
60%70%80%90%100%0.5xCSV2x3xcost per call, relative to CSV (log scale)GPT-6 Luna, CSV: 73.3% correct, 1.00x CSV's cost per call ($0.0036)CSVGPT-6 Luna, MarkItDown: 74.1% correct, 1.28x CSV's cost per call ($0.0046)MarkItDownGPT-6 Luna, Agent + shell on .xlsx: 97.5% correct, 1.00x CSV's cost per call ($0.0036)Agent + shell on .xlsxGPT-6 Luna, Cellprose in the prompt: 93.8% correct, 0.72x CSV's cost per call ($0.0026)Cellprose in the prompt

Cellprose 93.8% against CSV 73.3%, at 0.72x CSV's cost; the agent scores 97.5%.

GPT-6.1 Sol ↖ up and left is better
60%70%80%90%100%0.5xCSV2x3xcost per call, relative to CSV (log scale)GPT-6.1 Sol, CSV: 84.3% correct, 1.00x CSV's cost per call ($0.0364)CSVGPT-6.1 Sol, MarkItDown: 86.1% correct, 1.65x CSV's cost per call ($0.0599)MarkItDownGPT-6.1 Sol, Agent + shell on .xlsx: 99.2% correct, 0.92x CSV's cost per call ($0.0336)Agent + shell on .xlsxGPT-6.1 Sol, Cellprose in the prompt: 99.3% correct, 1.15x CSV's cost per call ($0.0419)Cellprose in the prompt

Cellprose 99.3% against CSV 84.3%, at 1.15x CSV's cost; the agent scores 99.2%.

Real-world set, 30 workbooks, 3 repeats, run of 2026-11-11, list price, every token uncached. Four of the six arms are drawn; the table under "Every arm, mean per call" below lists all of them. Each model ran at one reasoning setting, so there is no effort curve. Hover a point for its numbers.

Verdicts

Cost per correct answer, against each alternative.

Ratio of Cellprose's cost per correct answer to the baseline's, with its 95% paired-bootstrap interval over workbooks. A win needs the whole interval below 1 and no more than one point of accuracy lost; a loss needs it wholly above 1.

Real-world: 30 workbooks, 163 questions, 15 Damodaran valuation models and 15 Enron trading workbooks; five question types. 3 repeats.

AgainstAccuracy (Cellprose / baseline)Cost ratio95% intervalVerdict
MarkItDown99.2% / 81.1%0.440.34 to 0.55win
Agent + shell99.2% / 98.8%0.680.47 to 0.92win
Agent + viewer99.2% / 97.4%0.720.52 to 0.95win
CSV99.2% / 82.6%0.580.50 to 0.67win

Every arm, mean per call

ArmAccuracyTokens per callCost per callCost per correctTurns
Cellprose in the prompt99.3%24,388$0.0563$0.05671.0
MarkItDown81.4%36,155$0.1047$0.12861.0
CSV82.7%22,396$0.0804$0.09721.0
Agent + shell on .xlsx98.7%35,175$0.0830$0.08414.6
Agent + formula viewer97.3%33,170$0.0771$0.07924.7
Agent + shell on Cellprose file98.3%37,671$0.0865$0.08804.7
No workbook (guess)23.2%the control: what a model gets right with no workbook at all

Question-averaged accuracy here; the verdicts above average by workbook, so the two can differ by a tenth of a point. Prices are list prices with every token uncached.

Accuracy by question type

Where formulas decide the answer.

CSV and MarkItDown carry values only. The gap opens on the questions that need a formula: what a cell is computed from, what a change would move, and which numbers were typed in where a formula belongs.

Claude Sonnet 5.5, Real-worldLookupCell addressFormula traceImpactTyped-in vs formula
Cellprose in the prompt100.0%100.0%100.0%95.8%100.0%
MarkItDown100.0%95.2%83.9%71.4%62.0%
CSV98.6%96.8%82.1%74.0%70.4%
Agent + shell on .xlsx100.0%100.0%100.0%93.0%100.0%
Agent + formula viewer100.0%96.8%99.9%87.8%99.6%
Agent + shell on Cellprose file98.6%100.0%97.9%95.8%100.0%
Addendum A

Cellprose with its own tools.

An agent given the workbook's index and Cellprose's read, find, trace and what-if tools, with no shell, against the same two shell agents. Frozen and posted separately; 2 repeats. The decision to run it was made after the main results were seen, so read it as a second result, not part of the first.

AgainstAccuracy (own tools / baseline)Cost ratio95% intervalVerdict
Agent + shell99.5% / 99.8%0.490.35 to 0.67win
Agent + viewer99.5% / 99.9%0.500.35 to 0.71win

Across three overlapping views (the real-world comparisons appear in two of them, so these are not 24 independent tests): 14 wins of 24, no losses, cost ratios 0.29–0.79. The remaining "no difference" verdicts come from accuracy, mostly address questions on Haiku 5.5 and GPT-6 Luna. Impact questions are left out of the primary view because the what-if tool shares its method with the ground truth for them.

Method and limitations

Read before quoting a number.

  • 30 real-world workbooks (163 questions) and 28 SpreadsheetBench workbooks (111 questions), reported separately and never pooled. With this few workbooks the intervals are wide, Haiku's widest.
  • The questions are generated by rule from the workbooks, not written by their authors. SpreadsheetBench tasks are restated as value questions, where CSV already scores well and Cellprose has less to add.
  • Four models on one date: Claude Haiku 5.5, Claude Sonnet 5.5, GPT-6 Luna, GPT-6.1 Sol. Nothing is claimed for a model that was not run.
  • Every token is priced as uncached input or output at list price. The billed amount, with caching, was lower; see the full results.
  • Cellprose sends more tokens than CSV (1.31× on Claude, 1.60× on GPT). It wins on cost per correct answer only where it is right more often.
  • A one-shot call and an agent loop spend tokens differently. A win over an agent is on cost per correct answer, which is what a user pays, and says nothing about which reads a workbook better at equal cost.
  • The benchmark workbooks average about 20,000–25,000 tokens. Very large models do not fit one call; for those the index-then-tools route is the one to use, and it has not been measured on large files.
  • Full tables, failures, resumes and spend: RESULTS.md. How runs are done and scored: the benchmark protocol. Every number on this page is generated from the result files by benchmarks/site_run.py, and a test checks it against RESULTS.md.

Cellprose is an open format for spreadsheet workbooks. Benchmarks: pre-registered run of 11 Nov 2026, 58 held-out workbooks, four models, three repeats. Privacy

CellproseGitHub ↗
A1fx
Ready