GPT vs Claude vs Gemini on Company Filings: Why Model Benchmarks Mislead in Finance

If you are choosing a language model to read company filings, the public leaderboards will not tell you what you need to know. They rank general reasoning and clean-text fluency, and neither one decides whether a model reads a balance sheet correctly.

The honest answer to “GPT or Claude or Gemini for filings?” is that on ordinary pages they are all competent, and on the hard pages the winner depends entirely on the documents you actually work with. The gap that matters is not the model’s IQ. It is how numbers, units, and context are handled around the model.

What leaderboards actually measure

Most well-known benchmarks test a model on tidy, self-contained problems: multiple-choice knowledge, grade-school and competition math, code snippets, short reading passages. The inputs are curated. The questions have one clean answer. The English is well-formed.

That is a fair way to compare general capability. It is a poor proxy for financial document work, because a filing is almost the opposite of a benchmark item. It is long, inconsistent, densely numeric, and unforgiving of small errors. A model can top a reasoning leaderboard and still misread a footnote that changes a revenue figure.

A benchmark rewards a model for being clever on a clean question. A filing punishes a model for being careless on a messy one. These are not the same skill.

The leaderboard measures the first. Your workflow depends on the second.

Filings break the assumptions benchmarks rely on

Company filings violate almost every convenience that makes a benchmark tractable. A few examples that anyone who has parsed an annual report will recognise:

• Units drift within a single document. One statement is in crore, a note is in lakh, a subsidiary schedule is in millions, and a per-share figure is in rupees. Get the unit wrong and the number is off by orders of magnitude while still looking plausible.
• Line items are named inconsistently. “Revenue from operations,” “net sales,” “total income,” and “turnover” can mean subtly different things across companies, sectors, and years, and sometimes within the same filing.
• Restatements and reclassifications quietly change history. A prior-year figure in this year’s report may not match the figure that was originally filed, because the company restated or regrouped it. A model that trusts the first number it sees will carry the error forward.
• Layout is adversarial. Numbers live in merged table cells, rotated columns, scanned images, and multi-page tables that split awkwardly. The reading order a human infers visually is not the order the text arrives in.
• Context is long and load-bearing. An earnings-call transcript or a full report can run tens of thousands of words, and the sentence that qualifies a number (“excluding the one-time gain”) may sit pages away from the number itself.

None of these appear on a general leaderboard. All of them appear on Tuesday.

The gap is on edge cases, not average pages

Here is the uncomfortable part for anyone hoping a leaderboard will pick their model for them: on clean, well-structured pages, frontier models are mostly interchangeable for this work. If a page is a plain P&L in consistent units with standard labels, they will all read it about as well as each other.

The differences that decide real quality show up on the edge cases. Does the model notice the units switched mid-document? Does it prefer the restated figure or the original, and does it tell you which one it used? Does it hold a qualifier from paragraph three when it reports a number from paragraph nine? Does it refuse to invent a value when the cell is genuinely blank?

These are exactly the situations a general benchmark does not contain, because curated benchmarks are built to remove ambiguity, and ambiguity is the whole job in finance.

General benchmark task	Real filing task
Answer a clean multiple-choice question	Reconcile “total income” across three differently labelled statements
Solve a self-contained math problem	Track units that change from crore to lakh to millions inside one report
Read a short, well-formed passage	Hold a footnote qualifier across a 40,000-word transcript
One unambiguous correct answer	A number that is only correct relative to a restatement and a citation
Well-formed printed English	Merged cells, rotated columns, scanned tables, split pages

A model that wins the left column can lose the right column, and the left column is what the leaderboard reports.

Most of the quality lives around the model, not in it

There is a second reason leaderboards mislead. Even when a model is capable, the output quality on filings is dominated by the data handling that surrounds it: how the document is parsed and reconstructed, how tables are extracted, how units and periods are normalised, how the model is asked to cite its source cell, and how the answer is checked against arithmetic and against the original filing.

Two teams using the identical model can get very different accuracy on the same filing, because one feeds it a clean, structured, well-scoped page with a strict instruction to cite and verify, and the other pastes in a raw blob of scanned text and hopes. The leaderboard score is the same for both teams. The results are not.

This is the practical takeaway hidden inside the vendor debate. Swapping GPT for Claude for Gemini changes less than most buyers expect, once the page is clean and the instructions are strict. Improving the extraction, normalisation, and verification around whichever model you choose changes far more. At Altys Labs this is the part we spend our time on, because it is the part that actually moves accuracy on real documents.

How to benchmark models for finance

If you want to know which model to trust on filings, stop reading general leaderboards and build a small, honest test on your own documents. A workable recipe:

1. Use your real filings. Pull actual reports, statements, and transcripts from the companies and sectors you cover. Public, but yours.
2. Include the hard cases on purpose. Unit switches, restatements, ambiguous line items, blank cells, scanned tables, and long transcripts. If your test set is all clean pages, every model will pass and you will have learned nothing.
3. Define correct answers up front. Write down the exact numbers and the source location for each question, so scoring is objective rather than “the answer sounded right.”
4. Score exactness and citation, not fluency. A confident wrong number is worse than a flagged uncertainty. Reward the model that says “not present in the provided text” over one that guesses.
5. Test the whole pipeline, not the raw model. Benchmark the model together with your parsing, normalisation, and verification steps, because that is what will actually run in production.
6. Re-run it when models or documents change. New model versions and new filing formats both shift results. A benchmark is a habit, not a one-time purchase.

Do this and the vendor question tends to shrink to its real size. On most pages the frontier models are close. On the pages that matter, your own test set will tell you which model, and which surrounding pipeline, gets the number right and shows its work. That is the only benchmark that pays your bills, and no public leaderboard can run it for you.